The place to start is with HOMER itself. There are over 300 help documents, many linked through context. There is an excellent search function available that will likely lead you to the equation or explanation you are seeking.
One of the most frequently asked questions that we get from HOMER users, especially students, is “What are the algorithms that HOMER uses to calculate the least-cost system?”
The computer code that is behind HOMER is proprietary and protected by international copyright law. However, the basis of the way in which HOMER solves problems is freely available in the following publication:
Micropower System Modeling with HOMER, by T. Lambert, P. Gilman, and P. Lilienthal. Published in: Integration of Alternative Sources of Energy, by F. Farret and M. Simões. Copyright © 2006 by John Wiley & Sons, Inc.