I would like to include balance of systems in the cost model (like a charge regulator and solar panel rack in the PV module).

In fact you can account for such costs in HOMER's cost tables.  For example, if the PV array costs $7/W and the PV mounting hardware will cost $500 regardless of the size of the PV array, you can model that scenario as follows:
 

That says the PV array costs $500 (for the mounting hardware) plus $7/W.  By adding more rows to the cost table, you can model more complex scenarios.  For example, you might assume that the mounting hardware will cost $500 for an array up to 1 kW, $750 for an array between 1 kW and 3 kW, and $1250 for a larger array.

Any costs that you cannot assign to one particular component should go into the system fixed capital cost or system fixed O&M cost.  You can find those inputs in the Economics window.

As a general rule, you should account for all cost associated with a particular component in that component's cost table.  That allows HOMER to make an accurate estimate of the total system cost, and to compare the costs of different components when searching for the optimal system configuration.