I am currently using HOMER to optimize the off-grid residential PV system. I put hourly based electric load profiles by months, chose the size, and other properties of each component(PV, converter, battery). However, I am not sure how I can optimize the system size or capacity of each component automatically. Since I am now putting these numbers by myself, I am not sure whether these numbers are really optimzed or not, though HOMER model tells me a sign of whether the system is feasible or not.  

HOMER lets you input multiple sizes of each component, and it simulates all combinations of the sizes you enter.  The Optimization Results tab lists all the feasible system configurations, sorted in order of increasing life cycle cost.  So of the system configurations you have told HOMER to simulate, the optimal one will appear at the top of the list.

It can be difficult to determine whether that top system is the true optimum though.  One thing to check is where the top-ranked system fits within your 'search space,' which is the set of system configurations that you have told HOMER to simulate.  To determine that, take a look at the Search Space window, and click Show Winning Sizes.  The bottom table shows where the winning sizes fall within the ranges you specified.

The scenario below shows essentially the best-case scenario, where the winning sizes are about in the middle of the range, and the search space is evenly spaced and fairly high resolution.  If the optimal PV array size was 0.3 kW but I hadn’t considered any sizes smaller than that, then I would want to add a few smaller sizes just to be more confident in that number.  Similarly, if that was the largest size I had considered, I would want to add a few larger sizes since 0.4 kW or 0.5 kW might be better yet.  In the same way, if I had tried 0, 48, and 128 batteries, and 48 turned out to be the best of those sizes, I might want to refine the resolution by adding, say, 36, 42, 54, 60, and 72 batteries to the search space.

Another issue is whether you have considered the appropriate components.  For example, you could do a lot of work to optimize a pv-battery system when in fact a pv-diesel-battery system might be preferable.  If you did not think of adding a diesel generator to the schematic, you might not realize that it improves the system considerably.  Similarly, you might try modeling very large batteries and conclude that a battery bank does not make sense, when in fact a small battery bank with much smaller batteries might be very beneficial.  You might find that you require some modeling experience before you feel comfortable with these issues.  HOMER can help by enabling you to experiment with different component types and sizes.