I am trying to determine the optimum size for an additional diesel generator. I have aggregated the hydros, and entered 2 of the existing generators as GEN1 & GEN2. The question I have is with the proposed GEN3. I have entered the "sizes to consider" and the "fuel consumption data for GEN3 but I am unsure how HOMER distinguishes fuel consumption differences between the "sizes to consider"? The different sizes I am considering are different RPMs and efficiencies, so I doubt that the fuel consumptions are a linear relationship between sizes.

If they have different RPMs they probably have different lifetimes too.  You are right to worry about modeling those generators with a single generator model in HOMER.  Unfortunately I think you'll have to do multiple runs with HOMER to model those different generators.  In other words, in the first run you could use GEN3 to model the smallest of the possible new generators, then in the second run you could use GEN3 to model the next larger of the possible new generators, and so on.  You would have to compare costs manually to determine which of the possible new generators would minimize the total NPC.