It sort of depends on what weight you can handle, and what you are doing, and for how long. In general (leaving detailed work out of it) the heavier the head the easier the hammer will do its job. You let the mass of the hammer head the job instead of your arm.
Last weekend my son had a long handled hatchet out for root cutting and I about exhausted myself trying to get the job done with that. I went and got one of my biggest single edge axes and it was actually easier to manage as the weight of the ax was doing a lot of the work instead of me trying to force the smaller head to do the work.
The job and swing angle will have a lot to do with it also. An overhead swing is a lot less fatiguing than a side swing, for most people. So depending on what you are doing, you may need a lighter head just to manage the swing.
I think you are looking at having a couple of hammers, and use the one right for the job situation. Of course you must be able to swing the thing, which is primarily getting it lifted over your head. Go to a store and try seeing what weights you can manage for the lifting part, and go from there.
My basic recommendations on a WAG would be for you to have maybe a 3 lb engineers hammer (mini sledge head) a 24 oz ball peen and a 16 oz ball peen, assuming you can manage the lift for the bigger ones.
This is assuming you need something heavy for basic bashing, and little finesse, and the PBs for lighter bashing and getting into more detailed work where a simple flat face may not be appropriate.