A lot depends on what kind of projects you work on. Even though all light duty vehicles sold in the US have been metric for 10-20 years, I don't stock any metric fasteners. I started with 50 each of USS (coarse) and SAE (fine) nuts, washers and lock washers from 1/4' to 1/2". No bolts, because I just don't have the room to store them.
I do have a mixture of 1/4", 3/8" and 1/2" stainless for boat projects and for outdoor equipment.
The last couple of years I have been building my collection downward. #12, #10, #8 and #6. Started again with 50 nuts, washer and lock washer in each size, but I have added a few machine screw and sheet metal screws in a couple of length, in #6, #8, #10 and #12.
Also buy yourself a 100 pack of hollow wall anchors in size #6-8 and #10-12.
Red Head Poly Set Anchors in size #10-12 are great in block walls, concrete, etc. If you can find them, buy them without screws. I always seem to need longer screws or stainless
(bolted a laundry tub to the floor).
I am lucky that I have a "real" fastener store close by that will sell me any quantity. If I go there for 5-10 whatever, I try to buy a whole box (25, 50, 100).
You can find some decent deals on eBay, including FREE shipping, if you buy a sufficient dollar amount ($50-$100). On the 1/4" and up, make certain that they are
grade 5 ! Below grade 5 is junk and above grade 5 is really only needed in a few applications.