When you go to get some hardware, are you usually thinking I need 1/4" or I need a bolt or I need something metric? Can you identify a pattern?
I sorted by type. Screws, bolts, washers, nuts, pins all got binned. I did not sort or label them by size.
I would go with one drawer for metric...