1-4mm in 0.5mm increments plus 5-12mm in 1mm increments will get you very far. 3.3 for M4 is pretty useful, too. Next I would add 8.5 for for M10 and maybe 6.8 for M8.
I'd rather go bigger than 12mm than fill in more 0.5 increments.
This is from my experience as a hobbyist, not as a professional.