I believe this might be an issue of perspective. Say for example, you have no distinction in your thought processes between SAE and metric. All you see in your perspective approach is a series of wrenches in graduating sizes, and likewise a series of graduating fasteners with the simple task before you of matching one to the other. Within this limited context there is almost a child-like simplicity that I can appreciate. In such a system, not only would having two measuring systems be overly complicated, but even having size markings on the wrenches would be unnecessarily distracting.
Say you were in a post-apocalyptic world like in one of those movies, and you're an offspring of the survivors having no formal education. You're confronted with machines of all kinds you need to fix and have access to a shop fully equipped with all kinds of tools. Remember, you don't know the wrench markings from squat. From that perspective, you would most likely proceed exactly as the OP suggested. This child-like simplicity would indeed be the most efficient approach. So I think those who are overly critical of the OP's thought are lacking some imagination and insight.
HOWEVER: this is NOT the way we are taught to think about machines and their maintenance, either from a theoretical or a practical perspective. Envisioning geometry and dimension lies at the very heart of working on machines. And this kind of inner-envisioning extends far beyond simple mating of a fastener head with a specific size tool.
For example, let's say we need to replace a set of piston rings. Not only are we envisioning the location, attachment and dimensional wear on the rings relative to the cylinder wall, we are also envisioning these same elements with the new replacements. This is long before we actually get to see the cylinders and pistons with our own eyes. We're also going to need to envision the dimensions of the rings, clearances, boring the cylinders to a specification, etc. All of this requires thinking about exact sizes in precise measurements.
Another factor is the ability of a mechanic--whether a professional or a shade tree--to anticipate each step in the process. Not unless you want to re-invent the wheel every time you have a repair. Sizing systems are fundamental to this anticipatory thinking, and to the envisioning of the entire procedure prior to beginning. Even down to knowing what tools you'll need before the job starts to insure you have them on hand. Then you need a system to organize those tools so you can access what you need, when you need it. All this is fundamental to being a good repairman, whether amateur or professional--that is, if you aspire to more than just being a hack. Understanding SAE and metric is a fundamental part to this entire way of thinking.
If the OP is only concerned with hacking away at something, then why even bother with any kind of tool organization? Organizing tools is part of the above-described processes. Any organization is done according to a standard. If you ar5e going to make up your own unique and novel standard, why ask advice from anyone schooled in the established standard???? They will be no help to you in a one-man world of your own creation.
My advice? Learn how to do it right, for all the above reasons, wherein matching a fastener size with a specific tool is only a small fragment of the process. A kid who picks up a guitar and starts hacking away at it can get pretty good at it, but when they begin to take professional lessons, the teacher needs to have them forget everything they know and start anew from scratch with the right way to hold your thumb, finger position, picking the strings, etc, because only with these correct techniques can one advance beyond a certain proficiency. Otherwise, you'll always be a hack. If that's your goal, then ask advice in forums populated by all other hacks. That's not the Garage Journal. Try Flea Market Flip or something like that.
Learn how to do it right. What's "right" is right for a reason.