I agree with most of what you say, but the metric system does have some arbitrariness about it. The fundamental unit of length (the Meter) is itself arbitrary. Likewise for the gram. There's nothing fundamental in nature that defines the length of the meter or the mass of the gram. The meter was essentially picked to be similar to the yard in length. Subsequently, SI units have been redefined in ways that tie them to fairly fundamental things but this is sort of a retrospective affair.
There are systems of measurement that are tied to fundamental constants of the universe called "Natural Units". Physicists use them quite frequently since they simplify many calculations (for example, E=mc^2 becomes E=m in many natural units systems since the speed of light "C" is 1 in these systems). Natural units are typically defined in terms of Plank's constant, the speed of light in a vacuum, the fundamental electrical charge, Boltzman's constant, etc.. Unfortunately the scale of these units don't typically lend themselves to measure macroscopic things like the size of a bolt.
While I was raised on inches, feet, miles, pounds and tons, I do agree that the metric system is easier to work in. There's no computation required to know what size comes between 13mm and 15mm, all you've got to do is count.