If you work around batteries, you can't always shut things off. In the simplest case, an bare metal wrench can short a battery. The results aren't pretty. Insulated tools avoid this risk.
I also work around capacitors. Many of which have some pretty high voltages. And DC caps are especially fun, because after you discharge them down to 0V, you can watch them climb back up in voltage over the next several hours due to polarization effects.
As mentioned elsewhere, you may not be working in a panel on a neutral or ground bus, but in proximity to live stuff. No reason to take chances. Or like you said, in case you drop one.
Then, there's always the scenario that you're working in a box that you're sure is dead, but something turns out to be live. Perhaps three circuits meet in one box. Or you've turned off the switch, but voltage is still present because it's going through a lighted switch, or something stranger.