I sunk mine into a wooden file handle... MUCH harder to lose that way (it's also how I handle the key for my ancient Black and Decker Model 1 1/2-inch drill).
On the drill press, I then put a small eye hook into the end of the key's wooden handle, and attacked it to a janitor's spring-loaded key ring cable... Lou the Janitor kept it on his belt with a belt clip like a tape measure: I keep it on the drill press column with a big hose clamp just under the head of the machine.
Pull it down, tighten, it retracts out of the way, and it can't ever be borrowed and lost.
I like the wooden handle, because it lets me get more even pressure while tightening the chuck.
-Brad