Finally!
So I finally go around to drilling and tapping my pieces for the punch holder. I forgot I didn't have a collar to hold the punch on the press pin so I went to TS and got a 1 1/2" pipe coupling. I welded that to the top of the holder. I drilled and tapped a 1/4" hole in it so I'd have a set screw. Tried it on a piece of 1/8" flat bar and it worked great!
Hopefully I didn't inhale too much of the fumes from welding that galvanized ****.
I have no idea how well the collar will hold. The coupling is cast and I was using the same MIG I use for mild steel.

To top it off, the I tacked right side of the collar and, in the 5 seconds it took me to move to the left side, it lifted about a 1/16". I didn't notice it so it got welded a little crooked. **** it.
View media item 68298