Yep, a zombie thread....please forgive me.
Regarding the wobble with hole saws, when the drill press has a Morse Taper socket, and the hole saw has 1/2-20 threads, I would consider a Morse Taper arbor with 1/2-20 threads (intended to mount a drill chuck). Plenty of them to be found on ebay...