Rudy Kouhoupt wrote a series of articles in one of the model engineering magazines a while back about converting a small drill press into a light duty milling machine. I remember reading it several years ago. I don't remember what was involved, but I do recall being impressed with what he accomplished. Then again, he was building small scale model parts, not gunsmithing, and Rudy was a hell of a machinist.
I've seen a few conversions over the years. The ones I've seen do away with the chuck altogether, and bore the spindle to fit one specific size of milling cutter (3/8" shank for most that I've seen). To get any sort of accuracy, the spindle needs to be bored on a lathe. The milling cutters are typically held in with set screws. Sometimes the quill bearings are replaced with ones that will take an axial load, but not always. An X-Y table or vise is also needed.
At the end of the day, you can't beat physics. My Walker Turner drill press is perhaps a couple of hundred pounds. It's a great old drill press, not heavy duty but still a better machine than even a pretty good modern consumer-grade drill press. I used it for some really light duty milling (slots and light passes in Delrin and MDF). Then I found a deal on a well-used Asian mill/drill. It weighs well over 600 pounds. I think the table alone weighs more than all of the castings for the Walker Turner.
Milling on a drill press is a bit nerve-wracking. In the back of your mind, you're always worried about the cutter chattering because of the lack of rigidity, or digging in and wrecking the quill or bearings. You need to do a fair bit of prep work to the drill in order to make it moderately usable and moderately safe, and the going will be slow. I wouldn't use it for any gunsmithing work at all unless you can tolerate a good chance of ruining the receiver.
No matter how strapped you are for cash, you're almost always better off waiting for a deal on Craigslist. Set up an alert and be ready to jump on the offer the minute you get the email from Craigslist.