I started with a drill press I got from my father. Probably bought at Spag's in Massachusetts, if anyone from that area remembers that place. Best place to shop. Anyhow, it's probably the equivalent of a typical Harbor Freight entry level drill press. Stepped pulleys. I used to change speeds, one for drilling wood, another for hole saws, even slower for metal. Not hard to do, but yeah a bit of an invconvenience. But if drilling metal, a slower speed helps so you don't overheat and smoke the drill.
Several years ago I picked up an old school Delta Rockwell 17-600, 3-phase. Put a VFD on it and as others mentioned already, a VFD is a dream on a multi-use drill press. I use that almost exclusively for metal these days, but still change speed especially for large drills.
A couple years ago I picked up a small benchtop Delta old school drill press. Step pulleys, I use that for metal but only with small drills, less than 1/4" or so, and because of that I set the belts and leave it there.
Big thing? Metal. If you're drilling metal, watch your rpm. Drill wood at an undesireable speed you might get tearout or smoke the hole, burning the wood. Drill metal at an undesireable speed and you can overheat or even fracture the drill. "Drill" meaning the "bit", not the machine itself.
So the clear answer? You need several drill presses!