It's probably the drills themselves. What brand are they? 20 years ago I bought a 3/8 drill for work, think it was Irwin or VA. That put holes in more car bumpers than I could reasonably count. Every couple cars I'd give the shop foreman a dollar (less than a third what it cost new) to stick it in his DD. It's at least a good half inch shorter now than it was when I bought it but so long as you did your part when putting the drill in the chuck it cut as good as the day I peeled it out of the plastic, hell I probably still have it somewhere. However getting it seated properly in the chuck is critical. The DD is a fairly simple piece of equipment but not getting the drill seated properly in the chuck makes it totally worthless. Seriously go on youtube, watch a couple videos, and practice. It's not difficult to do nor is it difficult to mess it up. Make sure you do your part and the DD will do it's part.
Next find a set of Made in the USA drill bits. Look for Viking, Norseman, Chicago Latrobe, Cleveland, Atlas, Hanson, Ohio Drill & Tool to name a few. Make sure they are actually made in the US and don't just have "USA" or "America" in the company/product name. Most drill bits you find today are made in China/India/Brazil/Mexico from whatever metal they got a deal on last week and they might be heat treated, somewhat . . . sort of, maybe. Big box stores generally carry this cheap garbage.
My boss at work wanted to put a hole in some stainless tubing, had a brand new still in the package VA drill bit that he'd managed to make a dimple with after good minute+ of drilling. I went home, snagged my Hanson set I've had for years and zipped a hole right where he wanted it in seconds. The next week I found a great deal on a 10pc Viking set and ordered it for him. Some things it just doesn't pay to save a buck on. You can spend a pretty penny on drill bits, but you buy one quality set and you should rarely need to buy another drill bit again.