I may have approached this from a more advanced user standpoint, you and Seber both bring up good points.
If we are talking Tiffany townhome owner or Dave the duplex guy, yeah a impact might be a better solution. Definitely better at driving 3" drywall screws into a 4x4 to make that sweet shipping pallet coffee table
BUT.... if we are talking serious DIY, a drill is still a better more versatile tool.
hex bits more precise then a twist bit, c'mon, get out.
Sure the bit may be precise, but please show me who is making a zero tolerance non wobble hex chuck. Its the chuck thats the problem, not the bits.
Plus you are limited to hex shank for everything, counter sinks, self centers, large diameter twist bits, hole cutters, plug cutters. Good luck finding a USA made 82* zero flute 1/2" counter sink HEX drive at your local outfit
Perhaps im just using more stuff then you?
I think we're not communicating well (much easier to have happen in a forum vs. over a beer). Here's my attempt at a breakdown (apologies in advance for the length):
tl;dr Overall Point: At their primary tasks, each is non-trivially better than the other. For the secondary tasks, the impact is a far better (basic) drill than the drill is a driver.
Long-form
A drill is superior for drilling tasks, especially heavy-duty type stuff (incl. large-diameter twist bits), precision holes or things like counter-sinks, etc.). That kind of thing is exactly why I have a drill. (M18 brushless-but-not-FUEL)
A drill is absolutely inferior in every way to a good, modern impact for fasteners. The chuck works to mitigate some of the downsides, but it is a band-aid over sucking chest wound. It's never going to do the job well, just well-enough. This is where I was talking about precision - I agree hole-making precision is an easy win for the drill.
Net: A drill is made for drilling. It's functional for fasteners, but a shadow of what a good impact is.
A modern impact has modes that sense the state of the fastener. It doesn't need a chuck. It will drive fasteners that give drills fits or requires pre-drilling. It goes from constant rotation to impacting automatically and with the correct amount of force, stopping when the fastener is done (at least if you're in the right mode). No cam-out or other stupidness that leads to a damaged fastener or workpiece. It also has repeatability you just can't get with a drill when it comes to driving. A story: one of the early tasks for my M12 Surge was a bunch of decking. I'd started the day before. My old C3 impact was working on vintage batteries and between that and the noise/vibration, I gave up on it and switched to my drill. Despite being a shiny-new M18 brushless, it wasn't fun. I went that night and got the Surge, and the next day was awesome - fast and repeatable. I thought I wasn't going to finish until late afternoon, but ran through the rest of the work really fast. (Beyond that, the SURGE is really quiet and far smoother/low vibration than any impact I've ever used). I've used it to assemble all kids of stuff from work tables to IKEA stuff and it did a great job of the kinda-delicate task of swapping out all of the cabinet hinges in our kitchen (the screws are skinny and easy to cam out or snap). The impact driver made all of that just seamless. There are zero situations I've experienced or can think of for fasteners where a drill is better than my Surge. That probably wasn't true of my C3 impact - I can't even remember if it had modes, etc. - but it was such a buzzy vibration machine that I mainly used it for tougher tasks and used a drill for the rest. But that ended the day the Surge arrived. I haven't used other impacts, but I'm sure the DeWalt/Makita/RIGID/Ryobi/etc. products can be similarly effective because they have the modes and modern circuitry to do the same things mine does (though I don't think anything is doing it as quietly or gently, at least based on what I've seen). I haven't had any issues at all getting hex bits to do anything I'm using the tool for, nor have I had any fail/twist/whatever. But I do the homework and invest in good stuff. I'd also note that I've never used my drill for automotive stuff (other than drilling out bolts/etc.) and the Impact sees a fair bit of use there too.
For drilling, an impact driver can be fine for light-to-medium stuff. I've used it for installing shelves and variety of other basic jobs. I'd switch to a drill for spade/forstner/countersink/etc. bits, but I consider myself a moderately-advanced DIYer and those aren't things I'm doing frequently.
Net: An impact driver is made for driving. And it does a fine job for basic drilling.
Outside of better and worse individual products (e.g. Brand X makes a great drill but a crappy impact) that might shape perspectives, I can't see much room here outside of sticking to what you've always done or experience with old impacts that weren't good at much beyond difficult fasteners, but I'm lucky to have excellent examples of both products. I've also never been a "I've always done it that way" sort of guy - I'm happy to embrace change that improves.
I'd guess that hex bits beyond basic Phillips/torx/etc. bits had a rough start, but these days there is no lack of availability for high-quality bits for just about any task.
Overall (and last) Point: For the basic DIY homeowner asking what the best tool is to have if they're just going to have one, the answer is an impact. The tasks faced these days for most are a lot more fasteners than holes, and the need for holes is generally simple enough than an impact will be 100% fine. That describes my oldest in the first apartment exactly (and most basic DIY homeowners), so I put a Surge in the apartment-warming tool kit. Beyond that basic, one-tool scenario (and pretty much everyone on GJ is beyond that) lies the real answer: Both. If you only have an impact and you're eyeballing forstner or hole bits, add a drill to your kit.
(I'll shut up now)