Use the drill for drill bits, wire wheel attachments, mixing paddles, etc. The driver is for driving in lags/screws, and removing them. Dont leave home without it! ;-)
I'm going to add that if the drill you have has a torque limiting clutch setting, that's also useful for driving screws. Adjust the setting until you set the screw head flush or at whatever level you like, and then all the rest of your screws will be at a similar depth. (Variations within your wood piece does cause some variability). I'll usually use the drill to drive screws rather than the impact when I don't want to split wood, or when using pocket holes.
On the other hand, any time you're putting a screw in that requires a lot of torque, the impact is the way to go. When the drill is set on full drill mode, if you're not careful, the torque can surprise you and pull the drill out of your hands.
Link has some guy showing the difference:
