Speaking from several decades of experience as an inside machinist (meaning I worked in a machine shop and set up, operated, and programmed machine tools, thereby making thousands of machined parts) the above-pictured tool or any of similar type is the best choice for deburring holes. It will also work on edges, but a file is usually a better choice for that type of deburring.
Note that the above tool takes a bit of getting used to. The blades are a bit fragile and will snap off if you aren't careful. A sharp blade and a light touch will usually do the job.
As for tapped holes the correct procedure is to centerdrill/spotdrill the hole to give the drill a place to start cutting. Next is to drill the hole. Then countersink the hole. That will also remove the burr around the hole if the countersink is sharp. Lastly, tap the hole. All this is usually best done on a drill press (or other machine tool) but it is possible to do it with a powered drill. The tap should be driven with a T-handle tap wrench if possible to minimize side deflection that will likely break the tap, etc.
As an item of miscellaneous info kerosene is the lubricant of choice when drilling/tapping aluminum if available. There are specialized cutting fluids available as well. In the end I usually just give the tool and/or workpiece a shot of WD-40. Works on steel and just about everything else.