A 4" holesaw can work well if you can run it slowly. As mentioned above, there is a risk of things going pear-shaped when the periphery makes contact. Things can also get wobbly as the pilot drill gets pushed around and shaves off the inside of the pilot hole. Expect to end up with an oversized hole with a pretty rough edge. Whenever possible, I use the holesaw to cut a hole through some scrap material (usually 1/2" MDF), clamp the scrap to the hole location, remove the pilot drill and cut the hole using the scrap as an external "pilot".
The first time I did this was because I needed a disk of stainless steel without a center hole. It worked so well that I've used the method whenever a fairly tidy holesawed hole is needed.
Some folk drill the 1/4" pilot hole and then use a piece of 1/4" drill rod as the pilot. I find this slightly better than just using the pilot drill, but not as good as the clamp-on-the predrilled scrap method.
The carbide-toothed holesaws, whether the SM-TCT range from Starret or the remarkably-similar-looking cheap ones from ebay or Amazon, are pretty good: I find I can get a much more accurately-sized hole with them and they cope with stainless steel better than the normal holesaws. However, they are intended for sheet metal so the depth of cut is very limited and there is usually a larger diameter ridge (safety stop collar) which means the 1/2" MDF is too thick. For these, I use thinner scrap. I've used them to 95mm (3 3/4") through 6mm (1/4") stainless steel in a drill press and to 2" through 1/16" steel with a cordless drill.
The Starrett CDT-TCT range has a 1" depth-of-cut and there are also very-similar-looking ones available elsewhere. The cheapies have performed fine for me , as have the Starrett ones, but I've not used any above about 32mm (1 1/4"). That was with a cordless drill.
