So. The hole saw itself is threaded and screws onto the arbor and is located with a couple of pins which locate in the top flange. The movement comes because the threaded section doesnt snug up tight because the primary retention mechanism is the pins.
Im only drilling at 90degs so I am just clamping the tube which is fine.
The fundamental design leads me to think that its very unlikely to be a rigid mount (the hole saw size can be changed using a single arbor).
Ive got a few more to do and I have done a few already but I was wondering how tight the other brands sit.
That movement is movement in the same line of travel that the drill bit spins (in other words, it is not a wobble). All hole saws I have ever used don't snug flush when using the pins. The pins, keeping it off of 'snug,' will help you get the bit off when you are done. Because of the pins you aren't over-tight and the spinning bit isn't getting tighter and tighter on the arbor as you drill. You can 'lock' that blade on that arbor easily without pins on the big holes.
Your results will be fine.
No part of this discussion matters for notching tube or pipe with a hole saw though. The act itself only requires some precision on the cut angle. A little wobble at the end of blade will have little to no effect on your notch. And if it was off a little, all would be corrected at fit-up and tack.
Looking at your picture above, If you want a better way to snug round stock in a mill or drill press vise, get a small V-block and lay it on its side in the vise with the tubing in the V. Then clamp the vise shut with one jaw on bottom of v-block and the other on the tubing. No chance of spin. Also works to hold round stock straight up in the vise. A good investment for the shop
BTW, I think the new Milwaukee Ice Hardened bimetal hole saws that they carry at HD ****. No bueno.