If you bolt it down, a rubber pad underneath along with another rubber pad between the leg and the nut. Think like a shock absorber stem mount with isolation on top as well as the bottom.
The relief valve on my gas water heater goes into the outside wall and exits on the sidewalk along side the garage. It has been that way since the house was built over 20 years ago.
Since your shaft bearing plates are bolted to the horizontal rail instead of being riveted, you should be fine removing it. I would add a mounting plate to attach the bearing plate to the vertical mount just to prevent it from twisting.