My dad before had similar issues with a warehouse he had. What he did was remove the handle from the door, so all it had was the hole. He then covers the hole with a plug. In the event of an emergency that you have to use it, remove the plug and insert the handle back in.
Only downside if...