What's the plan to cover the walls ? Do you plan on heating it and maintaining a temp duringthe winter ?
If so, I'd consider furring out the wall to match a furring strip on the concrete stem wall. Put some 3/4" foam insulation between the furring strips and cover it but don't go all the way down to the floor, give yourself 4-6".
If you're not going to heat it except when you are out there, I wouldn't worry about covering it. But, you could still do it and plan for the future.
As far as losing heat when the door is opened for 5-10 minutes, it doesn't matter much. The heat (or any temparature in the concrete) will make the interior space recover quickly once the door is closed again. If you heat and maintain a temp, the floor will act as a big heat sink and help retain the heat, that's why I would cover the stem wall. If you don't, the concrete exposed to the exterior cold, will draw the heat right out.