One son had a house with a slab from a former garage (long gone), and he couldn't build on all of it because it is too close to property lines.
I bought a south Florida CBS single-story house, the cheapest house on the block, and for blocks around. It was > 50 years old then, built between WW I & WW II. Now, if it was still standing, it would be nearly 100 years old. There was a one-car wood garage at the back alley, and in violation of current code for setbacks. A previous owner knocked it down, just before I bought the house. I never saw the garage, but neighbors said it was in poor shape, and the slab was too-low, after heavy rain, the slab would become submerged, so unfit for use. After I bought the house, I contracted to have the slab demo'ed, and the hole from it filled and graded.
In many jurisdictions, 'being grandfathered-in' does not mean you can raze the existing structure and replace it with something new on the old setbacks. The liberal interpretation would be that you can make repairs to an existing non-conforming wall, but you cannot remove it and use the non-conforming setbacks to re-build. You could attempt to obtain a variance for the parts of the structure in the setbacks, in rebuilding, if you removed the non-conforming structure but such work is usually referred-to as a 'self-imposed hardship,' and is grounds for disallowing a variance request.
We demo'ed an existing attached 2-car garage and had it rebuilt on a new floorplan, meeting current setbacks of the time. It's entirely CBS or solid poured concrete, and would be a safe space in the event of a hurricane. It's designed to withstand 180 mph wind gusts. The renovation was an addition, designed with a loft for heavy storage, like motorcycle engines and parts. It has clear-span rafters, not trusses, though it has a low ceiling to keep costs down and the loft is considered 'not-habitable space.' A Munchkin could fit in there, I suppose, but the only means of access is a ladder.
I am now retired, but was a FL licensed plans examiner, a planner, and a lifesafety inspector.