That and you'd have to check with you county setback lines. A structure can only come so far forward, back or sideways according to the county.
So say the setback says 30 feet from sidewalk end, and you encroach and only give it 20 feet, they may make you tear it down.
I had a similar problem...