My SO and I fell in love with a house, but ended up not buying it because there was a lot of land behind it that could have been used one day for a house lot--there was a "paper street" that ran between the house we liked and a next door neighbor... that would be the rear lot driveway, it would pass only about 10' from the living room and rear 3-season room of the house we liked... what was a backyard w/ a lovely view of trees could have become a view of someone else's front yard.
Both my realtor, the selling realtor and the seller all tried to convince me that nothing would ever be built back there, but the fact was that there was no zoning reason that the lot could not be used... I went to town hall and discussed it w/ the town engineer and some building inspectors, all agreed, the paper street and lot *could* one day be developed. That land was town owned, so I even inquired about buying it, they said it was not for sale at this time.
...so we passed on the house... Now, maybe I'll drive by in 20 years, see nothing was ever built, and kick myself... but either way I can sleep at night because I know that I headed off a potentially bad situation.
The neighbor behind you didn't perform due diligence... he didn't understand the lot layouts, he didn't understand the setback and didn't conceive of anyone building in front of his house. 100% his fault, not yours.
To those that say karma will bite you, maybe so, but the fact is that the neighbor was assuming no one would ever build there, it was his gamble (or ignorance) and he lost... perhaps karama is biting him for some past offense.
Enjoy your garage, smile at the neighbor, try to keep your property clean and quiet as part of being a good neighbor.