While I think the 3' variance would go through without any issues, getting the new garage built within 2.5 feet of the property line might be a harder press of the variance. Having an elevation difference of only 2-3' is nothing and would easily pass, as long as the materials used for construction, along with the initial design, blend in with the existing structure.
What the OP missed out on was the value of extra privacy being add to his backyard and rear deck. Or, perhaps, even more, a hidden reason for the neighbors to want to build a buffer between his backyard and theirs due to the OP's potentially offensive use of his deck and backyard. In another of our homes, we ended up with a privacy feature, as the neighbors backyard was used for a dumping ground for debris from his landscaping business, making our time on our newly built deck not enjoyable at all. Same went for their children, who we loved dearly, but seemed to lack attention from the parents, and also got dumped into their backyard, not allowing us some quiet time as they sought our attention, or the attention of our company, in a constant manner whenever we were outside.