A lot to unpack here.
Adverse possession is real, and if your building is on the neighbor's property this is a perfect example of it. To create a valid claim to permanent property rights requires notorious possession (such as a building or improvement) be on the property for an extended period of time. A quick Google search says in your state its 20 years. I wouldn't go that route, but if it was in your mind basically at this point you'd just wait to see if the neighbor came to you and said "You're on my property. Move it." If that was after 20 years from the day the carport was built and you can document it, you'd end up in court and you'd have a chance to win.
But...what I would do instead is spend the money to have the property surveyed.
While someone pointed out that GIS lines on the county property report can be off a lot. It's also true that old-time surveyors would often make errors when computing corners or reference points and those errors could add up from point to point until a fence is off quite a lot.
Now...if the fence is over on you, your neighbor could make the claim that the fence IS the property line regardless, because that was the agreed line and it is notorious.
I got into this for work several years ago. Had a remote property that my employer owned. The neighbor built a fence well onto that property. I got a surveyor and had it surveyed about 3 months after the fence showed up and the neighbor got all pissy about moving his fence. "You guys don't even need that ground!" My response was that there was a lot to my job, but giving away property the company owned wasn't something I was authorized to do. I gave him 2 weeks to take down the fence or we'd take it down for him.
If we'd have allowed that fence to stay up, eventually he could have claimed adverse possession.
The OP's story says that previous owners had a "handshake deal". What about that? That's no ADVERSE possession. A handshake deal can be cancelled by either party, or the successor property owner at any time.
Back to my story about getting the property surveyed. While the fence was on my employer's property, the actual corners weren't anywhere near where we thought they were forever before the survey. Turned out the city street wasn't centered in the right-of-way and our "layman" measurements that assumed it was were off by about 5 feet. It was nice to get some pins in the ground to give peace of mind and clarity.
That's where the OP should start. Hire a surveyor.
Final point...the other time this will come back to haunt you is when you sell. It's likely that it will be noted that your improvement is either off the property, in the setback or both. If that carport was mine and I was moving it? I'd move it inside the setback where it should have been put to begin with.
Final Final point...it just occurred to me that even if you were able to establish adverse possession, that would likely only go to the edges of the carport....so YOU'D STILL be in the setback. You could win an adverse possession case after spending 10's of thousands in legal fees and the neighbor could go to the county and make you move your carport off the setback anyway.
, some of this hand ringing is out of control.