Not right he would charge you for that time?
Any time at site is billed time. I work in consulting Engineering, and all my time, including travel, is billable. If I have to go to your location and you want to chit chat, you're paying about $5.50 a minute for that. If you call me about the file, that time gets billed as well. That's industry standard.
I can't fathom why it wouldn't be right for him to charge that time, he's on your site, it's your responsibility to control access.
If he makes a business arrangement with the neighbor to see him about something during your site visit, then that chat time should be charged to the neighbor. However yes, it's totally reasonable for the engineer to charge time on your site to you, whether or not someone else is wasting your time.