I would have simply said I don't loan tools out because way too many people are idiots.
Yea, IMO, you don't have to be super blunt about it and cause a family issue, but IMO the fact that he presumed you were willing kind of tells me that he doesn't respect tools the way that people here, or people who make their living with them do.
I didn't respect my dad's tools as a teenager and I think karma (not from my dad, he is not like that, just tool karma) is coming back to bite me now. One of my employees is allowed to work on his stuff in my shop after hours and separately, our landlord who I am friends with who works on his stuff in our shared space both use my tools when they need something they don't have. Which I don't mind in concept but I have to lecture them both that it's like camping, there should be no trace left that you were here. I keep certain tools in my cart and some in my box and other specialty stuff on racks/shelves (only "tech" in the shop) and invariably after they use something, it's gone, or in the wrong place or whatever. Fu#*$* me, I hate reaching for something that should be in a place and having to stop and find it even though I don't work any kind of flat rate or even in a hurry.
THe landlord borrowed a bunch of swivel 13mm stuff last week trying to get some torque converter bolts out (had to pull engine from kids 4.0 SOHC explorer to fix timing chains) and he proudly left a pile of returned stuff on top of my box when I came in the next day. Except it had several of his sockets in the stack, one of his swivels and not my 13mm short impact swivel.
Both great people and certainly don't do it on purpose, but the employee is very young (21) and neither have had to make a living with tools and as such, learned to respect it in the **** way that we do here.
DaveW