I've done work myself on houses that we've owned; had permits for some, but not all jobs. I've also had contractors do jobs, and at least one told me that I could get a permit or not. They didn't care, said the work would pass inspection anyway.
What annoys me about permits is that our current town, and maybe others as well, bases the permit fee on the cost of the job. So if I wanted an expensive top quality siding job for 2x the price of a cheap job, the permit costs me twice as much. Makes no sense to me, since they're still just inspecting the same amount of siding work. Which they never inspected anyway, by the way.
We needed a permit to replace our asphalt driveway a few years ago since we were adding to its size. The town office said it would take 2-3 weeks to review the application & issue the permit, but if I wanted to leave a blank check with them so they could fill in the permit fee, they'd review it the next day & issue the permit. I didn't understand the rationale for that, but I did it.
I understand the need for permits to insure safety, proper functioning, etc., but I think towns abuse the rules & turn the office into a profit center.