My house currently is running on 100AMP service. I have inquired about upgrading to 200AMP service but the cost is frightening at the moment.
How "frightening"?
Much depends on the local utility (and bureaucratic climate); but typically, the PoCo will upgrade the feeder, meter, etc., gratis, as long as they can see a legitimate "need" for the higher-capacity service; and in this day and age, it doesn't take much to justify 200A. That would leave just the breaker panel itself, meter pan, some conduit & other misc. hardware, and of course the labor, on your nickel.
I am looking at buying a compressor and a heater + ac for the garage.
All of which are, as you've surely noted, rather high-load devices.
The compressor requires a 30AMP breaker @ 240V. Runs at about 22AMPs
The heater requires a 40AMP breaker @ 240V. Runs at 35 or so amps.
Not to mention my polishers and lights (I detail cars)...
I was originally recommended to run a 60AMP subpanel to the garage but after looking at my usage I dont really know if that'll work.
It would be an "edge case", at best.
So do I run a 100AMP sub from my 100 AMP box?
I'd probably go for 90A (which is the limit for AL 2-2-2-4 MHF, which in turn is likely to be about your most cost-effective feeder cable. You didn't mention whether this garage is attached or detached; but especially if the latter, this is likely your only really practical (again, read that as "cost-effective") choice.
The compressor isn't on all the time.. But if it comes on while the heater is running will it trip right away?
You mean if they're both fed from a common 60A breaker (which in turn feeds the garage sub-panel)? It certainly wouldn't surprise me, particularly if you had other misc. loads (lights, stereo system, electric hand tools, etc.) running at that moment. Like I said above: "Edge Case."
As a secondary issue... Even if you were to go with a 90/100A sub-panel, thus providing more than adequate electrical capacity to the garage itself, there's still the matter of the 100A service to the home as a whole. AFAIK, there is no rule which states you can't run a 100A sub-panel off a 100A main panel; but the basic arithmetic still stands: If the various "stuff" in the garage is drawing, say, 60-80A, that doesn't leave all that much capacity left over to run the house. Do you perchance have an electric range or clothes dryer? Is the house air-conditioned? Can you imagine yourself working in the garage while your wife is cooking Dinner and/or doing the laundry? Are you starting to appreciate the problem?
How do I not blow up my house?
That's one thing you really SHOULDN'T need to worry about, presuming the installation of the sub-panel (and the service upgrade, if you also do that) is done properly. But with only a 60A feeder, you CAN look forward to a lot of nuisance trips.
Switch to a natural gas or propane heater
That would help somewhat, at least during heating season. But it would NOT be the whole solution, particularly as Summer rolls around and that Air Conditioner he mentioned starts getting active.