#4 would be good for a 100 amp service, but your right, to run 100 amp from your house panel it would have to be #2 copper.
Three #2 THHN or THWN copper conductors, two hot and one neutral with a #8 equipment grounding conductor.
1 1/4 PVC to run them in.
If you run the grounding conductor...