Rip it out now!!! I had a tree growing in the middle of my driveway for many years and it caused all of the previously mentioned problems. Yes, you will hit it, graze it, scuff it, and cuss it. Trust me! You can plant a new one somewhere else if it makes you feel better but please kill that tree!