I built a barn back in Ohio in the early 1990's.. clay seems to be the norm.
In answer to your question: Backfill with the material that came out when you dug the holes.
Think about this for a minute...
- If you've dug holes into clay soil for your columns, and it is truly clay, and then you fill the hole with gravel, it will be like a big plastic container full of gravel that will hold water. You've built a sump.... and that's not at all what you want
Back filling with whatever was there originally means it will drain the same as it did before you installed the columns.
Make sure the building is higher than the surrounding elevation, and make sure your finish grading has a slope away from the building. And be sure to add gutters and downspouts that shoot the water away from the building. Do that and it'll be fine.
You didn't ask about the concrete "cookie" concept in the bottom of the hole... but it is a good idea... it helps distribute the downward force over a larger area and minimizes the chance of the column sinking into the soil.