Set the poles, then fill. That way you know that your poles are set in good soil, not fill. If the filling and grading is done with any level of care at all, there will be no danger of the perma columns shifting, so long as they are set in the proper depth of undisturbed soil. I built a 20'...