First up, make sure you're using a somewhat decent material, for the excercise aluminium will probably do, as it's easy to achieve a good finish, and that might help pick where your problem is.
Make sure you have minimal stick out of the work piece, 2xdiameter is a general rule of thumb, unless...