I found a couple of my pictures that will help. These were from a refurb I completed last November.
The photo below is the as found condition. Notice that the pulley shaft is recessed into the pulley probably 3/32".
During reassembly, the pulley shaft would not go much higher into the pulley than before. There was still vertical movement like you're experiencing. Upon further investigation, I noticed a slight ridge at the top of the pulley bore. This ridge prevented the needed vertical adjustment of the shaft.
In the photo below, you may be able to see the bevel at the top of the pulley bore I cut with a large countersink bit. I removed that ridge and was the able to eliminate that vertical slop. Notice that the top of the pulley shaft now protrudes through the pulley. I also filed small grooves in the top of the pulley shaft to mark the location of the centers of the flats. I also marked the locations of the set screws on the pulley with a sharpie. Alignment was a no-brainer.
