This is what mine looked like. Once you remove that screw in the spindle. You see there's a washer in the front you just have to rotate it and then you should be able to take the whole spindle out.
So I think the issue for me is the washer is too small. As it spins perfectly around and those washer tips do not make contact with the outside edges on the vise. When I first got it it was caked on with old paint and grease so that's probably what was holding it on in the first place.
Thanks everyone for replying. I just wanted to make sure the washer did not have to be flipped a certain way. That looks to not be the case here. I got the vise for $45. I thought you can't go wrong for that price.