Impact sockets will work fine with hand tools, the opposite is true: Chrome sockets are harder than impacts, and cannot be safely used on impacts as they can shatter.
The drawback to using impacts as hand sockets are two fold: One is that they aren't as rust resistant, and seem to get grimier (chrome wipes clean easily) and they are slightly heavier/thicker so tight clearance areas can be a problem. The benefit is that they are stronger because of thickness, and are cheaper. A lot of guys that use an impact keep their costs down by buying impacts only, or impacts first before they get chrome. Others just buy impacts to get their acquisition costs down.
Adding a postscript: I personally use chrome sockets always unless I'm using an impact. I just prefer the lighter weight, thinner clearance, and easier to clean up. But, I'm in a different circumstance than some; I've acquired my tools over many years and don't have to worry about initial cost anymore. I already own both impact and chrome, so can pick from what I have, I don't have to consider what to buy.
