I had this very problem. First with a couple of Droids, then with my first two iPhones. Got a third iPhone within the first year of switching over because of this issue.
Yes, it is metal dust getting into the speaker where you listen. Speaker=magnet.
I got a Lifeproof case (found them cheaper...