Hmmmm. From what you describe then I'm leaning towards YES - you have a problem with amphipods.
They are more of a pain on the eyes then anything else, but You definitely DON'T want them on your corals!
If so, then a goby or two might be in order! They love to munch on them!