If it is growing between the polyps, and not on them, it could be a type of sponge, which is very common and shouldn't hurt them. If the polyps seem irritated or are deteriorating, then its not likely a sponge.
As Reef Junke stated, A picture would be best to give you positive ID and help solve it if its a problem.