That brown your seeing might be some type of sponge or tunicate. In most cases these are harmless, but there are some varieties that can smother your corals. I wouldn't worry about it, but if you want it gone you might want to place the coral in a small bowl of water and use a stiff brush to remove it.
Btw what you have there is Sympodium. Clove polyps is just a generic name given to MANY different kinds of corals that look similar.