I had the same problem with some hairy rhodactis, They kill my sps when they touch very aggressive little beasties.
I try to pick them off with a fingernail. that seems to work the best, for they also get into those little nooks and crannies. If you cut them up with a razor blade, all those little pieces will grow into one....
Also a toothbrush works to get the rest of the crap off....But you have to take the rock out and scrub it....
It took me a few days too.... I still have them, but they are under control with the low nutrient sps tank....
When I had higher nutrients, my tank exploded with xenia and mushrooms.... scary.