I actually asked a similar question on reining in my neon GSP. I got some advice that I did. He told me to use a scraper to get as much off as I like, and then use a wire brush.
So I took my barbecue wire brush, placed the large rock in a water bucket and scrubbed for about 10 minutes. I still see the remnants in areas that I tried to get rid of, but I am unsure if it will come back in that area. Until it spreads from the "wanted area".
However, based on this thread, I am contemplating getting rid of the whole thing. Presently it is on the top quarter of a large live rock, so I guess that's it, I got to get rid of it.