truthfully, you probably will never remove it all. best bet is to take the rock it is on and leave it outside in the sun for a week or so and then scrub off any remains.
i've tried rubbing it out with my thumb, ripping it out, etc. if you leave a tiny sliver of mat on the rock, they'll grow back. goodluck, man!