I used to use a brush i bought at a shoe store. The handle had curved ends that fit around your fingers.
The best way to clean it off your rocks is to kill it. A refugium lit 24/7 with lots of macro algae, good flow, a turkey baster to blow off the rocks, and a light feeding schedual will make it so you never have algae again. I haven't had one spec in over 3 years.