I have never seen any evidence that they actually damaged any of my clams, but they make them close by irritating them by climbing around the shell picking at algae.
I removed most of mine just to be nice. (and, so I get to see my clams open into their full glory without intimidation!)