I have heard that hermits, worms and stuff will leave a clam alone as long as it is healthy. If it starts to die, it will be a race to get to it.
I have blue leg, red leg and hawaiian zebra leg hermits, and they don't ever mess with my clams.
You do have to be careful for pyram snails that can do severe damage to clams.