I've had (for what it's worth) different kinds of hermits, plus the occasional bristle worm, and never had issues with any of my clams.
Remember, these creatures can "sense" when a clam is sick, so unless it's in good health, you may find them attacked and eating the clam, which may look like it's aggression, when it's just Nature in action.