IMO, not if you just have the standard issue blue-legged and scarlet hermits. I have many blue legged hermits, the ones most notorius for being agressive and they don't do anything but clean the shell of algea and detritus.
It's possible for any hermit to take advantage of a clam that's on the verge of death, but they wouldn't be the cause, unless you have some other type of hermits that could be a problem. Did you ever see hermits bothering the clam before it died?