I don't know what kind of hermits those are, but they're cool.
Can you get a not-blurry pic of the thing? It's either a baby hairy squat lobster (actually a very small species of crab, harmless) or a gall crab. A gall crab burrows into the skeleton of a coral and hides there. It isn't usually harmful to the coral, though it can occasionally irritate the coral slightly by existing. Either way, it's not a bad thing, in fact it's something interesting.
There's a tiny chance it's a baby something else, so do get a better picture, but in the meantime don't kill it.