A great point, Adam. I too got rid of my hermits. The acro crab was the only thing I kept, and I watched it like a hawk for 2 months before relaxing a bit.
I believe that there is far more we don't know than what we do, but critters (provided we have some control over them) should be given the benefit of the doubt, which requires they be kept under watchful eye.
By control, I refer to the ease of removal (crabs) vs. how difficult it would be to remove an unknown swimming worm that you may never see again.
Hitchers come with every piece of rock. Watch your stuff when you put new pieces in, and watch it at NIGHT too. Inventory your corals and know their health. Then you can determine if there is something really going on. There is risk in EVERY add you make to your tank. Luckily, this one can be managed easily, and maybe you got a new, free, cool critter.
I don't pay much attention to smooth vs. bumpy type descriptions. Those are just short-cuts we all sometimes use because we really don't know the answer.
I say, give the little dude a chance. Watch him like a hawk. See what happens.
This is a very fun part of the hobby.