Usually these are in a sand bed because they make their tubes (castings) by gluing sand particles together. If you can move the casting without destroying it, then you are good. Otherwise...
I never had any corals close because of these worms, and I have a million of them. Perhaps it was just the corals got used to them?
Anyway, I would be more concerned about those white things in the upper right hand corner of the picture near the top of the zoa colony that look a bit like zoanthid-eating nudibranch tentacles. If you can remove the colony and check it for egg cases and/or adult nudibranchs, that may ease your mind a bit.