Probably sponge.
You can scrape them off with a sharp fine knife.
Those polyps are pretty strong and I don't think that the sponge could harm them, but it doesn't look nice when there is a whole in the middle.
Just take the sponges off. Not a big deal.
Protect yourself against the palytoxin!
Goggles, gloves and close your mouth!!!!!
Grandis.