How long have you had the coral? I would suspect that the the culpret is brown jelly disease. It is very common in fleshy LPS like Euphyllia(frogspawn, hammer, torch), Catalaphyllia(elegance), and Gonipora.
There is not much you can do to rescue the coral. Strong current sometimes will help but be carefull as it is contageous and may spread to other corals.
The worms living on the underside are most like just inocent bistanders. Unless you have actually caught them red handed.