So I have been to the ER with palytoxin poisoning.
I wouldn't keep what you have in my house if you paid me. Those are the same type or similar that hit me. I was cleaning a part of my overflow they had grown on and made a bad judgement call. For some stupid reason I used hot water to clean the overflow wier. The effect was rapid and I was in the er within short order. I'll spare the sob story but palytoxin is not something to take lightly.
That said, not all of them have the toxin, but they all do have the ability to contain it.
In regards to how fast acting it is, the biggest sign you have been hit is common with many poisons. You taste aluminum in your mouth. At this point you need to seek help immediately regardless of your exposure level.
I was lucky, I was only in the ER for 18 hours which is how long they wanted to hold me. The longest time to death recorded after exposure was at 18 hours from sudden heart failure. There is no antitoxin, they can only treat the symptoms.