Some zoas and palys (very little is known how much and which ones aside from Protopalythoa toxica) contain palytoxin. Some gorgonians contain small amounts, too. Leathers don't contain palytoxin. They do contain other nasties that can cause issues, if you are sensitive to it (each person is different).
If you use common sense and wash hands before putting your hands in your mouth/on food, etc., you will be fine. That's a good rule to adopt for contact with any animal, esp. anything in our aquariums. With so many unknown bacteria and unknown counteractions with the numerous bacteria, it's always better safe than sorry.
Many have said that they had palytoxin poisoning, but the fact of the matter is that if that were true, they'd most likely be dead... The research the Army did on palytoxin as a weapon showed that when subjected to it, the animal's systems shut down and death occurred somewhat rapidly. However, since so little was known about where to find the toxin and since generally so little is present (if at all) in most zoas, the experiments were dropped due to infeasibility of isolating enough of it for practical purposes of weapon use.