You can see some side effects from the palytoxin, but the worst cases I've heard have been people getting blood poisoning (assuming it came from something on the zoas they were fragging). Their hands might swell up and get white streaks leading towards the shoulder. The worst I've ever gotten is a numb arm and a dizzy feeling (two occasions after hours of fragging). It will enter through a small cut or mucous membrane, typically, so people recommend using gloves and protective eyeware. I've not heard of anyone getting sick just from touching and moving them around. You do want to wash your hands before touching your mouth after being in the tank, but that is a given. Under normal circumstances, I'd say your likely going to be fine.
On related note, it turns out that some people have tried to rid their aquariums of palys and have had major coral loses as a result. That said they can have an effect if you were to take a wire brush and scrub away to remove them from your tank (not sure who ever thought that was a good idea...). Seemed like common sense not to scrub animals out in display water, but that goes to show that under extremes, they can be dangerous.
In the recent Coral magazine, there was mention of there only needing to be one millionth of a gram of palytoxin introduced to the bloodstream to kill a 200-220 pound person. I'd love to know where that information came from, as all the info I've seen shows (if memory serves) that it takes more than that to kill a mouse. If that information were even remotely correct, I think we would have seen hundreds of zoa related deaths in America, given tha mount of interaction with these animals here. Apparently, someone with an agenda and publishing abilities is using scare tactics, as he has in the past...
All that said, palytoxin is very dangerous, but under normal circumstances, you are not likely to have issues (unless you happen to be allergic to it, etc.). Use good hygiene, and you should be fine. Wash after putting hands in the tank/touching corals and use gloves and safety glasses/goggles and you will greatly reduce any risk. I wash after having hands in the tank, but have never used gloves. I'll be one to learn the hard way...