I personally have been keeping and handling sea anemone's for about 28 years now.
I've for the most part not had any real issues all these years w/ the exception of two instances, one was my H. Crispa I had for 5.5 years, never an issue until one time it wove itself into my rockwork, and I had to get a little firm in trying to get it back where I really wanted it to be and away from PH's.
That day I did show red dots on my forearm for the first time after years of handling it or working around it.
Recently I had a fairly bad reaction to a Hadonni, a species I had never kept before.
Now that was more severe, headache, nausea, and w/ very little contact.
So I think it not only varies depending on species, but also your own personal body reaction, and very probably how much that animal may feel disturbed or threatened in how it was handled.
We all should probably be more careful in handling them as well as many other corals we keep.