I don't know, maybe some people are more sensitive to it than others. Whenever I've been stuck (always a fingertip) I've had a reaction. It's not that bad - less than a bee sting - but it was definitely uncomfortable. A little swelling, redness, itch, and a little painful.
There's a debate about whether they've really got venom or whether it's just a physical insult (tiny fibers lodged in your skin). Whatever, I still avoid handling them and I'm careful when picking up things where they may be hiding.