Technically, you should really wear gloves. There are lots of corals that have toxins and you never know which you might have an allergic reaction to. Also, say you slightly scratch your skin on a piece of live rock, well, there is lots of bacteria in the water that could cause you to get really sick. It has happened to several reefers. Yes, you should wear gloves, but the reality of it is, most don't.
I tried wearing gloves, I couldn't get a good grip on things and water always got into my gloves.
Amanda