On the Nikon camera I use setting A, and run the Aperture from 5.6 to 7.8 to get a better depth of field. I started with the 3100 and its shutter speed is a little slow so a tripod is a must. A 60mm Macro lens will give you better coral pictures, and a look down box allows top down shots for the best coral color. Turn the white balance to the highest settings. You can find good deals on used lenses on E bay to pick up a Macro it will net you better pictures. My tanks are lit with T5 and It puts out a very even light that in my opinion makes taking pictures easier, if you are all LED it is much tougher to get the colors you are looking for that are natural. A little post editing can shift the colors to a more natural look, or give it that high end pop you see many vendors using
I use Aperture on the Mac, Photoshop is another good choice if you are PC. Post editing is a must unless you have mad camera skills. With the advice above take it with a grain of salt as it comes from a begginer. Play with settings and get a tripod for a start, I started with a N3100, but upgraded to a D7000 and found that I can take better pictures because of a much faster shutter speed. Here is a shot I took of my favorite coral at the moment. Rainbow Crush.
Play with settings and take many test photos and you will learn your equipment much quicker!
