Shoot RAW as Misled said.
-----------------------------------------------------------------------------------------
I'm guessing by your post that you're new or somewhat to DSLR's.
The D300, while not a new model, is still a capable camera for things like macro shooting.
Some things you might try -
1) turn off/cover all other light sources other than the tank lights (especially fluorecent lights).
2) Use a tripod
3) Minimum Focus distance - Make sure you're not closer than the min focus distance of the lens.
Eg. - are you shooting 8" away from the tank but the min focus distance for the lens you are using is 2 feet?
4)Shoot RAW - you can correct white balance with a button click afterwards (too blue)
5) What is your depth of field? This is how much of what your looking at is in focus. How much is in focus is governed by distance from the subject and how much of the lens iris is opened ....or... closed (refered to as "f stop" in camera lingo)-
http://www.dofmaster.com/dofjs.html - you might try this link and plug in your camera and lens to see
6) Shutter speed - is the camera snapping the picture fast enough to freeze motion?
---------------------------------------------------------------------------------------------------------------------------
These are just few simple things you can look at and adjust but keep in mind that whole books are available about Macro photography.
Picture taking is a worthwhile and rewarding hobby. Don't be put off by jargon or "bad" pictures along the way.