Trial and error works if you have some basic guidelines. Some good tips have been posted in this forum if you look around. Read the sticky posts:
- Use a tripod
- Take the photo perpendicular to glass/acrylic. Any angle distorts the image and it will not focus properly
- Manually focus so you can sharpen exactly what you are trying to highlight
- Learn how aperture setting affects depth of field and shutter speed by practicing with various settings. Smaller aperture (higher number) will give you more depth of field but slower shutter speed so harder to get a really crisp image. I typically use Aperture fixed mode (I select aperture, camera auto selects shutter speed).
- Practice on corals before fish. Macro images of fish take a lot of time and effort and a bit of luck to get the fish stationary enough to snap a good pic. It's always a tradeoff between aperture/shutter speed and depth of field.
- Use at least basic post processing software to adjust brightness and contrast to optimal level. I don't adjust color because I am color blind so never know what to adjust.
- Learn how to under/overexpose on your camera. I usually underexpose by one stop or so since it is easier to adjust lighting on a slightly underexposed image afterwards with a photo editor.
- Use mirror lockup if you are using a DSLR. This reduces any camera vibration during the photo and sharpens the image.
- When you get the basics down and are looking for the highest quality, switch to raw mode. I am not there yet, so I still use high res JPEG since it is so quick and easy to take a few pictures and then look at them on my PC, then I can decide which are worthy of post processing.
If you already know all this, then as Titusvilesurfer suggests you keep practicing.