I have a 5 Gallon Hex tank that I turned into a reef tank. It works just fine, and I really like the look of the hex tanks.
The AC 30 will be fine. I run an AC 20 on the back of my 5 gallon full blast and the sand doesn't move at all. Just slide the filter so the output is all the way to the corner so you will have some deflection off the front corner and not just current running straight down the front pane to the sand.
I think you might be better served with two pouches of SeaChem Purigen in the filter rather than Chaeto and live sand. It's a bit small for all of that and the modding and lighting will add to the expense. The Purigen will remove the nitrogenous wastes and they can be cleaned by soaking them in a 50/50 mix of water and bleach overnight, then rinsing them and soaking them overnight again with a few drops of chlorine remover. Will save you a ton of money in the long run. I bought 3 pouches and swap one out every two weeks.
In my 15 High I just stuff a ball of chaeto behind a rock with good lighting and flow. It grows well and is doing it's thing as well as if it was in a refugium without the expense or hassle.
So you have the tank- very cool tank BTW. You have the circulation and the filtration. (You should be fine just buying one pouch of Purigen to use at first, then the other two every two weeks to spread out the expense.
As far as sand goes... Do you have any in the 55 tank that you already have established? If so, just scoop enough out of it for about a half inch deep sand bed, just enough to show slightly above the bottom frame. That would be great to use sand that is already live from your other tank, unless you have ever used any copper based medications. Then you might want to take a sample of your water from the 55 to your LFS and ask them to test it for any traces of copper.
Same thing with the live rock. Are there any smallish, intricate and interesting pieces that you could easily remove and use in this tank. I'd look for one that is more long and narrow than flat. Most of the corals that you can put in this tank will come attached to rocks that are rather small and flat, i.e. zoos, polyps, and mushrooms. So you don't need to fill it originally or there wont be any room for the coral covered rocks that you will add later. You just need a central piece or two to give some verticalness to your aquascape.
My 5 is so small that I looked for a single piece of Tonga branch that had a good height and an open, arched shape. I placed it back from dead center to give a little depth with a sand bed in front of it, and it hides the heater, temp senser and the intake of the AC 20.
You do have a number of options for lighting. None of them are too expensive. I have the Coralife dual 9 watt fixture over my 5 Hex. It is a solid little light. It is 12" from the light to my sand bed and the small frags of Dragon Eye Zoos that I have on the sandbed expand out flat, grow and are producing new polyps- a sign that they are happy with the light and they are in the dimmest area of the tank. Any other frags that need a bit more light get moved up onto the rock closer to the light. This will take some experimentation. It's a little over $30 and replacement bulbs are about $10 each.
When I bought mine I told the guy at the LFS what I had in mind; and that I would need a piece of eggcrate light diffuser to cover the top of the tank, so I could set the light directly on it. He threw in a piece that I cut down to size, thus saving myself about $10 at Lowes for a 2' by 4' sheet that I didn't need. It looks pretty cool, will keep your fish from jumping out, will keep your cat out, and it will allow for great air flow to keep the tank and light cooler and the water oxygenated. The light has an acrylic splash guard and is well sealed, so you shouldn't have to worry about any moisture problems unles you touch it with salty fingers.
"I'll buy this light if I can get a piece of eggcrate to set it on over my tank."
Your other option might be one of the new Perchlights or gooseneck lights by Red Sea. They are both power compacts and clip to the side of your tank. You could use one or two.
Or you might put two actinic bulbs into the Coralife Mini and a 50/50 into either gooseneck lights. Then you can double your lighting and do dusk/dawn simulations as well. Maybe something that you work towards.
The Super Bling-Bling lighting option might be a custom canopy with a PC retrofit kit. Or an HQI pendant. Serious overkill and major bank busters.
I'm with ya! Hex tanks make cool nano reefs.
One more thing: be sure to paint one side of it either black or dark blue before you start. It will help to hide your heater, and you don't want to be able to see the HOB filter thru the tank. My preference would be black to hide the 50 Watt Visitherm Stealth you might need to add this winter. My light and AC 20 add only one degree above ambient on my 5 Hex with the open top.
Reef On!