Help!: Apex failure = crash in progress

[entropygain]

I'd greatly appreciate the help of the community on managing a crash in progress.

240g new reef tank with corals and a few small fish transferred from old 140g reef tank transferred 2-3 weeks ago. All stable, perfect parameters for 2 weeks.
New system was upgraded to Apex system controls, including control for two heaters. The tank is on a backup power system in an office setting.

04:00 yesterday
--the power went off and generators kicked in.
--According to APEx history tank temp started to climb shortly thereafter.
--No alarms received.
--Tank reached 95F by 16:00 and climbed slowly to 96F

11:45 this morning the problem was discovered by our facilities manager.
--Apex was off-line (though might have been our routers) but apparently powered on and locked the heaters on.


12:00 today rushed to office
-- unplugged heaters
--added ice to sump, put fan on surface
-- temp back to 90 and falling

Preparing for water change and running carbon
Haven't had a chance to test parameters, but who cares if the temp is 95

Any other suggestions? Maybe completely cooked but would like to salvage what I can.

thanks!

 

[CAPT_Dave]

Oxygen solubility is inversely proportional to temperature. That means the warmer the water the less oxygen. Make sure you're aerating. If you have a skimmer that should do but if not direct your powerheads at the surface to get more churn.


Sent from my iPad using Tapatalk

 

[entropygain]

Dave - thx
Yes Skimmer + 2 power heads + return gives good aeration

I added zeolites in media bags to help blunt any ammonia spike, but don't know how well they might work. The water is somewhat cloudy so I presume bacterial bloom and skimmer is going nuts. Hopefully, this will limit the spike.

Temp down to 85 and falling.

 

[scuzy]

On your heaters do you have a secondary controller like a ranco or built in thermistor? This will prevent issues like this from happening. Also what's you apex statement for the heaters?

I run a ranco with apex as backup. The ranco controls the heaters and the Apex will shut the plug if tempgoes whacked.


Sent from my iPhone using Tapatalk Pro

 

[entropygain]

Thx Scuzy

I screwed up and didn't have a backup controller on the heaters. My bad! Let Murphy have an angle and he bites you in the butt.

but I still don't understand why the Apex didn't shut them down.

I don't understand what you mean by apex statement. Unfortunately, I didn't have the alarm configured properly, but it still should have shut off the outlet which was set for 78-79.5 with a fallback of OFF

 

[scuzy]

Ok wanted to see your fallback statement. That looks correct unless your probe want sending the right temperature reading.


Sent from my iPhone using Tapatalk Pro

 

[entropygain]

When I rebooted the Apex the history was horrible:

The tank looks terrible with huge sinularia shrunk and shedding slime, all my hammers completely retracted.

 

[ca1ore]

One thing to check is whether apex created a new address for the power bar. I had this happen with the new apex when the brain lost power. Upon restoration it had re-addressed the power bar and the heater outlets were set to ON rather than AUTO. I use EJ with built in thermostats so cannot cook the tank. I now run the apex brain off a UPS so it never loses power.

 

[entropygain]

Partial apologies to Apex.

Everything is on backup power which kicks in a few seconds after a power loss (but not un-interruptable). When the backup power kicked in, the power bar didn't restore. It should have.

However, the overheat isn't Apex fault. Apparently, my facilities crew responding to the power outage found the power bar unlit and the tank inactive. To fix it they unplugged everything from the Apex and plugged it into the wall - including heaters! Obviously my crew training was inadequate. They were afraid to call me in the middle of the night and should have just left it alone...

Still a sad outcome

 

[IUfan]

Sorry to hear this happened. You should connect to apex fusion and turn Heartbeat ON. Then as soon as the apex loses connectivity to fusion (power failure, router failure etc.) it triggers an alarm and an email/text alert.

Sometimes it's just the router resetting, sometimes it's a power failure. Either way, good to know it via an email alert.


Sent from my iPhone using Tapatalk

 

[scuzy]

Help!: Apex failure = crash in progress

One thing to check is whether apex created a new address for the power bar. I had this happen with the new apex when the brain lost power. Upon restoration it had re-addressed the power bar and the heater outlets were set to ON rather than AUTO. I use EJ with built in thermostats so cannot cook the tank. I now run the apex brain off a UPS so it never loses power.

I had an issue with my fmk it somehow added a new one and it added 1 gallon of kalk into my tank.


Sent from my iPhone using Tapatalk Pro

 

[Im14abeer]

For future reference, the thermostat on your heater should be set one degree higher than your controller parameter. This way if you lose control the heater will still shut itself off before things get out of hand. It's nice that heaters have this built in redundancy. It won't help if the contact welds itself closed, but it would have prevented this particular situation. Hope everything recovers for you, this hobby is a constant battle against Murphy.