Rebuild and recovery of a dead tank

[jason2459]

So, I couldn't help myself this morning. This is to much fun. lol

ATO has been kicking on plenty so I went ahead and bumped up the limewater dosing to 1 second every minute. That slowed down the ATO noticeably. So, I will halt and hold back on increasing any dosing and watch.

 

[jason2459]

I think I posted a while back some parts of what I planned on doing for maintanence on my skimmer once I got the swabbie and Davey Jones Locker. I just posted this in another thread and figured I'd log it here.

My skimmer maintanence is extremely minimal. I use a swabbie neck cleaner that has done an excellent job cleaning the neck keeping it squeaky clean.

Then that red tube connects down to a 5 gallon bucket. That 5 gallon bucket has a horizontal float switch that alerts and tells me I need to dump it. Then it also has a pressure switch above that level that again alerts me saying you idiot why didn't you listen to me before and then it also shuts down my skimmer.

Now the maintanence on that is me bringing up Apex Fusion on any web browser and turning a pump on that I also have sitting at the bottom of that bucket. That pump has a hose attached that feeds to a floor drain next to it. The beauty of it is I don't have to time it when to turn it off. I turn it on for a second and right back off. It then siphons the rest out.

So, far I've spent 3 seconds of my time every so often on maintenance. If I want I have a IR night vision type webcam I can remotely control to get a visual of what's going on or can see how full the skimmate bucket is and dump it sooner than alerted if I cared too.

The next maintanence chore on it is cleaning out the collection cup. I'm far to lazy to bring it up to the sink. So, I put my finger on the air intake which then floods the collection cup to the skimmate bucket flushing it all out. I also have a new saltmix container that has a pump in it directly to my sump. I turn it on for a few seconds so my ATO doesn't replace the saltwater that just flushed out the skimmer's collection cup. I fill up the bucket and release my finger. I then take my 3 seconds to turn on the bucket's pump for a second to empty it.

That maintanence is a real chore and takes me a good 60 seconds to fill up that bucket. Not to mention the 5 seconds to turn on the new salt mix pump AND back off. Then the bother to take the final 3 seconds to turn on and off the bucket's draining pump.

It's a drag and I do that every TWO weeks! Or really on schedule now when I scrape my ATS screen which I do now on the 1st and 15th of the month so its easier to remember then every two weeks.

The Bucket

As for cost. If the pump goes out its a reliable and efficient yet inexpensive sicce based pump.

I am looking to try out an MTC skimmer which I already have a fairly inexpensive mag1800 pump to go with it.

Which I already have spares of both. As I also run a mag 1800 for my return pump. I always keep spares of stuff.

But yes, some skimmer pumps are crazy expensive but very nice.

 

[jason2459]

My urchin was in a new hiding spot yesterday. It's growing quickly. Notice my starfish climbing around towards it to.

 

[jason2459]

I think I'm set now with tweaking out the limewater/kalk dosing.

It's set to 1 second every minute 24/7. pH as taken a nice little bump up and hanging around 8.1's to low 8.2's. Nice little swing and I would assume extremely low to non-existent alk swing. ATO still kicks on but not nearly as much. Come spring I'll need to monitor more to make sure the ATO still kicks in at least every few hours. If anything the limewater would be shutdown if it raises the water to a high float switch.

 

[jason2459]

I got back my Avast Marine 25ml/m peristaltic pump. It broke last year and I finally got around to contacting them for warranty repair. Technically it was about 8 days out of warranty but they said send it on in and they'll fix it. They replaced the motor and shipped it back.

I was going to switch my vodka/vinegar to dose 2x per day from the 6x it's doing now. But I don't know if I'm comfortable doing so. I just ran the pump and it doesn't seem like it operates very smoothly and has this click that happens every revolution. I think I'll just stick with the BRS 1.1ml/m dosing pump that's been running solid now for the past 6 years or so. I probably should replace the inner tubing though. I keep forgetting to do that for about the past 5 years.

 

[jason2459]

Here's the Apex programming that it looks like I'm finally settling on between my ATO of RO/DI, newsalt water adjustment/backup, and lime water dosing.


I have a couple pH probes and they are referenced in a few spots for a low and high value. So, if I want to adjust my low and high threshold I only have to do it at one virtual outlet instead of searching for and forgetting someplace that's referencing the pH. Kind of a like a global variable in programming or a central config. The virtual outlets also allow for a defer on the pH value incase they are getting interferences and bouncing up and down.

pH High and Low triggers central configs
[ vPH_High_C7 ] ( Cntl_C7 )
Program Type: Advanced
Display Icon: Up/Down Arrows
Logging: Enabled
Program:
Set OFF
If DT_pH > 8.50 Then ON
If BM_pH > 8.50 Then ON
Defer 001:00 Then ON


[ vPH_Low_C8 ] ( Cntl_C8 )
Program Type: Advanced
Display Icon: Up/Down Arrows
Logging: Enabled
Program:
Set OFF
If DT_pH < 7.85 Then ON
If BM_pH < 7.85 Then ON
Defer 001:00 Then ON


kalk/limewater

[ Lime_7_3 ] ( 7_3 )
Program Type: Advanced
Display Icon: Hourglass
Logging: Enabled
Program:
Fallback OFF
Set OFF
OSC 000:59/000:01/000:00 Then ON
If Outlet vPH_High_C7 = ON Then OFF
If FeedD 230 Then OFF

That's very simply saying dose for 1 second every minute 24/7. Unless the pH hit's what I have my vPH_High_C7 virtual outlet to trigger on at. I calculate this out to around 1.5 gallons every day of fully saturated lime water based on the stand alone spectrapure pump that's used. The FeedD timer is for when I fill up the 29g tote and mix up the kalk into the fresh ro/di. I don't want it to dose until it's all settled out. I turn on the feedD which does nothing but make sure this is used for a few hours. Then I don't have to forget to turn off or back on the outlet. It will turn back on automatically.


Now for the ATO components. I have a low and high triggers on top of the tsunami AT1's that I have my pumps plugged into for the ultra high failsafes but those aren't in my apex as they are separate. I like having a few things like that separate in the off chance the apex outlet gets stuck on which is not common.

High virtual outlettriggered by a float switch and backed up by a few Tsunami AT1s. One for each new salt pump, lime water pump, freshwater ato pump, and AWC pumps.

[ vSumpHigh_A7 ] ( Cntl_A7 )
Program Type: Advanced
Display Icon: Up/Down Arrows
Logging: Enabled
Program:
Set OFF
If SmpH_2 OPEN Then ON
If SmpH_2 CLOSED Then OFF
Defer 000:01 Then ON

Low virtual outlettriggered by float switches. I have two setup in serial working together one failsafe higher then the normal operating one. This one triggers the ATO.
[ vSumpLow_A6 ] ( Cntl_A6 )
Program Type: Advanced
Display Icon: Up/Down Arrows
Logging: Enabled
Program:
Set OFF
If SmpL_1 CLOSED Then ON
If SmpL_1 OPEN Then OFF
Defer 000:02 Then ON


Now that I have my lime water split and I have my conductivity probe stabilized I can do a little logic to decide to top off with RO/DI or my new salt mix used in AWC. But I also have a failure to operate override so the opposite will kick in. I'm also more comfortable doing this with the conductivity probe now that my Top off is so much less now. If one or the other is used in error it's not that much at all and will easily catch it as I calculate it would take just over a week of the wrong one being used to switch the salinity by ~.5ppt. Otherwise I would not feel comfortable with using the conductivity probe that can easily be thrown off by something as simple as an air bubble.

The ATO warning that something is wrong with one or the other top of pumps. I have a mj1200 in my 50g brute new saltmix container and a mj1200 in the bottom of the SlimJim ro/di container. (I some how have like 100 mj1200 pumps. )

[ vATOWarn_B8 ] ( Cntl_B8 )
Program Type: Advanced
Display Icon: Hourglass
Logging: Enabled
Program:
Set OFF
If Outlet vSumpLow_A6 = ON Then ON
Defer 034:00 Then ON

That defer is there for a reason which is coming now and where the order of the lines of code really matter. Here's the Fresh RO/DI ATO outlet programming that references the Low/High switches, pH, and ATO warning. And this will need a lot of comments.

[ ATO_6_4 ] ( 6_4 )
Program Type: Advanced
Display Icon: Up/Down Arrows
Logging: Enabled
Program:
Fallback OFF
Set OFF
OSC 015:00/001:00/000:00 Then ON
(I do not like unrestricted continuous flow. It's not needed and 1 minute every 15 minutes should be more then enough time to top off. This is where the ATOWarn virtual outlet kicks in if it's not and will trigger the new salt mix pump to top off. )

If Outlet vSumpLow_A6 = ON Then ON
(Simple enough.)

If Condx9 < 34.5 Then OFF
(Now here is where the order of programming lines come in. This will cause the New Salt pump to kick in to top off.)

If Outlet vATOWarn_B8 = ON Then ON
(IF for some reason there's an issue and the new salt pump doesn't kick on this will turn this pump back on. Look above. vATOWarn has a defer timing of 34 minutes. The ATO and new salt pump outlet are 1 minute every 15 minutes. If that's not enough time to complete for some reason with in two cycles that means there could be something wrong and this ATO pump should kick in to help out regardless of what the conductivity/salinity probe says.)

If Outlet vSumpLow_A6 = OFF Then OFF
(This turns the entire thing off as soon as it's topped off.)

If Outlet vSumpHigh_A7 = ON Then OFF
(High mark failsafe which there's also the Tsunami AT1s in case this even fails. )

If Outlet return_7_1 = OFF Then OFF
(no need to top off as the vSumpHigh will be triggered as well but again incase it does this will turn it off too.)

If Outlet vATOEmpty_B6 = ON Then OFF
(Run Dry protection. I'm looking at the Avast 400ml/m Diaphragm pump to eliminate this worry. Right now I have a ton of spare mj1200s so using those. This is also part of my email alert so I know to top off my top off container. This is a horizontal float switch at the bottom of the SlimJim.)

If Outlet vATOFull_B7 = ON Then OFF
(I also have a horizontal float switch at the top to trigger this and an alert. Basically I can now have my ATO run as normal while I'm filling up the ATO SlimJim container. It has a float valve connected to my RO/DI which has an ASO that will shutdown the RO/DI once the container is full. BUT I do not want the ATO to continue running once full. I don't like the idea of a continuous unlimited supply of water from the water company being available. I also don't want to RO/DI to start/stop continually causing TDS creep wasting away my DI very quickly. So, once this is triggered it will shutdown my ATO and alert me via text and email. I will then manually shutoff the RO/DI unit and turn off the vATOFull virtual outlet resuming the ATO operation. Once the switch is cleared I can turn that virtual outlet back on.)


OK, so now for the new salt mix backup/ salinity adjustment. I would not normally advise this kind of control as the conductivity probe in a sump continuously is susceptible to a lot of different interference from electricity, bubbles, slime, precipitation, bacteria growth, coralline growth, algae growth, etc. etc. BUT like stated above my risk if very very minimal now with the lime water dosing taking up the majority of my top off needs and this is a good backup to incase the pump for the freshwater ATO fails for some reason.

[ NewSalt_6_7 ] ( 6_7 )
Program Type: Advanced
Display Icon: Hourglass
Logging: Enabled
Program:
Fallback OFF
Set OFF
OSC 015:00/001:00/000:00 Then ON
(Just like above this shouldn't need near a minute to top off what's needed. Top off needs are not that often so every 15 minutes should be plenty.)

If Outlet vSumpLow_A6 = ON Then ON
(same as above)

If Condx9 > 35.5 Then OFF
(here is the opposite from the fresh water. If above 35.5 don't run.)

If Outlet vATOWarn_B8 = ON Then ON
(EXCEPT if for some reason the freshwater ATO pump can't keep up or is having problems for some reason. Something is wrong this will be a backup to that regardless of what the salinity is.)

If Outlet vSumpLow_A6 = OFF Then OFF
If Outlet vSumpHigh_A7 = ON Then OFF
If Outlet return_7_1 = OFF Then OFF
(Same as above)

If NSE9_4 OPEN Then OFF
(This is a horizontal float switch in the 50g brute can. I'm also looking to replace the mj1200 in here for the 400ml/m diaphragm pump.)

Defer 000:20 Then ON
(This little defer statement allows the freshwater ATO pump be the primary.)

I also have a ATO fail virtual outlet. Which will shutdown my return pump and in effect my skimmer as well.
[ vATOFail_A5 ] ( Cntl_A5 )
Program Type: Advanced
Display Icon: Hourglass
Logging: Enabled
Program:
Set OFF
If Outlet vSumpLow_A6 = ON Then ON
Defer 102:00 Then ON


[ return_7_1 ] ( 7_1 )
Program Type: Advanced
Display Icon: Spigot
Logging: Enabled
Program:
Fallback ON
Set ON
If DT_Sw1 OPEN Then OFF
If Outlet vfeed_A8 = ON Then OFF
If FeedB 000 Then OFF
If Outlet vATOFail_A5 = ON Then OFF
Defer 015:00 Then ON


The dual inline low and high switches with the pressure switches to tsunami all in behind that nastiness.

another one of the several of the pressure switches in use

redundant failsafe high switch

The SlimJim

All three containers

 

[jason2459]

And with that above things are stable and awesome. This was all influenced by TMZ.

The ATO kicking in as needed I think just enough but not to much is nice.

But here's some charts I had to pull off the Apex directly as fusion doesn't go back far enough. These show how the CO2 scrubber helped and now with the consistent lime dosing its not needed and protects against one off daily spikes from events where humidity significantly drops.

Here's a graph of my system with using my kalk/lime as ATO no CO2 scrubber. It would dip just under 7.8 regularly at night. It most likely wasn't an issue and I could have continued with that most likely indefinitely

Limewater ATO alone

Edit: fyi those big dips 4 times there are my automated 4x per day feedings where my Apex shuts down my return pump/skimmer and feeds a mix of NLS Marine/AlgaeMax pellets.




But I personally would rather maintain above 8.0 if there's a reasonable and easy way which the CO2 scrubber is. This kept my pH above 8.0 consistently. I picked these dates as it shows what happens when humidity drops a lot and ATO kicks in more. It snowed quite heavily on the 24th here. pH would normally range with this setup between 8.0 and 8.3

Limewater ATO + CO2 scrubber.

Now that I have my ATO and Limewater dosing separate for some reason I do not need the CO2 scrubber and maintain in what I see as about as perfect range as I could be in with little effort of 8.1-8.2. Small swing and I image little to no alk spikes. Alk has maintained at 10 as it has been. The past few days have been perfect as well as its been snowing the past 2 days. So it shows 1 day of no snow and two days with snow with no significant pH spike like there was above on the 24th with limewater as ATO up to 8.5.

Limewater dosing 1s per minute alone. ATO separate. No CO2 scrubber. Snow last two days

 

[jason2459]

I think this is the same starfish but looks slightly different but it could be I'm just getting a good view of it. I still think I only have one left out of the two I got in.

I also picked up this thing.

It has a couple bald spots

My goniopora is looking like a bit more growth. It's slow.

 

[jason2459]

I tried bumping the water flow higher on the ATS again. Same thing happened again. Very poor production and very light coloration.

I have no idea why I didn't take front and back pictures. I thought I did but my mind's all over the place this week so must not have since I don't see them.

I did get this picture of the little bit that was scraped off and can see it's not all nice and green like I've had before.

I've reduced the flow a bit again.

 

[biecacka]

Jason, what are you feeding the ATS with? I'm getting ready to put mine online soon and feeding it with a line from my return pump. How many GPH are you shooting for?

Corey

 

[jason2459]

Jason, what are you feeding the ATS with? I'm getting ready to put mine online soon and feeding it with a line from my return pump. How many GPH are you shooting for?

Corey

I don't know the exact gph but its being fed by a RIO pump. I think rio1700? I'll have to look back to see. Every time I try to go wide open my screen production goes way down. I cut it back very slightly. Not much but seems to be just enough. I don't like touching my return line for anything else as I run a bean animal setup and my siphon is tuned to the return.

 

[jason2459]

Yep, RIO 1700. It's listed for around 642gph but that's with 0 head so something shy of that with around 1' head and then cut back very slightly with a ball valve.

 

[jason2459]

Back home and time to get my tank back on track. I've been gone about a week and have to say about all the failsafes I've setup were tested. Thank goodness for my Apex.

So, just before I left I toped off everything that could be toped off, cleaned everything that could be cleaned , and emptied everything that needed emptied.

I had everything ready to be gone for a month. I woke up to get ready to head to the airport and I notice a bunch of text messages and emails from my Apex. I run down to the basement and the return and skimmer are off and the return section is drained.

Investigate. 30m before having to leave.

I turn on the new salt pump and it took a few seconds before it starts sucking air. I open up the 50g brute and its near empty from being comoletely full the night before and new salt added. I go to fill it up with fresh water from my ATO slimjim I just filled up the night before. A couple seconds into it and the pump start sucking air. I open it up and its empty.

Keep Calm. 25 minutes before having to leave.

I look around the floor for water. Over 70g of water. I look around the tank, under the tank in the basement, and around the sump room. All dry.

Keep calm. Where could it have gone. 20 minutes before having to leave.

Think.

Floors all dry so no leaks. New salt mix brute is empty so it didn't siphon into it and its below the sump so couldn't siphon out. ATO is empty so same thing.

Think quick. Keep calm. 10m before having to leave.

AWC. Look in the sump. Not around it.

Quick look at parameters. pH plummeted. Temp in sump fine. Temp at tank no idea but I'm sure much lower then normal. Salinity 30ppt.

Act quick and think. 5 minutes before having to leave.

Think of a solution that will keep my system going at least for the next 5 days before I get back and as little interaction needed by tank sitter.

So, I pull the AWC tubing out completely and unplug the spectrapure. If it wasn't a siphon problem and a problem with the pump stuck on this will resolve both situations. I'll figure out which was the issue later if possible. Spectrapure pumps are supposed to be anti siphon...

I turn on the valve to the RODI and the valves to both the ATO and new salt mix brute. Leave instructions to shut off the gray thing with blue handle. aka the main shutoff to the RODI. I ran it around to the front of everything so it was obvious.

I dump in 8 or 10 cups of salt into the new salt mix brute. This is not a good thing to do and I new it. This will cause a lot of precipitation. But my goal at this point will be to get the sump filled with salty water and get the return going so heat will be delivered to the tank. Then let the Apex keep topping off the system with that salty water slowly raising the salinity back up.

Run out the door for the airport and hope it all comes together.


26 hours later.

Apex alert. One of my energy bars has lost power. I look for which on it is and it runs my return pump....

Keep calm at least gyres are running.

32 hours later

I get someone to check the gfci outlet. It was tripped for some reason and now reset. Return pump back in action and gfci hasn't tripped again since. I'll look for stray voltage later.


34 hours later

Apex is alerting me my skimmer bucket needs dumped. I turn on the dump pump.

20 minutes later. Same alert and I dump it again.

10 minutes later. Another alert that the skimmate bucket needs dumped again. Skimmer is going nuts.


Keep calm.

I just shut down the skimmer and left it off. I do not want to worry about that the entire time I'm away. Also, don't want to waste the salt water. Need to get salinity up anyway. Evidently the skimmer doesn't like the conditions my tank is in and the saltwater being used to top off now. It's definitely not ideal saltwater.

So, now I'm depending on the ATS to provide oxygen and remove nutrients. Bacteria should have a field day.

BUT I can't see it now. My laptop happen to be plugged into the same powerbar as the return that went out. Laptop must not have a good battery. Can't see of anything is dead or anemones getting shredded up.

Can't worry about that now. Just have to let the Apex take over from here and let it do its thing. AWC and Skimmer are now off and don't have to worry about those two causing problems.


Failsafes that worked.

1. my run low ATO alert kept kicking in my new salt pump and fresh water pumps keeping the sump going until they were empty.

2. I don't keep enough fresh water to cause my salinity to go bellow 30ppt or 1.023 which is low but not leathal.

3. My low switches in both containers didn't allow pumps to run dry

4. My ATO fail switch and programming didn't allow my return to run dry as it got shutoff at I think I saw in the Apex graphs around 4am. (Alerts were going off all night and I had my phone on silent. I won't ever do that again) rest of that section siphoned out so it could have totally burned that return pump out.

5. Large float switch in new salt container shutdown AWC (which leads me to believe there was a siphon created in the spectrapure pump so I need to talk to them about that)

6. pH high shutdown my lime dosing while sump was offline for a long period of time.

7. Heater outlet was on as the display tank was getting colder. Sump temp maintained perfect temp as the Ranco controller did not allow it to over heat and took over as the primary temp controller. It is set to 80 it looks like.

8. Skimmer bucket and apex integration with it saved me from a run away skimmer.

9. Power outage alerting allowed me to know exactly which strip and outlet was out. If the outlet was bad I could have had the return plugged into a different outlet.

10. I'm sure there's more but can't think of it right now. I guess the high full levels in the top off and new salt containers allowed me to know when they were full and the main water supply to the RODI system could be shut off. That was nice.

Home now and it looks like everything is alive. Most the LPS corals look ticked off. Glass is covered in bacteria as the skimmers been off and no water changes have been happening like normal. I didn't turn off the carbon dosing or the lime water dosing.


I did a quick set of parameter checks. Over the past several days the salinity has slowly climbed back up and right now at 33ppt.

Nitrates ~.2

Calcium: a low 380

Alk: a very high 14 and probably why the LPS are not happy at all.

Phosphates: 4ppb converts to 0.012ppm

So, now I have the skimmer back online and it seems fine now. I'll have to keep an eye on it. I added some salt to the new salt mix. It's salinity was at 15ppt. It's now up to 35ppt. I also turned back on the AWC and will be watching that closely as well.

Of course it all had to happen as I was leaving and while I was gone. But man I would have basically bee. With out anything in my tank but some powerheads the entire time I was away. Most likely the tank would have had some losses. I would have definitely had to restart a new sump. Over 5 days of being off would not have smelled good either.

 

[Floyd R Turbo]

Oh DANG!!! Did you ever figure out what the problem was in the first place?? Where did 70g of water disappear to??

 

[jason2459]

Oh DANG!!! Did you ever figure out what the problem was in the first place?? Where did 70g of water disappear to??

I have to believe it went down the drain. Over 70g could not have evaporated in just a couple hours with out boiling the tank which didn't happen. Over 70g would be very obvious if leaked or drained out onto the floor somewhere. Couldn't have siphoned out into any of the containers. ATO was empty, brute new salt mix was empty, and the lime water tote was still full so would have had over 70g spilling over onto the floor.

Only thing that could put it into the drain is the automatic water change setup. But that was a lot of water to drain out in a few hours. It had to be done between around 11pm and 6am. By 3 am it would have been below my ATO/AWC alert. By ~4am my ATO /AWC failure programming kicked in and shutdown the return pump. Then between that time and 6am the rest was removed out of the return area and down to the baffles in the other two sections.

I don't know how fast a siphon could work through a 1/4" rodi line that's restricted by a peristaltic pump. Seems unlikely. So, leads me to believe the spectrapure stuck on possibly and AFAIK the fastest it can pump is 250ml/m.

I'll look closer at a the graphs and timing when I have time. I've had 0 time this last week as it was a very busy business trip. The tank was the last thing I wanted to worry about.

 

[slief]

I have to believe it went down the drain. Over 70g could not have evaporated in just a couple hours with out boiling the tank which didn't happen. Over 70g would be very obvious if leaked or drained out onto the floor somewhere. Couldn't have siphoned out into any of the containers. ATO was empty, brute new salt mix was empty, and the lime water tote was still full so would have had over 70g spilling over onto the floor.

Only thing that could put it into the drain is the automatic water change setup. But that was a lot of water to drain out in a few hours. It had to be done between around 11pm and 6am. By 3 am it would have been below my ATO/AWC alert. By ~4am my ATO /AWC failure programming kicked in and shutdown the return pump. Then between that time and 6am the rest was removed out of the return area and down to the baffles in the other two sections.

I don't know how fast a siphon could work through a 1/4" rodi line that's restricted by a peristaltic pump. Seems unlikely. So, leads me to believe the spectrapure stuck on possibly and AFAIK the fastest it can pump is 250ml/m.

I'll look closer at a the graphs and timing when I have time. I've had 0 time this last week as it was a very busy business trip. The tank was the last thing I wanted to worry about.

Wow. That's just crazy. I ran the Spectrapure pumps for many years and never had any shipon issues. Which pump is this? Is it one of the Litermeter III slave pumps with a separate AC Adapter and not the full LM III? If so, do you have that AC Adapter plugged into one of the relay outlets (outlet 4 or 8)? Those pumps draw minimal wattage and if you have them plugged into one of the triac outlets, that could have caused the pump to stay on as the EB8 can't properly sense a load on the outlet. Anything below 6 watts should really be on a relay outlet.

 

[jason2459]

Wow. That's just crazy. I ran the Spectrapure pumps for many years and never had any shipon issues. Which pump is this? Is it one of the Litermeter III slave pumps with a separate AC Adapter and not the full LM III? If so, do you have that AC Adapter plugged into one of the relay outlets (outlet 4 or 8)? Those pumps draw minimal wattage and if you have them plugged into one of the triac outlets, that could have caused the pump to stay on as the EB8 can't properly sense a load on the outlet. Anything below 6 watts should really be on a relay outlet.

Its the standard LMIII with remote pump plugged into the LMIII (B port). The LMIII is plugged into a Tsunami AT1 pressure switched outlet (High mark in sump) which is plugged into a piggy back plug that plugs into 6_8 (AWC_6_8) which would be EB6 outlet 8.
https://www.reeftronics.net/ja/jason2459/apex-status

That piggy back plug is a large tethered float switch in my 50g brute new salt mix container. When I got down to the basement that morning the LMIII was shut off. Which a couple things can shut it off. The high AT1 pressure switch in sump, the high Apex float switch in sump, the programming saying ATO or AWC has failed due to low float switch being closed for too long, and the big tethered float switch in the brute on run down.

I'm looking at the apex graphs now for the morning of the 13th.

1AM
Looks like my ATO and new salt pumps were kicking in like mad starting at 1:00AM they are only allowed to run for 1 minute every 15 minutes. That probably saved everything from happening faster then it did. Maybe.

2AM
By 2:26am my new salt pump is shutdown due to low switch opening up in my new salt brute saying it needs filled. Need to fill Alert is going off.

At 2:50am 34.3ppt salinity level and Condx/Salinity starts plummeting due to no more new salt mix and just straight ATO fresh water going in while this mysterious disappearance of tank water is being removed. Condx Alert is going off

2:59.58AM ATO Warning alert is going off.

3AM
Salinity at 31.9ppt
At 3:12am ATO low float opens up and alerts out it's empty and shuts down that pump. Need to fill alert is going out along with all the other alerts repeatedly being sent out every 60 minutes is what I have resends set to.

4AM
4:07AM ATO Fail Alert goes off and shutdown the return pump which also shuts down my skimmer.

6AM I get up to my phone full of text and email alerts. Kicking myself for putting my phone on silent which I rarely if ever do.

 

[jason2459]

So, the LMIII's outlet is pretty much ON all the time. The LMIII then handles the ~1% water changes daily through it's on board controller spread out I think 155 times a day. The Apex is there to shut it down or shut other things down like it did if there's an AWC failure which it seems there was. Just bad timing on me leaving out of town when it happened and me not hearing my system Alerting the crap out of me while it was happening.

 

[jason2459]

Have to say though it was stressful and several things failed. But it seems like it all failed the way it's supposed to and alert how it was all supposed to. So, I guess I got a good real world test on all the alerting and failsafes working together and everything is alive in the tank.

So, now the only thing that needs tested is the run high failsafe and alerts. Those were all the run low and empty failsafes and alerts kicking in. That would be if the ATO stuck on or the other AWC litermeter pump stuck on and kept pumping water into the sump. There's several high float switches and all pumps are plugged into a pressure switch as well which would also shut them off incase the apex doesn't.

 

[Floyd R Turbo]

There still has to be a hole in the logic somewhere which allowed all that tank water to go down the drain. This is the primary cause of the issue, until you find that problem, it could happen again!