Rebuild and recovery of a dead tank

[jason2459]

There still has to be a hole in the logic somewhere which allowed all that tank water to go down the drain. This is the primary cause of the issue, until you find that problem, it could happen again!

Yeah I've been thinking about that and reviewing what I might want to change. I'm fairly happy with how it went down. Not happy it happened of course. I think I hit worse case minus the tank blowing out.

Problem is if it was a siphon that was the cause this would still be the same outcome. If it was the spectrapure pump stuck on with in its own controller then I'll think of something to check it from running freely. Simple resolution to that problem is the Apex DOS as it ties in to the Apex directly.

 

[jason2459]

So, this is the outlet that the AWC is connected to.

Fallback ON
Set ON
If Outlet vSumpHigh_A7 = ON Then OFF
If Outlet vSumpLow_A6 = ON Then OFF
If Condx9 < 30.0 Then OFF
If Condx9 > 37.5 Then OFF
Defer 060:00 Then OFF

I'll have to look at the timing even closer but this may help tell me definitely if it was a siphon. Not necessarily if it wasn't. But one thing I see I can change if it wasn't is reduce that defer time down by half. That defer time was based on the litermeter standalone single pump used for the limewater ATO. Right now I have mj1200's for the top off. I could even cut that down to 20 minutes based on their programming.

I think I'll also change that to fallback off. That and change that condx statement higher to like 32 now that I've corrected the large salinity swings it was having before.

I do want to change those mj1200's out for the 400ml/m diaphragm pumps from avast. That will help in some situations too but I've mentioned that before so that's not a new change based on this situation and don't thing it would have helped other then possibly delaying the final outcome a bit more.

 

[jason2459]

OK, new programming set. And will think of what else to prevent a rouge pump continuously running. Can't stop the siphon.

Fallback OFF
Set ON
If Outlet vSumpHigh_A7 = ON Then OFF
If Outlet vSumpLow_A6 = ON Then OFF
If Condx9 < 32.0 Then OFF
If Condx9 > 37.0 Then OFF
Defer 018:00 Then OFF


Edit: actually I could do something with solenoid shutoff valves to stop the siphon. Kick those on when AWC_6_8 outlet = off.

 

[jason2459]

I haven't looked at the graphs more and need to but I did some math. I just don't see it being the Spectrapure Litermeter. I had a hard time thinking it could be a siphon from it. Being that it does have 3 rollers would make it very hard for a siphon to even be allowed through if not impossible with out it breaking a roller. All rollers were in tact. I did check that.

So, only thing I could think of if it was the Sepctrapure LMIII is if the controller on it didn't shut off the pump. The pump is set to 190ml/m right now as calibrated. BUT say it stuck on and the on board controller stuck it on at full speed which is 250ml/m.

That's 15,000ml or 3.96 gallons per hour. This started around 1am and removed most all the water by 4am. But again lets factor in it happen to just finish off right at 6am when I got up. And lets say it actually started at 12am for everything to start kicking on at 1am.

At a full 6 hours of running at 250ml/m could only remove 23.78 gallons. I lost close to 50g from my new salt mix brute, close to 23 gallons in the SlimJim ATO, and all water that would be above the baffles and in return section of my sump so around another 20 gallons. So, I have to still figure out the source of water loss. Unless a restricted siphon through 1/4" RODI tubing could move more water then that. No way.

I'm thinking now somehow my skimmer. Now to look at those graphs as it's integrated with Apex as well. At least the water removal component is and figure out how it emptied out of the bucket with out me turning on the pump inside it to empty it.

 

[jason2459]

Well, this certainly lines up and helps fill in the picture. New entries in bold.

1AM
Looks like my ATO and new salt pumps were kicking in like mad starting at 1:00AM they are only allowed to run for 1 minute every 15 minutes. That probably saved everything from happening faster then it did. Maybe.

1:06AM to 1:47AM Float switch in Davey Jones Locker opens and triggers the alert that it needs dumped.

1:29AM Pressure switch in Davey Jones Locker trips alert that the Bucket is Full

1:29AM Skimmer is turned off due to the Davey Jones Locker being full.

That should have done it. No more skimmer from that point on. That would have been about 3 gallons of water removed and 3 gallons of new saltwater added. A nice little mini water change. I would have woke up at 6am with those alerts and dumped the bucket. If there were actual issues then I would have found out then and not lost 70+gallons of water.

BUT!!!


1:31AM Skimmer turns back on and doesn't turn back off until 4:07AM when the return shut down due to the ATO fail alert.

2AM
By 2:26am my new salt pump is shutdown due to low switch opening up in my new salt brute saying it needs filled. Need to fill Alert is going off.

At 2:50am 34.3ppt salinity level and Condx/Salinity starts plummeting due to no more new salt mix and just straight ATO fresh water going in while this mysterious disappearance of tank water is being removed. Condx Alert is going off

2:59.58AM ATO Warning alert is going off.

3AM
Salinity at 31.9ppt
At 3:12am ATO low float opens up and alerts out it's empty and shuts down that pump. Need to fill alert is going out along with all the other alerts repeatedly being sent out every 60 minutes is what I have resends set to.

4AM
4:07AM ATO Fail Alert goes off and shutdown the return pump which also shuts down my skimmer.

6AM I get up to my phone full of text and email alerts. Kicking myself for putting my phone on silent which I rarely if ever do.

 

[jason2459]

New Theory that I should be able to replicate tonight easily but how I have no idea. When I test this theory out tonight I should find out how.

I remember one annoying thing all weekend was me not being able to see my skimmer's neck. I have a plastic sheet over my sump to reduce evaporation during the winter. This sheet was in the way of seeing the skimmers neck from my basement camera I have setup to view it all remotely. I also remember coming home that Monday late night and moving it out of the way. I don't remember if it was right on top of the air intake.

I have a feeling it was very near that air intake and got sucked onto it. From fiddling around the night before I left with getting the ATS screen cleaned off I shifted that sheet around to get to it. I normally make sure it's not in the way of the skimmer or the camera to see it's operation. I don't remember doing it that night. I did that pretty late in the evening on the 12th into the 13th. Probably around midnight which is lining up with that 1am event.

So, around midnight I'm shifting things around. By 1am has slowly drooped close enough to the skimmers air intake and gets sucked onto it causing the skimmer to overflow. The Apex alerts that it needs dumped shoots out and soon after it shoots out that it's full.

This all makes sense to me. What doesn't is what follows. I have a pump in the bucket that I have 1/2" clear tubing ran to a floor drain right next to it. When I get that DJ Dump alert I'll pull up the sump camera to get a quick visual and turn that pump on for a second and back off. This creates a siphon that drains out the bucket. But once drained the siphon is broken.

Then the cycle continues. Eventually I get the alert a week later that it needs dumped again and I turn that pump on again.

What appears to have happened is that bucket went past the Dump switch level and hit Full shutting down the skimmer. Then somehow siphoned out. The pump never turned on. I don't have any programming to turn it on. It's manually set to off and it does have SET OFF and DEFAULT OFF as the only programming on the outlet.

Since it never hit full stage again and the skimmer turned back on and lost 70+ gallons of water until 4am when ATO failed alert came on I can only think that some how

1. the siphon started brought the bucket down below the full stage (have to figure out the HOW this happened)
2. skimmer starts back up and still overflowing due to restricted air intake
3. the siphon never breaks and keeps up with the overflowing skimmer
4. skimmer along with the bucket siphoning out into the drain empties my ATO and new salt mix brute
5. ATO fail alert kicks in and shuts it all down once both reservoirs are emptied.

Now I have to figure out how the heck that siphon got started with out the pump turning on to start it.

In the mean time I have closed some gaps and tightened up the timing on all the alerts so it all gets shutdown much much sooner. I can't stop that siphon if it occurs again but I can shutdown the skimmer that's supplying it. Have to figure out how that siphon started so it doesn't do it again though.


Here's my biggest gap in logic that caused it all to siphon out.

I thought I had the ATO Warn to shutdown the skimmer in place and the ATO Fail to shutdown the return in place. The return ATO fail is in place which did stop my return from burning out. So, that was good.

Here was my skimmer programming
Fallback OFF
Set ON
If Power SumpEB8_7 Off 002 Then OFF
If Outlet vSumpHigh_A7 = ON Then OFF
If Outlet return_7_1 = OFF Then OFF
If Outlet vLockerF_A3 = ON Then OFF
Defer 002:00 Then ON

What I thought was in there was my ATO Warn if on then OFF statement. Notice the vSumphigh statement in there is there and I did take out the vSumpLow statement but what was supposed to be put in it's place was the vATOWarn line as it has a delay in it otherwise the skimmer was turning on and off every time the ATO kicked on and off.

So, here's the new Skimmer programming

Fallback OFF
Set ON
If Power SumpEB8_7 Off 002 Then OFF
If Outlet vATOWarn_B8 = ON Then OFF
If Outlet vSumpHigh_A7 = ON Then OFF
If Outlet return_7_1 = OFF Then OFF
If Outlet vLockerF_A3 = ON Then OFF
Defer 002:00 Then ON


Tightening up the timeing the vATOWarn would have went on before 2am saving a lot of water. But figuring out WHY that started draining will save all but 3 gallons of water if that ever happens again.

 

[jason2459]

It's driving me nuts not knowing how that skimmer bucket started draining. Basically I'll just put my finger over the air intake and let the bucket fill up and see what happens.

Can see that plastic sheet in this picture I posted earlier in this thread.

 

[jason2459]

Oh yeah, and I still have to look for stray voltage and why my GFCI tripped which was the next issue during my trip away. If I don't find any I will change out the outlet as they could themselves go bad.

 

[SpectraPure]

There still has to be a hole in the logic somewhere which allowed all that tank water to go down the drain. This is the primary cause of the issue, until you find that problem, it could happen again!

The drain line for the drain pump should only be 1 to 2 inches below the water surface. This way only 1 to 2 inches can be drained out if something happens.

Jeremy

 

[jason2459]

The drain line for the drain pump should only be 1 to 2 inches below the water surface. This way only 1 to 2 inches can be drained out if something happens.

Jeremy

Yep it was. I've moved it into the return section and below the low sump switches that turn on the ATO. It was over by the drain pipes which 1-2 inches there means it's drained out the return section and everything down to the baffles plus another 1-2" in the other two sections.

I've pretty much ruled out it wasn't the SpectraPure LMIII as the math didn't add up to how much water was lost compared to how much the LMIII could have possibly pulled out.

 

[jason2459]

Salinity is back up close to 34ppt now. 1 more ppt over the next few days.

Anemones are looking nice and full today. At least what I can see through the webcam that I love watching through out the day.

 

[jason2459]

Alk spike recovery update. Most of the lps is coming back. Looks like the couple SPS I have and was showing growth before this look "burnt". Frogspawn is in worse shape then anything else and it may recover but may not. Goniopora looks great oddly. Bubble tip Anemones are full. That minimax looks like its growing well. The gorgonia looks rough and hope it recovers. Yellow Fiji looks normal ticked off right now as I'm taking pictures and will fill out again in about an hour I'm sure. Encrusting monty looks fine and is still super purple in person. Purplehaze I think it's called.


Terrible picture of course from me but all those stubs were new growth so hope it recovers

Couldn't get even a bad picture of the other SPS. Oh, mushrooms look fine.

Duncan looking better then it did the past couple days.

Coming out nicely

Really not looking good. I've had a hammer look like this before and it just withered away from there.

Oh yeah the alien egg looking coral looks normal

 

[jason2459]

Going to test out putting my finger over the air intake filling the Davey Jones Locker here soon and see how what happens and if I can replicate what happen the morning of the 13th.

 

[jason2459]

Well, mystery solved for the most part. At least how the siphon kicked in when the skimmate bucket filled up.

That night before I secured a few of the hoses together that go into the floor drain. This includes the water softener, awc old tank water, rodi waste water, and the skimmate bucket drain hose.

I then shoved it down into the drain.

Mistake. Do not do this.

What that did is put the drain tube into the water that's in the floor drain. When the siphon kicked in, which I did do that night before after doing that to the hoses to make sure the bucket was fully empty, is not allow the siphon to break fully.

So, pump turned on and emptied the bucket. Hose shoved into the water in the drain allowed that water to drain out of the bucket until it was empty. BUT once empty air got into the tube through the bucket side but was stopped just over half way. A good foot of the tube on the drain side was still filled with water.

Now, when the bucket was filling up there wasn't enough pressure to cause the siphon to keep going. The bucket just kept filling up. THEN, by the time the skimmate got to the top of the bucket over the water level in the tube that was left on the drain side there was enough pressure to start up the siphon again.

This just kept going in a vicious cycle. I basically turned my skimmate bucket into a toilet and my skimmer had diarrhea.

So, all I had to do is make sure the tubing is not shoved into the floor drain hole and into the water down there. Then the siphon fully breaks once the bucket is empty. So, I'd the skimmer ever goes crazy for some reason or the air intake gets clogged, which I think is what happened, it will now do what it was supposed to do which is shut down after the skimmer bucket is full. Which is only around 4 gallons at most. At 3 gallons I get alerted it needs dumped. At 4 gallons it shuts down the skimmer.

Or if for some reason it doesn't the ATO Warning alert will now shutdown the skimmer like I thought I had programmed in there which is about 20 minutes of run time. Not 4 hours like the 13th.

Here it was fully shoved in which up until the night of the 12th it never was before and never had this issue

And now pulled out like it was before and siphon fully breaks like it should.

 

[Floyd R Turbo]

Talk about Murphy's Law in action, dang!

You could add a dome-cover to the air intake of the skimmer, like a half-ball or a stand-off plate that prevents the intake from being occluded.

The drain thing though, that sucks...maybe you put a "T" in the line and extend a vent line up a few feet & zip-tie it to something to prevent that from happening. Kind of like a plumbing drain vent line. Removes any potential siphoning condition.

Glad you figured it out, sucks with the timing though!!!

 

[jason2459]

I need to shove the airline back up into the skimmer cup which totally prevents that cover issue. I've always had that before but again a recent change in dosing limewater separate from ATO had allowed me to remove the CO2 scrubber. So, I temporarily just removed all the tubing from the air intake. Have to route that back up.

The siphon break at the drain is a good idea. I totally did that to myself. I had to really shove that into the drain. Thinking I was preventing any of it from popping out and draining onto the floor. That would have been better as anything that would be pumped onto the floor would eventually make its way to the drain.

 

[jason2459]

This is what's temporarily been removed and need to be put back in place.

I finally got around to setting up the T to the collection cup. One side pulls in air that passes through my ozone generator and CO2 scrubber and the other pulls in air from the collection cup.

This does a few things for me. On the ozone/scrubber side it cuts down on the air drawn from those two. The ozone generator when running, which it hasn't been, doesn't need a lot of air pulled through it to be effective.

The scrubber I don't want a lot of air going through it as it would exhaust the soda lime a lot faster making it more expensive which wasn't what I wanted to do otherwise I'd just punch a whole in the wall and pull from outside. I also don't need or want to increase the pH that much at all. I just wanted to establish an easy, cheap, no risk of outside pollutants, and effective way to increase pH steadily above 7.9. It's done all of that so far.


The second line going to the lid and into the collection cup does a few things as well. First is it draws in smelly air so it can't escape which is even worse now that I don't clean my collection cup much at all anymore with the swabbie installed.

Second is it pulls in that smelly air but also if my ozone is running would pull back in any residual ozone not allowing it to escape potentially creating a health risk. It does require any other opening in the lid to be closed off. This way I don't need to worry about paying escaped air through carbon which could still allow ozone to pass through if it's channeling or the carbon is exhausted.

Third that air line will shut down skimming and reduce the chance of an over flow as it sucks that overflowing water back down into the skimmer. This could happen if day the drain line clogs up and somehow I don't notice it or I'm away for a week or the skimmer goes volcano foam crazy.

The avast swabbie lid has a lot of little holes. I closed them up with gorilla tape and had to drill out on so a 3/8 line would fit in. 1/4 wouldn't allow enough air or water if needed draw and could clog up easier.

 

[jason2459]

AND Now my neighborhood is with out power.

Gotta love reeftronics for these notifications. Evidently fusion still thinks things are fine. My apex can't send out a notification it's down. I need to setup a UPS for my DSL router so then the Apex can. I have the 12v Adapter for the apex and a UPS that one of the Energy Bars are plugged into. The 12v Adapter is plugged into a wall outlet. So if power goes out the Apex will know and alert out. But I need to get that other UPS going.

At least I have the battery backup for the gyre going my tank is still getting circulation for the next 36 hours or so.

 

[jason2459]

Power was out for about 4.5 hours today. My tank hates me about now.

 

[jason2459]

Getting things ordered for a "Big 3" test kit shootout. So far I've got the Triton kit ready to go, Saliftert, API (alk/cal only), and just ordered all three for Nyos. I might do AWT one more time with this and will get all three with Elos and call it good I think.