OK, not calcerous, then. I have tons of sponges in my tank. Hard little yellow ones and soft pink ones, some gray that started out black and makes these little towers in the caves LOL, and some nasty, soft gray stuff that comes right back when I pull it out. The pink started out kind of invasive then backed off. The hard yellow ones don't seem to do any harm, the black either. That soft gray stuff isn't overtaking anything, but it sure grows fast & won't go away!
That's a pretty good sized mass you have going, I'd definitely keep an eye on it & if it starts encroaching on corals maybe try a 10:1 tank water/3% peroxide dip & see if that helps.