Definitely not a sponge
Is it hard to scrape off? It looks like green coralline to me. Purple is the most common, but it comes in a bunch of colors.
You say "worse with water changes". Do you dose calcium? This implies its growth is limited by a lack of calcium, your water changes supply a fresh dose.
Its either a sponge, or dead organic matter. Did it just show up, or did you just notice it? Pull a piece off and smell it, that will solve the mystery. Macroalgae and organic sponges/dead stuff, have very different smells lol.
...if the tufts are really tough, it may be a Schizothrix sp.