You definitely may be skimming out the nutrients that the sponge needs. Based on the livestock you listed, your bioload doesn't look high at all for the size of your tank (not that you should stop skimming.)
Also, I think that going to a daily feeding routine is also the right thing to do for this type of sponge. Unlilke ball sponges and others from shallower water that receive some of their energy from zooxanthellae, as far as I know the frilly sponges get all of their energy from filter feeding.
Changing the amount of nutrients you're putting into your system too quickly could lead to nitrite or ammonia spikes, so just test your water regularly while you're tweaking your feeding routine. (I alternate zooplankton feedings with phytoplankton to try to keep the nutrients reasonably low.)
Good luck!