I have a bb tank with chaeto in the fuge and have done so for years. Regardless of whether you use a filter sock or not, if you have pods, worms etc (and that would be a good thing) in your chaeto, you are going to have detritus. They have to eat too and they will creat waste. Just like if you were raising copepods, the bottom has to be cleaned periodically. Personally I use a brine shrimp net which works good. It is never totally clean but my PO4 and nitrates test zero so I am not worried about collecting a little detritus.
As a whole though, I think it is healthy to have chaeto or other macroalgae. I use it mainly to create a home for microfauna for the tank. Especially (but not in your case) if you have a bb main tank it is more important since there is no sandbed for them to grow in.