It is not likely a concern. I had a tank emergency years ago, and I intentionally dumped GAC into the sump to give a blast of cleaning. If you want and can easily siphon it out, that is fine, but it won't hurt anything staying there forever if you cannot get it all.