What you are most likely seeing is the molt from the crabs (they shed their external skeleton as they grow). If the numbers are decreasing, the crabs themselves are the likely perpetrators, they will fight for shells, it is a good idea to keep a few extra slightly larger sea shells for them to grow into.