It depends on the animal that you keep in the 10 gal. A 2-3 inch gonodactylid such as Neogonodactylus wennerae coming out of the Gulf Coast of Florida would be perfect. So would a Gonodactylus smithii, G. platysoma, or G. chiragra from the Indo-Pacific. This is also a good set-up for a 3 " Odontodactylus havanensis (Florida) or O. brevirostris (Hawaii) - both hard to find. You could also keep a larger spearer in such a tank. A 4 inch Pseudosquilla ciliata would be great.
However, if you are thinking of a larger smasher such as an Odontodactylus scyllarus (Peacock or Painted Mantis), you could run into trouble with a large one - say over 5 inches. A juvenile in the 3-4 inch range will probably be alright, but a larger one could put a stress on the capacity of the system and there is a slight possibility that it could break the tank. I have kept over a couple of hundred O.s. over the years and many were held temporarily in 5 - 20 gal glass tanks. I can only remember animals breaking three, but it does happen - typically (if you can say anything is typical with a sample size this small) when the animal tries to dig in a corner and starts striking at the obstruction.
We are currently measuring the striking forces delivered by O. s. and I hope to have some accurate measurements soon. The data that we have so far suggest that my original estimate of an impact equivalent to a 22 cal. short at mussel velocity was a bit high - but not much.
The bottom line is that if you are using a standard glass 10 gal, you would be be safest with a smasher no larger than 4 inches. Than is not to say that a 5 inch O.s. going to ruin the carpet, but there is a risk.
Roy