If you are gong to go through all the trouble of removing the rocks, then yes, some fresh water CAN drive them out at a cost of very serious stress to the shrimp and everything else.
May I make a couple of suggestions?...
#1...Find out where it lives and try spot feeding it. If its eating stuff you provide, ,it will be less likely to want to kill to eat.
This will also
A-save stress on the tank, time spent flushinig every nook and cranny and B- provide another addition to the tank that will grow on ya.
#2...if you really cant accept the thing living in your tank, then try to remove it from the rocks in a 'kind' manor. If you can keep from killing it, I would be willing to bet you can get enough money from selling it to one of the many people looking for one to off-set the cost of replenishing your clean-up crew.
I keep a mantis in a VERY mixed reef. My tank only has a few snails, ZERO hermits, pom pom crab, emerald crab, skunk shrimp, 6 line wrasse, and a maroon clown. The clown is by far the most fierce occupant of my reef. She will actually jump out of the water as your hand gets close and its an all out assult until you remove your hand. My mantis is lazy. I have spoiled it by usual feedings of frozen squid and various other stuff. Its tricky to locate the little fella sometimes. Once placed into the tank with all my other stuff, I have yet to see anything come up "MIA", but this is just my exp. I wouldnt mind if it took out a snail or two, maybe even the crabs- just so it all gets eaten. No wasting stuff around here!