While unable to identify the specific species, I have come to learn that most nudibranch and flatworms have two strategies as to how they appear, they will be either vividly colored to warn off predators that they are poisenous or they mimic the coloration and pattern of their usualy single specific prey/food source. As such, the one shown in your photo appears likely that it COULD be a coral predator as many such predators do mimic the brown/mottled appearance of the coral they feed upon, at the same time, it could be a harmless herbivoure that is found in muddy, sandy locations, in short, I would play on the side of caution and not return them to the tank. Very few nudibranch are actualy herbivours so the odds are against you.
Chuck