Coral banded shrimp will aggravate them to death by picking off the papillae or pieces of the mantle bit by bit. Some cleaners may do the same. I can't think of anything else that would be problematic unless you have a musselcracker fish over about 50 pounds or a large ray in your tank