Pruning it back occasionally is the only practical preventative I know. I've had it happen when there was a big change in tank environment (moving caulerpa between tanks). I've also had it happen when I let my refugia get really packed with macroalgaes. Some specie seem more prone to it than others. Feather caulerpa is, IME, especially bad.
I've pretty much rid my system of caulerpa due to it leaking caulerpins into the water. Sometimes I regret it as caulerpa seems to be an especially fast grower under certain conditions. Anyway, if you are growing it in a tank where you can float the algae, you might try Chaetomorpha, instead. Some of the browns grow fairly quick and don't "go sexual" or spread readily by fragmentation. Sargassum is a brown that is easy to float in a sump and grows fast.