I guess, I think it depends on the tank. Xenia wont grow at all in my tank, my clove i got is not doing great either. Brains and acans on the other hand do fine same with mushrooms.
I think invasive is a subjective term. If you like a coral why not have a tank full. If you had sps grow out and fill your entire tank would you call it invasive? Xenia and some other softies typically just grow quicker than others so people deem them invasive.