Dan, some good questions. I keep a mixed reef with refugium and have found that, yes, it is possible to keep them together, but I find it not optimal. Certain corals require different conditons, which are not reproducable in the small confines of a tank, when one attempt to favour the NPC.
The first item I've noted is the current. Most Dendronephthya require a very brisk current, one that is so strong that most SPS won't open well. The solution it to attempt flow-rate region, which is possible in a bigger tank.
I have rarely had problems with too much waste product due to feeding. Actually, I find there is a balance whereby the NPC help reduce the organic load through their feeding. This is a big jump of the top of the cliff, but I find one can find a balanced group of organisms that compliment each other, despite their different feeding/lighting/current requirements. Not all organisms are possible, but I find that Scleronephthya and many gorgoniana are compatable with SPS. Also, Crinoids, many sponges and tuncates.
One problem I see it proximal poisoning, a type of defense we cannot calculate in an aquarium. Some soft corals and gorgonians are known to exude toxic substances to defend their territory. Such animals need to be identified and documented. In an aquarium they can make life for other animals impossible. Sea cucumbers and certain sponges are a classic example.
I don't think any of these problems are insurmountable, simply we fail the information to deal with them, when we manage to identify the problems. We are still at the begining of a long tale.
