A lot of it has to do with the fact that they are horrible shippers. Being able to spot a healthy specimen is very difficult. In the times I worked in fish retail and as a hobbyist, I would say that I ran into a specimen worth buying a couple of times a year at the most.
They require lots of flow, strong lighting and exact attention to detail. A tank that would be amenable to the most exacting sps corals would befit a magnifica anemone.