I think that there's a critical mass... Not sure what it is. In small tanks, it's impossible to achieve so there will always be hostility.
I have 7 yellow tangs and one purple tang. These would usually not get along, but seem ok.
I think the key word is this: if it's two fish, then one fish will want to dominate and they will fight. This is because the aggressor has nothing to fear... It's one on one.
With three, the aggressor can still be a pain, but it has to keep two other fish in line assuming they're both smaller. This distraction means that each of the two smaller fish has less damage. Also, if it's two big fish and one small, the big fish will have less ability to pick on the small because they each also have to watch their own back from the other big fish.
It's this constant fear about becoming attacked while preoccupied with a weaker fish that keeps all fish in a group calm. I think that very small or very big fish can still be an issue, but if they're all relatively close, they will defer to accepting that the territory they're living on can't belong to any one fish- they must share.
Also, tanks can usually be designed like prison blocks where all fish see all other fish all the time. That pretty unnatural and means that weak fish really can't avoid bigger fish. It's a glass prison with a hostile inmate who never stops looking at you! The weird rock placement and shapes makes it so there's more places to hide than be visible. If the fish get spooked, they can all disappear in a flash. Not one fish is visible - looks like an empty rockscape even though there's two dozen fish in there.
1 very large blue hippo tang
1 large two stripe rabbitfish
1 large desjardini sailfin tang
1 medium make naso tang
1 medium purple tang
1 medium Indian Ocean sailfin tang
7 small-medium yellow tangs
1 large dragon goby
1 large starry blenny
1 dusky jawfish
1 medium melanarus wrasse
2 medium six line wrasses
1 medium yellow wrasse
1 medium female lyretail anthias
2 medium green chromis
1 large male green mandarin dragonet
1 medium copperband butterfly
2 false percula clownfish (mated in RBTA)
In 27" x 96"
It's just my take on it, but I'm sure there's a critical number or ratio that controls school behavior for tangs and other fish.