I've found phosphate, unless ridiculously outlandish, to have less of a direct impact on color in my experience. Stability of Alkalinity seems to be the more sensitive level for color for me. I would blame higher phosphate for slower coral growth if anything.
And, the presence of phosphate does encourage algae growth and it becomes important to keep measurements low to avoid nuisance algae from causing problems. For myself, I know phosphates have become a problem when I'm wiping my walls too often. I should be able to get 4 days or so in between wipes if everything is in check.