I'm late to tell you to not remove the rocks; periphyton is now dead, and will take a year to re-grow. But this still might be of interest.
Your original problem is similar to most people's: the rock has accumulated phosphate for 12 to 24 months, because the previous export was not enough. During the first 12 to 24 months of many tanks, it may seem that your filters etc are "working", but that's because your rock is doing the absorbing for you. In 12 to 24 month, they fill up, then you have what you have now.
Phosphate flow out of rocks
Many people, when they get their nutrients low for the first time, get worried when more (not less) algae starts to grow on their rocks. It seems really strange, especially when nitrate and phosphate have gone lower than before. What is happening is that phosphate is coming out of the rocks. Remember, phosphate is invisible, so you can only see the effects of it, and it always "flows" from higher concentrations to lower concentrations (just like heat does).
Example: If your room is warm, and you put a cold object on the floor, heat from the air in the room will "flow" into the object until the object and the air are the same temperature. Example 2: If you put a hot object on the floor, heat will "flow" out of the object and go into the air in the room, again, until the air and the object are the same temperature. Now suppose you open your windows (in the winter). The warm air in your room will go out the windows, and it will get colder in the room. The object on the floor is now warmer than the air, so heat will flow out of the object and into the air, and then out the window.
Think of phosphate as the heat, and your rocks as the object, and your windows as the scrubber. As the scrubber pulls phosphate out of the water, the phosphate level in the water drops. Now, since the phosphate level in the water is lower than the phosphate level in the rocks, phosphate flows from the rocks into the water, and then from the water into the scrubber. This continues until the phosphate levels in the rocks and water are level again. And remember, you can't see this invisible flow.
This flow causes an interesting thing to happen. As the phosphate comes out of the rocks, it then becomes available to feed algae as soon as the phosphate reaches the surface of the rocks where there is light. So, since the surface of the rocks is rough and has light, it starts growing MORE algae there (not less) as the phosphate comes out of the rocks. This is a pretty amazing thing to see for the first time, because if you did not know what was happening you would probably think that the algae was mysteriously being added to your tank. Here are the signs of phosphate coming out of the rocks:
1. The rocks are older, and have slowly developed algae problems in the past year.
2. Your filters have recently started to work well, or you made them stronger.
3. Nitrate and phosphate measurements in the water are low, usually the lowest they have been in a long time.
4. Green hair algae (not brown) on the rocks has increased in certain spots, usually on corners and protrusions at the top.
5. The glass has not needed cleaning as much.
Many people have never seen the effects of large amounts of phosphate coming out of the rocks quickly. But sure enough it does if you keep nutrients low enough in the water. How long does it continue? For 2 months to a year, depending on how much phosphate is in the rocks, how much feeding you do, and how strong your filters are. But one day you will see patches of white rock that were covered in green hair the day before; this is a sure sign that the algae are losing their phosphate supply from the rocks and can no longer hold on. Now it's just a matter of days before the rocks are clear.
And...
What is Periphyton?
Periphyton is what turns your rocks different colors. You know... the white rocks you started with in SW, or the grey rocks (or brown wood) you started with in FW. After several months or years, the rocks become a variety of different colors and textures. Why? Because the periphyton that has grown on it is a mix of different living things, of different colors, and thicknesses. And the important part is: It is LIVING.
That's right: The colored stuff that has coated your rocks is all living organisms. Sponges, microbes, algae, cyano, biofilms, and of course coralline. After all, "peri" means "around the outside", and "phyto" means "plant". Ever slipped in a slippery puddle? That's probably periphyton that made it slippery. It's a very thin coating on the rocks, sometimes paper thin.
There is a lot of photosynthetic organisms in periphyton, and this of course means that they need light; but they need nutrients too (ammonia, nitrate, phosphate). And as you might figure, they will be on the lighted portions of the rocks. And they will grow to intercept food particles in the water, based on the water flow. Just think about how sponges orient their holes for water flow; the micro sponges in periphyton do it too but on a tiny scale.
What about under the rocks, in the dark areas? Well these periphyton don't get light, so they are primarily filter feeders. So they REALLY grow and position themselves to be able to intercept food particles. And they don't really need to fight off algae, because algae does not grow in the dark, so they have no need for anti-algae tactics like plants in illuminated areas have.
Reef studies have shown that at certain depths, more of the filtering of the water comes from periphyton and benthic algae than comes from the phytoplankton which filters the deeper water. And in streams, almost all the filtering is done by periphyton. So, what you have on rocks that are "mature" or "established" is a well-developed layer of periphyton; and all the things that comes from it.
This is why mandarin fish can eat directly off the rocks of an "established" tank (tons of pods grow in the periphyton), but not on the rocks of a new tank. Or why some animals can lay their eggs on established rocks, but not new ones. Or why established tanks seem to "yo-yo" less than new ones. Even tangs can eat periphyton directly when it's thick enough. Yes periphyton can also develop on the sand, but since the sand is moved around so much, the periphyton does not get visible like it does on rocks. So thick periphyton on established rocks is your friend. And totally natural too. Keep in mind though I'm not referring to nuisance algae on rocks; I'm only referring to the very-thin layer of coloring that coats the rocks.
But what happens when you "scrape the stuff off your rocks"? Well you remove some of the periphyton, which means you remove some of your natural filter and food producer. What if you take the rocks out and scrub them? Well now you not only remove more of your natural filter and food producer, but the air is going to kill even more of the microscopic sponges in it. And what if you bleach the rocks? Well, goodbye all filtering and food producing for another year. It's an instant reduction of the natural filtering that the periphyton was providing.
However, what if you just re-arrange the rocks? Well, some of the periphyton that was in the light, now will be in the dark; so this part will die. And some of the periphyton that was in the dark will now be in the light, so it will not be able to out-compete photosynthetic growth and thus will be covered and die too. And even if the light stays the same, the direction and amount of water flow (and food particles) will change; sponges that were oriented to get food particles from one direction will now starve. So since the light and food supply is cut off, the filtering that the periphyton was providing stops almost immediately, due only to the re-arranging of the rocks.
Starvation takes a little longer. The periphyton organisms won't die immediately, since they have some energy saved up; but instead, they will wither away over several weeks. So on top of the instant reduction in filtering that you get by just moving the rocks, you get a somewhat stretched-out period of nutrients going back into the water. And after all this, it takes another long period of time for the periphyton to build up to the levels it was at before: 1 to 2 years. Even changing the direction of a powerhead will affect the food particle supply in the area it used to be pointed at.
So a good idea is to try to keep everything the same. Pick your lighting, flow, layout, and try to never move or change anything. It's a different way of thinking, but you should have a stronger natural filter and food producer because of it.