IMO it depends mostly on your display and interest. I have never cleaned anything other than the front glass of my tank (and I can only clean 75% of that since a whopper huge T. Deresa clam is leaning aganst the rest of it). The one thing that you may want to consider is that you are talking about a huge surface area. Over the years, along with the C. algae, it will host a large volume of cryptic organisms. These nifty little guys are fantastic water scrubbers. I will always take every little bit of natural filtration that mother nature can provide.
I have always tried to make my tanks look like a window to another world. The heavily encrusted look adds depth to my field of vision (other than staring through to the wall, a royal blue background or one of those cheesy petstore fish backgrounds!) and hides my overflow box from sight.