They both look like sponges. However, I had a big algae problem a couple of months ago and had this stuff just like your bottom picture that grew over the rocks. It wasn't hard, but you could peel it off in small sheets using tweezers. I had to keep removing it by hand and it eventually went away. If it is growing noticeably then you may have the same problem.
-Mike