You could try a razor blade to peel it away slowly, though they don't seen to attach like a mushroom. As Sugar_Magnolia stated, your best option is likely to chisel it off.
Depending on your rock density you may try a strong knife. My rock is porous enough that I can actually force a knife into it. Worth a shot, just don't stab yourself!