Ken- that's a beautiful clam you've got.. and your shot does it justice I imagine! your color edit was a great improvement!
your edging looks great for the majority of the mantle.. except on the left hand side and a few other bumps.. that last slope looks especially jagged to me..
there are lots of ways to smooth out your lines.. layering/blurring/smudging/blending could all potentially help.. If you look at your first image, the rim of the mantle looks very smooth.. not sharp and jagged like the second.. mostly I think this was because of the smart brush selection which cannot come close to the sensitivity of our eyes..
what editing program are you using? Im not sure what your options are.. but just zoom in till you cant stand it any more and go for a manual selection of your background if possible- that way you can maintain control of pixel selection. It takes way more time yes.. but its worth it in the end I think... on CS3 you can create layer masks and blend multiple layers with great control and results to avoid a 'tweaked' look.. I honestly haven't done much layer masking myself, but it could be effective for this image.
just my thoughts.. thanks for sharing