We have both Ilyanassa Obsoleta and Nassarius Vibex here in North Carolina. I've had both in my tank and I can tell you what I've learned from both.
First let me say that I've never had a problem with either, but I've never had a clam in the same tank either.
Ilyanassa Obsoleta have dark brown shells, sometimes so dark they appear brown. Their shells are also smooth. They will sometimes bury in the sand and act just like a Nassarius Vibex. I've never lost anything to them except some snail on snail violence against each other.
Nassarius Vibex have a gray shell, usually on the lighter side. Their shells are nobby at the ends. The easiest way to tell them apart from the whelk is the length of the foot. This obviously is not the tell tale sign of a vibex but its the easiest way for me to make sure its not a whelk. The Vibex foot should easily be seen sticking out past the aft end of the shell. Vibex snails to not always hide in the sand, especially when being introduced into a new tank. Once they are comfortable with the new tank then they will hide in the sand. I just added some today and about 25% of them are on the glass right now.
Not sure if that helps, hope so though.
