I would also be concerned with using terrestrial food not naturally available to marine fish, but would be VERY concerned with what Glove stated. Earthworms eat dirt and so anything present in the soil would be a possible contaminant, and it doesn't necessarily have to be ag. based. I am not at all sure at what levels the contaminants would be present in the worms but would be concerned.
Besides ag based pollution you could possibly have high levels of heavy metals and maybe even copper, also soil is chock full of microbes you may not want to introduce.
I just wouldn't risk it when good marine based food is so easily available and cheap too.