It's a lot more than just speculation.
In the case of the lionfish, genetic tests have confirmed that the entire Atlantic population traces its roots to about 6 females (and an unknown number of males), all from one small region in Indonesia. If these fish were brought in by ballast water it would be the first case of ANY scorpionfish becoming established anywhere in the world through a ballast water introduction in the entire history of shipping. Previously, there are only 3 other cases of single individual scorpionfish being transported via ballast water. It's a rare event. The chances of introducing not one, but 6 females, or larvae from 6 females, all from the same place, to a single location (and nowhere else in the world) at around the same time, are unfathomably low. Ballast water introduction is simply not a plausable explanation.
However, we know that same part of Indonesia where these fish came from is one of the world's biggest exporters of aquarium fish. We also know that around 6 lionfish are known to have escaped into Biscayne Bay in 1992, probably all from Indonesia. By the early 2000s, lions were established all the way up the East Coast, but no further south than Miami. Given the direction of the currents, that suggests Miami as the point of introduction. There are known aquarium introductions and they fit the evidence neatly as the source of introduction, whereas ballast water doesn't.
The first Caulerpa infestation was found directly adjacent to the Monaco aquarium and genetic tests have since confirmed that the Mediterranean algae is identical to the variety found in the Monaco Aquarium and the Stuttgart Zoo (where the Monaco aquarium got their Caulerpa. That genotype is not found anywhere in nature.