Do you have anything else with them?
If not, they may see the lack of other fish as evidence there is a predator nearby (everyone is "hiding", i.e. that early scene in Finding Nemo).
You may need a dither fish or two. Otherwise just give them some time.