I had a nice sized angler (5"+) in a 24 gallon tank (aquapod). It lived for a good year and a half before I gave it away.
I probably wouldn't go with a tank smaller than the one I used, unless you are really good at maintaining water parameters. They need large prey items for food and therefore tend to create a sizeable bioload. That's really the limitation, not the swimming space. Anglers don't really need a lot of room. They'll find their favorite spot and basically sit there waiting for food.
The only fish that IMO are suitable with frogfish are large non-piscivorous eels, like a zebra moray. Most every other fish is either small and has a good chance of getting eaten by the frogfish (even fish double the size of the frogfish can fall into this category), large and will eat the frogfish, or large and will possibly nip at the frogfish. The last category even includes tangs. Frogfish are exceptionally still and camouflaged to look like their surroundings. Tangs can nip at them thinking they're merely a rock to clean algae off.