I've seen this few times now. Here is an excerpt from my Advanced Marine Aquarium Techniques book:
Hyperextension of mouth: In an unusual medical case, a blue line angelfish, Chaetodontoplus septentrionalis bit so vigorously at some algae that its mouth became stuck in a hyperextended position. While the fish could still breath, it could not feed normally. As one tenet of medicine is, "œFirst, do no harm" the fish was left to its own devices to se if it would recover on its own. After a day with no improvement, the fish was anesthetized using MS-222, and the mouth was gently manipulated. The mouth seemed completely locked in the wide open position. Eventually, a small bone was located at the top of the isthmus between the jaws and the fish's head. Depressing this, (like the locking spine on a triggerfish) allowed the entire mouth structure to return to its normal position. The fish began feeding normally the next day.
To add to that, after this was written, the angel got it again, and then got better on its own. I've also had a Serranus basslet get this, and it got better on its own.
Did this come on suddenly or over time? If suddenly, then it is probably hyperextension. If it developed over weeks, then it is probably a tumor (usually hyperplasia of the thyroid).
Jay Hemdal