It's not truly surprising. Apple frequently makes format/database changes with upgrades. The resulting files are almost never convertible 'downhill', there is simply no provision to be able to do that.
There appears to be a 'get out of jail free' card for Photos, though, using iCloud as an intermediary. There is a long [& outdated] explanation on Downgrading a Photos Library to a Previous Version of Photos with the Help of iCloud Photos which I have never tested, so cannot comment on whether it will still work from Big Sur right the way back to Mojave.
I can confirm that deleting the Big Sur Library, creating a new Photos Lib, making it your System Library and turning on iCloud Photos will download a new Mojave compatible library.
This does rather emphasise the need for a solid backup strategy, to prevent issues like this in future. Personally, not only do I keep backups of everything, but before I do any major OS upgrade, I clone my entire boot drive as a 'belt & braces' for if I decide to go back. There are frequently issues trying to go 'downhill' using Time Machine, so I consider this to be a vital step.