This piece from Apple Platform Security Guide (HTML, PDF) seems to suggest that only users authenticated on this particular Mac can boot it from an external drive:
A Mac with Apple silicon doesn’t require or support a specific media boot policy, because technically all boots are performed locally. If a user chooses to boot from external media, that operating system version must first be personalized using an authenticated reboot from recoveryOS. This reboot creates a LocalPolicy file on the internal drive that’s used to perform a trusted boot from the operating system stored on the external media.
This means the configuration of booting from external media is always explicitly enabled on a per operating system basis, and already requires user authorization, so no additional secure configuration is necessary.
Another piece: as of M1, volumes have ownership. When creating a second boot drive, a consent is required from a user on the default boot drive to hand off Ownership to the users on the second boot drive.
Which suggests that users who "own" an external boot drive could be not allowed to boot someone else's Mac as they are not on the same hierarchy with users on that Mac.
But I would very welcome someone more knowledgeable to comment on that.