The Apple Silicon Macs all have encryption of the drive enabled at all times - no matter if you enable FileVault or not.
This means that the contents of the drive is encrypted (i.e. not immediately readable) - and you won't be able to just take out the drive (which is in itself a more difficult operation than usual) and read it from another computer. It is certainly not something that an average home user would ever be able to do.
The practical difference between having a password-protected system with FileVault disabled, and a password-protected system with FileVault enabled is how difficult it is to get to your files without knowing your password.
For a password-protected system with FileVault enabled, it is practically impossible for the average home user or even IT professional to get to your files without knowing your password. If you're directly targeted by a nation state, circumstances might be different - but that's not the practical world most of us live in.
For a password-protected system with FileVault disabled, you will ordinarily be required a password to login. However, there are ways to circumvent that and access your files, that could be accessible to home users. You're certainly exposed to flaws (bugs) in lots of different software packages that could help a malicious person gain access to your files.
A user would be able to just boot the Mac holding down Opt-R to enter Recovery Mode and gain access to the files. On non-Apple Silicon systems, this access can be somewhat blocked by enabling a Firmware Password, which means that booting into Recovery Mode requires a password. However, for some older machines it is possible for Apple (and possibly others) to reset that Firmware password. On recent Intel machines, removing the Firmware Password automatically erases the contents of the disk.
On your Apple Silicon system, there's no concept of a Firmware Password as such. Instead the system automatically protects access to the Recovery Mode with your password when you have enabled File Vault.
So all in all, I would highly recommend enabling FileVault on your computer.
There's absolutely no performance impact in doing so (remember that your drive is encrypted no matter what). The main risk or concern involved in enabling FileVault is that you might forgot your password and recovery key - this can be somewhat mitigated by letting the system store a recovery key with your iCloud account.