Yes, different file system have different advantages.
Reliability:
For example if you compare your current ExFAT with the macOS standard file format APFS or the older HFS+, you could gain several advantages by switching:
In terms of safety against "lost files" (where the drive is actually still good) both APFS and HFS+ have the advantage of being journaled file systems. This means that they track changes to metadata (i.e. not the file contents) in a form of log on the drive.
This means that if a data transfer is not completed due to for example power loss, yanking out the drive or software error, then the file system can "roll back" the change using the information from the log. Essentially this means that although the file transfer didn't succeed, you wouldn't end up with a corrupted drive and lost access to your files.
If the card is physically damaged or "broken" then no software file system can fix that of course - that's worth keeping in mind. Always keep backups (this is true of any storage medium).
Note that there's a big difference in the quality of SD-card. Don't buy the cheapest ones - they're usually not that reliable. However you can actually get very reliable SD-cards - they're just in a different price range. I've been using industrial SD-cards for many years in a commercial setting where they're exposed to high and low temperatures, vibrations, high endurance (i.e. SLC instead of MLC), etc. and you really need high quality SD-cards in those cases. I've had good experience with cards from Cactus.
Compatibility:
One of the disadvantages of APFS/HFS+ is that they're not natively supported by for example Windows. If you try to use the SD-card with Windows, the files won't show up without installing extra drivers and support software.
Regarding your desire to use Time Machine, remember that this requires the drive to be formatted as HFS+ on Catalina or older. On Big Sur you can use APFS.
Safety:
In terms of safety in a broader perspective, the APFS file system supports full disk encryption. This means that you could have the data on your SD-card encrypted while using it as any other SD-card. If the SD-card is lost at some points, others would not be able to your read your data without your passphrase.
Speed:
Regarding speed there is a difference between the file systems. A file system such as APFS or HFS+ is usually slightly slower than FAT32 for example as they have to do extra work in terms of saving logs, checksums, etc. However in practical use, the average user wouldn't be able to tell them apart in terms of speed on a modern SD-card.
Modern SD-cards can actually provide data at very high speeds. Today you can easily buy SDXC cards of the UHS-II variety, which have data transfer rates in the range of 300 MB/s (the faster ones cost about 100$ or more depending on capacity - for example cards from Angelbird).
Newer specifications for SD card interfaces have also been released in the even faster UHS-III and even SD Express variety. This could theoretically mean data rates of up to 624 MB/s and 3938 MB/s respectively in the future.
Take note that how you connect the SD card to your Mac can affect performance. Are you using an internal card reader on an older Mac, check which card types it supports. If you're using a USB-connected card reader, ensure that you have a USB 3.0 type in order to support the fastest SD-cards at their full speed. A USB 2.0 connected card reader usually tops off at about 40-50 MB/s, so if you have faster cards than that, you would want a USB 3.0 reader.