I have a simple question: I bought a NAS from Synology and I copy my files from macOS (my filesystem is AFPS) to the NAS with Samba, formatted in ext4. Sometimes I also copy files from AFPS to NTFS (I'm using the Paragon Software). I never had any problems, but I want to be sure: Can there be problems due to these different file systems for example that files won't be copied, because the target system doesn't support the filename? macOS would tell me, if this is the case?
Files content is just bytes and the FS job is to handle it and that's all, by copying a file you just copy the bytes and not how it being handled. SMB send the data as a raw data and the "other side" receive it as a raw data and handle this data according to it's rules.
So to answer your question, the FS does not matter.
This article suggests that the illegal character issue is a function of the OS, more than the file system. So you can write chars like * and | to an NTFS filename using MacOS, but it will cause trouble when viewed on a Windows platform.
Of course, NASes use an OS to serve the files: Synology uses a Linux-based system. As Unixy cousins, things should be similar. If Windows were in the mix somewhere, then there might be problems.
To answer your last question: MacOS can only tell you what the server OS provides to it. If the server OS misunderstands the filename and parses it differently, that may not necessarily generate an error, even if the file doesn't save as expected.
Another possible problem when passing files across different file systems and OSes is whether extended attributes remain associated with the file. I would expect a Synology NAS to handle this correctly, but it's worth checking.