First, a warning: unsupported backup targets are unsupported for a reason. I don't know the details, but apparently there's some special sauce (or at least a newer version of the AFP protocol) needed on the server to cleanly handle things like getting disconnected partway through a backup. As a result, backups to unsupported network servers seem to get corrupted more than they should, which isn't good for a backup system.
Now about security: the backup is stored on the server in the form of an encrypted disk image (in sparse bundle format). The image is mounted on the client (your computer), which means that the encryption/decryption takes place on the client, and all that goes over the network (and is seen / stored by the server) is encrypted blocks of the image.
To answer your questions directly:
- Yes, the backup is encrypted before being sent to the server.
- No, the AFP protocol is not particularly secure, but it doesn't matter because the data being transferred over AFP is encrypted.
- Yes, the TM data on the NAS is encrypted.
BTW, be sure to store your backup password somewhere safe -- If your HD dies, and you don't remember the backup encryption password, there'll be no way to get your data back.