My solution is not perfect, but it is better than hearing some private voice memo when the music library is played among people... It is also better than having to remember to delete them all the time.
The solution is the following:
1) go to "/Users/yourusername/Music/iTunes/iTunes Media"
2) right click on the "Voice Memos" folder and select "Info"
3) in the "General Properties" section select "Locked"
That's it. This prevents all apps from accessing the Voice Memos folder. This also means that iTunes cannot copy voice memos to that place.
There is only one slight inconvinience with this solution, however. When trying to synchronize new voice memo iTunes shows an alert complaining that it was unable to access the disc. But this only happens during the first synchronization after new memos have been added. Until You add new memos, the alert box will not appear. So I would say this is only a slight inconvenience compared to necessity of remembering that memos have been added and deleting them. Or listening to Your private memos being played to people listening to Your music...