Voice Memos are synced through iTunes.
First you have to turn on music syncing. You don't have to actually sync any music, but that's where the option for Voice Memos is.
When your device is plugged in, click on its icon in iTunes then click on Music on the left-hand sidebar. Check the box next to Sync Music, then ensure Include Voice Memos is also checked.
If you don't want to actually put any music on the device, click the "Selected playlists ..." radio button and then don't check anything in the Playlists, Genres, Artists or Albums areas.
Click on Apply (or Sync) toward the bottom right. Your voice memos will transfer to your computer.
Once on your computer, they'll be in your iTunes Media folder (usually at ~/Music/iTunes/iTunes Media) under Voice Memos. Copy them from that location into a safe place.
Then go back into iTunes, select all on the automatically-created Voice Memos playlist (under Music), and press and hold Option while you right-click and choose Delete. You will be prompted to confirm you want to delete the items, and then to move them to the Trash. I would suggest you do this, since you made a copy of them above.
Sync again and they will be removed from the device as well.