Mountain Lion VPN fails to connect but works on Windows VM

Question

I want to setup a VPN on my mac but I can't make it work. If I start my Windows 7 VM, the VPN connects fine (from Windows) but if I try to connect from my Mac it fails…

First of all, here is the system log:

Oct  7 15:18:45 Leonardo-Ferreiras-iMac.local configd[18]: SCNC: start, triggered by System Preferen, type L2TP, status 0  
Oct  7 15:18:45 Leonardo-Ferreiras-iMac.local pppd[3385]: pppd 2.4.2 (Apple version 596.15.2) started by leo, uid 501  
Oct  7 15:18:45 Leonardo-Ferreiras-iMac.local pppd[3385]: L2TP connecting to server 'vpn.aec.com.br' (186.249.4.10)...  
Oct  7 15:18:45 Leonardo-Ferreiras-iMac.local pppd[3385]: IPSec connection started  
Oct  7 15:18:46 Leonardo-Ferreiras-iMac.local racoon[192]: Connecting.  
Oct  7 15:18:46 Leonardo-Ferreiras-iMac.local racoon[192]: IPSec Phase1 started (Initiated by me).  
Oct  7 15:18:46 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Initiator, Aggressive-Mode message 1).  
Oct  7 15:18:47 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:18:47 Leonardo-Ferreiras-iMac kernel[0]: Sandbox: sandboxd(3386) deny mach-lookup com.apple.coresymbolicationd  
Oct  7 15:18:48 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:49 --- last message repeated 2 times ---  
Oct  7 15:18:49 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Phase1 Retransmit).  
Oct  7 15:18:50 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:18:50 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:51 --- last message repeated 2 times ---  
Oct  7 15:18:51 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:18:52 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:52 --- last message repeated 1 time ---  
Oct  7 15:18:52 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Phase1 Retransmit).  
Oct  7 15:18:52 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:52 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:18:53 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:54 --- last message repeated 2 times ---  
Oct  7 15:18:54 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:18:54 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:55 --- last message repeated 2 times ---  
Oct  7 15:18:55 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Phase1 Retransmit).  
Oct  7 15:18:55 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local pppd[3385]: IPSec connection failed  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local pppd[3385]: L2TP connecting to alternate server 'vpn.aec.com.br' (200.251.240.177)...  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local pppd[3385]: IPSec connection started  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local racoon[192]: Connecting.  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local racoon[192]: IPSec Phase1 started (Initiated by me).  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Initiator, Aggressive-Mode message 1).  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:56 --- last message repeated 2 times ---  
Oct  7 15:18:56 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:18:57 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:18:59 --- last message repeated 2 times ---  
Oct  7 15:18:59 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Phase1 Retransmit).  
Oct  7 15:19:00 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:19:00 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:19:01 --- last message repeated 2 times ---  
Oct  7 15:19:01 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:19:02 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Phase1 Retransmit).  
Oct  7 15:19:03 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:19:03 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:19:05 --- last message repeated 2 times ---  
Oct  7 15:19:05 Leonardo-Ferreiras-iMac.local com.apple.quicklook.satellite[3358]: [QL] No sandbox token for thumbnail request file://localhost/private/var/log/racoon.log, it will probably fail  
Oct  7 15:19:05 Leonardo-Ferreiras-iMac.local racoon[192]: IKE Packet: transmit success. (Phase1 Retransmit).  
Oct  7 15:19:05 Leonardo-Ferreiras-iMac.local sandboxd[3386] ([3358]): QuickLookSatelli(3358) deny file-read-data /private/var/log/racoon.log  
Oct  7 15:19:06 --- last message repeated 2 times ---  
Oct  7 15:19:06 Leonardo-Ferreiras-iMac.local pppd[3385]: IPSec connection failed  
Oct  7 15:19:06 Leonardo-Ferreiras-iMac.local racoon[192]: IPSec disconnecting from server 200.251.240.177  
Oct  7 15:19:06 Leonardo-Ferreiras-iMac.local racoon[192]: IPSec disconnecting from server 186.249.4.10  

Second: The Windows 7 VPN manual (the only one available) states that I should check the box "CHAP Protocol" & "CHAP v2" and I must select the option where encryption is required.

Third: I must also enter the pre-shared-key, which I deduce is the shared-secret option of mountain lion.

Still no luck at all… can anyone help?

EDIT 1: The dump file is here

Answer 1

It seems odd that your machine is attempting to use aggressive-mode in phase 1 of the IPSec connection. From what I can see, the default in 10.8.5 is to use main-mode. I'm wondering if you've modified the file at /etc/racoon/racoon.conf in trying to get this working? Can you post the contents of the file at /etc/racoon/racoon.conf?

Alternately, it may be possible to duplicate the configuration file that the L2TP/IPSec connection in System Preferences is using, and modify it slightly (configure to use main-mode) to see if that helps. Unfortunately, that configuration file is created at connect time, and is copied into /var/run/racoon/fqdn.of.host.conf. This file is also readable only by root (to protect the PSK that's stored inside of it). If you're quick, you can prepare a Terminal command that will copy that config file, initiate the VPN connection from System Preferences or the menu bar, and perform the copy command while the VPN is attempting to connect (you may have 5-10 seconds to perform the copy). Here's the command that you'd run right after initiating the VPN connection:

sudo cp -Rp /var/run/racoon/vpn.aec.com.br ~/Desktop

That file has permissions that disallow users from reading it. Use the command to make it readable:

sudo chmod 777 ~/Desktop/vpn.aec.com.br

NOTE: the file that you've copied onto the Desktop contains your PSK. You should not post this file (unmodified) to this website. If you would like to post this configuration file for troubleshooting, obfuscate the value of shared_secret before posting.

You can configure the VPN connection to use main-mode by modifying the configuration file that you've copied onto your Desktop. Open that file in any text editor, and locate the line that says exchange_mode. You may find that it says something like:

exchange_mode aggressive;

If so, see if you can change it so that it uses main-mode:

exchange_mode main;

Next, disable the built-in launchd script that launches racoon normally:

sudo launchctl unload /System/Library/LaunchDaemons/com.apple.racoon.plist

Then, you can temporarily launch racoon, and instruct it to use the configuration file that you exported:

sudo racoon -f ~/Desktop/vpn.aec.com.br -l /var/log/racoon.log

The racoon process will run in the background using the configuration file that you copied onto the Desktop. The process will also output to /var/log/racoon.log Next, you can initiate a VPN connection using racoonctl:

sudo racoonctl vpn-connect vpn.aec.com.br

The VPN menubar item will not update while this process is connecting. However, you should see output in /var/log/system.log and in /var/log/racoon.log that reflects the fact that the IPSec Phase 1 connection is moving along (i.e., Main Mode Message 2, 3, 4, etc.). If you see that the machine is able to establish the IPSec tunnel (look for "IPSec Phase1 established" in system.log), you're likely having problems with the exchange_mode that's configured.

How to undo this configuration

To put things back to the way that they were, you can disconnect (if the connection was successful) and then terminate the racoon process that's running:

sudo racoonctl vpn-disconnect

sudo killall racoon

Then, re-load the launchd config for racoon:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.racoon.plist

How to make this configuration permanent

If the racoon configuration file that you used in the tests above seems to be allowing phase 1 to complete, you can configure racoon to use that config file in the future:

Copy the config file that is on your Desktop into the folder /etc/racoon:

sudo cp ~/Desktop/vpn.aec.com.br.conf /etc/racoon/vpn.aec.com.br.conf

Make sure to secure the permissions on that file:

sudo chmod 600 /etc/racoon/vpn.aec.com.br.conf

Comment out the last line of the file at /etc/racoon/racoon.conf, and add a line that will load the configuration that you copied to /etc/racoon in the step above:

include "/var/run/racoon/*.conf" ;

Should be changed to:

#include "/var/run/racoon/*.conf" ;
include "/etc/racoon/vpn.aec.com.br.conf" ;

After you've made this change, you can save the file and attempt the VPN connection again from the menu bar.

Answer 2

1. Make sure that Back to My Mac is disabled under System Preferences → iCloud.

2. There are certain characters that OS X does not like in shared secrets. Make sure that your shared secret…

   • is shorter than 64 characters in length
   • does not contain ' (apostrophe)
   • does not contain ′, ″ or ‴ (prime, double prime or triple prime)

3. Allow access to your key stored in Keychain Access:

   1. Open Keychain Access (/Applications/Utilities/)

   2. Select the System keychain on the top-left list of keychains.

      

   3. Find the certificate relating to your VPN server and click the disclosure triangle to the left of it to show the Private Key (should have a key icon).

      

   4. Double-click it to open the Private Key properties, and select the Access Control tab.

   5. Select Allow all applications to access this item.

   6. Click Save Changes and enter your administrator password.