How to use VPN with OS X Server?

Question

I have recently set up a server on my Mac mini with the OS X Mountain Lion Server app. I would like to have remote access (from outside of my home's wifi network) to the files and folders in my home folder and shared locations (on the Mac mini), and I understand that to do this I need a VPN connection. I'm not familiar with these kinds of things, and I've been trying to set this up but I can't get it to work.

It's worth mentioning that I don't actually have a registered domain name - the host name is server.example.private (I've replaced "example" with something else). From any computer connected to my home's wifi network I can access the server's wiki by entering "192.168.0.(some number)" into the address box, but I cannot access it by using "server.example.private" (even from the wifi network) - this might indicate what the problem is. Furthermore, I can access my home folder and shared locations using Finder from a Macbook Air (I can only do this when connected to my home's wifi network) by connecting to my Mac mini and logging in - I would like to be able to do this remotely.

I'm probably just doing something really stupid, like not configuring some setting (or perhaps not setting up DNS properly...). Could anyone please explain the steps required to set up VPN (and maybe different aspects relating to/depending upon this)? All the tutorials I've seen so far ask me to register a domain name, and I'm not quite ready to do that yet (I hope that's not the problem). Please let me know if you need any more details.

Thanks.

Answer 1

OSX Server does have several VPN servers in it. You can configure these using Server.app however, if your internal name resolution is broken this is likely to be more trouble.

You'll also need to make sure your Mac Mini's VPN port is available to the internet on a static IP (or dynamic host name).

As an interim step I suggest looking at 'Back To My Mac' which is free as part of iCloud. Technically it uses adhoc ipsec vpn tunnels and gives you an DNS name for each mac set up, effectively bypassing the current internal name resolution issue. e.g. server.12345678.members.btmm.icloud.com Services that you run e.g. web and ssh will accessible.

They're btmm DNS address are not internet routable and don't, generally, require any port forward or static IP addresses. For a couple more technical details check here: https://apple.stackexchange.com/a/53776/46039

Answer 2

I found the solution! The three problematic factors were as follows, and these may also apply to other people trying to set up VPN for Mac OS X Server:

1. I had to ensure that port forwarding was properly set up, as I didn't really see this as an important thing - once the VPN service was working, I disabled the port forwarding for the various VPN ports (500, 1701 and 4500, I think) and it stopped working.
2. I had to use the server's host name on its VPN's config page (server.example.private), BUT I had to use the external/public IP address (like the one you get from whatsmyip.org) on the client's VPN's config page.
3. DON'T ENTER THE GROUP NAME ON THE CLIENT'S MACHINE UNLESS YOU REALLY HAVE TO! I entered my group's name, and it didn't work (maybe because I am an administrator as well...)

I hope this helped, at least it's working for me now, and please let me know if you need any more help with any problems that occur while setting up VPN!