Take the 2-minute tour ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.
  • I'm running OS X 10.9 with Server 3.0.1 on a Mac residing on a private subnet sitting behind a router whose WAN port is plugged into a cable modem, hence the ISP is a well known cable internet service provider.

  • This server has its DNS correctly configured (i.e., the result of "sudo changeip -checkhostname" on the server via command line yields perfect results).

  • This server is running an Open Directory Master.

  • Dynamic DNS is correctly configured for the router, and resolves the public IP address assigned by my ISP to the same domain name as the server (I forward any required ports for services I run).

  • This server also has installed a server certificate signed by a trusted CA (i.e., Go Daddy) and is working perfectly for all OS X Server services including Open Directory.

  • This server also has the Mail service configured (SMTP and IMAP) with no problems (I can send and receive mail to/from the server.

  • This server server also has push notifications enabled and has a push notification certificate installed perfectly (obtained from Apple's push notification certificate portal, just renewed a few days ago).

  • I have some iOS devices running iOS 7.0.4. I have configured Mail on these iOS devices to send and receive mail to/from the aforementioned server for a few different user accounts on the server. This works fine (tested, can send and receive mail no problems).

  • The aforementioned iOS devices mail settings for the aforementioned server have been configured to receive push notifications when mail is received to said user accounts on the server.

With all of that said, the iOS devices are sometimes able to receive push notifications from the Apple Push Notification Service (APNS "cloud") in situations when the iOS devices reside on the same private subnet as the Mac server (via Wi-Fi), and when they reside on the public Internet (via cellular data networks or public Wi-Fi networks such as coffee shops).

Thus, push notifications do work when mail messages on the server are received, but not always. After a period of time has elapsed on the server whereby no email messages have been received (seems to be several hours but I haven't yet been able to pinpoint it precisely), the server apparently loses what is supposed to be a persistent connection with the APNS gateway. OS X Server unfortunately does not log when this connection is lost. Then, when a new email message finally arrives again after several hours and is received by the server, the iOS devices do not receive their expected push notifications and instead OS X Server consistently logs an error message like this one (the only differences being of course the process ID and time/date stamp):

11/26/13 5:48:11.762 AM push_notify[181]: stream: received error: The operation couldn’t be completed. Connection reset by peer on: incoming stream: APN to host: gateway.push.apple.com:2195

Once the above error type is recorded in the logs, a subsequent email message sent to the server successfully generates a push notification message on the configured iOS devices so long as the subsequent message is sent before minimal time has elapsed (i.e., several hours). I do not port forward ports 2195 or 2196 from the router to the Mac server because Apple's support document implies these ports are for outgoing traffic (from the server to the APNS gateway) unless I've misunderstood.

An excerpt in Apple's Mac Developer Library Technical Note TN2265 caught my attention with respect to idle:

An occasional disconnect while your provider is idle is nothing to be concerned about; just re-establish the connection and carry on. If one of the push servers is down, the load balancing mechanism will transparently direct your new connection to another server assuming you connect by hostname and not by static IP address.

Is OS X Server (the "provider" in this context) essentially just "carrying on" by "re-establishing" the connection to APNS after being "idle" for several hours as noted in the logs per the stream error aformentioned?

Someone I spoke with about this posited the above problems may be due to the router's WAN port not being assigned a static IP address by my ISP, but of all the Apple developer documentation and support docs I've looked at about push notifications with OS X Server do not state a static IP address is required.

note: I have also tested this with the same hardware and settings but running OS X 10.8.5 Mountain Lion with Server app 2.2.1 with essentially the same results but IMHO better log verbosity, as in:

11/29/13 11:16:55.713 PM push_notify[11951]: stream: received error: The operation couldn’t be completed. Connection reset by peer on: incoming stream: APN to host: gateway.push.apple.com:2195

11/29/13 11:16:55.722 PM push_notify[11951]: Disconnected from apn server gateway.push.apple.com for topic com.apple.mail.XServer.2a132c32-dda4-45a1-68e1-b3cca3865c12: error Connection reset by peer

11/29/13 11:16:55.722 PM push_notify[11951]: will attempt to reconnect stream APN to host gateway.push.apple.com:2195 in 15 seconds

Any help or suggestions resolving this would be greatly appreciated, it may be something simple I've overlooked.

share|improve this question

3 Answers 3

up vote 1 down vote accepted

This problem was been resolved. The ASUS "Dark Knight" router which was providing the private LAN (NAT) and port forwarding to the Mac running OS X Server has a firmware bug. The bug manifest by DROPPING the ESTABLISHED TCP connection on port 2195 between the Mac running OS X Server and APNS, after two hours of quiescence. The router should not have dropped this connection, there was no firewall rule instructing it to do so. The lesson learned is to be much more selective and wise with regard to choosing routers (especially consumer off-the-shelf variety) for use with servers even for servers run in a small business context (like a Mac Mini Server).

Heading

share|improve this answer
    
Great solution, how did you determine it was the router dropping the connection? (In my case the router is also the Internet gateway, so I can't simply remove it from the equation). –  Josh Jun 20 at 23:00
1  
Apple Enterprise was very helpful in assisting me with debugging this situation. It took a while to figure this out because neither of us expected the router to drop packets. To finally isolate the problem, Apple Enterprise support asked me for a packet trace between the router and my ISP's modem. I connected a hub (not a switch) between the router and modem, and connected another computer to the hub. On this other computer I ran: sudo tcpdump -s 65535 -w ~/router_modem.wireshark port 2195 or port 2196 or port 5223 or port 1640 This allowed us to finally diagnose it was the router (Asus). –  user3051849 Jun 22 at 13:21

My first steps in diagnosis would be to forward both those ports and see what happens (I might even put the machine into a DMZ for a couple of hours).

I'd then put a packet sniffer on the 17.0.0.0/8 address block and see what traffic on what ports is going across the net when it does work and when it doesn't.

share|improve this answer

I believe that @user3051849 has the root cause, but in my case I can't swap out the router. So, enclosed are my two workarounds. As of OSX Server 3.1.2 this problem remains.

Use Apple's launchctl to create a job that runs every 2 hours to restart the mail server. So, this should only be used on mail servers that are not busy. Create a plist file in /Library/LaunchDaemons/MailRestart.plist, the contents should be:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>mailrestart</string>
    <key>ProgramArguments</key>
    <array>
        <string>/etc/periodic/daily/500.restart-mail</string>
    </array>
    <key>StartInterval</key>
    <integer>7200</integer> <!-- Every 2 hours -->
    <key>StandardOutPath</key>
    <string>/var/log/daily.out</string>
</dict>
</plist>

Note that this launch agent runs Apple's /etc/periodic/daily/500.restart-mail script every 2 hours (7200 seconds). In case Apple removes the restart mail script, here is a copy for your convenience.

As an aside, it seems Apple is aware of connection problems with their push service, as they created the periodic script to restart the mail server during the daily run of the periodic service. However, their fix is rather weak, because in my case the periodic service runs daily, but my connection to the push service is dropped after 2 hours of inactivity. So, I need something more frequent than a daily mail server restart.

Next from a shell invoke:

sudo launchctl load /Library/LaunchDaemons/MailRestart.plist

This command loads the script into the Launch Control service causing the script to run every 2 hours, logging to /var/log/daily.out (the same destination that the periodic service logs too). In my case I needed to change this to under an hour, rather than every 2.

A Better Approach An alternative solution is to leave the mail daemon alone and kill the process that handles push notifications, ie. push_notify. This has the added benefit of not killing the mail daemons which may interrupt imap connections. For example, the command:

sudo launchctl stop com.apple.push_notify

Stops the push_notify service, launchd immediately restarts the service afterwards. So, the new process will have a fresh connection to Apple's notification servers. I'll leave it as an exercise to the reader to wrap the above into a script and invoke it from a modified plist as above.

share|improve this answer
    
Thanks for contributing a workaround for those who are unable to swap out their router. In your workaround you wrote It seems Apple is aware of the problem, but their fix is rather weak, as the periodic service runs daily. Are you referring to a different problem other than what I originally posted about (push notification service dropping)? I replaced the Asus with Ubiquiti EdgeRouter but haven't yet re-instated OS X Server's Mail service (first I wanted to run the EdgeRouter with just the basic to confirm its configuration is stable). I can now re-test Mail service on OS X Server 3.1.2. –  user3051849 Jun 22 at 13:30
    
Apple's mail restart script is commented with Restart mail, otherwise our push notifications get hung up. Which seems to acknowledge the problem that push_notify can fail (either through our dumb routers dropping connections, or some other root cause. However, as I describe in my edited answer their script falls short. If you run into this problem again I recommend using launchctl to restart push_notify, which seems to be working for me. –  Josh Jun 22 at 17:07
    
Using a Ubiquiti EdgeRouter, which is an enterprise class router, not a "dumb" router, I can confirm that this same problem with Mail and push notifications persists just as it had with my previous Asus "dumb" router. This is disappointing, a workaround should not be required. Apple should fix this, especially with OS X 10.10 Yosemite and Server 4.0 waiting in the wings (blaming this problem on routers is mostly a red herring). I will next approach a workaround per your (Josh's) suggestion, thanks for helping identify a workaround solution. –  user3051849 Jul 26 at 4:31
    
Josh, I can't locate Apple's /etc/periodic/daily/500.restart-mail script on my instances of OS X Server, running OS X 10.9.4 and Server version 3.1.2. I checked my archived snapshots of OS X 10.8.4, 10.8.5 and previous Server versions. I checked /Applications/Server.app/Contents/ServerRoot/etc. No luck. What version of OS X / Server are you using? I would like to see a copy of 500.restart-mail, can you post it please? btw on OS X 10.9.4 / Server 3.1.2 the path to push_notify launchd is /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.a‌​pple.push_notify.plist –  user3051849 Jul 26 at 15:30
    
@user3051849 See the revised answer with a link to the script. The Apple script does occasionally soft fail for me, as the Mail server doesn't always stop/start in time. Instead I've been restarting the push notify service which has been working well. I agree that this is a bug in Apple's push/notify not to recover gracefully from disconnects, any idea how to open a bug report with Apple? According to apple.com/support/osxserver/contact I'm past their support time period. –  Josh yesterday

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.