Intriguing connection issue on OS X

Question

I've been recently having this issue with my internet connection on my MacBook Pro Early 2011 running OS X 10.8.3: from time to time the connection "freezes" for about 5 seconds and then comes back.

It happens both over Wi-Fi or over Ethernet cable, and it only happens to my machine when it's running OS X (it won't happen when running Windows 7 on the same machine, or on any other machine/device). It makes Skype drop calls every 2 minutes or so, so it's very frustrating.

Pinging Google.com looks like this when running OS X (there are hundreds of packets that return in less than 100ms (with a few in the 130 range), then a drop off for several seconds):

64 bytes from 173.194.34.196: icmp_seq=694 ttl=48 time=71.463 ms
64 bytes from 173.194.34.196: icmp_seq=695 ttl=48 time=68.362 ms
64 bytes from 173.194.34.196: icmp_seq=696 ttl=48 time=69.056 ms
64 bytes from 173.194.34.196: icmp_seq=697 ttl=48 time=92.563 ms
64 bytes from 173.194.34.196: icmp_seq=698 ttl=48 time=130.814 ms
64 bytes from 173.194.34.196: icmp_seq=699 ttl=48 time=71.054 ms
64 bytes from 173.194.34.196: icmp_seq=700 ttl=48 time=73.588 ms
64 bytes from 173.194.34.196: icmp_seq=701 ttl=48 time=71.185 ms
64 bytes from 173.194.34.196: icmp_seq=702 ttl=48 time=72.161 ms
64 bytes from 173.194.34.196: icmp_seq=703 ttl=48 time=69.163 ms
64 bytes from 173.194.34.196: icmp_seq=704 ttl=48 time=73.425 ms
64 bytes from 173.194.34.196: icmp_seq=705 ttl=48 time=141.980 ms
64 bytes from 173.194.34.196: icmp_seq=706 ttl=48 time=226.818 ms
64 bytes from 173.194.34.196: icmp_seq=707 ttl=48 time=210.087 ms
Request timeout for icmp_seq 708
Request timeout for icmp_seq 709
Request timeout for icmp_seq 710
Request timeout for icmp_seq 711
Request timeout for icmp_seq 712
64 bytes from 173.194.34.196: icmp_seq=713 ttl=48 time=73.582 ms
64 bytes from 173.194.34.196: icmp_seq=714 ttl=48 time=70.994 ms
64 bytes from 173.194.34.196: icmp_seq=715 ttl=48 time=72.502 ms
64 bytes from 173.194.34.196: icmp_seq=716 ttl=48 time=70.467 ms
64 bytes from 173.194.34.196: icmp_seq=717 ttl=48 time=68.470 ms
64 bytes from 173.194.34.196: icmp_seq=718 ttl=48 time=70.767 ms
64 bytes from 173.194.34.196: icmp_seq=719 ttl=48 time=69.078 ms

Note: my machine's Wi-Fi MAC is 68:a8:6d:29:cf:8a (static IP 192.168.1.250) and its Ethernet address is 3c:07:54:5a:e0:44 (static IP 192.168.1.251). The router's LAN IP is 192.168.1.1 and its WAN IP is 85.61.155.224.

In the next screenshot one can see, during a Skype call:

• ping 192.168.1.1 in the upper left.
• ping 85.61.155.224 in the bottom left.
• ping google.com in the bottom right.
• the arp -an and arp -ad commands executed.

When I executed the arp -ad command at a time when the connection was lost, the list didn't show any addresses. It looked like this:

Miguels-MacBook-Pro:~ Ai$ sudo arp -ad
192.168.1.1 (192.168.1.1) deleted
192.168.1.4 (192.168.1.4) deleted
192.168.1.255 (192.168.1.255) deleted
Miguels-MacBook-Pro:~ Ai$ arp -an
Miguels-MacBook-Pro:~ Ai$

I don't have sufficient knowledge to follow mike's instructions on how to obtain and compile the source of the mtr command.

This is how things look when it's worse:

Running netstat -s gives:

Miguels-MacBook-Pro:mtr-0.84 Ai$ NETSTAT -s
tcp:
    18246745 packets sent
        1119644 data packets (502840461 bytes)
        43704 data packets (23125605 bytes) retransmitted
        1 resend initiated by MTU discovery
        11219994 ack-only packets (80633 delayed)
        0 URG only packets
        10 window probe packets
        5446529 window update packets
        419140 control packets
        0 data packets sent after flow control
    25777361 packets received
        1284807 acks (for 502390806 bytes)
        222223 duplicate acks
        2 acks for unsent data
        21993647 packets (3385435972 bytes) received in-sequence
        85441 completely duplicate packets (85927570 bytes)
        189 old duplicate packets
        6141 packets with some dup. data (1633845 bytes duped)
        2225930 out-of-order packets (3047304289 bytes)
        2 packets (0 bytes) of data after window
        0 window probes
        7324 window update packets
        63837 packets received after close
        56 bad resets
        9 discarded for bad checksums
        0 discarded for bad header offset fields
        0 discarded because packet too short
    200907 connection requests
    118631 connection accepts
    110736 bad connection attempts
    1273 listen queue overflows
    220132 connections established (including accepts)
    335687 connections closed (including 10893 drops)
        4086 connections updated cached RTT on close
        4086 connections updated cached RTT variance on close
        1485 connections updated cached ssthresh on close
    44620 embryonic connections dropped
    1178835 segments updated rtt (of 1308648 attempts)
    76481 retransmit timeouts
        189 connections dropped by rexmit timeout
        0 connections dropped after retransmitting FIN
    17 persist timeouts
        0 connections dropped by persist timeout
    2015 keepalive timeouts
        1 keepalive probe sent
        1409 connections dropped by keepalive
    127007 correct ACK header predictions
    21519356 correct data packet header predictions
    5021 SACK recovery episodes
    5638 segment rexmits in SACK recovery episodes
    6044752 byte rexmits in SACK recovery episodes
    33658 SACK options (SACK blocks) received
    2125185 SACK options (SACK blocks) sent
    0 SACK scoreboard overflow
udp:
    28584263 datagrams received
    0 with incomplete header
    0 with bad data length field
    84 with bad checksum
    4216 dropped due to no socket
    239052 broadcast/multicast datagrams dropped due to no socket
    729188 dropped due to full socket buffers
    0 not for hashed pcb
    27611723 delivered
    28323341 datagrams output
ip:
    61548853 total packets received
    4 bad header checksums
    0 with size smaller than minimum
    0 with data size < data length
    0 with ip length > max ip packet size
    0 with header length < data size
    0 with data length < header length
    0 with bad options
    0 with incorrect version number
    103276 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    51420 packets reassembled ok
    61383903 packets for this host
    32 packets for unknown/unsupported protocol
    0 packets forwarded (0 packets fast forwarded)
    105 packets not forwardable
    112953 packets received for unknown multicast group
    0 redirects sent
    53953058 packets sent from this host
    155 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    3748 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 tunneling packets that can't find gif
    3 datagrams with bad address in header
    0 packets dropped due to no bufs for control data
icmp:
    4216 calls to icmp_error
    0 errors not generated 'cuz old message was icmp
    Output histogram:
        echo reply: 202
        destination unreachable: 4216
    0 messages with bad code fields
    0 messages < minimum length
    168 bad checksums
    0 messages with bad length
    0 multicast echo requests ignored
    0 multicast timestamp requests ignored
    Input histogram:
        echo reply: 7013069
        destination unreachable: 14133
        echo: 202
        time exceeded: 289
    202 message responses generated
    ICMP address mask responses are disabled
igmp:
    0 messages received
    0 messages received with too few bytes
    0 messages received with wrong TTL
    0 messages received with bad checksum
    0 V1/V2 membership queries received
    0 V3 membership queries received
    0 membership queries received with invalid field(s)
    0 general queries received
    0 group queries received
    0 group-source queries received
    0 group-source queries dropped
    0 membership reports received
    0 membership reports received with invalid field(s)
    0 membership reports received for groups to which we belong
    0 V3 reports received without Router Alert
    16 membership reports sent
ipsec:
    0 inbound packets processed successfully
    0 inbound packets violated process security policy
    0 inbound packets with no SA available
    0 invalid inbound packets
    0 inbound packets failed due to insufficient memory
    0 inbound packets failed getting SPI
    0 inbound packets failed on AH replay check
    0 inbound packets failed on ESP replay check
    0 inbound packets considered authentic
    0 inbound packets failed on authentication
    0 outbound packets processed successfully
    0 outbound packets violated process security policy
    0 outbound packets with no SA available
    0 invalid outbound packets
    0 outbound packets failed due to insufficient memory
    0 outbound packets with no route
ip6:
    151513 total packets received
    0 with size smaller than minimum
    0 with data size < data length
    0 with bad options
    0 with incorrect version number
    0 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    0 fragments that exceeded limit
    0 packets reassembled ok
    5555 packets for this host
    0 packets forwarded
    145711 packets not forwardable
    0 redirects sent
    2608 packets sent from this host
    0 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    4578 output packets discarded due to no route
    23 output datagrams fragmented
    46 fragments created
    0 datagrams that can't be fragmented
    0 packets that violated scope rules
    145711 multicast packets which we don't join
    Input histogram:
        hop by hop: 2327
        TCP: 244
        UDP: 142524
        ICMP6: 6416
    Mbuf statistics:
        244 one mbuf
        two or more mbuf:
            lo0= 2215
        149054 one ext mbuf
        0 two or more ext mbuf
    0 packets whose headers are not continuous
    0 tunneling packets that can't find gif
    0 packets discarded due to too may headers
    0 failures of source address selection
    0 forward cache hit
    0 forward cache miss
    0 packets dropped due to no bufs for control data
icmp6:
    0 calls to icmp_error
    0 errors not generated because old message was icmp error or so
    0 errors not generated because rate limitation
    Output histogram:
        router solicitation: 50
        neighbor solicitation: 19
        neighbor advertisement: 19
        MLDv2 listener report: 59
    0 messages with bad code fields
    0 messages < minimum length
    0 bad checksums
    0 messages with bad length
    Input histogram:
        neighbor advertisement: 245
    Histogram of error messages to be generated:
        0 no route
        0 administratively prohibited
        0 beyond scope
        0 address unreachable
        0 port unreachable
        0 packet too big
        0 time exceed transit
        0 time exceed reassembly
        0 erroneous header field
        0 unrecognized next header
        0 unrecognized option
        0 redirect
        0 unknown
    0 message responses generated
    0 messages with too many ND options
    0 messages with bad ND options
    0 bad neighbor solicitation messages
    0 bad neighbor advertisement messages
    0 bad router solicitation messages
    0 bad router advertisement messages
    0 bad redirect messages
    0 path MTU changes
ipsec6:
    0 inbound packets processed successfully
    0 inbound packets violated process security policy
    0 inbound packets with no SA available
    0 invalid inbound packets
    0 inbound packets failed due to insufficient memory
    0 inbound packets failed getting SPI
    0 inbound packets failed on AH replay check
    0 inbound packets failed on ESP replay check
    0 inbound packets considered authentic
    0 inbound packets failed on authentication
    0 outbound packets processed successfully
    0 outbound packets violated process security policy
    0 outbound packets with no SA available
    0 invalid outbound packets
    0 outbound packets failed due to insufficient memory
    0 outbound packets with no route
rip6:
    0 messages received
    0 checksum calcurations on inbound
    0 messages with bad checksum
    0 messages dropped due to no socket
    0 multicast messages dropped due to no socket
    0 messages dropped due to full socket buffers
    0 delivered
    0 datagrams output
pfkey:
    0 requests sent to userland
    0 bytes sent to userland
    0 messages with invalid length field
    0 messages with invalid version field
    0 messages with invalid message type field
    0 messages too short
    0 messages with memory allocation failure
    0 messages with duplicate extension
    0 messages with invalid extension type
    0 messages with invalid sa type
    0 messages with invalid address extension
    0 requests sent from userland
    0 bytes sent from userland
    0 messages toward single socket
    0 messages toward all sockets
    0 messages toward registered sockets
    0 messages with memory allocation failure

Running netstat -I en1 gives:

Miguels-MacBook-Pro-2:mtr-0.84 Ai$ netstat -I en1
Name  Mtu   Network       Address            Ipkts Ierrs    Opkts Oerrs  Coll
en1   1500  <Link#5>    68:a8:6d:29:cf:8a 72539835     0 63847581     0     0
en1   1500  fe80::6aa8: fe80:5::6aa8:6dff 72539835     - 63847581     -     -
en1   1500  192.168.1     192.168.1.250   72539835     - 63847581     -     -

Running ifconfig -a gives:

Miguels-MacBook-Pro-2:mtr-0.84 Ai$ ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
    options=3<RXCSUM,TXCSUM>
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
    inet 127.0.0.1 netmask 0xff000000 
    inet6 ::1 prefixlen 128 
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=2b<RXCSUM,TXCSUM,VLAN_HWTAGGING,TSO4>
    ether 3c:07:54:5a:e0:44 
    media: autoselect (none)
    status: inactive
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    ether 68:a8:6d:29:cf:8a 
    inet6 fe80::6aa8:6dff:fe29:cf8a%en1 prefixlen 64 scopeid 0x5 
    inet 192.168.1.250 netmask 0xffffff00 broadcast 192.168.1.255
    media: autoselect
    status: active
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
    ether 0a:a8:6d:29:cf:8a 
    media: autoselect
    status: inactive
fw0: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 4078
    lladdr a4:b1:97:ff:fe:ec:f0:80 
    media: autoselect <full-duplex>
    status: inactive

---

What I think:

• It's not a Wi-Fi issue because it happens over cable too.
• It's not a router/ISP issue because other devices and machines have no problem.
• It's not a machine issue because it only happens when running OS X.
• It must therefore be an OS X issue.

What I tried:

• Restart, shutdown.
• Turn AirPort on and off, different Ethernet cables.
• Repair permissions.
• Reset the PRAM.
• Clear all system and user caches with Onyx.

Weird note: For some strange reason the problem seems to worsen when a skype call is taking place.

I would kindly appreciate ideas on how to approach this issue.

Answer 1

When your connections starts timing out, can you do arp -an in Terminal.app and see if you still have all MAC addresses in the ARP table? as in - your router's MAC address, or the host you're trying to ping?

If you do (and you have the time before it starts working again), can you flush the arp table (sudo arp -ad) and then see if your router's MAC address shows up in the ARP table again?

Also, try running a ping to your router's LAN IP address in one Terminal session, and maybe a ping to your router's WAN IP address in another while you're on Skype. See if all of them start timing out or just one of them. One more tool that I find useful is mtr - you may need to get the source and compile it yourself or use fink / macports or other package manager. When you get it, just run it to a destination somewhere on the Internet and it will show you which hop stops responding.

How to install software from sources (such as mtr) Requires Xcode to be installed:

• download the source archive (typically .tar.gz or .tar.bz2)
• uncompress the downloaded file (e.g. in Terminal.app run gzip -dc filename.tar.gz | tar -xvf -, which will typically create a new directory in the current directory, and put the content of the archive in there)
• navigate to the obtained folder in terminal
• run ./configure --prefix=/usr/local (please note, I like to install software from source into /usr/local to keep it away from binaries installed as part of the system; the --prefix=/usr/local option to configure will do just that)
• run make
• run sudo make install
• done!

Answer 2

Could you first check that you are really using the network interface you should:

ifconfig -a

Could you look at the output of the following commands (if en0 is the network interface name of your Ethernet card):

netstat -I en0

To help locate the problem could you make a specific Location with just your Ethernet card activated and if possible only using either IPv4 or IPv6 but not both:

Could you run the following extract of potential hardware or driver errors:

grep ' en[012]' /var/log/kernel.log

(don't be frightened you may find a lot of Wi-Fi channels information).

The following message exhibited by your netstat:

44620 embryonic connections dropped

means you are actually the target of a silly tcp syn flooding (which is a denial of service (DOS) attack).

When your:

ping 192.168.1.1

chokes for 6s, could you run:

netstat -m

Answer 3

First, I see dropbox running in your menu bar; have you disabled that, yet?

Second, try removing any other startup/login items. Look in:

Login:

1. ~/Library/LaunchAgents/
2. ~/Library/LaunchDaemons/
3. System Preferences > Users & Groups > Login Items

Startup:

1. /Library/LaunchAgents/
2. /Library/LaunchDaemons/
3. /Library/StartupItems/
4. /Library/Preferences/com.apple.loginitems.plist (rarely exists)

Answer 4

I've had this problem for a long time now (beginning after an upgrade to Mavericks) and, after months of research, I think I finally found a fix.

First of all, there are quite a number of people with the very same problem in the Apple forums:

• Why in OS X Mavericks internet is so slow?
• Mavericks network issue

So this is a known issue and I really don't know why Apple has not yet provided a fix for this. In the threads listed above, there are many suggestions to fix this, but most of those did not work. Some fix the problem temporarily:

• Disconnect and reconnect the network
• The old friend: reboot
• Remove the folder containing the network config: sudo rm -rf /Library/Preferences/SystemConfiguration

After these measures, the network connection feels much better and I do not experience drops for several hours or sometimes even days. But the problems always come back.

This question and the hints that the problem might be related to ARP brought me to start further research and I found this page, which describes the bug in detail and also contains a patch, which I cite here:

sudo su
touch /etc/sysctl.conf
echo net.link.ether.inet.arp_unicast_lim=0 >> /etc/sysctl.conf
chown root:wheel /etc/sysctl.conf
chmod 0644 /etc/sysctl.conf

Please refer to the provided link for an in-depth explanation of the fix, which is supposed to be included in a future OS update for Yosemite by Apple. It disables unicast ARP requests, which cause confusion with some network equipment like your home router.

After applying the fix and rebooting, it should be checked if

sudo sysctl -a | grep net.link.ether.inet.arp_unicast_lim

returns net.link.ether.inet.arp_unicast_lim: 0. If the number is not equal to zero, the fix was not applied correctly.

Afterwards, I found another thread at the apple communities that contains the same solution: Mavericks and Failed ARP causing network drops! Well, after you know what the problem is, finding the correct solution is a lot easier.

Answer 5

There is a great deal of information here on the troubleshooting and diagnostic end of things, but sometimes when troubleshooting it's fun to return to basics and question some assumptions.

As I mentioned in a comment, this looks very much like a QOS router kicking in due to your machine temporarily exceeding some bandwidth or packet rate cap.

What if you are doing different patterns, volumes and amounts of network traffic while on OS X as opposed to Windows and that's the real cause, not the hardware drivers or the software?

I would expect running OS X is correlated with your observations, but what if it's not the cause of the temporary network pauses.

Have you tried researching what if any QOS filters and routing changes are implemented by your network provider? Have you considered tunneling all traffic to another computer (ssh or VPN) so that you can rule out trivial filters. (If the provider is doing deep packet inspection, or destination and true rate limiting - you may not be able to escape these short timeouts.)

I hope there's an answer you can find by looking at the details of the network (and we'll all learn something from exploring those options) - but be sure you also consider that your measurement tools and added traffic to ping / poke at things could be affecting the traffic counts and making it more likely Skype will drop for you. The routers I set up are programmed to drop ICMP traffic before all other traffic since when capacity gets tight - I'd rather the ping fail and other packets get through. Your ISP and network provider might have set things up similarly.

Answer 6

In addition to all the stuff here, you might want to make sure Auto Proxy Discovery is not on (as well as Automatic Proxy Configuration). That tends to cause more problems than not and it's often not needed.

Answer 7

With all the great diagnostic information in this question you have narrowed down the possibilities greatly.

To begin with, your pings to 192.168.1.1 greatly isolate the problem to either your router, computer, or LAN. This is not a problem with DNS or your ISP.

I'm most disturbed by the results of your ping tests to 192.168.1.1. Did you do something weird in setting them up?

For example, you have successful pings with ICMP sequence numbers of 24267, 24268, and 24269, then 3 timeouts, then success again with ICMP 24273. So the numbers of the successes seem right. However, the numbers of the timeouts are completely different. I'd expect to see request timeouts from ICMP 24270, 24271, and 24272 but instead the timeouts report ICMP 89806, 89807, and 89808. I've never seen that before and so to me it suggests you have a broken network stack on that computer. Perhaps one too many extensions. Any chance you have Netgear Genie installed? Or maybe VPN software?

In any case, I'd say it's time to start disabling "enhancements" to see if you can find a culprit installed on the computer.

Edit

OK, mystery solved. The ICMP sequence number is a 16 bit field. Treated as an unsigned integer, that means it has a maximum value of 65,535 and then wraps around to zero. So if the local ping program is maintaining a 32 bit integer counter (which it probably would by default), it could report a 32 bit integer number for missing packets. However, when reading replies, the reply will necessarily only have the last 16 bits of the counter. So the reply to sequence number 89805 will be 89505 & 0xFFFF which is 24269.

Answer 8

Curious problem considering it persists of ethernet. I had a similar issue but found WiFi interference from other networks to be the problem. Switching to a 5GHz band fixed my problem, which is guess is worth a shot.

Answer 9

Mac OSX Hints http://hints.macworld.com/article.php?story=20080605143917233 on dropped connections because DNS lookups fail pending DCHP identification of a router ..

try configuring your Mac to use the OpenDNS (OpenDNS.ORG) servers 
instead of your ISPs DNS servers. 

It is most likely the DNS and/or a acceleration setting in your modem settings and bypassing that DNS should help resolve your issue.

Answer 10

This smells like another device on your network is trying to use the same IP as you, or some trouble with DHCP.

Could you see if you can still reproduce it after assigning yourself a static IP?

Goto Network Preferences, choose your Ethernet interface, advanced, TCP/IP

Change the "Configure IPv4" dropdown to "Manually"

IPv4 Address: 192.168.1.150 (something unique, not what DHCP had assigned you before) Subnet Mask: 255.255.255.0 Router: 192.168.1.1

Save

Then try to reproduce the issue again. When doing this test, make sure your Wi-Fi is off so only your Ethernet is in use. This will help narrow it down.

---

If you're still having the issue, you should download Wireshark (http://www.wireshark.org/) start a capture, reproduce the issue, save the dump and let us take a look.

Also, what Router/AP are you using?

Answer 11

Two things to check that correlate to this being caused by increase LAN traffic due to new roommates.

1. Are there QoS (Quality of Service) settings on the router, and if so, how are they set? Skype traffic would be prioritized and if the WAN is getting saturated the router might respond by temporarily shutting of lower priority connections.
2. Is the router CPU simply getting overloaded? When I upgraded from 1 Gbs DSL to 5 Gbs cable service I found that my router simply could not keep up with the increased traffic and had to buy a new one. Investigate the performance of your router and see if this could be a problem. Most routers have detailed performance reviews available on the internet; check and see how your router is rated compared to your internet service capacity.

Answer 12

I had a similar problem and in my case it seems to be caused by Tunnelblick, even when VPN was not connected. I uninstalled it (with the uninstaller, not just drag to Trash) and problem went away.