2

Hoping to get some insight regarding a bizarre issue that is being seen at a few of our networks, as well as others (based off reddit and NANOG posts). The short story of what is happening is that Apple devices (primarily MacBooks as well as some iMacs) that are sleeping will hijack the default gateway IP for the wireless VLAN that they are in, as well as responding to ARP requests asking who the default gateway is. This behavior makes the AP's log a message "CAPWAP detected next hop MAC changed from correct_DG_mac_address to apple_mac_address."

We have tickets open with Cisco who is blaming Apple, and Apple who is stating we need logs that are utterly impossible to get as we cannot recreate this behavior. Apples current workaround? Push out a script to completely disable sleep on all Apple devices.

More specifics:

  • This is only affecting 3802i WAPs that are in flexconnect mode, and we are seeing this at multiple sites, on two different wireless controllers. The 3802i's do not reboot - they lose connectivity, recover, repeat.

  • Wireless subnets are generally as large as /20 and contain the AP's as well as all clients. We plan to move AP's to a different subnet as a test, but we are hesitant due to the fact that if AP's no longer log the CAPWAP message/drop from Prime - then we will not know when this behavior occurs making it harder to track.

  • The ARP entry on the switches NEVER changes. For this reason, I believe DAI/DHCP snooping would not be an effective solution as I've seen others suggest.

  • We think this behavior may be related to sleep proxy that can we seen with out of the box Apple TV's, but have yet to prove that.

My question is - has anyone ran into this type of behavior before. In addition, does anyone have any suggestions as far as how to remedy this behavior so it no longer occurs?

  • 1
    This is unfortunately way above my head - however, I do recall a similar issue years ago related, I think, to SMB/Bonjour/DHCP. Macs would force an election on 'master browser' & always win, even though they were not supposed to. The 'fix' was to fully set the correct PC to master browser in Windows, preventing Macs from winning any election they forced. – Tetsujin Apr 30 at 7:28
  • Here is some additional documentation,, We were thinking maybe if we disabled Bonjour Service on each of the devices we could stop this behavior. mailman.nanog.org/pipermail/nanog/2019-March/100109.html – mscwheat May 7 at 17:53
1

We've been experiencing this issue and we're running Cisco 3802 Flex Connect with a WLC 8500.

We are going to attempt DIA This weekend but from both Cisco and Apple they are saying there isn't much we can do. Cisco is saying they will have a patch for the Wireless Controller to introduce ARP security for this in about a month and Apple won't release to the public what is the issue or when their patch would be out to fix it.

Some Options that have been floated to us as a Temp Solution -- Downgrade the WLC to 8.2 from the current recommended version 8.5.140 -- Upgrade the WLC to 8.5.140.11 a special release not provided without support consent that will require DHCP reassociation on every device when roaming. Not a great Trade off. -- Remove Flex Connect and re design your WLAN to Central which would tunnel the ARP traffic to the WLC and use built in security for ARP protection. "We are not going to do this"

What we will be trying! Enable DIA on the Switching platform and log the drops. The hope is if this is from a sleeping Macbook then the getting dropped won't matter and we'll see some kind of containment.

This is our last option.

  • Thanks for your comment. Cisco also told us about the 8.5.140.12 special release, but clearly it has issues of its own like you mentioned. Are your 3802 WAPs on the same VLAN as the MacBooks? I also highly considered DAI, but my thought process is that it will not work as expected since we never have once seen the ARP entry change on the switch where the SVI is configured. Once the MacBooks incorrectly replies, the AP updates its next hop MAC address and falls apart there. If it does work for you, please let me know – A Concerned Admin May 5 at 2:22
  • The DAI did work in the sense that it blocked the ARP traffic. But we are finding that many of the End Users devices that are sending out the toxic ARP traffic are getting blocked from the wireless so it's still causing outages. We're just protecting the victim machines but still creating outages on Offenders. It appears to be less of a impact but we have so many devices broadcasting I'm not sure this will be a solution for us. – mscwheat May 7 at 17:44
  • This is the configuration we used cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/… – mscwheat May 7 at 17:50
  • From our Users Feedback over the Last Few Days, It appears that DAI is a significant improvement. It's not a fix as the offending devices are still getting denied access to the wireless but at least they aren't ARP'ing them selves as the Gateway and blocking a larger number of other devices. – mscwheat May 9 at 22:19
0

This is Cisco's public document on the topic. https://www.cisco.com/c/en/us/support/docs/wireless/aironet-3800-series-access-points/214491-arp-responses-for-default-gateway-ip-add.html

The issue is real. I have 1st hand experience with it at large scale. I have seen actual packet captures of the behavior described in the article.

The workarounds listed solve the problem. I have deployed them 1st hand at scale.

8.5.151 was just released on CCO on 06/21.

  • Hey There! can you please edit the answer to be more self sustained in case the link changes. – ankiiiiiii Jun 21 at 9:35
  • Thanks. We deployed 8.5.151 on a few sites, and we are still seeing the issue which we have verified using the grep include "IP address" "show client summary ip username ssid devicetype" on the WLC CLI , even though the config flexconnect arp-caching disable is configured. – A Concerned Admin Jun 25 at 19:19
-1

Seeing exactly the same issue here with meraiki APs.

There was a recent firmware update. OTOH apple has updates all the time. So we are not sure if it was the FW upgrade or not at this stage

Meraiki 25.13 OSX 10.14.4

We managed to mitigate the problem by enabling client isolation on the SSID.

Muttley.

Some of the information contained in this post requires additional references. Please edit to add citations to reliable sources that support the assertions made here. Unsourced material may be disputed or deleted.

  • Can you document any specific firmware update of one specific device? Meraki or apple? I’m thinking you mean to say apple changed their devices, but it’s not clear without a reference. – bmike May 1 at 11:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .