Opened 13 months ago

Last modified 4 weeks ago

#1080 assigned request

NAT64 network doesn't work for first ~15 seconds after connection

Reported by: lorenzo@… Owned by: jclarke@…
Priority: tbd Milestone: ietf-098
Component: incoming Keywords:
Cc: ek@…, panda@… My Current Location: Montreux 3
My MAC Address: 64:bc:0c:50:38:2c My OS: Android O preview

Description

Due to IPv6 issues, all my Android devices can't use the ietf or ietf-nat64 network for the first 10-15 seconds after connecting.

The following example is taken on "ietf-nat64", but the same appears to occur on "ietf" as well. The device will complete 802.1x negotiation, do DAD, get an RA with a global address, but then nothing works for the first 10-15 seconds. After that time, I get a very weird DAD packet sent by a Cisco box for my own IPv6 address, and then things start working. Here's an annotated tcpdump of all port 53 and ICMPv6 packets, except omitting some periodic RAs and NUD packets:

  1. Get on network, perform DAD, send RS, get RA with prefix and DNS server:

09:42:24.823617 64:bc:0c:50:38:2c > 33:33:ff:50:38:2c, ethertype IPv6 (0x86dd), length 78: :: > ff02::1:ff50:382c: ICMP6, neighbor solicitation, who has fe80::66bc:cff:fe50:382c, length 24
09:42:25.827362 64:bc:0c:50:38:2c > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: fe80::66bc:cff:fe50:382c > ff02::2: ICMP6, router solicitation, length 16
09:42:25.979313 08:81:f4:8a:a7:f0 > 64:bc:0c:50:38:2c, ethertype IPv6 (0x86dd), length 150: fe80::1998:1 > ff02::1: ICMP6, router advertisement, length 96
09:42:25.983529 64:bc:0c:50:38:2c > 33:33:ff:50:38:2c, ethertype IPv6 (0x86dd), length 78: :: > ff02::1:ff50:382c: ICMP6, neighbor solicitation, who has 2001:67c:370:1998:66bc:cff:fe50:382c, length 24
09:42:25.983857 64:bc:0c:50:38:2c > 33:33:ff:6c:c6:2b, ethertype IPv6 (0x86dd), length 78: :: > ff02::1:ff6c:c62b: ICMP6, neighbor solicitation, who has 2001:67c:370:1998:41a5:38e3:906c:c62b, length 24

  1. Immediately use the IPv6 address I got to send DNS queries for connectivitycheck.gstatic.com, www.google.com, ipv4only.arpa (NAT64 detection):

09:42:26.102495 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 109: 2001:67c:370:1998:41a5:38e3:906c:c62b.2211 > 2001:67c:370:229::6.53: 21398+ AAAA? connectivitycheck.gstatic.com. (47)
09:42:26.106525 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.13619 > 2001:67c:370:229::6.53: 28989+ AAAA? www.google.com. (32)
09:42:26.128679 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 93: 2001:67c:370:1998:41a5:38e3:906c:c62b.24946 > 2001:67c:370:229::6.53: 23380+ AAAA? ipv4only.arpa. (31)

  1. After 5 seconds of no response, retransmit the DNS queries:


09:42:31.108741 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 109: 2001:67c:370:1998:41a5:38e3:906c:c62b.17817 > 2001:67c:370:229::7.53: 21398+ AAAA? connectivitycheck.gstatic.com. (47)
09:42:31.110206 08:81:f4:8a:a7:f0 > 64:bc:0c:50:38:2c, ethertype IPv6 (0x86dd), length 78: fe80::1998:1 > fe80::66bc:cff:fe50:382c: ICMP6, neighbor advertisement, tgt is fe80::1998:1, length 24
09:42:31.111151 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.22410 > 2001:67c:370:229::7.53: 28989+ AAAA? www.google.com. (32)
09:42:31.135125 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 93: 2001:67c:370:1998:41a5:38e3:906c:c62b.21404 > 2001:67c:370:229::7.53: 23380+ AAAA? ipv4only.arpa. (31)

... and again:

09:42:36.114986 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 109: 2001:67c:370:1998:41a5:38e3:906c:c62b.2211 > 2001:67c:370:229::6.53: 21398+ AAAA? connectivitycheck.gstatic.com. (47)
09:42:36.117228 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.13619 > 2001:67c:370:229::6.53: 28989+ AAAA? www.google.com. (32)
09:42:36.141290 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 93: 2001:67c:370:1998:41a5:38e3:906c:c62b.24946 > 2001:67c:370:229::6.53: 23380+ AAAA? ipv4only.arpa. (31)

  1. What? A Cisco box sends a fake DAD probe for my IPv6 address?

09:42:36.937825 00:3a:7d:71:93:89 > 64:bc:0c:50:38:2c, ethertype IPv6 (0x86dd), length 78: :: > ff02::1:ff6c:c62b: ICMP6, neighbor solicitation, who has 2001:67c:370:1998:41a5:38e3:906c:c62b, length 24
09:42:36.938541 64:bc:0c:50:38:2c > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: 2001:67c:370:1998:41a5:38e3:906c:c62b > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:67c:370:1998:41a5:38e3:906c:c62b, length 32

  1. Retransmit DNS queries again, and again. 10 seconds pass:

09:42:41.121230 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 109: 2001:67c:370:1998:41a5:38e3:906c:c62b.17817 > 2001:67c:370:229::7.53: 21398+ AAAA? connectivitycheck.gstatic.com. (47)
09:42:41.123489 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.22410 > 2001:67c:370:229::7.53: 28989+ AAAA? www.google.com. (32)
09:42:41.147377 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 93: 2001:67c:370:1998:41a5:38e3:906c:c62b.21404 > 2001:67c:370:229::7.53: 23380+ AAAA? ipv4only.arpa. (31)
09:42:46.129534 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 109: 2001:67c:370:1998:41a5:38e3:906c:c62b.13974 > 2001:67c:370:229::6.53: 1761+ AAAA? connectivitycheck.gstatic.com. (47)
09:42:46.130552 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.7285 > 2001:67c:370:229::6.53: 32390+ AAAA? www.google.com. (32)
09:42:46.130645 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.17098 > 2001:67c:370:229::6.53: 22387+ AAAA? www.google.com. (32)
09:42:46.153967 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 93: 2001:67c:370:1998:41a5:38e3:906c:c62b.14337 > 2001:67c:370:229::6.53: 21476+ AAAA? ipv4only.arpa. (31)

  1. Finally get an answer for one query. Time taken: 22 seconds.

09:42:46.774636 08:81:f4:8a:a7:f0 > 64:bc:0c:50:38:2c, ethertype IPv6 (0x86dd), length 280: 2001:67c:370:229::6.53 > 2001:67c:370:1998:41a5:38e3:906c:c62b.13974: 1761 1/4/4 AAAA 2607:f8b0:4009:80c::2003 (218)

  1. Now other queries work as well:

09:42:51.137833 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.1986 > 2001:67c:370:229::7.53: 22387+ AAAA? www.google.com. (32)
09:42:51.138124 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 94: 2001:67c:370:1998:41a5:38e3:906c:c62b.10805 > 2001:67c:370:229::7.53: 32390+ AAAA? www.google.com. (32)
09:42:51.145327 08:81:f4:8a:a7:f0 > 64:bc:0c:50:38:2c, ethertype IPv6 (0x86dd), length 258: 2001:67c:370:229::7.53 > 2001:67c:370:1998:41a5:38e3:906c:c62b.10805: 32390 1/4/4 AAAA 2607:f8b0:4009:800::2004 (196)
09:42:51.145916 08:81:f4:8a:a7:f0 > 64:bc:0c:50:38:2c, ethertype IPv6 (0x86dd), length 258: 2001:67c:370:229::7.53 > 2001:67c:370:1998:41a5:38e3:906c:c62b.1986: 22387 1/4/4 AAAA 2607:f8b0:4009:800::2004 (196)
09:42:51.157360 64:bc:0c:50:38:2c > 00:00:5e:00:02:c6, ethertype IPv6 (0x86dd), length 93: 2001:67c:370:1998:41a5:38e3:906c:c62b.19511 > 2001:67c:370:229::7.53: 21476+ AAAA? ipv4only.arpa. (31)
09:42:51.160038 08:81:f4:8a:a7:f0 > 64:bc:0c:50:38:2c, ethertype IPv6 (0x86dd), length 239: 2001:67c:370:229::7.53 > 2001:67c:370:1998:41a5:38e3:906c:c62b.19511: 21476 2/4/0 AAAA 64:ff9b::c000:aa, AAAA 64:ff9b::c000:ab (177)

Change history (19)

comment:1 Changed 13 months ago by panda@…

Owner: changed from llynch@… to panda@…
Status: newaccepted

Hi Lorenzo,

Thank you for the report.

The MAC address 00:3a:7d:71:93:89 is our Cisco Wireless LAN Controller. I will check the configuration and investigate this issue.

Thank you.
Hirochika Asai

comment:2 Changed 13 months ago by llynch@…

updating due to trac notification issue for reporter

comment:3 Changed 13 months ago by jim@…

[Resending due to a problem with outbound ticket emails]

Hi Lorenzo,
Thank you for the report.
The MAC address 00:3a:7d:71:93:89 is our Cisco Wireless LAN Controller. I will check the configuration and investigate this issue.
Thank you.
Hirochika Asai

comment:4 Changed 13 months ago by panda@…

Hi Lorenzo,

We could reproduce this issue.

The NS from Wireless Lan Controller (WLC) is due to the feature that WLC converts multicast packets to unicast with its forwarding database cache to mitigate multicast packets that consume radio time slot much. We suspect the cause of this issue is that this forwarding database needs several seconds to get updated after your association. Currently, we cannot find any way to disable this feature, but we are trying to find alternative way to solve the packet losses (filtered by WLC) for first several seconds.

We will continue to work on this issue and let you know when we find anything.

Thank you.
Hirochika Asai

comment:5 Changed 13 months ago by panda@…

Hi Lorenzo,

We enable the unknown address multicast NS forwarding on WLC so that NS packets can be forwarded before WLC completes proxy DAD. This does not disable WLC's weird DAD but should fix the issue that you loses packets for the first several seconds.

Could you please check if the issue is solved?

Thank you.
Hirochika Asai

comment:6 Changed 13 months ago by jim@…

Lorenzo,

Just a quick ping. Jen submitted a ticket on this as well, and she's confirmed that Asai-san's fixes resolved her issue. We just wanted to double check with you that you're seeing it as fixed before we resolve this ticket. Could you take a few seconds to test?

Thanks!

  • Jim

comment:7 Changed 13 months ago by jim@…

Resolution: fixed
Status: acceptedclosed

Lorenzo,

For ticket queue cleanliness, I'll close this ticket, as we believe things are fixed. A reply from you will re-open if you find something outstanding. Let us know if you still have issues!

  • Jim

comment:8 Changed 5 months ago by Rick Alfvin

Milestone: ietf-98ietf-098

Milestone renamed

comment:9 Changed 4 weeks ago by Hans Kuhn

Cc: Clemens Schrimpe added

comment:10 Changed 4 weeks ago by Hans Kuhn

Cc: hak@… added

comment:11 Changed 4 weeks ago by lorenzo@…

Cc: Clemens Schrimpe hak@… removed
Resolution: fixed
Status: closedreopened

This is still happening. It happened in Singapore (100) and is now happening in London (IETF 101) as well.

(Note: in Singapore we also had a bad interaction of this behaviour with a bug in the IPv6 stack of Pixel 2 that caused IPv6 not to work at all. Ticket #1171 covers that issue.)

comment:12 Changed 4 weeks ago by Clemens Schrimpe

Owner: changed from panda@… to Clemens Schrimpe
Status: reopenedaccepted

We'll conduct the "Linux Laptop with O-Bit turned off" experiment later today in the NOC, after we setup the ISOC wireless network, which we'll use for this test.

comment:13 Changed 4 weeks ago by Clemens Schrimpe

Test network is ready now!

If I'm not in the NOC: Just swing by, hook on to SSID "ISOC" (username: isoc, password: isoc) and enjoy!

comment:14 in reply to:  14 ; Changed 4 weeks ago by lorenzo@…

I'll try to make it over there at 11:30.

comment:15 Changed 4 weeks ago by lorenzo@…

Steps to reproduce on Ubuntu 14.04 using kernel 4.4:

  1. Disconnect from wifi.
  1. Clear the association from the controller.
  1. If stable IPv6 addresses are in use, ensure that the stable IPv6 address formed on connect are not the same as the ones from the previous connection. This is because if the IPv6 addresses are already in the controller's cache, things start working immediately.

addr=1234; for i in {1..7}; do addr=$addr$(printf :%04x $((RANDOM % 65536))); done && echo $addr | sudo tee /proc/sys/net/ipv6/conf/default/stable_secret

  1. Bring down wifi and bring it up again. This ensures that the temporary IPv6 addresses formed on connect are not the same as the ones from the previous connection. NOTE: the command to use here depends on which wifi driver the system is using. My system is using iwlmvm.

sudo rmmod iwlmvm && sudo modprobe iwlmvm

  1. This will cause NetworkManager? to connect to wifi.
  1. As soon as the device connects to wifi, start a ping:

ip -6 a l dev wlan0 | grep inet6.*glob; ping6 -n 2001:4860:4860::8888

  1. Notice that it takes several seconds for ping to start replying.

comment:16 Changed 4 weeks ago by lorenzo@…

Forgot to mention: the above steps assume that the network has disabled the O bit in the router advertisements. If the O bit is set to 1, the host will do DHCPv6, and when the controller sees the DHCPv6 packet (even if it's an empty info-req) it allows the client on the network.

comment:18 Changed 4 weeks ago by Clemens Schrimpe

Owner: changed from Clemens Schrimpe to jclarke@…
Status: acceptedassigned

@joe: Please try to (re-?)open a ticket for this. Thanks

comment:19 Changed 4 weeks ago by lorenzo@…

Cc: panda@… added

Per debugging today with panda@, this also affects Mac OS X (I think panda@ was using High Sierra).

Steps to reproduce on Mac OS X:

  1. Configure network to send RA with M=0, O=0 and SLAAC.
  2. Connect Mac to network.
  3. Attempt to send packets to the Internet.

Packets will be dropped for the first 15 seconds after connection.

comment:20 Changed 4 weeks ago by jclarke@…

Cisco bug CSCvi62746 has been filed to track this issue.

Note: See TracTickets for help on using tickets.