Opened 4 years ago

Closed 4 years ago

Last modified 5 weeks ago

#642 closed request (fixed)

some IPv6 not working from Dover/Tennyson

Reported by: mcr@… Owned by: cdoyle@…
Priority: tbd Milestone: ietf-088
Component: incoming Keywords:
Cc: My Current Location:
My MAC Address: My OS:

Description

my work desktop:
dooku-[~] mcr 5555 %host knothole.gatineau.credil.org
knothole.gatineau.credil.org has address 10.10.5.40
knothole.gatineau.credil.org has IPv6 address 2001:4830:116e:1:224:21ff:fea7:5e27

my publically visible server (same IPv6 subnet, sixxs tunnel)
dooku-[~] mcr 5556 %host orlando.gatineau.credil.org 
orlando.gatineau.credil.org has address 132.213.238.235
orlando.gatineau.credil.org has IPv6 address 2001:4830:116e:1::212

Despite what I said in person, the problem is not wired vs wireless, but
seems to have something to with the lower /64.  Note that I don't get past
the third hop, so it's not a problem with the LAN there.

Reverse traceroute:
orlando-[~] mcr 8121 %mtr -6 -r -n 2001:67c:370:128:21a:6bff:fe6a:a96a
HOST: orlando.gatineau.credil.org Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 2001:4830:116e:1::1           0.0%    10    0.6   1.0   0.6   4.9   1.4
  2. 2001:4830:1100:154::1         0.0%    10   31.0  30.8  30.7  31.0   0.1
  3. 2001:4830:e1:b::2             0.0%    10   30.5  30.8  30.5  31.4   0.3
  4. 2001:4830:e1:b::1             0.0%    10   31.0  30.9  30.6  31.2   0.2
  5. 2001:4830:ff:b100::1          0.0%    10   30.9  31.3  30.7  32.3   0.5
  6. 2001:4830:ff:b101::1          0.0%    10   36.4  36.8  36.4  37.4   0.3
  7. 2001:4830:ff:f150::2          0.0%    10   44.2  44.2  43.8  44.5   0.2
  8. 2001:4830:ff:1234::2          0.0%    10  102.3 102.3 101.8 103.4   0.5
  9. 2001:4830:ff:1001::1          0.0%    10  112.9 113.1 112.6 114.0   0.5
 10. 2001:4830:ff:f201::1          0.0%    10  144.2 144.6 144.2 145.3   0.4
 11. 2001:4830:ff:f200::2          0.0%    10  145.5 144.8 144.1 145.5   0.4
 12. 2001:504:16::354              0.0%    10  145.7 145.1 144.5 145.9   0.5
 13. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0



my laptop on ietf-1:
dooku-[~] mcr 5551 %ifconfig wlan0
wlan0     Link encap:Ethernet  HWaddr 00:19:d2:c0:9f:1e  
          inet addr:31.133.176.72  Bcast:31.133.183.255  Mask:255.255.248.0
          inet6 addr: 2001:67c:370:176:219:d2ff:fec0:9f1e/64 Scope:Global
          inet6 addr: fe80::219:d2ff:fec0:9f1e/64 Scope:Link
          inet6 addr: 2001:67c:370:176:a4de:d671:c818:8e5f/64 Scope:Global

dooku-[~] mcr 5553 %mtr -n -r -6 knothole.gatineau.credil.org
HOST: dooku                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:67c:370:176::2        0.0%    10    4.8   3.4   0.7   6.5   1.9
  2.|-- 2001:569:fff::a            0.0%    10   41.4  25.5   2.5 109.3  32.0
  3.|-- 2001:568:1::50d            0.0%    10    6.4   9.1   5.3  12.5   2.4
  4.|-- 2001:504:16::7577          0.0%    10    6.4  10.0   6.4  17.2   3.4
  5.|-- 2001:4830:ff:f200::1       0.0%    10    9.1   8.5   4.7  11.8   1.9
  6.|-- 2001:4830:ff:f201::2       0.0%    10   40.5  39.9  36.4  45.0   2.6
  7.|-- 2001:4830:ff:1001::2       0.0%    10   49.6  51.5  48.0  55.9   2.6
  8.|-- 2001:4830:ff:1234::1       0.0%    10  106.3 109.5 106.3 115.1   2.5
  9.|-- 2001:4830:ff:f150::1       0.0%    10  120.1 116.8 113.4 129.2   4.8
 10.|-- 2001:4830:ff:b101::2       0.0%    10  119.5 123.0 119.5 130.5   3.5
 11.|-- 2001:4830:ff:b100::2       0.0%    10  120.7 122.9 119.4 130.7   4.3
 12.|-- 2001:4830:e1:b::2          0.0%    10  124.2 122.2 120.0 125.0   1.8
 13.|-- 2001:4830:1100:154::1      0.0%    10  119.7 123.1 119.0 145.5   8.0
 14.|-- 2001:4830:116e:1:224:21ff 10.0%    10  153.3 151.3 149.3 153.5   1.6

dooku-[~] mcr 5554 %mtr -n -r -6 orlando.gatineau.credil.org 
HOST: dooku                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:67c:370:176::2        0.0%    10    1.9   3.8   0.7   7.9   2.8
  2.|-- 2001:569:fff::a            0.0%    10    1.3  18.7   1.3  59.6  20.2
  3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

WIRED:

my laptop on wired:

dooku-[~] mcr 5542 %ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:1a:6b:6a:a9:6a  
          inet addr:31.133.132.29  Bcast:31.133.135.255  Mask:255.255.248.0
          inet6 addr: 2001:67c:370:128:21a:6bff:fe6a:a96a/64 Scope:Global
          inet6 addr: fe80::21a:6bff:fe6a:a96a/64 Scope:Link

dooku-[~] mcr 5559 %mtr -n -r -6 orlando.gatineau.credil.org
HOST: dooku                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:67c:370:128::3        0.0%    10    0.3  53.4   0.3 531.7 168.0
  2.|-- 2001:67c:370:228::2        0.0%    10    0.3   0.3   0.3   0.4   0.0
  3.|-- 2001:569:fff::a            0.0%    10    1.0  10.6   0.7  69.4  21.5
  4.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
dooku-[~] mcr 5560 %mtr -n -r -6 knothole.gatineau.credil.org
HOST: dooku                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:67c:370:128::3        0.0%    10    0.3   0.3   0.2   0.4   0.0
  2.|-- 2001:67c:370:228::2        0.0%    10    0.3   0.3   0.2   0.6   0.1
  3.|-- 2001:569:fff::a            0.0%    10    1.1  15.3   0.9  61.0  20.2
  4.|-- 2001:568:1::50d            0.0%    10    4.1   4.6   4.0   8.6   1.4
  5.|-- 2001:504:16::7577          0.0%    10    5.1   5.0   5.0   5.1   0.0
  6.|-- 2001:4830:ff:f200::1       0.0%    10    5.0   5.1   5.0   5.2   0.1
  7.|-- 2001:4830:ff:f201::2       0.0%    10   36.5  36.6  36.4  37.1   0.2
  8.|-- 2001:4830:ff:1001::2       0.0%    10   47.1  47.3  47.0  48.2   0.3
  9.|-- 2001:4830:ff:1234::1       0.0%    10  105.2 105.7 105.1 107.3   0.6
 10.|-- 2001:4830:ff:f150::1       0.0%    10  112.0 112.4 111.6 113.1   0.5
 11.|-- 2001:4830:ff:b101::2       0.0%    10  117.6 118.1 117.3 119.5   0.6
 12.|-- 2001:4830:ff:b100::2       0.0%    10  121.2 119.1 118.1 121.2   0.9
 13.|-- 2001:4830:e1:b::2         10.0%    10  118.6 118.7 118.2 119.4   0.4
 14.|-- 2001:4830:1100:154::1      0.0%    10  118.3 118.3 117.7 120.0   0.7
 15.|-- 2001:4830:116e:1:224:21ff 10.0%    10  147.7 148.5 147.7 150.8   1.0

untitled-part.bin

Attachments (1)

untitled-part.bin (307 bytes) - added by mcr@… 4 years ago.
Added by email2trac

Download all attachments as: .zip

Change history (10)

Changed 4 years ago by mcr@…

Attachment: untitled-part.bin added

Added by email2trac

comment:1 Changed 4 years ago by llynch@…

Owner: changed from llynch@… to cdoyle@…
Status: newassigned

comment:2 Changed 4 years ago by anonymous

We are currently working with our upstream provider to isolate and resolve a v6 routing through a transit AS.

comment:3 Changed 4 years ago by mcr@…

I want to mention that things are more functional, or least more deterministic: I find that either all addresses work, or none of them. Hitting up-arrow return enough times makes things work.
(I was using my aiccu/sixxs tunnel for awhile to work around this)

comment:4 Changed 4 years ago by Bill Fenner

There is clearly something bad going on; after the first hop in the Telus network we see 50% loss on some IPv6 paths.

My working theory is that there's ECMP but one of the paths is not actually forwarding IPv6 properly. This is one of the failure modes that can happen more easily when transporting IPv6 prefixes over IPv4 transport, because the BGP session doesn't share state with the IPv6. Of course, I have no idea what the IPv6 topology is inside Telus or if there's meant to be ECMP, it is just something that fits the 50% loss we see.

To Arista in Santa Clara (Level3):

traceroute6 to 2001:550:401::401 (2001:550:401::401) from 2001:67c:370:229::12, 64 hops max, 12 byte packets
 1  rtra  0.239 ms  0.415 ms  0.189 ms
 2  2001:569:fff::a  7.237 ms  0.830 ms  0.870 ms
 3  2001:568:1::50d  4.025 ms *  4.863 ms
 4  *
    2001:550:3::2e1  4.277 ms *
 5  2001:550::67:1113  25.087 ms  25.431 ms *
 6  2001:550::25  24.914 ms *  25.078 ms
 7  *
    2001:550::1040  25.100 ms *
 8  2001:550:2:3::8:1  25.094 ms *  25.215 ms
 9  *
    2001:550:2:3::8:2  24.288 ms *
10  2001:550:401::401  25.025 ms * *

When we hit this bad path, we start getting nearly exactly every other packet lost.

comment:5 Changed 4 years ago by Bill Fenner

The pings at

https://management.meeting.ietf.org/smokeping/smokeping.cgi?target=Links.External-IPv6.arista

and

https://management.meeting.ietf.org/smokeping/smokeping.cgi?target=Links.External-IPv6.mcr

show 100% loss. If my theory holds, this is because the pings are hashing to the half of the ECMP that's not working.

comment:6 Changed 4 years ago by Bill Fenner

Telus has fixed an issue inside their network, and all of my tests are passing now. mcr, want to re-test?

comment:7 Changed 4 years ago by cdoyle@…

Resolution: fixed
Status: assignedclosed

Telus identified a router issue. This issue has been resolved and all seems well in the world of v6.

comment:8 Changed 4 years ago by anonymous

All the connections that I could not make now seem to work.
Thanks for pursuing this.

comment:9 Changed 5 weeks ago by Rick Alfvin

Milestone: ietf-88ietf-088

Milestone renamed

Note: See TracTickets for help on using tickets.