Skip to contentSkip to conversation

Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later

DISCUSSION Patrick Boutilier 3 participants 6 messages Mar 17, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I have an IPIP tunnel setup between two hosts in different buildings. The Linux router they route through causes extreme slowness in the tunnel when running kernels from 3.18 on . tcpdump shows many cksum errors which don't show up in the 3.17 and earlier kernels. Speed goes back to normal when rx-checksumming and tx-checksumming are turned off using ethtool on the Linux router . Would this be an effect of bulk network packet transmission that was introduced in 3.18 or some other issue?

eth2
driver: igb
version: 5.2.15-k
firmware-version: 1.5.1
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

eth6
driver: ixgbe
version: 3.19.1-k
firmware-version: 0x80000389
bus-info: 0000:41:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Example of cksum from tcpdump:

Flags [.], cksum 0x2237 (incorrect -> 0xbb02), seq 424119200:424120648, ack 1, win 115, options [nop,nop,TS val 303173784 ecr 3971495454], length 1448

Results of nuttcp test with rx-checksumming and tx-checksumming on:

Testing upload inside tunnel

    0.0625 MB /   5.00 sec =    0.1048 Mbps    79 retrans
    0.0625 MB /   5.00 sec =    0.1048 Mbps    59 retrans
    0.0625 MB /   5.00 sec =    0.1049 Mbps    27 retrans
    0.2096 MB /  16.11 sec =    0.1092 Mbps 0 %TX 0 %RX 186 retrans 

0.47 msRTT

Testing download inside tunnel

    0.5000 MB /   5.00 sec =    0.8389 Mbps   112 retrans
    2.6250 MB /   5.00 sec =    4.4040 Mbps   412 retrans
    0.8750 MB /   5.00 sec =    1.4680 Mbps   156 retrans
    4.0243 MB /  15.48 sec =    2.1809 Mbps 0 %TX 0 %RX 705 retrans 

0.47 msRTT

Testing upload outside tunnel

  423.0000 MB /   5.00 sec =  709.4949 Mbps   275 retrans
  442.0625 MB /   5.00 sec =  741.7008 Mbps    53 retrans
  418.5625 MB /   5.00 sec =  702.1329 Mbps   133 retrans

1287.0080 MB / 15.08 sec = 715.9252 Mbps 1 %TX 30 %RX 461 retrans
0.43 msRTT

Testing download outside tunnel

  552.4375 MB /   5.00 sec =  926.8347 Mbps    26 retrans
  555.1875 MB /   5.00 sec =  931.4499 Mbps    17 retrans
  553.0000 MB /   5.00 sec =  927.7802 Mbps    16 retrans

1664.2149 MB / 15.03 sec = 928.6737 Mbps 9 %TX 5 %RX 60 retrans 0.46
msRTT

Results of nuttcp test with rx-checksumming and tx-checksumming off:

Testing upload inside tunnel

  440.0000 MB /   5.00 sec =  738.0112 Mbps   117 retrans
  433.0000 MB /   5.00 sec =  726.4892 Mbps   135 retrans
  447.9375 MB /   5.00 sec =  751.5714 Mbps   112 retrans

1321.0069 MB / 15.04 sec = 736.7213 Mbps 3 %TX 21 %RX 364 retrans
0.51 msRTT

Testing download inside tunnel

  494.0625 MB /   5.00 sec =  828.8973 Mbps   335 retrans
  471.0000 MB /   5.00 sec =  790.2050 Mbps   219 retrans
  469.8750 MB /   5.00 sec =  788.3174 Mbps   220 retrans

1435.1875 MB / 15.00 sec = 802.4203 Mbps 89 %TX 12 %RX 774 retrans
0.46 msRTT

Testing upload outside tunnel

  431.2500 MB /   5.00 sec =  723.2808 Mbps    33 retrans
  446.3750 MB /   5.00 sec =  748.9268 Mbps    10 retrans
  475.4375 MB /   5.00 sec =  797.7044 Mbps     0 retrans

1355.0834 MB / 15.06 sec = 754.6438 Mbps 1 %TX 33 %RX 43 retrans
0.53 msRTT

Testing download outside tunnel

  424.8125 MB /   5.00 sec =  712.7061 Mbps   102 retrans
  456.5625 MB /   5.00 sec =  765.9872 Mbps    80 retrans
  454.0000 MB /   5.00 sec =  761.6935 Mbps   116 retrans

1335.7823 MB / 15.01 sec = 746.6959 Mbps 4 %TX 5 %RX 298 retrans
0.55 msRTT

begin:vcard
fn:Patrick Boutilier
n:Boutilier;Patrick
org:;Nova Scotia Department of Education adr:;;2021 Brunswick Street;Halifax;NS;B3K 2Y5;Canada email;internet:[email protected]
title:WAN Communications Specialist tel;work:902-424-6800
tel;fax:902-424-0874
version:2.1
end:vcard

#1 lore raw
Quoted 8 lines · Patrick Boutilier

I have an IPIP tunnel setup between two hosts in different buildings. The Linux router they route through causes extreme slowness in the tunnel when running kernels from 3.18 on . tcpdump shows many cksum errors which don't show up in the 3.17 and earlier kernels. Speed goes back to normal when rx-checksumming and tx-checksumming are turned off using ethtool on the Linux router . Would this be an effect of bulk network packet transmission that was introduced in 3.18 or some other issue?

If this is happening on a router then the only case where checksums are pertinent should be if GSO/GRO is enabled. Please try disabling these. Also, are these showing up as checksum errors on the receiver (netstat -s).

Quoted 146 lines

eth2
driver: igb
version: 5.2.15-k
firmware-version: 1.5.1
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

eth6
driver: ixgbe
version: 3.19.1-k
firmware-version: 0x80000389
bus-info: 0000:41:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Example of cksum from tcpdump:

Flags [.], cksum 0x2237 (incorrect -> 0xbb02), seq 424119200:424120648, ack 1, win 115, options [nop,nop,TS val 303173784 ecr 3971495454], length 1448

Results of nuttcp test with rx-checksumming and tx-checksumming on:

Testing upload inside tunnel

    0.0625 MB /   5.00 sec =    0.1048 Mbps    79 retrans
    0.0625 MB /   5.00 sec =    0.1048 Mbps    59 retrans
    0.0625 MB /   5.00 sec =    0.1049 Mbps    27 retrans
    0.2096 MB /  16.11 sec =    0.1092 Mbps 0 %TX 0 %RX 186 retrans 0.47

msRTT

Testing download inside tunnel

    0.5000 MB /   5.00 sec =    0.8389 Mbps   112 retrans
    2.6250 MB /   5.00 sec =    4.4040 Mbps   412 retrans
    0.8750 MB /   5.00 sec =    1.4680 Mbps   156 retrans
    4.0243 MB /  15.48 sec =    2.1809 Mbps 0 %TX 0 %RX 705 retrans 0.47

msRTT

Testing upload outside tunnel

  423.0000 MB /   5.00 sec =  709.4949 Mbps   275 retrans
  442.0625 MB /   5.00 sec =  741.7008 Mbps    53 retrans
  418.5625 MB /   5.00 sec =  702.1329 Mbps   133 retrans

1287.0080 MB / 15.08 sec = 715.9252 Mbps 1 %TX 30 %RX 461 retrans 0.43
msRTT

Testing download outside tunnel

  552.4375 MB /   5.00 sec =  926.8347 Mbps    26 retrans
  555.1875 MB /   5.00 sec =  931.4499 Mbps    17 retrans
  553.0000 MB /   5.00 sec =  927.7802 Mbps    16 retrans

1664.2149 MB / 15.03 sec = 928.6737 Mbps 9 %TX 5 %RX 60 retrans 0.46
msRTT

Results of nuttcp test with rx-checksumming and tx-checksumming off:

Testing upload inside tunnel

  440.0000 MB /   5.00 sec =  738.0112 Mbps   117 retrans
  433.0000 MB /   5.00 sec =  726.4892 Mbps   135 retrans
  447.9375 MB /   5.00 sec =  751.5714 Mbps   112 retrans

1321.0069 MB / 15.04 sec = 736.7213 Mbps 3 %TX 21 %RX 364 retrans 0.51
msRTT

Testing download inside tunnel

  494.0625 MB /   5.00 sec =  828.8973 Mbps   335 retrans
  471.0000 MB /   5.00 sec =  790.2050 Mbps   219 retrans
  469.8750 MB /   5.00 sec =  788.3174 Mbps   220 retrans

1435.1875 MB / 15.00 sec = 802.4203 Mbps 89 %TX 12 %RX 774 retrans 0.46
msRTT

Testing upload outside tunnel

  431.2500 MB /   5.00 sec =  723.2808 Mbps    33 retrans
  446.3750 MB /   5.00 sec =  748.9268 Mbps    10 retrans
  475.4375 MB /   5.00 sec =  797.7044 Mbps     0 retrans

1355.0834 MB / 15.06 sec = 754.6438 Mbps 1 %TX 33 %RX 43 retrans 0.53
msRTT

Testing download outside tunnel

  424.8125 MB /   5.00 sec =  712.7061 Mbps   102 retrans
  456.5625 MB /   5.00 sec =  765.9872 Mbps    80 retrans
  454.0000 MB /   5.00 sec =  761.6935 Mbps   116 retrans

1335.7823 MB / 15.01 sec = 746.6959 Mbps 4 %TX 5 %RX 298 retrans 0.55
msRTT

#3 lore raw
In reply to Patrick Boutilier
Quoted 7 lines · Patrick Boutilier

I have an IPIP tunnel setup between two hosts in different buildings. The Linux router they route through causes extreme slowness in the tunnel when running kernels from 3.18 on . tcpdump shows many cksum errors which don't show up in the 3.17 and earlier kernels. Speed goes back to normal when rx-checksumming and tx-checksumming are turned off using ethtool on the Linux router . Would this be an effect of bulk network packet transmission that was introduced in 3.18 or some other issue?

I just sent out a likely fix, would you mind testing it?
https://patchwork.ozlabs.org/patch/599213/

#2 lore raw
In reply to Tom Herbert
Quoted 9 lines · Tom Herbert
Quoted 8 lines · Patrick Boutilier

I have an IPIP tunnel setup between two hosts in different buildings. The Linux router they route through causes extreme slowness in the tunnel when running kernels from 3.18 on . tcpdump shows many cksum errors which don't show up in the 3.17 and earlier kernels. Speed goes back to normal when rx-checksumming and tx-checksumming are turned off using ethtool on the Linux router . Would this be an effect of bulk network packet transmission that was introduced in 3.18 or some other issue?

Quoted 4 lines

If this is happening on a router then the only case where checksums are pertinent should be if GSO/GRO is enabled. Please try disabling these. Also, are these showing up as checksum errors on the receiver (netstat -s).

Yes, turning off GSO/GRO also fixes the issue.

Not sure which value of netstat I am looking for.

Compiling a new kernel now with Jesse's patch to test. Will report back in a bit.

Quoted 142 lines
Quoted 142 lines

eth2
driver: igb
version: 5.2.15-k
firmware-version: 1.5.1
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

eth6
driver: ixgbe
version: 3.19.1-k
firmware-version: 0x80000389
bus-info: 0000:41:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Example of cksum from tcpdump:

Flags [.], cksum 0x2237 (incorrect -> 0xbb02), seq 424119200:424120648, ack 1, win 115, options [nop,nop,TS val 303173784 ecr 3971495454], length 1448

Results of nuttcp test with rx-checksumming and tx-checksumming on:

Testing upload inside tunnel

     0.0625 MB /   5.00 sec =    0.1048 Mbps    79 retrans
     0.0625 MB /   5.00 sec =    0.1048 Mbps    59 retrans
     0.0625 MB /   5.00 sec =    0.1049 Mbps    27 retrans
     0.2096 MB /  16.11 sec =    0.1092 Mbps 0 %TX 0 %RX 186 retrans 0.47

msRTT

Testing download inside tunnel

     0.5000 MB /   5.00 sec =    0.8389 Mbps   112 retrans
     2.6250 MB /   5.00 sec =    4.4040 Mbps   412 retrans
     0.8750 MB /   5.00 sec =    1.4680 Mbps   156 retrans
     4.0243 MB /  15.48 sec =    2.1809 Mbps 0 %TX 0 %RX 705 retrans 0.47

msRTT

Testing upload outside tunnel

   423.0000 MB /   5.00 sec =  709.4949 Mbps   275 retrans
   442.0625 MB /   5.00 sec =  741.7008 Mbps    53 retrans
   418.5625 MB /   5.00 sec =  702.1329 Mbps   133 retrans

1287.0080 MB / 15.08 sec = 715.9252 Mbps 1 %TX 30 %RX 461 retrans 0.43
msRTT

Testing download outside tunnel

   552.4375 MB /   5.00 sec =  926.8347 Mbps    26 retrans
   555.1875 MB /   5.00 sec =  931.4499 Mbps    17 retrans
   553.0000 MB /   5.00 sec =  927.7802 Mbps    16 retrans

1664.2149 MB / 15.03 sec = 928.6737 Mbps 9 %TX 5 %RX 60 retrans 0.46
msRTT

Results of nuttcp test with rx-checksumming and tx-checksumming off:

Testing upload inside tunnel

   440.0000 MB /   5.00 sec =  738.0112 Mbps   117 retrans
   433.0000 MB /   5.00 sec =  726.4892 Mbps   135 retrans
   447.9375 MB /   5.00 sec =  751.5714 Mbps   112 retrans

1321.0069 MB / 15.04 sec = 736.7213 Mbps 3 %TX 21 %RX 364 retrans 0.51
msRTT

Testing download inside tunnel

   494.0625 MB /   5.00 sec =  828.8973 Mbps   335 retrans
   471.0000 MB /   5.00 sec =  790.2050 Mbps   219 retrans
   469.8750 MB /   5.00 sec =  788.3174 Mbps   220 retrans

1435.1875 MB / 15.00 sec = 802.4203 Mbps 89 %TX 12 %RX 774 retrans 0.46
msRTT

Testing upload outside tunnel

   431.2500 MB /   5.00 sec =  723.2808 Mbps    33 retrans
   446.3750 MB /   5.00 sec =  748.9268 Mbps    10 retrans
   475.4375 MB /   5.00 sec =  797.7044 Mbps     0 retrans

1355.0834 MB / 15.06 sec = 754.6438 Mbps 1 %TX 33 %RX 43 retrans 0.53
msRTT

Testing download outside tunnel

Quoted 6 lines
Quoted 6 lines
   424.8125 MB /   5.00 sec =  712.7061 Mbps   102 retrans
   456.5625 MB /   5.00 sec =  765.9872 Mbps    80 retrans
   454.0000 MB /   5.00 sec =  761.6935 Mbps   116 retrans

1335.7823 MB / 15.01 sec = 746.6959 Mbps 4 %TX 5 %RX 298 retrans 0.55
msRTT

begin:vcard
fn:Patrick Boutilier
n:Boutilier;Patrick
org:;Nova Scotia Department of Education adr:;;2021 Brunswick Street;Halifax;NS;B3K 2Y5;Canada email;internet:[email protected]
title:WAN Communications Specialist tel;work:902-424-6800
tel;fax:902-424-0874
version:2.1
end:vcard

#4 lore raw
In reply to Jesse Gross
Quoted 9 lines · Jesse Gross
Quoted 7 lines · Patrick Boutilier

I have an IPIP tunnel setup between two hosts in different buildings. The Linux router they route through causes extreme slowness in the tunnel when running kernels from 3.18 on . tcpdump shows many cksum errors which don't show up in the 3.17 and earlier kernels. Speed goes back to normal when rx-checksumming and tx-checksumming are turned off using ethtool on the Linux router . Would this be an effect of bulk network packet transmission that was introduced in 3.18 or some other issue?

Quoted 3 lines

I just sent out a likely fix, would you mind testing it?
https://patchwork.ozlabs.org/patch/599213/

Yes, that patch fixes the problem.

Thanks.

begin:vcard
fn:Patrick Boutilier
n:Boutilier;Patrick
org:;Nova Scotia Department of Education adr:;;2021 Brunswick Street;Halifax;NS;B3K 2Y5;Canada email;internet:[email protected]
title:WAN Communications Specialist tel;work:902-424-6800
tel;fax:902-424-0874
version:2.1
end:vcard

#5 lore raw
Quoted 17 lines · Patrick Boutilier
Quoted 16 lines · Jesse Gross
Quoted 11 lines · Patrick Boutilier

I have an IPIP tunnel setup between two hosts in different buildings. The Linux router they route through causes extreme slowness in the tunnel when
running kernels from 3.18 on . tcpdump shows many cksum errors which don't
show up in the 3.17 and earlier kernels. Speed goes back to normal when rx-checksumming and tx-checksumming are turned off using ethtool on the Linux router . Would this be an effect of bulk network packet transmission
that was introduced in 3.18 or some other issue?

Quoted 5 lines
Quoted 3 lines

I just sent out a likely fix, would you mind testing it?
https://patchwork.ozlabs.org/patch/599213/

Yes, that patch fixes the problem.

Great, thanks a lot for testing.