When a TCP segment is split up (to be sent through a TUN device with a small MTU) who should recalculate the checksum?
Subsystems (2)
I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.
office1 LAN--------office1 tunnel machine
|
| openvpn tunnel
|office2 tunnel machine------office2 LAN
What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.
But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.
For example, here is a large packet coming in on the NIC of the machine running the tunnel, followed by a smaller packet ("caronia" is on the office 1 LAN, "olympic" is on the office 2 LAN):
11:59:23.020426 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 6770 11:59:23.041072 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354
Then the packet gets sent out on the tunnel as 5 smaller packets:
11:59:23.020449 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020534 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020536 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020539 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020543 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.041086 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354
And this is what the receiving system sees:
11:59:23.025658 IP (tos 0x8, ttl 62, id 42831, offset 0, flags [DF], proto TCP (6), length 1406)
caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0xb1b9), seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.025907 IP (tos 0x8, ttl 62, id 42832, offset 0, flags [DF], proto TCP (6), length 1406)
caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x871c), seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.025990 IP (tos 0x8, ttl 62, id 42833, offset 0, flags [DF], proto TCP (6), length 1406)
caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x97dd), seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.026183 IP (tos 0x8, ttl 62, id 42834, offset 0, flags [DF], proto TCP (6), length 1406)
caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x9961), seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.026231 IP (tos 0x8, ttl 62, id 42835, offset 0, flags [DF], proto TCP (6), length 1406)
caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x6a2a), seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.046163 IP (tos 0x8, ttl 62, id 42836, offset 0, flags [DF], proto TCP (6), length 1406)
caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0xd237 (correct), seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354The receiving system is, of course, unhappy about that and complains that it hasn't got 3073:9843
11:59:23.045040 IP olympic.calvaedi.com.ssh > caronia.CalvaEDI.COM.33232: Flags [.], ack 3073, win 1933, options [nop,nop,TS val 1199882514 ecr 215919290,nop,nop,sack 1 {9843:11197}], length 0
So, when the 6770 byte segment is split up into five 1354 byte segments who is supposed to recalculate the checksums?
(This is Debian bug 729567).
Signature · 1 line
John Hughes, SARL Atlantic Technologies.
Quoted 58 lines · John Hughes
I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.
office1 LAN--------office1 tunnel machine
| | openvpn tunnel |office2 tunnel machine------office2 LAN
What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.
But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.
For example, here is a large packet coming in on the NIC of the machine running the tunnel, followed by a smaller packet ("caronia" is on the office 1 LAN, "olympic" is on the office 2 LAN):
11:59:23.020426 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 6770 11:59:23.041072 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354
Then the packet gets sent out on the tunnel as 5 smaller packets:
11:59:23.020449 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020534 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020536 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020539 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020543 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.041086 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354
And this is what the receiving system sees:
11:59:23.025658 IP (tos
0x8, ttl 62, id 42831, offset 0, flags [DF], proto TCP (6), length 1406)caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0xb1b9), seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.025907 IP (tos 0x8, ttl 62, id 42832, offset 0, flags [DF], proto TCP (6), length 1406) caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x871c), seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.025990 IP (tos 0x8, ttl 62, id 42833, offset 0, flags [DF], proto TCP (6), length 1406) caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x97dd), seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.026183 IP (tos 0x8, ttl 62, id 42834, offset 0, flags [DF], proto TCP (6), length 1406) caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x9961), seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.026231 IP (tos 0x8, ttl 62, id 42835, offset 0, flags [DF], proto TCP (6), length 1406) caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x6a2a), seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.046163 IP (tos 0x8, ttl 62, id 42836, offset 0, flags [DF], proto TCP (6), length 1406) caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0xd237 (correct), seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354The receiving system is, of course, unhappy about that and complains that it hasn't got 3073:9843
Quoted 8 lines
11:59:23.045040 IP olympic.calvaedi.com.ssh > caronia.CalvaEDI.COM.33232: Flags [.], ack 3073, win 1933, options [nop,nop,TS val 1199882514 ecr 215919290,nop,nop,sack 1 {9843:11197}], length 0
So, when the 6770 byte segment is split up into five 1354 byte segments who is supposed to recalculate the checksums?
(This is Debian bug 729567).
Thanks for the report
It depends on the offload capabilities of the NIC forwarding packets
ethtool -k eth0
TCP checksums can be recomputed by tcp_gso_segment() (it was named tcp_tso_segment() in linux 3.10), unless the NIC told it was doing the checksum computation itself.
ethtool -K eth0 tx off
Should request stack to perform the cheksums.
What is the NIC doing the transmits ?
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · John Hughes
I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.
office1 LAN--------office1 tunnel machine
| | openvpn tunnel |office2 tunnel machine------office2 LAN
What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.
Quoted 8 lines
Quoted 5 lines
But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.
Thanks for the report
It depends on the offload capabilities of the NIC forwarding packets
The NIC is a kernel tun device.
Quoted 2 lines
ethtool -k eth0
# ethtool -k tun1 Features for tun1: rx-checksumming: off [fixed] tx-checksumming: off tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: off [requested on] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] ...
It seems someone has asked the tunnel to compute checksums (tx-checksum-ip-generic), but it can't do that.
Quoted 13 lines
TCP checksums can be recomputed by
tcp_gso_segment()(it was namedtcp_tso_segment()in linux 3.10), unless the NIC told it was doing the checksum computation itself.ethtool -K eth0 tx off
Should request stack to perform the cheksums.
What is the NIC doing the transmits ?
Quoted 97 lines · John Hughes
I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.
office1 LAN--------office1 tunnel machine
| | openvpn tunnel |office2 tunnel machine------office2 LAN
What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.
But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.
For example, here is a large packet coming in on the NIC of the machine running the tunnel, followed by a smaller packet ("caronia" is on the office 1 LAN, "olympic" is on the office 2 LAN):
11:59:23.020426 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 6770 11:59:23.041072 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354
Then the packet gets sent out on the tunnel as 5 smaller packets:
11:59:23.020449 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020534 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020536 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020539 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020543 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.041086 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354
And this is what the receiving system sees:
11:59:23.025658 IP (tos
0x8, ttl 62, id 42831, offset 0, flags [DF], proto TCP (6), length 1406)caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],cksum
0x1003(incorrect ->0xb1b9), seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.025907 IP (tos0x8, ttl 62, id 42832, offset 0, flags [DF], proto TCP (6), length 1406)caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],cksum
0x1003(incorrect ->0x871c), seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.025990 IP (tos0x8, ttl 62, id 42833, offset 0, flags [DF], proto TCP (6), length 1406)caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],cksum
0x1003(incorrect ->0x97dd), seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.026183 IP (tos0x8, ttl 62, id 42834, offset 0, flags [DF], proto TCP (6), length 1406)caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],cksum
0x1003(incorrect ->0x9961), seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.026231 IP (tos0x8, ttl 62, id 42835, offset 0, flags [DF], proto TCP (6), length 1406)caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],cksum
0x1003(incorrect ->0x6a2a), seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.046163 IP (tos0x8, ttl 62, id 42836, offset 0, flags [DF], proto TCP (6), length 1406)caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],cksum
0xd237(correct), seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354The receiving system is, of course, unhappy about that and complains that it hasn't got 3073:9843
11:59:23.045040 IP olympic.calvaedi.com.ssh > caronia.CalvaEDI.COM.33232: Flags [.], ack 3073, win 1933, options [nop,nop,TS val 1199882514 ecr 215919290,nop,nop,sack 1 {9843:11197}], length 0
Quoted 6 lines
So, when the 6770 byte segment is split up into five 1354 byte segments who is supposed to recalculate the checksums?
(This is Debian bug 729567).
Can you check to see if you have the following patch in your kernel
commit: 1cdbcb7957cf9e5f841dbcde9b38fd18a804208b Author: Simon Horman <[email protected]> Date: Sun May 19 15:46:49 2013 +0000
net: Loosen constraints for recalculating checksum in skb_segment()
This commit help if the forwarding system has to re-segment the data before transition. Especially if the receiving interface had GRO enabled with checksum offloading and the transmitting interface does not support checksum offloading.
-vlad
Quoted 6 lines · Vlad Yasevich
Can you check to see if you have the following patch in your kernel
commit: 1cdbcb7957cf9e5f841dbcde9b38fd18a804208b Author: Simon Horman <[email protected]> Date: Sun May 19 15:46:49 2013 +0000
Quoted 7 lines
net: Loosen constraints for recalculating checksum in
skb_segment()This commit help if the forwarding system has to re-segment the data before transition. Especially if the receiving interface had GRO enabled with checksum offloading and the transmitting interface does not support checksum offloading.
No, the Debian 3.10 kernel doesn't seem to have that commit:
Around line 2859 in skbuff.c I see:
if (fskb != skb_shinfo(skb)->frag_list)
continue;Quoted 15 lines · John Hughes
Quoted 6 lines · Vlad Yasevich
Can you check to see if you have the following patch in your kernel
commit: 1cdbcb7957cf9e5f841dbcde9b38fd18a804208b Author: Simon Horman <[email protected]> Date: Sun May 19 15:46:49 2013 +0000Quoted 7 lines
net: Loosen constraints for recalculating checksum in
skb_segment()This commit help if the forwarding system has to re-segment the data before transition. Especially if the receiving interface had GRO enabled with checksum offloading and the transmitting interface does not support checksum offloading.
Quoted 7 lines
No, the Debian 3.10 kernel doesn't seem to have that commit:
Around line 2859 in
skbuff.cI see:if (fskb != skb_shinfo(skb)->frag_list) continue;
Give it a try. It has helped me in similar situations. If it helps, we can get it into the stable tree.
-vlad
Quoted 8 lines · Vlad Yasevich
Quoted 4 lines · John Hughes
Quoted 3 lines · Vlad Yasevich
Can you check to see if you have the following patch in your kernel
commit:1cdbcb7957cf9e5f841dbcde9b38fd18a804208bGive it a try. It has helped me in similar situations. If it helps, we can get it into the stable tree.
Ok, I'll make myself a kernel with that patch.
Should be able to try it over the weekend.
Signature · 1 line
John.