Skip to contentSkip to conversation

When a TCP segment is split up (to be sent through a TUN device with a small MTU) who should recalculate the checksum?

DISCUSSION John Hughes 5 participants 7 messages Nov 15, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.

office1 LAN--------office1 tunnel machine

                            |
                            | openvpn tunnel
                            |

office2 tunnel machine------office2 LAN

What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.

But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.

For example, here is a large packet coming in on the NIC of the machine running the tunnel, followed by a smaller packet ("caronia" is on the office 1 LAN, "olympic" is on the office 2 LAN):

11:59:23.020426 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 6770 11:59:23.041072 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

Then the packet gets sent out on the tunnel as 5 smaller packets:

11:59:23.020449 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020534 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020536 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020539 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020543 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.041086 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

And this is what the receiving system sees:

11:59:23.025658 IP (tos 0x8, ttl 62, id 42831, offset 0, flags [DF], proto TCP (6), length 1406)

    caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0xb1b9), seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.025907 IP (tos 0x8, ttl 62, id 42832, offset 0, flags [DF], proto TCP (6), length 1406)
    caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x871c), seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.025990 IP (tos 0x8, ttl 62, id 42833, offset 0, flags [DF], proto TCP (6), length 1406)
    caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x97dd), seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.026183 IP (tos 0x8, ttl 62, id 42834, offset 0, flags [DF], proto TCP (6), length 1406)
    caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x9961), seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.026231 IP (tos 0x8, ttl 62, id 42835, offset 0, flags [DF], proto TCP (6), length 1406)
    caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x6a2a), seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.046163 IP (tos 0x8, ttl 62, id 42836, offset 0, flags [DF], proto TCP (6), length 1406)
    caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0xd237 (correct), seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

The receiving system is, of course, unhappy about that and complains that it hasn't got 3073:9843

11:59:23.045040 IP olympic.calvaedi.com.ssh > caronia.CalvaEDI.COM.33232: Flags [.], ack 3073, win 1933, options [nop,nop,TS val 1199882514 ecr 215919290,nop,nop,sack 1 {9843:11197}], length 0

So, when the 6770 byte segment is split up into five 1354 byte segments who is supposed to recalculate the checksums?

(This is Debian bug 729567).

Signature · 1 line
John Hughes, SARL Atlantic Technologies.
#1 lore raw
Quoted 58 lines · John Hughes

I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.

office1 LAN--------office1 tunnel machine

                             |
                             | openvpn tunnel
                             |

office2 tunnel machine------office2 LAN

What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.

But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.

For example, here is a large packet coming in on the NIC of the machine running the tunnel, followed by a smaller packet ("caronia" is on the office 1 LAN, "olympic" is on the office 2 LAN):

11:59:23.020426 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 6770 11:59:23.041072 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

Then the packet gets sent out on the tunnel as 5 smaller packets:

11:59:23.020449 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020534 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020536 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020539 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020543 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.041086 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

And this is what the receiving system sees:

11:59:23.025658 IP (tos 0x8, ttl 62, id 42831, offset 0, flags [DF], proto TCP (6), length 1406)

     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0xb1b9), seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.025907 IP (tos 0x8, ttl 62, id 42832, offset 0, flags [DF], proto TCP (6), length 1406)
     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x871c), seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.025990 IP (tos 0x8, ttl 62, id 42833, offset 0, flags [DF], proto TCP (6), length 1406)
     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x97dd), seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.026183 IP (tos 0x8, ttl 62, id 42834, offset 0, flags [DF], proto TCP (6), length 1406)
     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x9961), seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.026231 IP (tos 0x8, ttl 62, id 42835, offset 0, flags [DF], proto TCP (6), length 1406)
     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0x1003 (incorrect -> 0x6a2a), seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354
11:59:23.046163 IP (tos 0x8, ttl 62, id 42836, offset 0, flags [DF], proto TCP (6), length 1406)
     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], cksum 0xd237 (correct), seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

The receiving system is, of course, unhappy about that and complains that it hasn't got 3073:9843

Quoted 8 lines

11:59:23.045040 IP olympic.calvaedi.com.ssh > caronia.CalvaEDI.COM.33232: Flags [.], ack 3073, win 1933, options [nop,nop,TS val 1199882514 ecr 215919290,nop,nop,sack 1 {9843:11197}], length 0

So, when the 6770 byte segment is split up into five 1354 byte segments who is supposed to recalculate the checksums?

(This is Debian bug 729567).

Thanks for the report

It depends on the offload capabilities of the NIC forwarding packets

ethtool -k eth0

TCP checksums can be recomputed by tcp_gso_segment() (it was named tcp_tso_segment() in linux 3.10), unless the NIC told it was doing the checksum computation itself.

ethtool -K eth0 tx off

Should request stack to perform the cheksums.

What is the NIC doing the transmits ?

#2 lore raw
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · John Hughes

I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.

office1 LAN--------office1 tunnel machine

                              |
                              | openvpn tunnel
                              |

office2 tunnel machine------office2 LAN

What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.

Quoted 8 lines
Quoted 5 lines

But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.

Thanks for the report

It depends on the offload capabilities of the NIC forwarding packets

The NIC is a kernel tun device.

Quoted 2 lines

ethtool -k eth0

# ethtool -k tun1
Features for tun1:
rx-checksumming: off [fixed]
tx-checksumming: off
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: off [requested on]
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
...

It seems someone has asked the tunnel to compute checksums (tx-checksum-ip-generic), but it can't do that.

Quoted 13 lines

TCP checksums can be recomputed by tcp_gso_segment() (it was named tcp_tso_segment() in linux 3.10), unless the NIC told it was doing the checksum computation itself.

ethtool -K eth0 tx off

Should request stack to perform the cheksums.

What is the NIC doing the transmits ?

#3 lore raw
In reply to John Hughes
Quoted 97 lines · John Hughes

I have two offices, joined by a OpenVPN tunnel. I've upgraded the kernels in the machines running the tunnel to 3.10. All of a sudden I'm getting horrible transmission delays between the two offices.

office1 LAN--------office1 tunnel machine

                             |
                             | openvpn tunnel
                             |

office2 tunnel machine------office2 LAN

What seems to be happening is that packets are arriving at the LAN interface of the machine running the tunnel and being combined by generic-receive-offload. These packets then have to be split up again as they are too big for the tunnels MTU.

But when the packets are split the TCP checksum doesn't seem to be being recalculated, so the systems on the other end of the tunnel ignore them, forcing many retries and the observed delays.

For example, here is a large packet coming in on the NIC of the machine running the tunnel, followed by a smaller packet ("caronia" is on the office 1 LAN, "olympic" is on the office 2 LAN):

11:59:23.020426 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 6770 11:59:23.041072 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

Then the packet gets sent out on the tunnel as 5 smaller packets:

11:59:23.020449 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020534 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020536 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020539 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.020543 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.041086 IP caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.], seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

And this is what the receiving system sees:

11:59:23.025658 IP (tos 0x8, ttl 62, id 42831, offset 0, flags [DF], proto TCP (6), length 1406)

     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],

cksum 0x1003 (incorrect -> 0xb1b9), seq 3073:4427, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.025907 IP (tos 0x8, ttl 62, id 42832, offset 0, flags [DF], proto TCP (6), length 1406)

     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],

cksum 0x1003 (incorrect -> 0x871c), seq 4427:5781, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.025990 IP (tos 0x8, ttl 62, id 42833, offset 0, flags [DF], proto TCP (6), length 1406)

     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],

cksum 0x1003 (incorrect -> 0x97dd), seq 5781:7135, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.026183 IP (tos 0x8, ttl 62, id 42834, offset 0, flags [DF], proto TCP (6), length 1406)

     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],

cksum 0x1003 (incorrect -> 0x9961), seq 7135:8489, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.026231 IP (tos 0x8, ttl 62, id 42835, offset 0, flags [DF], proto TCP (6), length 1406)

     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],

cksum 0x1003 (incorrect -> 0x6a2a), seq 8489:9843, ack 2233, win 148, options [nop,nop,TS val 215919291 ecr 1199882508], length 1354 11:59:23.046163 IP (tos 0x8, ttl 62, id 42836, offset 0, flags [DF], proto TCP (6), length 1406)

     caronia.CalvaEDI.COM.33232 > olympic.calvaedi.com.ssh: Flags [.],

cksum 0xd237 (correct), seq 9843:11197, ack 2233, win 148, options [nop,nop,TS val 215919297 ecr 1199882508], length 1354

The receiving system is, of course, unhappy about that and complains that it hasn't got 3073:9843

11:59:23.045040 IP olympic.calvaedi.com.ssh > caronia.CalvaEDI.COM.33232: Flags [.], ack 3073, win 1933, options [nop,nop,TS val 1199882514 ecr 215919290,nop,nop,sack 1 {9843:11197}], length 0

Quoted 6 lines

So, when the 6770 byte segment is split up into five 1354 byte segments who is supposed to recalculate the checksums?

(This is Debian bug 729567).

Can you check to see if you have the following patch in your kernel

commit: 1cdbcb7957cf9e5f841dbcde9b38fd18a804208b
Author: Simon Horman <[email protected]>
Date:   Sun May 19 15:46:49 2013 +0000

net: Loosen constraints for recalculating checksum in skb_segment()

This commit help if the forwarding system has to re-segment the data before transition. Especially if the receiving interface had GRO enabled with checksum offloading and the transmitting interface does not support checksum offloading.

-vlad

#4 lore raw
Quoted 6 lines · Vlad Yasevich

Can you check to see if you have the following patch in your kernel

commit: 1cdbcb7957cf9e5f841dbcde9b38fd18a804208b
Author: Simon Horman <[email protected]>
Date:   Sun May 19 15:46:49 2013 +0000
Quoted 7 lines

net: Loosen constraints for recalculating checksum in skb_segment()

This commit help if the forwarding system has to re-segment the data before transition. Especially if the receiving interface had GRO enabled with checksum offloading and the transmitting interface does not support checksum offloading.

No, the Debian 3.10 kernel doesn't seem to have that commit:

Around line 2859 in skbuff.c I see:

                if (fskb != skb_shinfo(skb)->frag_list)
                        continue;
#5 lore raw
Quoted 15 lines · John Hughes
Quoted 6 lines · Vlad Yasevich

Can you check to see if you have the following patch in your kernel

commit: 1cdbcb7957cf9e5f841dbcde9b38fd18a804208b
Author: Simon Horman <[email protected]>
Date:   Sun May 19 15:46:49 2013 +0000
Quoted 7 lines

net: Loosen constraints for recalculating checksum in skb_segment()

This commit help if the forwarding system has to re-segment the data before transition. Especially if the receiving interface had GRO enabled with checksum offloading and the transmitting interface does not support checksum offloading.

Quoted 7 lines

No, the Debian 3.10 kernel doesn't seem to have that commit:

Around line 2859 in skbuff.c I see:

                 if (fskb != skb_shinfo(skb)->frag_list)
                         continue;

Give it a try. It has helped me in similar situations. If it helps, we can get it into the stable tree.

-vlad

#6 lore raw
Quoted 8 lines · Vlad Yasevich
Quoted 4 lines · John Hughes
Quoted 3 lines · Vlad Yasevich

Can you check to see if you have the following patch in your kernel
commit: 1cdbcb7957cf9e5f841dbcde9b38fd18a804208b

Give it a try. It has helped me in similar situations. If it helps, we can get it into the stable tree.

Ok, I'll make myself a kernel with that patch.

Should be able to try it over the weekend.

Signature · 1 line
John.