Skip to contentSkip to conversation

TCP keepalives ignored by kernel when the contain timestamps

DISCUSSION Charles Bearden 2 participants 11 messages Jun 9, 2011 – Jun 10, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I have come across a case that looks like it might be a kernel bug. It appears that tcp keepalives sent by a remote system are ignored when they contain tcp timestamps, but are ACKed when they don't. When they are ignored, the remote system resets the connection after a number of retries.

I have replicated this problem on both Ubuntu 10.04 with a 2.6.32-32-server kernel (x86_64) and CentOS 5.6 with a 2.6.18-238.12.1.el5 kernel. I'm sorry that I haven't had a chance to try to replicate the bug with a newer kernel, though a co-worker has looked through changelogs for more recent kernels and didn't find anything that looked relevant.

From either of these hosts I run an application that connects to a remote host for 2-3 minutes, and that for most of that time sends no application data back and forth. After 30 seconds of no data from the Linux host, the remote host sends a garden variety keepalive. When the remote host includes tcp timestamps in the keepalives, they are ignored by the Linux host, and the remote host resets the connection after 10 unACKed keepalives. When timestamps are absent from the keepalives, the Linux host ACKs each one, and all is copacetic.

Text output of a tcpdump trace of a connection that fails:
http://pastebin.com/v6CpteJ9

Text output of a tcpdump trace of a connection that succeeds:
http://pastebin.com/KVLb3Mzh

More details, in case you think they are relevant:

My application creates a JDBC connection to a remote MS SQL Server and executes a statement that does not return a result set, and so it doesn't need to pass application data back and forth while it executes. The statement takes 2 or 3 minutes to complete. I connect to two different remote hosts: a Win2003 machine, and a Win2008R2 machine. The Win2003 machine doesn't put timestamps in its keep-alives, so the application completes successfully when connecting to that host. If tcp timestamps are enabled on the Linux host, the Win2008 host includes them in its keepalives, and they are unACKed, so the connection is reset; if they are disabled on the Linux host, the Win2008 host doesn't include them in the keepalives, and the application completes successfully. I use (as you might expect) sysctl to disable tcp timestamps on the Linux hosts.

I have dumps for all permutations of CentOS/Ubuntu, Win200[38], and +/- timestamps on the Linux side, and I will post them if the developers think that they would be useful.

Thanks,
Chuck Bearden
Programmer Analyst IV
The University of Texas Health Science Center at Houston School of Biomedical Informatics
Email: [email protected]

#1 lore raw
Quoted 40 lines · Charles Bearden

I have come across a case that looks like it might be a kernel bug. It appears that tcp keepalives sent by a remote system are ignored when they contain tcp timestamps, but are ACKed when they don't. When they are ignored, the remote system resets the connection after a number of retries.

I have replicated this problem on both Ubuntu 10.04 with a 2.6.32-32-server kernel (x86_64) and CentOS 5.6 with a 2.6.18-238.12.1.el5 kernel. I'm sorry that I haven't had a chance to try to replicate the bug with a newer kernel, though a co-worker has looked through changelogs for more recent kernels and didn't find anything that looked relevant.

From either of these hosts I run an application that connects to a remote host
for 2-3 minutes, and that for most of that time sends no application data back and forth. After 30 seconds of no data from the Linux host, the remote host sends a garden variety keepalive. When the remote host includes tcp timestamps in the keepalives, they are ignored by the Linux host, and the remote host resets the connection after 10 unACKed keepalives. When timestamps are absent from the keepalives, the Linux host ACKs each one, and all is copacetic.

Text output of a tcpdump trace of a connection that fails:
http://pastebin.com/v6CpteJ9

Text output of a tcpdump trace of a connection that succeeds:
http://pastebin.com/KVLb3Mzh

More details, in case you think they are relevant:

My application creates a JDBC connection to a remote MS SQL Server and executes a statement that does not return a result set, and so it doesn't need to pass application data back and forth while it executes. The statement takes 2 or 3 minutes to complete. I connect to two different remote hosts: a Win2003 machine, and a Win2008R2 machine. The Win2003 machine doesn't put timestamps in its keep-alives, so the application completes successfully when connecting to that host. If tcp timestamps are enabled on the Linux host, the Win2008 host includes them in its keepalives, and they are unACKed, so the connection is reset; if they are disabled on the Linux host, the Win2008 host doesn't include them in the keepalives, and the application completes successfully. I use (as you might expect) sysctl to disable tcp timestamps on the Linux hosts.

Quoted 3 lines

I have dumps for all permutations of CentOS/Ubuntu, Win200[38], and +/- timestamps on the Linux side, and I will post them if the developers think that they would be useful.

Hi Charles

I could not reproduce the problem here, even using a quite old kernel as receiver (2.6.9)

15:54:33.566192 IP 192.168.20.108.55926 > 192.168.20.124.777: SWE 479814493:479814493(0) win 14600 <mss 1460,sackOK,timestamp 151666 0,nop,wscale 7>
15:54:33.566265 IP 192.168.20.124.777 > 192.168.20.108.55926: S 3714869381:3714869381(0) ack 479814494 win 5792 <mss 1460,sackOK,timestamp 54553041 151666,nop,wscale 2> 15:54:33.566274 IP 192.168.20.108.55926 > 192.168.20.124.777: . ack 1 win 115 <nop,nop,timestamp 151666 54553041>
15:54:33.566281 IP 192.168.20.108.55926 > 192.168.20.124.777: P 1:5(4) ack 1 win 115 <nop,nop,timestamp 151666 54553041> 15:54:33.566351 IP 192.168.20.124.777 > 192.168.20.108.55926: . ack 5 win 1448 <nop,nop,timestamp 54553041 151666>
15:54:33.566375 IP 192.168.20.124.777 > 192.168.20.108.55926: P 1:5(4) ack 5 win 1448 <nop,nop,timestamp 54553041 151666> 15:54:33.566380 IP 192.168.20.108.55926 > 192.168.20.124.777: . ack 5 win 115 <nop,nop,timestamp 151666 54553041>
15:54:43.577945 IP 192.168.20.108.55926 > 192.168.20.124.777: . 4:5(1) ack 5 win 115 <nop,nop,timestamp 152668 54553041> 15:54:43.578012 IP 192.168.20.124.777 > 192.168.20.108.55926: . ack 5 win 1448 <nop,nop,timestamp 54563053 152668,nop,nop,sack sack 1 {4:5} > 15:54:53.597946 IP 192.168.20.108.55926 > 192.168.20.124.777: . 4:5(1) ack 5 win 115 <nop,nop,timestamp 153670 54563053> 15:54:53.598012 IP 192.168.20.124.777 > 192.168.20.108.55926: . ack 5 win 1448 <nop,nop,timestamp 54573073 153670,nop,nop,sack sack 1 {4:5} >

Are you sure frame tcp checksums are OK when the 'faulty' linux receive
them ? (tcpdump -v)

#2 lore raw
Quoted 76 lines · Eric Dumazet
Quoted 40 lines · Charles Bearden

I have come across a case that looks like it might be a kernel bug. It appears that tcp keepalives sent by a remote system are ignored when they contain tcp timestamps, but are ACKed when they don't. When they are ignored, the remote system resets the connection after a number of retries.

I have replicated this problem on both Ubuntu 10.04 with a 2.6.32-32-server kernel (x86_64) and CentOS 5.6 with a 2.6.18-238.12.1.el5 kernel. I'm sorry that I haven't had a chance to try to replicate the bug with a newer kernel, though a co-worker has looked through changelogs for more recent kernels and didn't find anything that looked relevant.

From either of these hosts I run an application that connects to a remote host
for 2-3 minutes, and that for most of that time sends no application data back and forth. After 30 seconds of no data from the Linux host, the remote host sends a garden variety keepalive. When the remote host includes tcp timestamps in the keepalives, they are ignored by the Linux host, and the remote host resets the connection after 10 unACKed keepalives. When timestamps are absent from the keepalives, the Linux host ACKs each one, and all is copacetic.

Text output of a tcpdump trace of a connection that fails:
http://pastebin.com/v6CpteJ9

Text output of a tcpdump trace of a connection that succeeds:
http://pastebin.com/KVLb3Mzh

More details, in case you think they are relevant:

My application creates a JDBC connection to a remote MS SQL Server and executes a statement that does not return a result set, and so it doesn't need to pass application data back and forth while it executes. The statement takes 2 or 3 minutes to complete. I connect to two different remote hosts: a Win2003 machine, and a Win2008R2 machine. The Win2003 machine doesn't put timestamps in its keep-alives, so the application completes successfully when connecting to that host. If tcp timestamps are enabled on the Linux host, the Win2008 host includes them in its keepalives, and they are unACKed, so the connection is reset; if they are disabled on the Linux host, the Win2008 host doesn't include them in the keepalives, and the application completes successfully. I use (as you might expect) sysctl to disable tcp timestamps on the Linux hosts.

Quoted 3 lines

I have dumps for all permutations of CentOS/Ubuntu, Win200[38], and +/- timestamps on the Linux side, and I will post them if the developers think that they would be useful.

Hi Charles

I could not reproduce the problem here, even using a quite old kernel as receiver (2.6.9)

15:54:33.566192 IP 192.168.20.108.55926> 192.168.20.124.777: SWE 479814493:479814493(0) win 14600<mss 1460,sackOK,timestamp 151666 0,nop,wscale 7>
15:54:33.566265 IP 192.168.20.124.777> 192.168.20.108.55926: S 3714869381:3714869381(0) ack 479814494 win 5792<mss 1460,sackOK,timestamp 54553041 151666,nop,wscale 2> 15:54:33.566274 IP 192.168.20.108.55926> 192.168.20.124.777: . ack 1 win 115<nop,nop,timestamp 151666 54553041>
15:54:33.566281 IP 192.168.20.108.55926> 192.168.20.124.777: P 1:5(4) ack 1 win 115<nop,nop,timestamp 151666 54553041> 15:54:33.566351 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5 win 1448<nop,nop,timestamp 54553041 151666>
15:54:33.566375 IP 192.168.20.124.777> 192.168.20.108.55926: P 1:5(4) ack 5 win 1448<nop,nop,timestamp 54553041 151666> 15:54:33.566380 IP 192.168.20.108.55926> 192.168.20.124.777: . ack 5 win 115<nop,nop,timestamp 151666 54553041>
15:54:43.577945 IP 192.168.20.108.55926> 192.168.20.124.777: . 4:5(1) ack 5 win 115<nop,nop,timestamp 152668 54553041> 15:54:43.578012 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5 win 1448<nop,nop,timestamp 54563053 152668,nop,nop,sack sack 1 {4:5}> 15:54:53.597946 IP 192.168.20.108.55926> 192.168.20.124.777: . 4:5(1) ack 5 win 115<nop,nop,timestamp 153670 54563053> 15:54:53.598012 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5 win 1448<nop,nop,timestamp 54573073 153670,nop,nop,sack sack 1 {4:5}>

Quoted 2 lines

Are you sure frame tcp checksums are OK when the 'faulty' linux receive
them ? (tcpdump -v)

I will check when I get into the office and let you know.

#3 lore raw
Quoted 77 lines · Charles Bearden
Quoted 76 lines · Eric Dumazet
Quoted 40 lines · Charles Bearden

I have come across a case that looks like it might be a kernel bug. It appears that tcp keepalives sent by a remote system are ignored when they contain tcp timestamps, but are ACKed when they don't. When they are ignored, the remote system resets the connection after a number of retries.

I have replicated this problem on both Ubuntu 10.04 with a 2.6.32-32-server kernel (x86_64) and CentOS 5.6 with a 2.6.18-238.12.1.el5 kernel. I'm sorry that I haven't had a chance to try to replicate the bug with a newer kernel, though a co-worker has looked through changelogs for more recent kernels and didn't find anything that looked relevant.

From either of these hosts I run an application that connects to a remote host for 2-3 minutes, and that for most of that time sends no application data back and forth. After 30 seconds of no data from the Linux host, the remote host sends a garden variety keepalive. When the remote host includes tcp timestamps in the keepalives, they are ignored by the Linux host, and the remote host resets the connection after 10 unACKed keepalives. When timestamps are absent from the keepalives, the Linux host ACKs each one, and all is copacetic.

Text output of a tcpdump trace of a connection that fails:
http://pastebin.com/v6CpteJ9

Text output of a tcpdump trace of a connection that succeeds:
http://pastebin.com/KVLb3Mzh

More details, in case you think they are relevant:

My application creates a JDBC connection to a remote MS SQL Server and executes a statement that does not return a result set, and so it doesn't need to pass application data back and forth while it executes. The statement takes 2 or 3 minutes to complete. I connect to two different remote hosts: a Win2003 machine, and a Win2008R2 machine. The Win2003 machine doesn't put timestamps in its keep-alives, so the application completes successfully when connecting to that host. If tcp timestamps are enabled on the Linux host, the Win2008 host includes them in its keepalives, and they are unACKed, so the connection is reset; if they are disabled on the Linux host, the Win2008 host doesn't include them in the keepalives, and the application completes successfully. I use (as you might expect) sysctl to disable tcp timestamps on the Linux hosts.

Quoted 3 lines

I have dumps for all permutations of CentOS/Ubuntu, Win200[38], and +/- timestamps on the Linux side, and I will post them if the developers think that they would be useful.

Hi Charles

I could not reproduce the problem here, even using a quite old kernel as receiver (2.6.9)

15:54:33.566192 IP 192.168.20.108.55926> 192.168.20.124.777: SWE 479814493:479814493(0) win 14600<mss 1460,sackOK,timestamp 151666 0,nop,wscale 7>
15:54:33.566265 IP 192.168.20.124.777> 192.168.20.108.55926: S 3714869381:3714869381(0) ack 479814494 win 5792<mss 1460,sackOK,timestamp 54553041 151666,nop,wscale 2> 15:54:33.566274 IP 192.168.20.108.55926> 192.168.20.124.777: . ack 1 win 115<nop,nop,timestamp 151666 54553041>
15:54:33.566281 IP 192.168.20.108.55926> 192.168.20.124.777: P 1:5(4) ack 1 win 115<nop,nop,timestamp 151666 54553041> 15:54:33.566351 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5 win 1448<nop,nop,timestamp 54553041 151666> 15:54:33.566375 IP 192.168.20.124.777> 192.168.20.108.55926: P 1:5(4) ack 5 win 1448<nop,nop,timestamp 54553041 151666> 15:54:33.566380 IP 192.168.20.108.55926> 192.168.20.124.777: . ack 5 win 115<nop,nop,timestamp 151666 54553041>
15:54:43.577945 IP 192.168.20.108.55926> 192.168.20.124.777: . 4:5(1) ack 5 win 115<nop,nop,timestamp 152668 54553041> 15:54:43.578012 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5 win 1448<nop,nop,timestamp 54563053 152668,nop,nop,sack sack 1 {4:5}> 15:54:53.597946 IP 192.168.20.108.55926> 192.168.20.124.777: . 4:5(1) ack 5 win 115<nop,nop,timestamp 153670 54563053> 15:54:53.598012 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5 win 1448<nop,nop,timestamp 54573073 153670,nop,nop,sack sack 1 {4:5}>

Quoted 4 lines
Quoted 2 lines

Are you sure frame tcp checksums are OK when the 'faulty' linux receive them ? (tcpdump -v)

I will check when I get into the office and let you know.

You are correct: the checksums in the keepalives are broken, though they are correct in the other segments from the Win2008 server. I have updated the pastes linked to above with 'tcpdump -v' output. I apologize for missing that problem the first time around. Chuck

#4 lore raw
Quoted 4 lines · Charles Bearden

You are correct: the checksums in the keepalives are broken, though they are correct in the other segments from the Win2008 server. I have updated the pastes linked to above with 'tcpdump -v' output. I apologize for missing that problem the first time around.

Hmm, maybe its OK : If checksums are offloaded to NIC, tcpdump 'lies' telling checksum is not OK, since tcpdump get a copy of the packet before being handled by the NIC.

You should take a tcpdump on the receiving machine (the machine that apparently doesnt react to the keepalive probes)

#5 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 4 lines · Charles Bearden

You are correct: the checksums in the keepalives are broken, though they are correct in the other segments from the Win2008 server. I have updated the pastes linked to above with 'tcpdump -v' output. I apologize for missing that problem the first time around.

Quoted 6 lines

Hmm, maybe its OK : If checksums are offloaded to NIC, tcpdump 'lies' telling checksum is not OK, since tcpdump get a copy of the packet before being handled by the NIC.

You should take a tcpdump on the receiving machine (the machine that apparently doesnt react to the keepalive probes)

I should have made this clear before: these dumps were captured on the machine that is ignoring the keepalives ("Ubuntu.host" in the dumps) from the other host ("Win200[38].host" in the dumps). If I understand tcp checksum offloading correctly, it wouldn't play a role here.

#6 lore raw
Quoted 12 lines · Charles Bearden
Quoted 11 lines · Eric Dumazet
Quoted 4 lines · Charles Bearden

You are correct: the checksums in the keepalives are broken, though they are correct in the other segments from the Win2008 server. I have updated the pastes linked to above with 'tcpdump -v' output. I apologize for missing that problem the first time around.

Hmm, maybe its OK : If checksums are offloaded to NIC, tcpdump 'lies' telling checksum is not OK, since tcpdump get a copy of the packet before being handled by the NIC.

Quoted 8 lines
Quoted 2 lines

You should take a tcpdump on the receiving machine (the machine that apparently doesnt react to the keepalive probes)

I should have made this clear before: these dumps were captured on the machine that is ignoring the keepalives ("Ubuntu.host" in the dumps) from the other host ("Win200[38].host" in the dumps). If I understand tcp checksum offloading correctly, it wouldn't play a role here.

OK, thanks.

I am curious, could you check the payload byte included in the keepalive probes has the value of last sent byte on the session ?

(linux send keepalive probes with no data, so implementing this 'feature' would need to keep track of this value in tcp socket)

#7 lore raw
Quoted 22 lines · Eric Dumazet
Quoted 12 lines · Charles Bearden
Quoted 11 lines · Eric Dumazet
Quoted 4 lines · Charles Bearden

You are correct: the checksums in the keepalives are broken, though they are correct in the other segments from the Win2008 server. I have updated the pastes linked to above with 'tcpdump -v' output. I apologize for missing that problem the first time around.

Hmm, maybe its OK : If checksums are offloaded to NIC, tcpdump 'lies' telling checksum is not OK, since tcpdump get a copy of the packet before being handled by the NIC.

Quoted 8 lines
Quoted 2 lines

You should take a tcpdump on the receiving machine (the machine that apparently doesnt react to the keepalive probes)

I should have made this clear before: these dumps were captured on the machine that is ignoring the keepalives ("Ubuntu.host" in the dumps) from the other host ("Win200[38].host" in the dumps). If I understand tcp checksum offloading correctly, it wouldn't play a role here.

Quoted 7 lines

OK, thanks.

I am curious, could you check the payload byte included in the keepalive probes has the value of last sent byte on the session ?

(linux send keepalive probes with no data, so implementing this 'feature' would need to keep track of this value in tcp socket)

Each keepalive from the Win2008 machine has a 1-byte payload 0x00. The last byte of the last packet with a payload before that from the Win2008 host (at 14:40:18.166394 in the paste) is also 0x00. Is that what you were asking about?

#8 lore raw
Quoted 4 lines · Charles Bearden

Each keepalive from the Win2008 machine has a 1-byte payload 0x00. The last byte of the last packet with a payload before that from the Win2008 host (at 14:40:18.166394 in the paste) is also 0x00. Is that what you were asking about?

Yes, thats fine, thanks :)

#9 lore raw
Quoted 8 lines · Eric Dumazet
Quoted 4 lines · Charles Bearden

Each keepalive from the Win2008 machine has a 1-byte payload 0x00. The last byte of the last packet with a payload before that from the Win2008 host (at 14:40:18.166394 in the paste) is also 0x00. Is that what you were asking about?

Yes, thats fine, thanks :)

One other thing: when tcp timestamps are disabled on the Linux (receiving) end, so that the Win2008 host sends keepalives without timestamps, the checksums in the keepalives are correct. It's only when the keepalives also contain timestamps that the checksums are broken.

If you think this might be a Linux kernel issue, I'll be glad to keep working with you, but I don't want to spam this list if my problem isn't relevant. Thank you for your help in any case.

Chuck

#10 lore raw
Quoted 10 lines · Charles Bearden
Quoted 8 lines · Eric Dumazet
Quoted 4 lines · Charles Bearden

Each keepalive from the Win2008 machine has a 1-byte payload 0x00. The last byte of the last packet with a payload before that from the Win2008 host (at 14:40:18.166394 in the paste) is also 0x00. Is that what you were asking about?

Yes, thats fine, thanks :)

Quoted 9 lines

One other thing: when tcp timestamps are disabled on the Linux (receiving) end, so that the Win2008 host sends keepalives without timestamps, the checksums in the keepalives are correct. It's only when the keepalives also contain timestamps that the checksums are broken.

If you think this might be a Linux kernel issue, I'll be glad to keep working with you, but I don't want to spam this list if my problem isn't relevant. Thank you for your help in any case.

Well, if your linux machine receives frames with a bad checksum, linux is working well : These frames must be dropped.

It is a bug in the sender (Win2008 in this case).

I am not that surprised, given timestamps were disabled in microsoft products until very last releases (Windows Seven, and apparently winserver 2008)