Skip to contentSkip to conversation

TCP stack behaviour question

DISCUSSION Stuart MacDonald 3 participants 12 messages Sep 15, 2006 – Sep 20, 2006

lore
Share & tools
lore
#0 lore raw

I'm having some trouble with a network application I've written. I've done a lot of research the last few days; man 7 ip, man 7 tcp, kernel 2.4.31 source code, Stevens' Illustrated TCP/IP Vol 1 & 3 (for some reason we don't have Vol 2), Usenet, websites. I'm hoping someone here can help me out, or point me in the correct direction.

Distro: Debian 3.0 r2
Kernel: Stock 2.4.24

tcp_retries1 == 3
tcp_retries2 == 15

I have an application that setups up a TCP connection to a server. If the server has a power failure, TCP starts retransmitting the packet that wasn't ACKed. I see the exponential backoff.

Question 1: There's the original packet, plus 7 retransmitted packets for a total of 8, then TCP gives up. How is 7 (or 8) derived from the tcp_retries[12] settings?

Question 1a: The time between last and second-last retransmit packets is only about 27 seconds. I've read there's a maximum time, but also that it's usually 100 or 120 seconds. Where can I find that setting in /proc?

Question 1b: If RTO is that high, why is retransmit stopping?

Question 2: After the retransmit has given up, the app is still making an occasional write(), which succeeds! However, tearing down and attemting a new connection results in an immediate EHOSTUNREACH error. Why is the write() succeeding?

Question 2a: How can my app find out the EHOSTUNREACH error immediately? IP_RECVERR is not implemented on TCP, and SO_ERROR always reports no error (0).

..Stu

#1 lore raw
Quoted 5 lines · Stuart MacDonald

I'm having some trouble with a network application I've written. I've done a lot of research the last few days; man 7 ip, man 7 tcp, kernel 2.4.31 source code, Stevens' Illustrated TCP/IP Vol 1 & 3 (for some reason we don't have Vol 2), Usenet, websites. I'm hoping someone here can help me out, or point me in the correct direction.

Stevens is a good reference to BSD networking, but not necessarily to Linux.

Quoted 3 lines

Question 1: There's the original packet, plus 7 retransmitted packets for a total of 8, then TCP gives up. How is 7 (or 8) derived from the tcp_retries[12] settings?

Documented in tcp(7)

Quoted 4 lines

Question 1a: The time between last and second-last retransmit packets is only about 27 seconds. I've read there's a maximum time, but also that it's usually 100 or 120 seconds. Where can I find that setting in /proc?

It's fixed by the TCP specification.

Quoted 4 lines

Question 2: After the retransmit has given up, the app is still making an occasional write(), which succeeds! However, tearing down and attemting a new connection results in an immediate EHOSTUNREACH error. Why is the write() succeeding?

It goes all in the local buffer, until it starts blocking (or EAGAIN for non blocking sockets)

Quoted 4 lines

Question 2a: How can my app find out the EHOSTUNREACH error immediately? IP_RECVERR is not implemented on TCP, and SO_ERROR always reports no error (0).

Did you really read the manpages? It is implemented and it's documented.

-Andi (your new temporary manpage proxy)

#2 lore raw

From: On Behalf Of Andi Kleen

Thanks for replying, I appreciate it.

Quoted 2 lines

Stevens is a good reference to BSD networking, but not necessarily to Linux.

Which is why I posted here. I was wondering about the specific implementation.

What happened was this: I had a run where I captured output with tcpdump. My original post was based on that, and the results of the debug output from my app. For whatever reason, it appears the stack didn't generate all of the packets it should have. When the log showed a second-last to last retransmit time of about 27 seconds, and then a gap of about 400 to the very next packet of any kind, I assumed that meant the stack had given up on the retransmits when it appears something else was going on.

I did some digging into the kernel and on the next run found that all the expected retransmit packets were being generated, and that once the stack gives up on the retransmits then system calls return errors.

Quoted 7 lines
Quoted 2 lines

Question 2a: How can my app find out the EHOSTUNREACH error immediately? IP_RECVERR is not implemented on TCP, and

SO_ERROR always

Quoted 1 line

reports no error (0).

Did you really read the manpages? It is implemented and it's documented.

Yes I did and no it's not, according to the man page. I quoth:

# man 7 ip
..
               Note that TCP has no error queue; MSG_ERRQUEUE is
illegal on SOCK_STREAM sockets.  Thus all errors are returned by
socket function return or SO_ERROR only.

Maybe the man page is wrong? That's from my FC 3 install.

..Stu

#3 lore raw
Quoted 8 lines
# man 7 ip
..
               Note that TCP has no error queue; MSG_ERRQUEUE is
illegal on SOCK_STREAM sockets.  Thus all errors are returned by
socket function return or SO_ERROR only.

Maybe the man page is wrong? That's from my FC 3 install.

The sentence is correct, but TCP has a IP_RECVERR that works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)

-Andi

#4 lore raw

From: Andi Kleen [mailto:[email protected]]

Quoted 6 lines
Quoted 6 lines
# man 7 ip
..
               Note that TCP has no error queue; MSG_ERRQUEUE is
illegal on SOCK_STREAM sockets.  Thus all errors are returned by
socket function return or SO_ERROR only.
Quoted 6 lines
Quoted 1 line

Maybe the man page is wrong? That's from my FC 3 install.

The sentence is correct, but TCP has a IP_RECVERR that works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)

I read that too, but didn't know which one was correct, so I erred on the side of caution and believed ip(7).

Good to know, I'll look into that. Thanks.

..Stu

#5 lore raw
Quoted 9 lines · Stuart MacDonald

From: Andi Kleen [mailto:[email protected]]

Quoted 8 lines
Quoted 7 lines
# man 7 ip
..
               Note that TCP has no error queue; MSG_ERRQUEUE is
illegal on SOCK_STREAM sockets.  Thus all errors are returned by
socket function return or SO_ERROR only.

Maybe the man page is wrong? That's from my FC 3 install.

Quoted 7 lines
Quoted 4 lines

The sentence is correct, but TCP has a IP_RECVERR that works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)

I read that too, but didn't know which one was correct, so I erred on the side of caution and believed ip(7).

Ok maybe it's a bit misleading. Michael, you might want to clarify.

-Andi

#6 lore raw

Von: Andi Kleen <[email protected]>

Quoted 15 lines · Stuart MacDonald
Quoted 14 lines

From: Andi Kleen [mailto:[email protected]]

Quoted 12 lines
Quoted 7 lines
# man 7 ip
..
               Note that TCP has no error queue; MSG_ERRQUEUE is
illegal on SOCK_STREAM sockets.  Thus all errors are returned by
socket function return or SO_ERROR only.

Maybe the man page is wrong? That's from my FC 3 install.

The sentence is correct, but TCP has a IP_RECVERR that works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)

Quoted 4 lines
Quoted 2 lines

I read that too, but didn't know which one was correct, so I erred on the side of caution and believed ip(7).

Ok maybe it's a bit misleading. Michael, you might want to clarify.

Can some one of you propose a better text please?

Cheers,

Michael

Signature · 8 lines
Michael Kerrisk
maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 

Want to help with man page maintenance?  
Grab the latest tarball at
ftp://ftp.win.tue.nl/pub/linux-local/manpages/, 
read the HOWTOHELP file and grep the source 
files for 'FIXME'.
#7 lore raw

From: Michael Kerrisk [mailto:[email protected]]

Quoted 19 lines

Von: Andi Kleen <[email protected]>

Quoted 5 lines · Stuart MacDonald
Quoted 4 lines

From: Andi Kleen [mailto:[email protected]]

Quoted 3 lines
Quoted 3 lines
# man 7 ip
..
               Note that TCP has no error queue; 

MSG_ERRQUEUE is

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

illegal on SOCK_STREAM sockets. Thus all errors are

returned by

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

socket function return or SO_ERROR only.

Maybe the man page is wrong? That's from my FC 3 install.

The sentence is correct, but TCP has a IP_RECVERR that works differently without a queue. Basically it doesn't delay

the error

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)

Quoted 7 lines
Quoted 1 line
Quoted 1 line

I read that too, but didn't know which one was correct,

so I erred on

Quoted 3 lines
Quoted 1 line

the side of caution and believed ip(7).

Ok maybe it's a bit misleading. Michael, you might want to clarify.

Can some one of you propose a better text please?

Perhaps

Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. IP_RECVERR is valid for TCP, but all errors are returned by socket function return or SO_ERROR only.

?

..Stu

#8 lore raw

Von: "Stuart MacDonald" <[email protected]>

Quoted 11 lines
Quoted 3 lines
Quoted 1 line

Ok maybe it's a bit misleading. Michael, you might want to clarify.

Can some one of you propose a better text please?

Perhaps

Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. IP_RECVERR is valid for TCP, but all errors are returned by socket function return or SO_ERROR only.

?

Sound okay to you Andi?

Cheers,

Michael

Signature · 8 lines
Michael Kerrisk
maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 

Want to help with man page maintenance?  
Grab the latest tarball at
ftp://ftp.win.tue.nl/pub/linux-local/manpages/, 
read the HOWTOHELP file and grep the source 
files for 'FIXME'.
#9 lore raw
Quoted 8 lines · Michael Kerrisk

Von: "Stuart MacDonald" <[email protected]>

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Ok maybe it's a bit misleading. Michael, you might want to clarify.

Can some one of you propose a better text please?

Perhaps

Quoted 7 lines
Quoted 5 lines

Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. IP_RECVERR is valid for TCP, but all errors are returned by socket function return or SO_ERROR only.

?

Sound okay to you Andi?

Yes.
-Andi

#10 lore raw
In reply to Stuart MacDonald
Quoted 5 lines

Perhaps

Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. IP_RECVERR is valid for TCP, but all errors are returned by socket function return or SO_ERROR only.

Stuart -- thanks, added for man-pages-2.41.

Interestingly, at this point in the man pages source there is the following commented out text:

.\" FIXME . Is it a good idea to document that? It is a dubious feature. .\" On
.\" .B SOCK_STREAM
.\" sockets,
.\" .I IP_RECVERR
.\" has slightly different semantics. Instead of
.\" saving the errors for the next timeout, it passes all incoming .\" errors immediately to the user.
.\" This might be useful for very short-lived TCP connections which .\" need fast error handling. Use this option with care:
.\" it makes TCP unreliable
.\" by not allowing it to recover properly from routing
.\" shifts and other normal
.\" conditions and breaks the protocol specification.

Cheers,

Michael

Signature · 8 lines
Michael Kerrisk
maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 

Want to help with man page maintenance?  
Grab the latest tarball at
ftp://ftp.win.tue.nl/pub/linux-local/manpages/, 
read the HOWTOHELP file and grep the source 
files for 'FIXME'.
#11 lore raw
Quoted 3 lines

Interestingly, at this point in the man pages source there is the following commented out text:

Yes that was me. On second thought I suppose I was right back then that this feature is too dubious to be documented. So better keep it undocumented and drop the change.

-Andi

Quoted 19 lines

.\" FIXME . Is it a good idea to document that? It is a dubious feature. .\" On
.\" .B SOCK_STREAM
.\" sockets,
.\" .I IP_RECVERR
.\" has slightly different semantics. Instead of
.\" saving the errors for the next timeout, it passes all incoming .\" errors immediately to the user.
.\" This might be useful for very short-lived TCP connections which .\" need fast error handling. Use this option with care:
.\" it makes TCP unreliable
.\" by not allowing it to recover properly from routing
.\" shifts and other normal
.\" conditions and breaks the protocol specification.

Cheers,

Michael