TCP stack behaviour question
I'm having some trouble with a network application I've written. I've done a lot of research the last few days; man 7 ip, man 7 tcp, kernel 2.4.31 source code, Stevens' Illustrated TCP/IP Vol 1 & 3 (for some reason we don't have Vol 2), Usenet, websites. I'm hoping someone here can help me out, or point me in the correct direction.
Distro: Debian 3.0 r2
Kernel: Stock 2.4.24
tcp_retries1 == 3tcp_retries2 == 15
I have an application that setups up a TCP connection to a server. If the server has a power failure, TCP starts retransmitting the packet that wasn't ACKed. I see the exponential backoff.
Question 1: There's the original packet, plus 7 retransmitted packets for a total of 8, then TCP gives up. How is 7 (or 8) derived from the tcp_retries[12] settings?
Question 1a: The time between last and second-last retransmit packets is only about 27 seconds. I've read there's a maximum time, but also that it's usually 100 or 120 seconds. Where can I find that setting in /proc?
Question 1b: If RTO is that high, why is retransmit stopping?
Question 2: After the retransmit has given up, the app is still making an occasional write(), which succeeds! However, tearing down and attemting a new connection results in an immediate EHOSTUNREACH error. Why is the write() succeeding?
Question 2a: How can my app find out the EHOSTUNREACH error immediately? IP_RECVERR is not implemented on TCP, and SO_ERROR always reports no error (0).
..Stu
Quoted 5 lines · Stuart MacDonald
I'm having some trouble with a network application I've written. I've done a lot of research the last few days; man 7 ip, man 7 tcp, kernel 2.4.31 source code, Stevens' Illustrated TCP/IP Vol 1 & 3 (for some reason we don't have Vol 2), Usenet, websites. I'm hoping someone here can help me out, or point me in the correct direction.
Stevens is a good reference to BSD networking, but not necessarily to Linux.
Quoted 3 lines
Question 1: There's the original packet, plus 7 retransmitted packets for a total of 8, then TCP gives up. How is 7 (or 8) derived from the tcp_retries[12] settings?
Documented in tcp(7)
Quoted 4 lines
Question 1a: The time between last and second-last retransmit packets is only about 27 seconds. I've read there's a maximum time, but also that it's usually 100 or 120 seconds. Where can I find that setting in /proc?
It's fixed by the TCP specification.
Quoted 4 lines
Question 2: After the retransmit has given up, the app is still making an occasional
write(), which succeeds! However, tearing down and attemting a new connection results in an immediate EHOSTUNREACH error. Why is thewrite()succeeding?
It goes all in the local buffer, until it starts blocking (or EAGAIN for non blocking sockets)
Quoted 4 lines
Question 2a: How can my app find out the EHOSTUNREACH error immediately?
IP_RECVERRis not implemented on TCP, andSO_ERRORalways reports no error (0).
Did you really read the manpages? It is implemented and it's documented.
-Andi (your new temporary manpage proxy)
From: On Behalf Of Andi Kleen
Thanks for replying, I appreciate it.
Quoted 2 lines
Stevens is a good reference to BSD networking, but not necessarily to Linux.
Which is why I posted here. I was wondering about the specific implementation.
What happened was this: I had a run where I captured output with tcpdump. My original post was based on that, and the results of the debug output from my app. For whatever reason, it appears the stack didn't generate all of the packets it should have. When the log showed a second-last to last retransmit time of about 27 seconds, and then a gap of about 400 to the very next packet of any kind, I assumed that meant the stack had given up on the retransmits when it appears something else was going on.
I did some digging into the kernel and on the next run found that all the expected retransmit packets were being generated, and that once the stack gives up on the retransmits then system calls return errors.
Quoted 7 lines
Quoted 2 lines
Question 2a: How can my app find out the EHOSTUNREACH error immediately?
IP_RECVERRis not implemented on TCP, and
SO_ERRORalwaysQuoted 1 line
reports no error (0).
Did you really read the manpages? It is implemented and it's documented.
Yes I did and no it's not, according to the man page. I quoth:
# man 7 ip
..
Note that TCP has no error queue; MSG_ERRQUEUE is
illegal on SOCK_STREAM sockets. Thus all errors are returned by
socket function return or SO_ERROR only.Maybe the man page is wrong? That's from my FC 3 install.
..Stu
Quoted 8 lines
# man 7 ip .. Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. Thus all errors are returned by socket function return or SO_ERROR only.Maybe the man page is wrong? That's from my FC 3 install.
The sentence is correct, but TCP has a IP_RECVERR that works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)
-Andi
From: Andi Kleen [mailto:[email protected]]
Quoted 6 lines
Quoted 6 lines
# man 7 ip .. Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. Thus all errors are returned by socket function return or SO_ERROR only.
Quoted 6 lines
Quoted 1 line
Maybe the man page is wrong? That's from my FC 3 install.
The sentence is correct, but TCP has a
IP_RECVERRthat works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)
I read that too, but didn't know which one was correct, so I erred on the side of caution and believed ip(7).
Good to know, I'll look into that. Thanks.
..Stu
Quoted 9 lines · Stuart MacDonald
From: Andi Kleen [mailto:[email protected]]
Quoted 8 lines
Quoted 7 lines
# man 7 ip .. Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. Thus all errors are returned by socket function return or SO_ERROR only.Maybe the man page is wrong? That's from my FC 3 install.
Quoted 7 lines
Quoted 4 lines
The sentence is correct, but TCP has a
IP_RECVERRthat works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)I read that too, but didn't know which one was correct, so I erred on the side of caution and believed ip(7).
Ok maybe it's a bit misleading. Michael, you might want to clarify.
-Andi
Von: Andi Kleen <[email protected]>
Quoted 15 lines · Stuart MacDonald
Quoted 14 lines
From: Andi Kleen [mailto:[email protected]]
Quoted 12 lines
Quoted 7 lines
# man 7 ip .. Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. Thus all errors are returned by socket function return or SO_ERROR only.Maybe the man page is wrong? That's from my FC 3 install.
The sentence is correct, but TCP has a
IP_RECVERRthat works differently without a queue. Basically it doesn't delay the error reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)
Quoted 4 lines
Quoted 2 lines
I read that too, but didn't know which one was correct, so I erred on the side of caution and believed ip(7).
Ok maybe it's a bit misleading. Michael, you might want to clarify.
Can some one of you propose a better text please?
Cheers,
Michael
Signature · 8 lines
Michael Kerrisk maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 Want to help with man page maintenance? Grab the latest tarball at ftp://ftp.win.tue.nl/pub/linux-local/manpages/, read the HOWTOHELP file and grep the source files for 'FIXME'.
From: Michael Kerrisk [mailto:[email protected]]
Quoted 19 lines
Von: Andi Kleen <[email protected]>
Quoted 5 lines · Stuart MacDonald
Quoted 4 lines
From: Andi Kleen [mailto:[email protected]]
Quoted 3 lines
Quoted 3 lines
# man 7 ip .. Note that TCP has no error queue;
MSG_ERRQUEUEisQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
illegal on
SOCK_STREAMsockets. Thus all errors arereturned by
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
socket function return or
SO_ERRORonly.Maybe the man page is wrong? That's from my FC 3 install.
The sentence is correct, but TCP has a
IP_RECVERRthat works differently without a queue. Basically it doesn't delaythe error
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
reporting for incoming ICMPs to the last retransmit, but reports them immediately. This is documented in tcp(7)
Quoted 7 lines
Quoted 1 line
Quoted 1 line
I read that too, but didn't know which one was correct,
so I erred on
Quoted 3 lines
Quoted 1 line
the side of caution and believed ip(7).
Ok maybe it's a bit misleading. Michael, you might want to clarify.
Can some one of you propose a better text please?
Perhaps
Note that TCP has no error queue; MSG_ERRQUEUE is illegal on SOCK_STREAM sockets. IP_RECVERR is valid for TCP, but all errors are returned by socket function return or SO_ERROR only.
?
..Stu
Von: "Stuart MacDonald" <[email protected]>
Quoted 11 lines
Quoted 3 lines
Quoted 1 line
Ok maybe it's a bit misleading. Michael, you might want to clarify.
Can some one of you propose a better text please?
Perhaps
Note that TCP has no error queue;
MSG_ERRQUEUEis illegal onSOCK_STREAMsockets.IP_RECVERRis valid for TCP, but all errors are returned by socket function return orSO_ERRORonly.?
Sound okay to you Andi?
Cheers,
Michael
Signature · 8 lines
Michael Kerrisk maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 Want to help with man page maintenance? Grab the latest tarball at ftp://ftp.win.tue.nl/pub/linux-local/manpages/, read the HOWTOHELP file and grep the source files for 'FIXME'.
Quoted 8 lines · Michael Kerrisk
Von: "Stuart MacDonald" <[email protected]>
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Ok maybe it's a bit misleading. Michael, you might want to clarify.
Can some one of you propose a better text please?
Perhaps
Quoted 7 lines
Quoted 5 lines
Note that TCP has no error queue;
MSG_ERRQUEUEis illegal onSOCK_STREAMsockets.IP_RECVERRis valid for TCP, but all errors are returned by socket function return orSO_ERRORonly.?
Sound okay to you Andi?
Yes.
-Andi
Quoted 5 lines
Perhaps
Note that TCP has no error queue;
MSG_ERRQUEUEis illegal onSOCK_STREAMsockets.IP_RECVERRis valid for TCP, but all errors are returned by socket function return orSO_ERRORonly.
Stuart -- thanks, added for man-pages-2.41.
Interestingly, at this point in the man pages source there is the following commented out text:
.\" FIXME . Is it a good idea to document that? It is a dubious feature. .\" On
.\" .B SOCK_STREAM
.\" sockets,
.\" .I IP_RECVERR
.\" has slightly different semantics. Instead of
.\" saving the errors for the next timeout, it passes all incoming .\" errors immediately to the user.
.\" This might be useful for very short-lived TCP connections which .\" need fast error handling. Use this option with care:
.\" it makes TCP unreliable
.\" by not allowing it to recover properly from routing
.\" shifts and other normal
.\" conditions and breaks the protocol specification.
Cheers,
Michael
Signature · 8 lines
Michael Kerrisk maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 Want to help with man page maintenance? Grab the latest tarball at ftp://ftp.win.tue.nl/pub/linux-local/manpages/, read the HOWTOHELP file and grep the source files for 'FIXME'.
Quoted 3 lines
Interestingly, at this point in the man pages source there is the following commented out text:
Yes that was me. On second thought I suppose I was right back then that this feature is too dubious to be documented. So better keep it undocumented and drop the change.
-Andi
Quoted 19 lines
.\" FIXME . Is it a good idea to document that? It is a dubious feature. .\" On
.\" .BSOCK_STREAM
.\" sockets,
.\" .IIP_RECVERR
.\" has slightly different semantics. Instead of
.\" saving the errors for the next timeout, it passes all incoming .\" errors immediately to the user.
.\" This might be useful for very short-lived TCP connections which .\" need fast error handling. Use this option with care:
.\" it makes TCP unreliable
.\" by not allowing it to recover properly from routing
.\" shifts and other normal
.\" conditions and breaks the protocol specification.Cheers,
Michael