Bug? TCP shutdown behaviour when deleting local IP addresses

Hi all,

I sent this to the list yesterday (from another address) but didn't get any responses. Accordingly I'm expanding the receiver list to the listed maintainers for IPv4/IPv6.

I'm seeing some unexpected (to me, at least) behaviour with local TCP connections. The scenario goes as follows:

1) create new IP address and assign to eth device
2) TCP server starts listening on that IP address
3) TCP client connects to server
4) remove new IP address
5) kill server with ctrl-C. At this point it appears that because the address was removed the shutdown message isn't processed properly. netstat shows the server socket as FIN_WAIT1, but the client socket is still ESTABLISHED.
6) client writes to the connected socket (this passes with no error)
7) client waits for response from server, and waits forever or until keepalive expires

A few points:

This was originally seen on 2.6.27, but I've verified it on 2.6.35. I'll see about trying it on current git. I've got really simple client/server code if anyone wants to try reproducing.

If we don't remove the address in step 4, then step 5 results in the server socket going to FIN_WAIT2 and the client socket going to CLOSE_WAIT and step 7 returns right away with zero bytes.

It seems like the waiting forever behaviour in step 7 might be legitimate since the address was removed before shutting down the server, but it also seems like we should be able to do better given that everything is local. In the "remove IP address" case maybe step 6 should cause some sort of error since the IP address no longer exists?

Incidentally, if we do this sort of scenario with the client and server on different hosts then we get a "no route to host" error at step 6.

Curious how this is supposed to work...

Chris

Chris Friesen
Software Designer
GENBAND
www.genband.com

From: Chris Friesen <[email protected]>
Date: Wed, 17 Oct 2012 17:01:40 -0600

It can take more than 24 hours to get a response from people who are volunteers, and who will reply to your report because they want to rather than because they are obligated.

Posting it again just irritates such people, and will place your report at a much lower priority, just FYI.

Thanks for the pointers. I was worried that it had gone unnoticed, but I guess the level of traffic on netdev is small enough (compared to something like lkml) that that's generally not an issue.

Chris

I see no real problem here.

Its like you cut the cable somewhere in the path.

Only timeouts will apply.

And its not keeepalive timeouts in 7) but normal retransmits with exponential backoff.

Extract of Documentation/networking/ip-sysctl.txt :

tcp_retries1 - INTEGER
This value influences the time, after which TCP decides, that something is wrong due to unacknowledged RTO retransmissions, and reports this suspicion to the network layer. See tcp_retries2 for more details.

RFC 1122 recommends at least 3 retransmissions, which is the default.

tcp_retries2 - INTEGER
This value influences the timeout of an alive TCP connection, when RTO retransmissions remain unacknowledged. Given a value of N, a hypothetical TCP connection following exponential backoff with an initial RTO of TCP_RTO_MIN would retransmit N times before killing the connection at the (N+1)th
RTO.

The default value of 15 yields a hypothetical timeout of 924.6 seconds and is a lower bound for the effective timeout. TCP will effectively time out at the first RTO which exceeds the hypothetical timeout.

RFC 1122 recommends at least 100 seconds for the timeout, which corresponds to a value of at least 8.

While I agree generally, it's a bit unfortunate that we can't (as a quality of implementation thing) give an earlier notice of failure since the kernel knows about both ends of the connection even though the IP address is gone. On the other hand, I imagine that would mean special-casing things and presumably that would open a whole can of worms.

Thanks for the reply,

Chris

Really what is the difference between a cable cut and what you are doing ?

Some frames are lost (Dropped), and sender doesnt 'know' that is definitive or temporary failure.

If you want faster response, you need to send RST messages, not dropping frames.

So change your strategy, and add an iptables rule for example ?

After thinking about it for a while, I think you're right. Initially I was expecting that since we know the server side has been taken down we should be able to kill the client side, but then I considered the case where some sort of of high availability system may have moved the server to another host.

That's a good suggestion. I'll pass it on.

Thanks,
Chris

I had a patch for this a long time ago to speed up recovery. It was very useful with dial on demand ISDN links or DSL connections that hang up occasionally to force a new IP.

It was in SUSE kernels for a long time, but never made it into mainline

http://www.firstfloor.org/pub/ak/quilt-before-arch-merge/patches/iff-dynamic

-Andi

[email protected] -- Speaking for myself only

I'm a network engineer, as in I work primarily with IP routing. Ever since I started running Linux back in the mid 90ties I've had a love/hate relationship with how Linux handles disappearing network connectivity or IP addresses.

In my mind there are two ways to handle outage:

1. When a network connection (physical interface) goes down, keep everything as it is, it might come back up again shortly and then we can continue as if basically nothing happened. TCP was designed for this considering timeouts can be in hours.

2. When a network connection (physical interface) goes down, wait a few seconds, give up, reset all connectivity related to that connection, basically give up.

Now to my question for the netdev people:

Is there functionality in the kernel for a connection manager to easily accomplish 2, in that when it tries to deconfigure the IP address on the interface to also kill all TCP connections terminated at that IP? On my laptop, I regularily have to kill my ssh client after suspend/resume cycle, because it's been down for quite a while, and the ssh client doesn't know the TCP connection is now not functional anymore (TCP session is still up and retransmit won't happen for a while, so the TCP RST from the server (I use keepalives within SSH) isn't seen for a long time).

Without knowing what's in place right now, I see some behaviours that I'd like to have:

After resume (or otherwise network connectivity re-established), connection manager should be able to tell the kernel to:

a) kill all TCP/UDP/other sessions existing which doesn't currently have an active IP address on the machine. This is for the sake of local clients.
b) the TCP/SCTP sessions that *do* have an IP, should have their retransmit timers "reset", so that whatever needs to be sent, should be sent immediately (or shortly, within a few seconds).
c) tell the kernel to kill all TCP sessions bound to a certain IP, because the connection manager is going to remove it shortly. Send TCP RSTs or whatever and close the TCP session, so both ends know that network connectivity is going down.

This would mean that if I resume my laptop and it establishes network connectivity, I will then *know* within a few seconds what TCP sessions are still valid (the ones that aren't will be killed either because they're bound to an IP that is not available, or a TCP packet is sent out to which there will be a TCP RST reply from the other side if the TCP session is closed at that end).

All of this also has implications on IPv6 and autoconfiguration, I don't know if currently we are closing TCP sessions bound to IPv6 addresses that are going away due to the RA the addresses were autoconfigured based on, now doesn't have a valid lifetime anymore and the kernel is going to remove them. It also applies to both DHCPv6 and DHCPv4 when a lease is expiring and the IPv4/IPv6 address is going to be removed.

Right now I think the situation with a lot of "hanging" TCP sessions after connectivity is restored is suboptimal and negatively impacts user experience. Probably there should be knobs to turn for different user needs (server or desktop/laptop) because desired behaviour is different in different use cases.

Mikael Abrahamsson    email: [email protected]

Doing that is almost pointless and gives people false expectations.

Consider the simple network host-hub-hub-host.
If you disconnect the link between the two hubs then only timeouts can cause disconnects.
Such a remote fault is much more likely than the local one.

If the kernel takes down connections when a local interface goes down (as windows does) then people doing testing fail to test the correct scenario.

You also don't want a power glitch in the wiring cupboard to disconnect all your connections.

In the past I've also moved IP addresses between physical interfaces - modern HA systems might do the same.

David

My doing suspect/resume on my laptop causes more network interruptions than any link going down in the network outside my home.

I never asked for the kernel to take down connections, I asked for the connection manager to be able to do it.

In a server environment, most likely you're right. In some other environment, the opposite probably applies.

Yes, in your described scenario you're right, in my scenario (laptop suspend/resume) I don't agree with your view.

Mikael Abrahamsson    email: [email protected]

You do realize kernel has no idea that the loss of IP address is temporary or not ?

Some links can be slow to setup after a resume.

So they are going to force a close, even if the link becomes alive after 15 seconds. Too bad for some wireless setups, or tunnels.

Yes, why not. Why is it specific to linux I have no idea.

Indeed, that's why I wrote "connection manager tell the kernel to..", and didn't write "the kernel should do".

Absolutely.

I don't get what you're saying. I'm saying reset the retransmit timers, if there is no response, exponential backoff applies again. The reset of the timers is to avoid that the exponential backoff timer is now in "wait for several minutes to send the next packet" due to no connectivity during the first 10-20 seconds after coming out of resume.

On my Ubuntu 12.04 system, when I bring it out of resume and press enter in the terminal with an established ssh session, it takes minutes to discover that this session doesn't work and close it (it seems the TCP RST from the server isn't triggered for a long time). I usually end up killing the ssh process on the laptop and logging in again. What I want to happen is that within a few seconds of network connectivity being established, TCP should try to communicate and discover that the other end already closed the connection and close the local socket call so ssh quits and I can re-login again. One way of doing this would be for the connection manager to ask the kernel to reset (or severely lower) the retransmit timers on all established TCP connections.

Mikael Abrahamsson    email: [email protected]

FYI the idea was by Andi Kleen back in 2003. If was flag IFF_DYNAMIC on device (apparently, it should be per-interface sysctl instead or even a flag on specific address).

Andi suggested to hook netdev notifier and to reset tcp connections bound to addresses on this interface. He did not go so far to send resets before address is actually disabled (it was not a goal, normally address is already dead to the time when it is deleted),
but techically it is the same.

The problem with this was purely technical, the code has to scan through all the tcp hash table to search for connections to this address (grrr already :-)) and to take socket lock before making any actions. It is doable, but quite chumbersome and nobody was interested enough to finish the job.

With IPv6 and temporary addresses etc, and DHCPv4/v6 leases expiring, addresses going away can be a quite foreseeable event. When I suspend my laptop, the IP going away is also quite forseeable.

Are there hooks so that someone could write a userland application to do this job? I could imagine the connection manager doing this job as well (so it's configurable what connection to close or not).

What about the reset of the TCP retransmit timers, is that doable from userland? Or at least lowering them to 1s or something fairly sane (going to fast-retransmit doesn't make sense, neither does having a retransmit timer in the 10s of minutes).

Btw, what are the TCP retransmit timers when I come out of suspend? The machine has been "offline" for hours, what idea does TCP have of what's going on, will it keep the timers it had when machine went down or does TCP believe it hasn't seen a response for a long time and now the retransmit timers are very high?

Mikael Abrahamsson    email: [email protected]

Please don't fix that. I sometimes move my laptop from one ethernet socket to another. During the move I lose my DHCP-assigned IP address.

I do not want to lose my SSH sessions too.

/Benny

Would you mind if it was user configurable?

Would you mind if the TCP retransmit timers were severely lowered upon reconnection so your TCP sessions started working quicker? Or you don't see this problem at all?

Mikael Abrahamsson    email: [email protected]

No, I would not mind. But I am the wrong person to ask that question, I do not have to deal with supporting configuration options.

I would not mind that at all. I do not particularly recall seeing the problem, but that sounds like a great idea.

/Benny