TCP hangs in 2.4.20-pre11 (and 2.4.19)
I have one machine rsync'ing a debian mirror through another machine which runs a HTTPS proxy - rsync can use the HTTP CONNECT method to proxy-connected to a remote rsyncd.
On the gateway machine, the proxy consistantly hangs in a write(). I've replaced the squid proxy with a simple perl script + nc to make sure it isn't a squid-related problem..
# netstat -t | grep ftp.debian.nl Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 70832 14480 cust.94.89.adsl.c:33876 ftp.debian.nl:rsync ESTABLISHED
# ps -eo pid,tt,user,fname,tmout,f,wchan | grep nc 26782 ? nobody nc - 100 wait_for_tcp_memory
# strace -fp 26782 write(3, "\236\3754\"OU\331\6\36d\33;\270\3776\221\21\364Z\370/\346"..., 3384 <unfinished ...>
# tcpdump -v -e -i eth1 -n host ftp.debian.nl tcpdump: listening on eth1 17:36:40.946370 0:10:67:0:f8:8f 0:d0:b7:b2:67:82 0800 66: 195.64.82.85.873 > 195.64.94.89.33876: . [tcp sum ok] ack 2887846970 win 0 <nop,nop,timestamp 77491485 14953830> (DF) (ttl 61, id 19377, len 52) 17:36:40.946431 0:d0:b7:b2:67:82 0:10:67:0:f8:8f 0800 66: 195.64.94.89.33876 > 195.64.82.85.873: . [tcp sum ok] ack 1 win 0 <nop,nop,timestamp 14965835 77480589> (DF) (ttl 64, id 30547, len 52) 17:36:51.985379 0:d0:b7:b2:67:82 0:10:67:0:f8:8f 0800 66: 195.64.94.89.33876 > 195.64.82.85.873: . [tcp sum ok] ack 1 win 0 <nop,nop,timestamp 14966939 77480589> (DF) (ttl 64, id 30548, len 52) 17:36:52.032399 0:10:67:0:f8:8f 0:d0:b7:b2:67:82 0800 66: 195.64.82.85.873 > 195.64.94.89.33876: . [tcp sum ok] ack 1 win 0 <nop,nop,timestamp 77492593 14965835> (DF) (ttl 61, id 19378, len 52) 17:38:40.984930 0:10:67:0:f8:8f 0:d0:b7:b2:67:82 0800 66: 195.64.82.85.873 > 195.64.94.89.33876: . [tcp sum ok] ack 1 win 0 <nop,nop,timestamp 77503488 14965835> (DF) (ttl 61, id 19379, len 52) 17:38:40.984987 0:d0:b7:b2:67:82 0:10:67:0:f8:8f 0800 66: 195.64.94.89.33876 > 195.64.82.85.873: . [tcp sum ok] ack 1 win 0 <nop,nop,timestamp 14977838 77492593> (DF) (ttl 64, id 30549, len 52)
.. etc. It never recovers from this.
The machine has 2 eepro100 network cards. One is connected to the office network (which has its own 2 mbit connected to the outside world), the other is connected to an ADSL 8 mbit/sec line. Both connections to the outside world show the same problem. I've tried both eepro100 and e100 drivers with the same results. 2.4.20-pre11 and 2.4.19 both behave the same. There is no firewall in the path anywhere. And it doesn't matter which remote
rsync server I used, the connection always eventually hangs.
Wait - I do have a dual routing table setup, so that traffic originating from the ADSL IP goes out over the ADSL line:
# ip rule show 0: from all lookup local 32700: from 195.64.94.89 lookup adsl 32766: from all lookup main 32767: from all lookup default # ip route show table adsl default via 195.64.94.1 dev eth1
The machine is a PII/300 with 192 MB of RAM, of which it has plenty free.
What can this be ? Esp. the netstat output looks weird.
Mike.
Re: TCP hangs in 2.4 - blocking write() in wait_for_tcp_memory
In article <[email protected]>,
Quoted 3 lines · Miquel van Smoorenburg
On the gateway machine, the proxy consistantly hangs in a
write(). I've replaced the squid proxy with a simple perl script + nc to make sure it isn't a squid-related problem..
Right, I found the cause of the problem, but I'm not sure if the application of the kernel is wrong here.
On 2 machines do this:
machine1# socket -s 12345 < /dev/zero > /dev/null # server
machine2# socket -w machine1 12345 < /dev/zero # client
The first command starts a listening process on port 12345, that sends an infinite stream of zeros to the remote side and sinks all data received.
The second command connects to the first machine, sends an infinite stream of zeros, but never does a read() on the socket (the '-w' option).
The 'socket' program doesn't make the sockets non-blocking, it just does a select() loop to find out readability/writeability on the file descriptors.
This makes both socket programs hang in write(), in wait_for_tcp_memory. Shouldn't the kernel return a short write, instead of hanging both processes ? select() returned writeability.
As I described in my first mail, this happens in the real world as well - an application is writing lots of data to the remote side, while the remote side is sending data too - hang.
Oh, tested it on 2.4.19 and 2.4.20-pre11
Mike.
Re: TCP hangs in 2.4 - blocking write() in wait_for_tcp_memory
Quoted 3 lines · Miquel van Smoorenburg
This makes both socket programs hang in
write(), inwait_for_tcp_memory. Shouldn't the kernel return a short write, instead of hanging both processes ?select()returned writeability.
write(2) is allowed to do a short write on a blocking socket, but not mandated to do so. In fact I've only seen short writes under linux on non-blocking sockets.
SuSv3 says:
Blocking/immediate: Blocking is only possible with O_NONBLOCK clear. If there is enough space for all the data requested to be written immediately, the implementation should do so. Otherwise, the process may block; that is, pause until enough space is available for writing. The effective size of a pipe or FIFO (the maximum amount that can be written in one operation without blocking) may vary dynamically, depending on the implementation, so it is not possible to specify a fixed value for it.
...
Partial and deferred writes are only possible with O_NONBLOCK set.
Regards,
bert
Signature · 2 lines
http://www.PowerDNS.com Versatile DNS Software & Services http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
Re: TCP hangs in 2.4 - blocking write() in wait_for_tcp_memory
In article <[email protected]>,
Quoted 11 lines · bert hubert
Quoted 3 lines · Miquel van Smoorenburg
This makes both socket programs hang in
write(), inwait_for_tcp_memory. Shouldn't the kernel return a short write, instead of hanging both processes ?select()returned writeability.write(2) is allowed to do a short write on a blocking socket, but not mandated to do so. In fact I've only seen short writes under linux on non-blocking sockets.
SuSv3 says:
Ah, that's interesting:
Quoted 3 lines
Blocking/immediate: Blocking is only possible with
O_NONBLOCKclear. If there is enough space for all the data requested to be written immediately, the implementation should do so. Otherwise, the process may block; that is,
Okay, _may_ block. That's what I needed to know. So it's not a kernel bug, but a bug in applications like socket(1) and nc(1).
With squid I see corrupted downloads, that probably means squid does use non-blocking sockets but doesn't handle short writes correctly.
Quoted 1 line
Partial and deferred writes are only possible with
O_NONBLOCKset.
Thanks for the clarification.
Now I must go fix all those programs :/
Mike.