Skip to contentSkip to conversation

rsync over ssh on 2.4.2 to 2.2.18

DISCUSSION Russell King 8 participants 15 messages Feb 27, 2001 – Mar 23, 2001

lore
Share & tools
lore
#0 lore raw

Hi,

I'm seeing odd behaviour with rsync over ssh between two x86 machines - one if the is an UP PIII (Katmai) running 2.4.2 (isdn-gw) and the other is an UP Pentium 75-200 (pilt-gw) running 2.2.15pre13 with some custom serial driver hacks (for running Amplicon cards with their ISA interrupt- sharing scheme).

Please note: although I am using 2.2.15pre13, it is _not_ the cause of this problem, and I don't particularly want to touch the 2.2.15pre13 machine at the present time, which has been happy for the past year or so.

netstat on isdn-gw shows the following:

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 72868 0 isdn-gw.piltdown.a:1023 pilt-gw.piltdown.at:ssh ESTABLISHED

netstat on pilt-gw shows the following:

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 14372 pilt-gw.piltdown.at:ssh isdn-gw.piltdown.a:1023 ESTABLISHED

Looking on the network reveals:

	09:49:23.215852 P pilt-gw.ssh > isdn-gw.1023: . 1119640864:1119640864(0) ack 1175090336 win 31856 <nop,nop,timestamp 114834687 8389267> (DF) [tos 0x8]
	09:49:23.216095 P isdn-gw.1023 > pilt-gw.ssh: . 1:1(0) ack 1 win 0 <nop,nop,timestamp 8401267 114186976> (DF) [tos 0x8]

So isdn-gw is saying it can't accept more data, which ties up with what netstat is indicating. Note that this situation has persisted for around 12 hours with neither end moving forward.

When I straced the ssh process on isdn-gw this morning, things start moving, and in this instance everything completed normally, even after I stopped stracing ssh. However, there are times when I have to leave strace running to get the rsync to complete.

select(4, [3], [3], NULL, NULL) = 2 (in [3], out [3])
read(3, "\0\0\20\t\215R\241\355\201_o\227\4\371\340+\333_\333\34"..., 8192) = 8192 write(3, "\370\325|\355\232\256\247Kf\202\v\272\334@\231\317]\322"..., 135940) = 24616

On this instance, it appears that sshd has gotten stuck in select(). Maybe there is a race condition or missing wakeup in the TCP code?

Note that this problem also exists on 2.4.0 (I upgraded from 2.4.0 to 2.4.2 yesterday to see if the behaviour has changed).

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

#1 lore raw
Quoted 7 lines · Russell King

Hi,

I'm seeing odd behaviour with rsync over ssh between two x86 machines - one if the is an UP PIII (Katmai) running 2.4.2 (isdn-gw) and the other is an UP Pentium 75-200 (pilt-gw) running 2.2.15pre13 with some custom serial driver hacks (for running Amplicon cards with their ISA interrupt- sharing scheme).

[...]

I have seen behaviour similar to this (descibed in
http://marc.theaimsgroup.com/?l=linux-kernel&m=98262067309185&w=2)
locally on a 2.4.X machine where the rsync completes but fails to terminate, apparently because the child does not receive the KILLUSR1 (wild speculation)? Anyways, the parent process waits in wait4 and the child loops, waiting for the signal. This is not reproducable in 2.2.X (for me).

I have found that in order to reproduce this problem I need to make rsync do a certain amount of work. Ie., rsync linux/drivers new-dir will make rsync hang, but rsync linux/drivers/[a-p]* does not. Does the same thing apply for you? stracing the rsync does make it all work if I specify that rsync should trace the child, too.

Regards,
Rasmus

#2 lore raw
In reply to Russell King
Quoted 2 lines · Russell King

Please note: although I am using 2.2.15pre13, it is _not_ the cause of this problem

How do you know this? There are so many deadly TCP bugs fixed since 2.2.15pre13 I don't know how you can assert this.

Later,
David S. Miller
[email protected]

#3 lore raw
Quoted 2 lines · David S. Miller

How do you know this? There are so many deadly TCP bugs fixed since 2.2.15pre13 I don't know how you can assert this.

Through the evidence I gave.

Firstly, if the receiving side is saying that it has a window of 0, then it is not going to accept any more data.

Secondly, the receiving side has data waiting in the receive queue.

Thirdly, the receiving process is selecting on the socket, and dispite there being data waiting, select is saying that there is no data waiting.

All the data is pointing at the 2.4 kernel as being the culprit.

I'm surprised at your response given the amount of hard evidence I gave that supports my assertion.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

#4 lore raw
In reply to Russell King
Quoted 4 lines · Rasmus Andersen

terminate, apparently because the child does not receive the KILLUSR1 (wild speculation)? Anyways, the parent process waits in wait4 and the child loops, waiting for the signal. This is not reproducable in 2.2.X (for me).

rsync problems of this type abound across several UNIX platforms - I had problems like this on FreeBSD, but never under Linux. I suspect that rsync bends or at least stretches certain limits.

Regards,

bert

Signature · 3 lines
http://www.PowerDNS.com      Versatile DNS Services  
Trilab                       The Technology People   
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#5 lore raw
In reply to Russell King

Hi,

Quoted 5 lines

I'm seeing odd behaviour with rsync over ssh between two x86 machines - one if the is an UP PIII (Katmai) running 2.4.2 (isdn-gw) and the other is an UP Pentium 75-200 (pilt-gw) running 2.2.15pre13 with some custom serial driver hacks (for running Amplicon cards with their ISA interrupt- sharing scheme).

What version of ssh are you using? Older versions would use blocking IO which would result in deadlocks (and angry emails wrongly blaming rsync :)

Anton

#6 lore raw
Quoted 8 lines · Anton Blanchard
Quoted 5 lines

I'm seeing odd behaviour with rsync over ssh between two x86 machines - one if the is an UP PIII (Katmai) running 2.4.2 (isdn-gw) and the other is an UP Pentium 75-200 (pilt-gw) running 2.2.15pre13 with some custom serial driver hacks (for running Amplicon cards with their ISA interrupt- sharing scheme).

What version of ssh are you using? Older versions would use blocking IO which would result in deadlocks (and angry emails wrongly blaming rsync :)

Note that I proved that it was sitting in select(), and therefore can't be blocking.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

#7 lore raw
In reply to Russell King

Hello!

Quoted 4 lines

netstat on isdn-gw shows the following:

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 72868 0 isdn-gw.piltdown.a:1023 pilt-gw.piltdown.at:ssh ESTABLISHED

plus

Quoted 4 lines

select(4, [3], [3], NULL, NULL) = 2 (in [3], out [3])

Maybe there is a race condition or missing wakeup in the TCP code?

Moreover, even not _one_ wakeup is missing. At least two, because wakeups in read and write are separate and you have stuck in both directions. 8)8)

Well, if it was one I would start to dig ground inside tcp instantly. But as soon as two of them are missing, I have to suspect wake_up itself. At least, we had such bugs there until 2.4.0.

Alexey

#8 lore raw
Quoted 3 lines · [email protected] writes:

Moreover, even not _one_ wakeup is missing. At least two, because wakeups in read and write are separate and you have stuck in both directions. 8)8)

I'll see if I can strace it from the start until it hangs tomorrow.

Quoted 3 lines

Well, if it was one I would start to dig ground inside tcp instantly. But as soon as two of them are missing, I have to suspect wake_up itself. At least, we had such bugs there until 2.4.0.

I was running at one point a 2.4.0-test kernel, but I didn't see these effects back then (but then I wasn't monitoring the system as closely as I am at the moment).

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

#9 lore raw

Hello!

Quoted 1 line

I'll see if I can strace it from the start until it hangs tomorrow.

Please...

Also, try to make binary tcpdump.

Quoted 1 line

I was running at one point a 2.4.0-test kernel, but I didn't see these

Yes, it did not result in full stall. Lost wakeups were recovered f.e. by any keyboard activity. 8)

Wow! 2.2.15? rsync surely does work with 2.2.15. The first 2.2 fixing the same bugs, which solaris has now, was 2.2.17.

Alexey

#10 lore raw
In reply to Russell King

In article <[email protected]>,

Quoted 1 line · Russell King

I'm seeing odd behaviour with rsync over ssh between two x86 machines -

I've seen hanging rsync over ssh more than once, while sending much data from an x86 running Linux (late 2.3.x) to Sparc/Solaris2.5.1 over ssh 1.2.26. I had a feeling that it was triggered by certain data patterns because it often stopped at the same spot after restart (and therefore I attributed it to a bug in rsync). I did not investigate further.

Eugene

#11 lore raw

Hello!

Quoted 2 lines

I've seen hanging rsync over ssh more than once, while sending much data from an x86 running Linux (late 2.3.x) to Sparc/Solaris2.5.1

I remember this your report. However, recent news force to suspect that the reason was in Solaris yet. Actually, if you send tcpdump of failed session, this question can be answered.

Also REMEMBER!

rsync has __no__ chances to work, if one of sides is Linux-2.2 before 2.2.17 or Solaris of today (sunos-2.5.1 is enough old, probably, it still was right unlike subsequent kernels). These stacks have conciding set of fatal bugs, which prevent transmits with closed window.

To use rsync it is necessary to upgrade to >=2.2.17.

Alexey

#12 lore raw
In reply to Eugene Crosser
Quoted 3 lines · On Wed, Feb 28, 2001 at 11:20:22PM +0300, [email protected] wrote:

I remember this your report. However, recent news force to suspect that the reason was in Solaris yet. Actually, if you send tcpdump of failed session, this question can be answered.

Well, since I moved the rsync to 5pm, and then back to 9pm, I haven't seen this problem - everything is again working as expected (touch wood) with 2.2.15pre13 and 2.4.0.

This is odd, since it wasn't a one-off problem, but something that happened each and every day of a particular week. Anyway, if it starts happening again, I'll get a tcpdump of the session.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

#13 lore raw

Hello!

Quoted 7 lines

Well, since I moved the rsync to 5pm, and then back to 9pm, I haven't seen this problem - everything is again working as expected (touch wood) with 2.2.15pre13 and 2.4.0.

This is odd, since it wasn't a one-off problem, but something that happened each and every day of a particular week. Anyway, if it starts happening again, I'll get a tcpdump of the session.

Well... I can reproduce this not depending of day of week now. :-(

If I understood Andrew's mail correctly, rsync freezes when large amount of errors happen. Particularly, here ssh always freezes trying to write to stdout (pipe linked to rsync) some chunk of data (>64K), consisting of reports sort of "stat(usr/X11R6/bin/xmbind) : No such file" (my /usr has huge amount of stray symlinks...). rsync does not read this pipe, trying to write some more data to pipe, which is pipe linked to rsync stdin. Dead lock.

Note, that doing strace you do not see this write()! write() is interrupted by strace and you see succesful select(). You can see real place of lockup via ps axl or starting strace before lockup happened.

Andrew said about one more common case, when large amount of errors happens:
wrong permission on some target directory.

I have impression that long ago I observed the same affect, when disk space at target exhausted.

Why do I tall this? Well, probably, something changed at fs, wich you rsync and rsync generates less errors.

Alexey

#14 lore raw

In message <[email protected]> you write:

Quoted 2 lines

If I understood Andrew's mail correctly, rsync freezes when large amount of errors happen. Particularly, here ssh always freezes

Known hard-to-fix bug in rsync; too many errors in the pipe, and it locks up.

Rusty.
--
Premature optmztion is rt of all evl. --DK