New: TCP Communications gets blocked, then resetted
Subsystems (2)
Begin forwarded message:
Date: Mon, 20 Sep 2010 10:39:47 GMT
From: [email protected]
To: [email protected]
Subject: [Bug 18822] New: TCP Communications gets blocked, then resetted
https://bugzilla.kernel.org/show_bug.cgi?id=18822
Summary: TCP Communications gets blocked, then resetted
Product: Networking
Version: 2.5
Kernel Version: 2.6.32-24-generic #43-Ubuntu
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: IPV4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: NoCreated an attachment (id=30682)
--> (https://bugzilla.kernel.org/attachment.cgi?id=30682)
both machine dumps
having a freshly installed bacula server i could never get a backup from my working machine done to the bacula server.
The Connection transmits approximately 3 GiB, then locks up and resets the connection.
Transmission gets stuck at a certain point, then the bacula server does not reply to retransmitted packets on IPV4 stack. Retry Count on disks machine (my working machine) raises up to 13, then the Connection is gone. bacula server tries to send a Push packet (after KeepAlive timer runs out), and get the final RST packet from disks, because the connection is gone.
In the Attachment you will find the tcpdumps from both machines, actually the sending machine dropped some packets in the dump.
It might be a possible help: disks is running an 64 bit kernel whereas bacula is running 32 bit. I haven't looked into the option bits very well but it looks like there is a problem hidden:
last ack being ok:
09:32:56.142876 IP bacula.elkenet.bacula-sd > disks.elkenet.50766: Flags [.], ack 2005754588, win 9582, options [nop,nop,TS val 21825875 ecr 4530083], length 0
next ack packet:
09:32:56.144763 IP bacula.elkenet.bacula-sd > disks.elkenet.50766: Flags [.], ack 2005773412, win 9308, options [nop,nop,TS val 21825876 ecr 4530083,nop,nop,sack 1 {2005774860:2005776308}], length 0
root@disks:~# uname -a Linux disks 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:58:24 UTC 2010 x86_64 GNU/Linux
root@bacula:~# uname -a Linux bacula 2.6.32-24-generic-pae #43-Ubuntu SMP Thu Sep 16 15:30:27 UTC 2010 i686 GNU/Linux
Doing a 20GB backup on a debian server works fine
server:~# uname -a Linux server 2.6.32-5-486 #1 Sat Sep 18 01:43:00 UTC 2010 i686 GNU/Linux
Doing a 26 GB backup from a 32 bit Ubuntu works fine as well. Maybe its a 64 bit issue...
root@elke:~# uname -a Linux elke 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:17:33 UTC 2010 i686 GNU/Linux
If any further input is required, just let me know...
Signature · 6 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. --
Quoted 36 lines · Stephen Hemminger
Begin forwarded message:
Date: Mon, 20 Sep 2010 10:39:47 GMT
From: [email protected]
To: [email protected]
Subject: [Bug 18822] New: TCP Communications gets blocked, then resettedhttps://bugzilla.kernel.org/show_bug.cgi?id=18822
Summary: TCP Communications gets blocked, then resetted Product: Networking Version: 2.5 Kernel Version: 2.6.32-24-generic #43-Ubuntu Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: IPV4 AssignedTo: [email protected] ReportedBy: [email protected] Regression: NoCreated an attachment (id=30682)
--> (https://bugzilla.kernel.org/attachment.cgi?id=30682)
both machine dumpshaving a freshly installed bacula server i could never get a backup from my working machine done to the bacula server.
The Connection transmits approximately 3 GiB, then locks up and resets the connection.
Quoted 8 lines
Transmission gets stuck at a certain point, then the bacula server does not reply to retransmitted packets on IPV4 stack. Retry Count on disks machine (my working machine) raises up to 13, then the Connection is gone. bacula server tries to send a Push packet (after KeepAlive timer runs out), and get the final RST packet from disks, because the connection is gone.
In the Attachment you will find the tcpdumps from both machines, actually the sending machine dropped some packets in the dump.
If you didn't already, try with -w directly into a binary file and then post process to textual input with -r.
Quoted 4 lines
It might be a possible help: disks is running an 64 bit kernel whereas bacula is running 32 bit. I haven't looked into the option bits very well but it looks like there is a problem hidden:
Quoted 8 lines
last ack being ok:
09:32:56.142876 IP bacula.elkenet.bacula-sd > disks.elkenet.50766: Flags [.], ack 2005754588, win 9582, options [nop,nop,TS val 21825875 ecr 4530083], length 0
next ack packet:
09:32:56.144763 IP bacula.elkenet.bacula-sd > disks.elkenet.50766: Flags [.], ack 2005773412, win 9308, options [nop,nop,TS val 21825876 ecr 4530083,nop,nop,sack 1 {2005774860:2005776308}], length 0
...I fail to understand to what problem you're trying to point here with these two ACKs. Could you elaborate please (if you had something specific in mind)?
I went throught the logs... I cannot go through all the checking done because tcpdump without enough -v's hides the sequence numbers for pure ACKs (09:32:56.150182 shows only the ack seqno, not the other seqno which also is used by the validator), I think you need at least -vvv to show them nowadays. The last new data segment at 09:32:56.150132 was still received as it is reported in SACK, only the retransmissions that follow are discarded.
Quoted 13 lines
root@disks:~# uname -a Linux disks 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:58:24 UTC 2010 x86_64 GNU/Linuxroot@bacula:~# uname -a Linux bacula 2.6.32-24-generic-pae #43-Ubuntu SMP Thu Sep 16 15:30:27 UTC 2010 i686 GNU/LinuxDoing a 20GB backup on a debian server works fine
server:~# uname -a Linux server 2.6.32-5-486 #1 Sat Sep 18 01:43:00 UTC 2010 i686 GNU/Linux
Quoted 6 lines
Doing a 26 GB backup from a 32 bit Ubuntu works fine as well. Maybe its a 64 bit issue...
root@elke:~# uname -a Linux elke 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:17:33 UTC 2010 i686 GNU/Linux
Hmm, some ubuntu magic in these kernels.
Quoted 1 line
If any further input is required, just let me know...
MIBs might immediately tell what caused the segments between 09:32:56.150310 and 09:46:30.970140 to be discarded (e.g., take before and after snapshots of /proc/net/netstat and /proc/net/snmp and see what did increase).
Any TSO enabled?
Signature · 1 line
i.
Now with the original reported too among the receivers.
Quoted 107 lines · Ilpo Järvinen
Quoted 36 lines · Stephen Hemminger
Begin forwarded message:
Date: Mon, 20 Sep 2010 10:39:47 GMT
From: [email protected]
To: [email protected]
Subject: [Bug 18822] New: TCP Communications gets blocked, then resettedhttps://bugzilla.kernel.org/show_bug.cgi?id=18822
Summary: TCP Communications gets blocked, then resetted Product: Networking Version: 2.5 Kernel Version: 2.6.32-24-generic #43-Ubuntu Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: IPV4 AssignedTo: [email protected] ReportedBy: [email protected] Regression: NoCreated an attachment (id=30682)
--> (https://bugzilla.kernel.org/attachment.cgi?id=30682)
both machine dumpshaving a freshly installed bacula server i could never get a backup from my working machine done to the bacula server.
The Connection transmits approximately 3 GiB, then locks up and resets the connection.
Quoted 8 lines
Transmission gets stuck at a certain point, then the bacula server does not reply to retransmitted packets on IPV4 stack. Retry Count on disks machine (my working machine) raises up to 13, then the Connection is gone. bacula server tries to send a Push packet (after KeepAlive timer runs out), and get the final RST packet from disks, because the connection is gone.
In the Attachment you will find the tcpdumps from both machines, actually the sending machine dropped some packets in the dump.
If you didn't already, try with -w directly into a binary file and then post process to textual input with -r.
Quoted 4 lines
It might be a possible help: disks is running an 64 bit kernel whereas bacula is running 32 bit. I haven't looked into the option bits very well but it looks like there is a problem hidden:
Quoted 8 lines
last ack being ok:
09:32:56.142876 IP bacula.elkenet.bacula-sd > disks.elkenet.50766: Flags [.], ack 2005754588, win 9582, options [nop,nop,TS val 21825875 ecr 4530083], length 0
next ack packet:
09:32:56.144763 IP bacula.elkenet.bacula-sd > disks.elkenet.50766: Flags [.], ack 2005773412, win 9308, options [nop,nop,TS val 21825876 ecr 4530083,nop,nop,sack 1 {2005774860:2005776308}], length 0...I fail to understand to what problem you're trying to point here with these two ACKs. Could you elaborate please (if you had something specific in mind)?
I went throught the logs... I cannot go through all the checking done because tcpdump without enough -v's hides the sequence numbers for pure ACKs (09:32:56.150182 shows only the ack seqno, not the other seqno which also is used by the validator), I think you need at least -vvv to show them nowadays. The last new data segment at 09:32:56.150132 was still received as it is reported in SACK, only the retransmissions that follow are discarded.
Quoted 13 lines
root@disks:~# uname -a Linux disks 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:58:24 UTC 2010 x86_64 GNU/Linuxroot@bacula:~# uname -a Linux bacula 2.6.32-24-generic-pae #43-Ubuntu SMP Thu Sep 16 15:30:27 UTC 2010 i686 GNU/LinuxDoing a 20GB backup on a debian server works fine
server:~# uname -a Linux server 2.6.32-5-486 #1 Sat Sep 18 01:43:00 UTC 2010 i686 GNU/LinuxQuoted 6 lines
Doing a 26 GB backup from a 32 bit Ubuntu works fine as well. Maybe its a 64 bit issue...
root@elke:~# uname -a Linux elke 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:17:33 UTC 2010 i686 GNU/LinuxHmm, some ubuntu magic in these kernels.
Quoted 1 line
If any further input is required, just let me know...
MIBs might immediately tell what caused the segments between 09:32:56.150310 and 09:46:30.970140 to be discarded (e.g., take before and after snapshots of
/proc/net/netstatand/proc/net/snmpand see what did increase).Any TSO enabled?
Signature · 1 line
i.