Fix TCP problem
cover Fix TCP problem
The best reference I found for this was here:
https://www.evanjones.ca/tcp-stuck-connection-mystery.html
I'm resending this series because the first one had my Citrix email, which is probably not going to reach many people.
PATCH 1/5 daemonlib: Provide a "noop" command
From: Ian Jackson <[email protected]>
We are going to want clients to speak before waiting for the server banner. A noop command is useful for that.
Putting this here makes it apply to both ownerdaemon and queuedaemon.
--- tcl/daemonlib.tcl | 4 ++++ 1 file changed, 4 insertions(+)
tcl/daemonlib.tcl +4 -0
@@ -124,6 +124,10 @@ proc puts-chan {chan m} { puts $chan $m } +proc cmd/noop {chan desc} { + puts-chan $chan "OK noop" +} + #---------- data ---------- proc puts-chan-data {chan m data} {
Signature · 1 line
2.20.1
PATCH 2/5 TCP fix: Do not wait for queuedaemon to speak
In reply to Ian JacksonFrom: Ian Jackson <[email protected]>
This depends on the preceding daemonlib patch and an ms-queuedaemon restart.
--- Osstest/Executive.pm | 9 +++++++++ 1 file changed, 9 insertions(+)
Osstest/Executive.pm +9 -0
@@ -643,7 +643,16 @@ sub tcpconnect_queuedaemon () { my $qserv= tcpconnect($c{QueueDaemonHost}, $c{QueueDaemonPort}); $qserv->autoflush(1); + # TCP connections can get into a weird state where the client + # thinks the connection is open but the server has no record + # of it. To avoid this, have the client speak without waiting + # for the server. + # + # See A TCP "stuck" connection mystery" + # https://www.evanjones.ca/tcp-stuck-connection-mystery.html + print $qserv "noop\n"; $_= <$qserv>; defined && m/^OK ms-queuedaemon\s/ or die "$_?"; + $_= <$qserv>; defined && m/^OK noop\s/ or die "$_?"; return $qserv; }
Signature · 1 line
2.20.1
PATCH 3/5 TCP fix: Do not wait for ownerdaemon to speak
In reply to Ian JacksonFrom: Ian Jackson <[email protected]>
--- tcl/JobDB-Executive.tcl | 13 +++++++++++++ 1 file changed, 13 insertions(+)
tcl/JobDB-Executive.tcl +13 -0
@@ -414,7 +414,20 @@ proc become-task {comment} { set ownerqueue [socket $c(OwnerDaemonHost) $c(OwnerDaemonPort)] fconfigure $ownerqueue -buffering line -translation lf + + # TCP connections can get into a weird state where the client + # thinks the connection is open but the server has no record + # of it. To avoid this, have the client speak without waiting + # for the server. We tolerate "unknown command" errors so + # that it is not necessary to restart the ownerdaemon since + # that is very disruptive. + # + # See A TCP "stuck" connection mystery" + # https://www.evanjones.ca/tcp-stuck-connection-mystery.html + puts $ownerqueue noop must-gets $ownerqueue {^OK ms-ownerdaemon\M} + must-gets $ownerqueue {^OK noop|^ERROR unknown command} + puts $ownerqueue create-task must-gets $ownerqueue {^OK created-task (\d+) (\w+ [\[\]:.0-9a-f]+)$} \ taskid refinfo
Signature · 1 line
2.20.1
PATCH 4/5 TftiDiVersion: Update to latest installer for stretch
In reply to Ian JacksonThe stretch (Debian oldstable) kernel has been updated, causing our Xen 4.10 tests (which are still using stretch) to break. This update seems to fix it.
--- production-config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
production-config +1 -1
@@ -90,7 +90,7 @@ TftpNetbootGroup osstest # Update with ./mg-debian-installer-update(-all) TftpDiVersion_wheezy 2016-06-08 TftpDiVersion_jessie 2018-06-26 -TftpDiVersion_stretch 2020-02-10 +TftpDiVersion_stretch 2020-09-24 TftpDiVersion_buster 2020-05-19 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
Signature · 1 line
2.20.1
PATCH 5/5 Update TftpDiVersion_buster
In reply to Ian JacksonFrom: Ian Jackson <[email protected]>
--- production-config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
production-config +1 -1
@@ -91,7 +91,7 @@ TftpNetbootGroup osstest TftpDiVersion_wheezy 2016-06-08 TftpDiVersion_jessie 2018-06-26 TftpDiVersion_stretch 2020-09-24 -TftpDiVersion_buster 2020-05-19 +TftpDiVersion_buster 2020-09-28 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
Signature · 1 line
2.20.1