Skip to contentSkip to conversation

Fix TCP problem

PATCH RESEND New Ian Jackson 1 participant 6 messages Sep 28, 2020

lore
Share & tools
lore
#1 lore raw

PATCH 1/5 daemonlib: Provide a "noop" command

From: Ian Jackson <[email protected]>

We are going to want clients to speak before waiting for the server banner. A noop command is useful for that.

Putting this here makes it apply to both ownerdaemon and queuedaemon.

---
 tcl/daemonlib.tcl | 4 ++++
 1 file changed, 4 insertions(+)
tcl/daemonlib.tcl +4 -0
index 1e86d5f4..747deab1 100644
@@ -124,6 +124,10 @@ proc puts-chan {chan m} {
     puts $chan $m
 }
 
+proc cmd/noop {chan desc} {
+    puts-chan $chan "OK noop"
+}
+
 #---------- data ----------
 
 proc puts-chan-data {chan m data} {
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/5 TCP fix: Do not wait for queuedaemon to speak

In reply to Ian Jackson

From: Ian Jackson <[email protected]>

This depends on the preceding daemonlib patch and an ms-queuedaemon restart.

---
 Osstest/Executive.pm | 9 +++++++++
 1 file changed, 9 insertions(+)
Osstest/Executive.pm +9 -0
index 61a99bc3..80e70070 100644
@@ -643,7 +643,16 @@ sub tcpconnect_queuedaemon () {
     my $qserv= tcpconnect($c{QueueDaemonHost}, $c{QueueDaemonPort});
     $qserv->autoflush(1);
 
+    # TCP connections can get into a weird state where the client
+    # thinks the connection is open but the server has no record
+    # of it.  To avoid this, have the client speak without waiting
+    # for the server.
+    #
+    # See A TCP "stuck" connection mystery"
+    # https://www.evanjones.ca/tcp-stuck-connection-mystery.html
+    print $qserv "noop\n";
     $_= <$qserv>;  defined && m/^OK ms-queuedaemon\s/ or die "$_?";
+    $_= <$qserv>;  defined && m/^OK noop\s/ or die "$_?";
 
     return $qserv;
 }
Signature · 1 line
2.20.1
#3 lore raw

PATCH 3/5 TCP fix: Do not wait for ownerdaemon to speak

In reply to Ian Jackson

From: Ian Jackson <[email protected]>

---
 tcl/JobDB-Executive.tcl | 13 +++++++++++++
 1 file changed, 13 insertions(+)
tcl/JobDB-Executive.tcl +13 -0
index 29c82821..4fe85696 100644
@@ -414,7 +414,20 @@ proc become-task {comment} {
 
     set ownerqueue [socket $c(OwnerDaemonHost) $c(OwnerDaemonPort)]
     fconfigure $ownerqueue -buffering line -translation lf
+
+    # TCP connections can get into a weird state where the client
+    # thinks the connection is open but the server has no record
+    # of it.  To avoid this, have the client speak without waiting
+    # for the server.  We tolerate "unknown command" errors so
+    # that it is not necessary to restart the ownerdaemon since
+    # that is very disruptive.
+    #
+    # See A TCP "stuck" connection mystery"
+    # https://www.evanjones.ca/tcp-stuck-connection-mystery.html
+    puts $ownerqueue noop
     must-gets $ownerqueue {^OK ms-ownerdaemon\M}
+    must-gets $ownerqueue {^OK noop|^ERROR unknown command}
+
     puts $ownerqueue create-task
     must-gets $ownerqueue {^OK created-task (\d+) (\w+ [\[\]:.0-9a-f]+)$} \
         taskid refinfo
Signature · 1 line
2.20.1
#4 lore raw

PATCH 4/5 TftiDiVersion: Update to latest installer for stretch

In reply to Ian Jackson

The stretch (Debian oldstable) kernel has been updated, causing our Xen 4.10 tests (which are still using stretch) to break. This update seems to fix it.

---
 production-config | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
production-config +1 -1
index 6055bd18..0c135bcb 100644
@@ -90,7 +90,7 @@ TftpNetbootGroup osstest
 # Update with ./mg-debian-installer-update(-all)
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2018-06-26
-TftpDiVersion_stretch 2020-02-10
+TftpDiVersion_stretch 2020-09-24
 TftpDiVersion_buster 2020-05-19
 
 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
Signature · 1 line
2.20.1
#5 lore raw

PATCH 5/5 Update TftpDiVersion_buster

In reply to Ian Jackson

From: Ian Jackson <[email protected]>

---
 production-config | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
production-config +1 -1
index 0c135bcb..6f85a4df 100644
@@ -91,7 +91,7 @@ TftpNetbootGroup osstest
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2018-06-26
 TftpDiVersion_stretch 2020-09-24
-TftpDiVersion_buster 2020-05-19
+TftpDiVersion_buster 2020-09-28
 
 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
 
Signature · 1 line
2.20.1