Skip to contentSkip to conversation

ts-logs-capture: Cope better with unbootable host

PATCH New Ian Jackson 1 participant 2 messages Apr 2, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 ts-logs-capture: Cope better with unbootable host

If we cannot ssh to the host to fish out its own logs, do not try to do any of the other log captures which involve ssh'ing to the host.

This includes {fetch,extract}_logs_guest, which tolerated this situation - so then it's an optimisation.

But it also includes shutdown_guests, which was introduced in c5f8d41143ab "ts-logs-capture: Fish some logs out of guest filesystem" and is not tolerant enough. Since that commit, unbootable hosts have caused ts-logs-capture to wrongly declare jobs broken.

---
 ts-logs-capture | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
ts-logs-capture +8 -5
index c67856cd..7940aece 100755
@@ -190,7 +190,7 @@ sub fetch_logs_host () {
             1;
         }) {
             logm("host reboot failed, abandoning log fetches: $@");
-            return;
+            return 0;
         }
 	try_fetch_logs($ho, $logs);
     }
@@ -219,6 +219,8 @@ sub fetch_logs_host () {
          ) {
             try_cmd_output_save($cmd);
         }
+
+    return 1;
 }
 
 sub fetch_xenctx_guest ($) {
@@ -293,8 +295,9 @@ power_state($ho,1);
 find_guests();
 fetch_xenctx_guest($_) foreach @guests;
 serial_fetch_logs($ho);
-fetch_logs_host();
-fetch_logs_guest($_) foreach @guests;
-shutdown_guests();
-extract_logs_guest($_) foreach @allguests;
+if (fetch_logs_host()) {
+    fetch_logs_guest($_) foreach @guests;
+    shutdown_guests();
+    extract_logs_guest($_) foreach @allguests;
+}
 logm("logs captured to $stash");
Signature · 1 line
2.11.0
#1 lore raw

PATCH 2/2 ts-logs-capture: Cope better with unbootable host (2)

shutdown_guests might conceivably fail due to a flaky host. In that case we want not to declare the job broken so ts-logs-capture most not fail. But in that case we should skip fishing in-guest logs out of guest fs's.

---
 ts-logs-capture | 31 +++++++++++++++++++------------
 1 file changed, 19 insertions(+), 12 deletions(-)
ts-logs-capture +19 -12
index 7940aece..0320a5a5 100755
@@ -264,17 +264,23 @@ sub fetch_logs_guest ($) {
 }
 
 sub shutdown_guests () {
-    target_cmd_root($ho, <<'END', 180);
-        set -x
-        (
-            ( exec 2>/dev/null; sleep 30 ; echo y ) &
-            ( xl shutdown -a -F -w ; echo y ) &
-        ) | (
-            read x
-            xl list | awk '!/^Domain-0 |^Name / {print $2}' \
-            | xargs -t -r -n1 xl destroy ||:
-        )
+    if (!eval {
+	target_cmd_root($ho, <<'END', 180);
+	    set -x
+	    (
+		( exec 2>/dev/null; sleep 30 ; echo y ) &
+		( xl shutdown -a -F -w ; echo y ) &
+	    ) | (
+		read x
+		xl list | awk '!/^Domain-0 |^Name / {print $2}' \
+		| xargs -t -r -n1 xl destroy ||:
+	    )
 END
+    }) {
+	logm("cannot ensure no guests running, cannot fish their logs");
+	return 0;
+    }
+    return 1;
 }
 
 sub extract_logs_guest ($) {
@@ -297,7 +303,8 @@ fetch_xenctx_guest($_) foreach @guests;
 serial_fetch_logs($ho);
 if (fetch_logs_host()) {
     fetch_logs_guest($_) foreach @guests;
-    shutdown_guests();
-    extract_logs_guest($_) foreach @allguests;
+    if (shutdown_guests()) {
+	extract_logs_guest($_) foreach @allguests;
+    }
 }
 logm("logs captured to $stash");
Signature · 1 line
2.11.0