Skip to contentSkip to conversation

tests/qtest: Poll on waitpid() for a while before sending SIGKILL

PATCH Superseded Stefan Berger 3 participants 5 messages Jan 11, 2023 – Jan 12, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/qtest: Poll on waitpid() for a while before sending SIGKILL

To prevent getting stuck on waitpid() in case the target process does not terminate on SIGTERM, poll on waitpid() for 10s and if the target process has not changed state until then send a SIGKILL to it.

---
 tests/qtest/libqtest.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
tests/qtest/libqtest.c +17 -1
index 2fbc3b88f3..362b1f724f 100644
@@ -202,8 +202,24 @@ void qtest_wait_qemu(QTestState *s)
 {
 #ifndef _WIN32
     pid_t pid;
+    uint64_t end;
+
+    /* poll for 10s until sending SIGKILL */
+    end = g_get_monotonic_time() + 10 * G_TIME_SPAN_SECOND;
+
+    do {
+        pid = waitpid(s->qemu_pid, &s->wstatus, WNOHANG);
+        if (pid != 0) {
+            break;
+        }
+        g_usleep(100 * 1000);
+    } while (g_get_monotonic_time() < end);
+
+    if (pid == 0) {
+        kill(s->qemu_pid, SIGKILL);
+        TFR(pid = waitpid(s->qemu_pid, &s->wstatus, 0));
+    }
 
-    TFR(pid = waitpid(s->qemu_pid, &s->wstatus, 0));
     assert(pid == s->qemu_pid);
 #else
     DWORD ret;
Signature · 1 line
2.39.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines · Stefan Berger

To prevent getting stuck on waitpid() in case the target process does not terminate on SIGTERM, poll on waitpid() for 10s and if the target process has not changed state until then send a SIGKILL to it.

---
 tests/qtest/libqtest.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

Since this is a test suite and we know our CI system gets very heavily loaded, I think we should wait more than 10 secs, to ensure QEMU has time to flush pending I/O in particular which is most likely to delay things. If you bump the time to 30 secs then

Quoted 34 lines
tests/qtest/libqtest.c +17 -1
index 2fbc3b88f3..362b1f724f 100644
@@ -202,8 +202,24 @@ void qtest_wait_qemu(QTestState *s)
 {
 #ifndef _WIN32
     pid_t pid;
+    uint64_t end;
+
+    /* poll for 10s until sending SIGKILL */
+    end = g_get_monotonic_time() + 10 * G_TIME_SPAN_SECOND;
+
+    do {
+        pid = waitpid(s->qemu_pid, &s->wstatus, WNOHANG);
+        if (pid != 0) {
+            break;
+        }
+        g_usleep(100 * 1000);
+    } while (g_get_monotonic_time() < end);
+
+    if (pid == 0) {
+        kill(s->qemu_pid, SIGKILL);
+        TFR(pid = waitpid(s->qemu_pid, &s->wstatus, 0));
+    }
 
-    TFR(pid = waitpid(s->qemu_pid, &s->wstatus, 0));
     assert(pid == s->qemu_pid);
 #else
     DWORD ret;
Signature · 1 line
2.39.0

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
In reply to Stefan Berger
Quoted 14 lines · Stefan Berger

To prevent getting stuck on waitpid() in case the target process does not terminate on SIGTERM, poll on waitpid() for 10s and if the target process has not changed state until then send a SIGKILL to it.

---
  tests/qtest/libqtest.c | 18 +++++++++++++++++-
  1 file changed, 17 insertions(+), 1 deletion(-)
tests/qtest/libqtest.c +0 -0
index 2fbc3b88f3..362b1f724f 100644

tests/qtest/libqtest.c void qtest_wait_qemu(QTestState *s) · line 208

Quoted 7 lines
tests/qtest/libqtest.c +4 -0
@@ -202,8 +202,24 @@ void qtest_wait_qemu(QTestState *s)
  {
  #ifndef _WIN32
      pid_t pid;
+    uint64_t end;
+
+    /* poll for 10s until sending SIGKILL */
+    end = g_get_monotonic_time() + 10 * G_TIME_SPAN_SECOND;

Maybe we could use getenv() to allow tuning / using different value?

Quoted 17 lines
tests/qtest/libqtest.c +12 -1
+    do {
+        pid = waitpid(s->qemu_pid, &s->wstatus, WNOHANG);
+        if (pid != 0) {
+            break;
+        }
+        g_usleep(100 * 1000);
+    } while (g_get_monotonic_time() < end);
+
+    if (pid == 0) {
+        kill(s->qemu_pid, SIGKILL);
+        TFR(pid = waitpid(s->qemu_pid, &s->wstatus, 0));
+    }
 
-    TFR(pid = waitpid(s->qemu_pid, &s->wstatus, 0));
      assert(pid == s->qemu_pid);
  #else
      DWORD ret;
#3 lore raw
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Stefan Berger

To prevent getting stuck on waitpid() in case the target process does not terminate on SIGTERM, poll on waitpid() for 10s and if the target process has not changed state until then send a SIGKILL to it.

---
  tests/qtest/libqtest.c | 18 +++++++++++++++++-
  1 file changed, 17 insertions(+), 1 deletion(-)
Quoted 14 lines

tests/qtest/libqtest.c void qtest_wait_qemu(QTestState *s) · line 208

Quoted 12 lines
tests/qtest/libqtest.c +4 -0
index 2fbc3b88f3..362b1f724f 100644
@@ -202,8 +202,24 @@ void qtest_wait_qemu(QTestState *s)
  {
  #ifndef _WIN32
      pid_t pid;
+    uint64_t end;
+
+    /* poll for 10s until sending SIGKILL */
+    end = g_get_monotonic_time() + 10 * G_TIME_SPAN_SECOND;

Maybe we could use getenv() to allow tuning / using different value?

I'd rather we picked a value large enough that it will work reliably out of the box for all scenarios with no magic env required. We're just trying to prevent infinite waits if something unexpected happens. We don't need to use an aggressively short value, as most users will never hit this scenario. I think 30 seconds is large enough to be reliable but we could easily go higher to 60/120 if we want to be really really sure.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
Quoted 26 lines · Daniel P. Berrangé
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Stefan Berger

To prevent getting stuck on waitpid() in case the target process does not terminate on SIGTERM, poll on waitpid() for 10s and if the target process has not changed state until then send a SIGKILL to it.

---
   tests/qtest/libqtest.c | 18 +++++++++++++++++-
   1 file changed, 17 insertions(+), 1 deletion(-)
Quoted 14 lines

tests/qtest/libqtest.c void qtest_wait_qemu(QTestState *s) · line 208

Quoted 12 lines
tests/qtest/libqtest.c +4 -0
index 2fbc3b88f3..362b1f724f 100644
@@ -202,8 +202,24 @@ void qtest_wait_qemu(QTestState *s)
   {
   #ifndef _WIN32
       pid_t pid;
+    uint64_t end;
+
+    /* poll for 10s until sending SIGKILL */
+    end = g_get_monotonic_time() + 10 * G_TIME_SPAN_SECOND;

Maybe we could use getenv() to allow tuning / using different value?

Quoted 8 lines

I'd rather we picked a value large enough that it will work reliably out of the box for all scenarios with no magic env required. We're just trying to prevent infinite waits if something unexpected happens. We don't need to use an aggressively short value, as most users will never hit this scenario. I think 30 seconds is large enough to be reliable but we could easily go higher to 60/120 if we want to be really really sure.

I read your other comment later and I agree with you.