Skip to contentSkip to conversation

iotests: fix three spurious local failures

PATCH Superseded Denis V. Lunev 3 participants 8 messages Jul 10 – Jul 14

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover iotests: fix three spurious local failures

Three independent fixes found while chasing a clean './check -qcow2' run on a Python 3.14 host. Unrelated to each other, sent as one series since they came out of the same session.

Denis V. Lunev (3):
iotests: run the test pool with the 'fork' start method
iotests: skip FUSE tests when FUSE is not usable iotests/migrate-bitmaps-postcopy-test: drop the postcopy timing
assertion

 tests/qemu-iotests/common.rc                       | 14 ++++++++++++++
 tests/qemu-iotests/testrunner.py                   |  6 ++++--
 tests/qemu-iotests/tests/file-io-error             |  4 +---
 tests/qemu-iotests/tests/fuse-allow-other          |  2 ++
 tests/qemu-iotests/tests/fuse-mmap-shared          |  9 ++++++---
 .../tests/migrate-bitmaps-postcopy-test            |  1 -
 6 files changed, 27 insertions(+), 9 deletions(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH 1/3 iotests: run the test pool with the 'fork' start method

run_tests_pool() shares the runner via the class attribute TestRunner.shared_self, relying on worker processes to inherit it. That only works with the 'fork' start method. Python 3.14 switched the Linux default to 'forkserver', so workers see shared_self as None and parallel runs abort with:

assert runner is not None
AssertionError

Only reproduces with Python 3.14+ and 'check -jN' (N > 1); meson runs one test per process and never calls run_tests_pool(), so CI is unaffected.

Request get_context('fork') explicitly; it is available on all supported Python versions and a no-op before 3.14.

---
 tests/qemu-iotests/testrunner.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/qemu-iotests/testrunner.py +2 -2
index dbe2dddc32..5b2664d9f3 100644
@@ -26,7 +26,7 @@
 import json
 import shutil
 import sys
-from multiprocessing import Pool
+from multiprocessing import Pool, get_context
 from typing import List, Optional, Any, Sequence, Dict
 from testenv import TestEnv
 
@@ -125,7 +125,7 @@ def run_tests_pool(self, tests: List[str],
         assert TestRunner.shared_self is None
         TestRunner.shared_self = self
 
-        with Pool(jobs) as p:
+        with get_context('fork').Pool(jobs) as p:
             results = p.starmap(self.proc_run_test,
                                 zip(tests, [test_field_width] * len(tests)))
 
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH 1/3] iotests: run the test pool with the 'fork' start method

Am 10.07.2026 um 18:13 hat Denis V. Lunev geschrieben:

Quoted 37 lines

run_tests_pool() shares the runner via the class attribute TestRunner.shared_self, relying on worker processes to inherit it. That only works with the 'fork' start method. Python 3.14 switched the Linux default to 'forkserver', so workers see shared_self as None and parallel runs abort with:

assert runner is not None
AssertionError

Only reproduces with Python 3.14+ and 'check -jN' (N > 1); meson runs one test per process and never calls run_tests_pool(), so CI is unaffected.

Request get_context('fork') explicitly; it is available on all supported Python versions and a no-op before 3.14.

---
 tests/qemu-iotests/testrunner.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/qemu-iotests/testrunner.py +1 -1
index dbe2dddc32..5b2664d9f3 100644
@@ -26,7 +26,7 @@
 import json
 import shutil
 import sys
-from multiprocessing import Pool
+from multiprocessing import Pool, get_context
 from typing import List, Optional, Any, Sequence, Dict
 from testenv import TestEnv
 

tests/qemu-iotests/testrunner.py def run_tests_pool(self, tests: List[str], · line 130

Quoted 7 lines
tests/qemu-iotests/testrunner.py +1 -1
@@ -125,7 +125,7 @@ def run_tests_pool(self, tests: List[str],
         assert TestRunner.shared_self is None
         TestRunner.shared_self = self
 
-        with Pool(jobs) as p:
+        with get_context('fork').Pool(jobs) as p:
             results = p.starmap(self.proc_run_test,
                                 zip(tests, [test_field_width] * len(tests)))

Makes sense, but the Pool import is now unused:

=== pylint ===
+************* Module testrunner
+testrunner.py:29:0: W0611: Unused Pool imported from multiprocessing (unused-import)
=== mypy ===

Kevin

#3 lore raw

PATCH 2/3 iotests: skip FUSE tests when FUSE is not usable

In reply to Denis V. Lunev

file-io-error, fuse-allow-other and fuse-mmap-shared skip only when FUSE is not compiled in. When FUSE is built in but unusable at run time (no /dev/fuse access, fusermount lacking permissions), the export fails to mount with "Failed to mount FUSE session to export" and the tests report a spurious failure instead of skipping, like NBD tests already do for missing NBD support.

Add _notrun_on_no_fuse() to common.rc and use it in the shell tests. fuse-mmap-shared is Python, so it gets an equivalent inline check.

---
 tests/qemu-iotests/common.rc              | 14 ++++++++++++++
 tests/qemu-iotests/tests/file-io-error    |  4 +---
 tests/qemu-iotests/tests/fuse-allow-other |  2 ++
 tests/qemu-iotests/tests/fuse-mmap-shared |  9 ++++++---
 4 files changed, 23 insertions(+), 6 deletions(-)
tests/qemu-iotests/common.rc +14 -0
index 298bc483e0..62a2bc146b 100644
@@ -981,6 +981,20 @@ _require_drivers()
     done
 }
 
+# Skip if FUSE is unusable: not compiled in, or the export failed to
+# mount. $1 is the failing 'block-export-add' reply.
+_notrun_on_no_fuse()
+{
+    case "$1" in
+        *"Parameter 'type' does not accept value 'fuse'"*)
+            _notrun "No FUSE support"
+            ;;
+        *"Failed to mount FUSE session"*)
+            _notrun "FUSE not usable in this environment"
+            ;;
+    esac
+}
+
 # Check that we have a file system that allows huge (but very sparse) files
 #
 _require_large_file()
tests/qemu-iotests/tests/file-io-error +1 -3
index fb8db73b31..3cdb41f33d 100755
@@ -89,9 +89,7 @@ output=$(_send_qemu_cmd $QEMU_HANDLE \
     'return' \
     | grep -v 'option allow_other only allowed if')
 
-if echo "$output" | grep -q "Parameter 'type' does not accept value 'fuse'"; then
-    _notrun 'No FUSE support'
-fi
+_notrun_on_no_fuse "$output"
 echo "$output"
 
 echo
tests/qemu-iotests/tests/fuse-allow-other +2 -0
index eaa39f8f23..bcba191951 100755
@@ -77,6 +77,8 @@ fuse_export_add()
         _notrun "allow_other not supported"
     fi
 
+    _notrun_on_no_fuse "$output"
+
     echo "$output"
 }
 
tests/qemu-iotests/tests/fuse-mmap-shared +6 -3
index 52941a3bb6..b190105894 100755
@@ -28,9 +28,12 @@ def test_fuse_support(mount_point):
     })
     test_qsd.stop()
     if 'error' in res:
-        assert (res['error']['desc'] ==
-                "Parameter 'type' does not accept value 'fuse'")
-        iotests.notrun('No FUSE support')
+        desc = res['error']['desc']
+        if desc == "Parameter 'type' does not accept value 'fuse'":
+            iotests.notrun('No FUSE support')
+        if 'Failed to mount FUSE session' in desc:
+            iotests.notrun('FUSE not usable in this environment')
+        assert False, desc
 
 # Shared mmap when using direct IO is only supported for Linux kernels >= 6.6
 # with commit e78662e818f94 ("fuse: add a new fuse init flag to relax
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH 2/3] iotests: skip FUSE tests when FUSE is not usable

Am 10.07.2026 um 18:13 hat Denis V. Lunev geschrieben:

Quoted 25 lines

file-io-error, fuse-allow-other and fuse-mmap-shared skip only when FUSE is not compiled in. When FUSE is built in but unusable at run time (no /dev/fuse access, fusermount lacking permissions), the export fails to mount with "Failed to mount FUSE session to export" and the tests report a spurious failure instead of skipping, like NBD tests already do for missing NBD support.

Add _notrun_on_no_fuse() to common.rc and use it in the shell tests. fuse-mmap-shared is Python, so it gets an equivalent inline check.

---
 tests/qemu-iotests/common.rc              | 14 ++++++++++++++
 tests/qemu-iotests/tests/file-io-error    |  4 +---
 tests/qemu-iotests/tests/fuse-allow-other |  2 ++
 tests/qemu-iotests/tests/fuse-mmap-shared |  9 ++++++---
 4 files changed, 23 insertions(+), 6 deletions(-)
tests/qemu-iotests/common.rc +0 -0
index 298bc483e0..62a2bc146b 100644

tests/qemu-iotests/common.rc _require_drivers() · line 986

Quoted 6 lines
tests/qemu-iotests/common.rc +3 -0
@@ -981,6 +981,20 @@ _require_drivers()
     done
 }
 
+# Skip if FUSE is unusable: not compiled in, or the export failed to
+# mount. $1 is the failing 'block-export-add' reply.
+_notrun_on_no_fuse()

I'm not completely happy with the name, though this is not a blocker.

My first thought was to keep the name consistent with the other functions in common.rc and have something like _require_fuse. On second thoughts, this is probably confusing because it's different from the existing _require_* functions in that it takes an error message from the test case instead of doing its own probing.

So maybe at least remove the double negative? _notrun_on_fuse_error sounds slightly better to me.

tests/qemu-iotests/common.rc _require_drivers() · line 997

Quoted 11 lines
tests/qemu-iotests/common.rc +11 -0
+{
+    case "$1" in
+        *"Parameter 'type' does not accept value 'fuse'"*)
+            _notrun "No FUSE support"
+            ;;
+        *"Failed to mount FUSE session"*)
+            _notrun "FUSE not usable in this environment"
+            ;;
+    esac
+}
+

Adding these checks is a good idea, of course.

Kevin

#5 lore raw

PATCH 3/3 iotests/migrate-bitmaps-postcopy-test: drop the postcopy timing assertion

In reply to Denis V. Lunev

test_postcopy_success asserted downtime * 10 < postcopy_time to confirm the run spent meaningful time in postcopy. This is an unnormalized wall-clock heuristic tuned to old hardware (commit e80a4150a5), and fails on fast hosts where the bitmap payload now transfers in under a second while switch-over downtime stays fixed (e.g. downtime=0.6s, postcopy_time=0.8s).

It is also a weak proxy for the regression it guards against (postcopy degenerating to precopy): that is already covered by the bitmap sha256, mid-migration update and persistence checks, none of which depend on this timing.

Drop the assertion; keep printing the timings under debug.

---
 tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 1 -
 1 file changed, 1 deletion(-)
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test +0 -1
index c519e6db8c..f9c55b6395 100755
@@ -186,7 +186,6 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
         downtime = event_dist(event_stop, event_resume)
         postcopy_time = event_dist(event_resume, event_complete)
 
-        assert downtime * 10 < postcopy_time
         if debug:
             print('downtime:', downtime)
             print('postcopy_time:', postcopy_time)
Signature · 1 line
2.53.0
#6 lore raw

Re: [PATCH 3/3] iotests/migrate-bitmaps-postcopy-test: drop the postcopy timing assertion

Am 10.07.2026 um 18:13 hat Denis V. Lunev geschrieben:

Quoted 12 lines

test_postcopy_success asserted downtime * 10 < postcopy_time to confirm the run spent meaningful time in postcopy. This is an unnormalized wall-clock heuristic tuned to old hardware (commit e80a4150a5), and fails on fast hosts where the bitmap payload now transfers in under a second while switch-over downtime stays fixed (e.g. downtime=0.6s, postcopy_time=0.8s).

It is also a weak proxy for the regression it guards against (postcopy degenerating to precopy): that is already covered by the bitmap sha256, mid-migration update and persistence checks, none of which depend on this timing.

Quoted 5 lines

Drop the assertion; keep printing the timings under debug.

I agree that this assertion failure doesn't mean that the test failed. What it might mean is that maybe we didn't test the code as well as we would have liked to.

Should we try to slow down/throttle the migration artificially to trigger the cases we want to test here more reliably?

Kevin

Quoted 18 lines
 tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 1 -
 1 file changed, 1 deletion(-)
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test +0 -1
index c519e6db8c..f9c55b6395 100755
@@ -186,7 +186,6 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
         downtime = event_dist(event_stop, event_resume)
         postcopy_time = event_dist(event_resume, event_complete)
 
-        assert downtime * 10 < postcopy_time
         if debug:
             print('downtime:', downtime)
             print('postcopy_time:', postcopy_time)
Signature · 1 line
2.53.0
#7 lore raw

Re: [PATCH 3/3] iotests/migrate-bitmaps-postcopy-test: drop the postcopy timing assertion

Quoted 15 lines · Kevin Wolf

Am 10.07.2026 um 18:13 hat Denis V. Lunev geschrieben:

Quoted 14 lines

test_postcopy_success asserted downtime * 10 < postcopy_time to confirm the run spent meaningful time in postcopy. This is an unnormalized wall-clock heuristic tuned to old hardware (commit e80a4150a5), and fails on fast hosts where the bitmap payload now transfers in under a second while switch-over downtime stays fixed (e.g. downtime=0.6s, postcopy_time=0.8s).

It is also a weak proxy for the regression it guards against (postcopy degenerating to precopy): that is already covered by the bitmap sha256, mid-migration update and persistence checks, none of which depend on this timing.

Drop the assertion; keep printing the timings under debug.

Quoted 7 lines
Quoted 3 lines

I agree that this assertion failure doesn't mean that the test failed. What it might mean is that maybe we didn't test the code as well as we would have liked to.

Yes, the aim was to guarantee, that we don't simply pass the whole bitmap in downtime, not starting any postcopy at all.

Possible alternative is to do query-migrate exactly after RESUME, and check that remaining is significantly more than zero: this data is transferred in postcopy for sure.

Quoted 25 lines

Should we try to slow down/throttle the migration artificially to trigger the cases we want to test here more reliably?

Kevin

Quoted 18 lines
  tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 1 -
  1 file changed, 1 deletion(-)
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test +0 -1
index c519e6db8c..f9c55b6395 100755
@@ -186,7 +186,6 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
          downtime = event_dist(event_stop, event_resume)
          postcopy_time = event_dist(event_resume, event_complete)
 
-        assert downtime * 10 < postcopy_time
          if debug:
              print('downtime:', downtime)
              print('postcopy_time:', postcopy_time)
Signature · 1 line
2.53.0
Signature · 2 lines
Best regards,
Vladimir