Skip to contentSkip to conversation

iotests: fix three spurious local failures

PATCH Applied Denis V. Lunev 5 participants 9 messages Jul 15 – Jul 24

lore
Share & tools
lore
#0 lore raw

v2 cover iotests: fix three spurious local failures

Three independent fixes found while chasing a clean './check -qcow2' run on a Python 3.14 host. Unrelated to each other, sent as one series since they came out of the same session.

v2:
- patch 1: drop the now-unused 'Pool' import (Kevin)
- patch 2: rename _notrun_on_no_fuse to _notrun_on_fuse_error (Kevin)
- patch 3: replace the dropped assertion with a check that bitmap0's content on the destination has not reached its final value right after RESUME, throttling migration bandwidth

CC Kevin Wolf <[email protected]>
CC Hanna Reitz <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>

Denis V. Lunev (3):
iotests: run the test pool with the 'fork' start method
iotests: skip FUSE tests when FUSE is not usable iotests/migrate-bitmaps-postcopy-test: replace the timing assertion
with a content check

 tests/qemu-iotests/common.rc                      | 14 ++++++++++++++
 tests/qemu-iotests/testrunner.py                  |  4 ++--
 tests/qemu-iotests/tests/file-io-error            |  4 +---
 tests/qemu-iotests/tests/fuse-allow-other         |  2 ++
 tests/qemu-iotests/tests/fuse-mmap-shared         |  9 ++++++---
 .../tests/migrate-bitmaps-postcopy-test           | 15 ++++++++++++++-
 6 files changed, 39 insertions(+), 9 deletions(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH v2 1/3 iotests: run the test pool with the 'fork' start method

run_tests_pool() shares the runner via the class attribute TestRunner.shared_self, relying on worker processes to inherit it. That only works with the 'fork' start method. Python 3.14 switched the Linux default to 'forkserver', so workers see shared_self as None and parallel runs abort with:

assert runner is not None
AssertionError

Only reproduces with Python 3.14+ and 'check -jN' (N > 1); meson runs one test per process and never calls run_tests_pool(), so CI is unaffected.

Request get_context('fork') explicitly; it is available on all supported Python versions and a no-op before 3.14.

---
 tests/qemu-iotests/testrunner.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/qemu-iotests/testrunner.py +2 -2
index dbe2dddc32..cc36867719 100644
@@ -26,7 +26,7 @@
 import json
 import shutil
 import sys
-from multiprocessing import Pool
+from multiprocessing import get_context
 from typing import List, Optional, Any, Sequence, Dict
 from testenv import TestEnv
 
@@ -125,7 +125,7 @@ def run_tests_pool(self, tests: List[str],
         assert TestRunner.shared_self is None
         TestRunner.shared_self = self
 
-        with Pool(jobs) as p:
+        with get_context('fork').Pool(jobs) as p:
             results = p.starmap(self.proc_run_test,
                                 zip(tests, [test_field_width] * len(tests)))
 
Signature · 1 line
2.53.0
Reviewed-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] iotests: run the test pool with the 'fork' start method

Quoted 16 lines · Denis V. Lunev

run_tests_pool() shares the runner via the class attribute TestRunner.shared_self, relying on worker processes to inherit it. That only works with the 'fork' start method. Python 3.14 switched the Linux default to 'forkserver', so workers see shared_self as None and parallel runs abort with:

assert runner is not None
AssertionError

Only reproduces with Python 3.14+ and 'check -jN' (N > 1); meson runs one test per process and never calls run_tests_pool(), so CI is unaffected.

Request get_context('fork') explicitly; it is available on all supported Python versions and a no-op before 3.14.

Quoted 6 lines
---
 tests/qemu-iotests/testrunner.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
#3 lore raw

PATCH v2 2/3 iotests: skip FUSE tests when FUSE is not usable

In reply to Denis V. Lunev

file-io-error, fuse-allow-other and fuse-mmap-shared skip only when FUSE is not compiled in. When FUSE is built in but unusable at run time (no /dev/fuse access, fusermount lacking permissions), the export fails to mount with "Failed to mount FUSE session to export" and the tests report a spurious failure instead of skipping, like NBD tests already do for missing NBD support.

Add _notrun_on_fuse_error() to common.rc and use it in the shell tests. fuse-mmap-shared is Python, so it gets an equivalent inline check.

---
 tests/qemu-iotests/common.rc              | 14 ++++++++++++++
 tests/qemu-iotests/tests/file-io-error    |  4 +---
 tests/qemu-iotests/tests/fuse-allow-other |  2 ++
 tests/qemu-iotests/tests/fuse-mmap-shared |  9 ++++++---
 4 files changed, 23 insertions(+), 6 deletions(-)
tests/qemu-iotests/common.rc +14 -0
index 298bc483e0..bcb1ec50a9 100644
@@ -981,6 +981,20 @@ _require_drivers()
     done
 }
 
+# Skip if FUSE is unusable: not compiled in, or the export failed to
+# mount. $1 is the failing 'block-export-add' reply.
+_notrun_on_fuse_error()
+{
+    case "$1" in
+        *"Parameter 'type' does not accept value 'fuse'"*)
+            _notrun "No FUSE support"
+            ;;
+        *"Failed to mount FUSE session"*)
+            _notrun "FUSE not usable in this environment"
+            ;;
+    esac
+}
+
 # Check that we have a file system that allows huge (but very sparse) files
 #
 _require_large_file()
tests/qemu-iotests/tests/file-io-error +1 -3
index fb8db73b31..0d970c102f 100755
@@ -89,9 +89,7 @@ output=$(_send_qemu_cmd $QEMU_HANDLE \
     'return' \
     | grep -v 'option allow_other only allowed if')
 
-if echo "$output" | grep -q "Parameter 'type' does not accept value 'fuse'"; then
-    _notrun 'No FUSE support'
-fi
+_notrun_on_fuse_error "$output"
 echo "$output"
 
 echo
tests/qemu-iotests/tests/fuse-allow-other +2 -0
index eaa39f8f23..50a36601d7 100755
@@ -77,6 +77,8 @@ fuse_export_add()
         _notrun "allow_other not supported"
     fi
 
+    _notrun_on_fuse_error "$output"
+
     echo "$output"
 }
 
tests/qemu-iotests/tests/fuse-mmap-shared +6 -3
index 52941a3bb6..b190105894 100755
@@ -28,9 +28,12 @@ def test_fuse_support(mount_point):
     })
     test_qsd.stop()
     if 'error' in res:
-        assert (res['error']['desc'] ==
-                "Parameter 'type' does not accept value 'fuse'")
-        iotests.notrun('No FUSE support')
+        desc = res['error']['desc']
+        if desc == "Parameter 'type' does not accept value 'fuse'":
+            iotests.notrun('No FUSE support')
+        if 'Failed to mount FUSE session' in desc:
+            iotests.notrun('FUSE not usable in this environment')
+        assert False, desc
 
 # Shared mmap when using direct IO is only supported for Linux kernels >= 6.6
 # with commit e78662e818f94 ("fuse: add a new fuse init flag to relax
Signature · 1 line
2.53.0
Reviewed-by on v2 2/3
#4 lore raw

Re: [PATCH v2 2/3] iotests: skip FUSE tests when FUSE is not usable

Quoted 14 lines · Denis V. Lunev

file-io-error, fuse-allow-other and fuse-mmap-shared skip only when FUSE is not compiled in. When FUSE is built in but unusable at run time (no /dev/fuse access, fusermount lacking permissions), the export fails to mount with "Failed to mount FUSE session to export" and the tests report a spurious failure instead of skipping, like NBD tests already do for missing NBD support.

Add _notrun_on_fuse_error() to common.rc and use it in the shell tests. fuse-mmap-shared is Python, so it gets an equivalent inline check.

Quoted 6 lines
---
 tests/qemu-iotests/common.rc              | 14 ++++++++++++++
 tests/qemu-iotests/tests/file-io-error    |  4 +---
 tests/qemu-iotests/tests/fuse-allow-other |  2 ++
 tests/qemu-iotests/tests/fuse-mmap-shared |  9 ++++++---
 4 files changed, 23 insertions(+), 6 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
#5 lore raw

PATCH v2 3/3 iotests/migrate-bitmaps-postcopy-test: replace the timing assertion with a content check

In reply to Denis V. Lunev

downtime * 10 < postcopy_time was an unnormalized wall-clock heuristic (commit e80a4150a5) that fails on fast hosts, where the bitmap payload now transfers in under a second.

Check the actual invariant instead: right after RESUME, bitmap0's content hash on the destination must not yet match the fully migrated value. Throttle max-bandwidth first, since all-zero chunks skip the payload write and would otherwise let a fast host finish the transfer before the check runs.

CC Kevin Wolf <[email protected]>
CC Hanna Reitz <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>
---
 .../tests/migrate-bitmaps-postcopy-test           | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test +14 -1
index c519e6db8c..33ff2b861f 100755
@@ -160,12 +160,26 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
 
         self.vm_b.cmd('migrate-set-capabilities', capabilities=caps)
 
+        # Throttle so the chunks covering our discards (the only ones
+        # not skipped by the all-zero fast path) can't outrun the check
+        # below.
+        self.vm_a.cmd('migrate-set-parameters', max_bandwidth=16536)
+
         self.vm_a.cmd('migrate', uri='exec:cat>' + fifo)
 
         self.vm_a.cmd('migrate-start-postcopy')
 
         event_resume = self.vm_b.event_wait('RESUME')
         self.vm_b_events.append(event_resume)
+
+        # bitmap0 can't already have its final content: that requires
+        # the bit data to have actually arrived.
+        result = self.vm_b.qmp('x-debug-block-dirty-bitmap-sha256',
+                               node='drive0', name='bitmap0')
+        assert result['return']['sha256'] != all_discards_sha256
+
+        self.vm_a.cmd('migrate-set-parameters', max_bandwidth=0)
+
         return (event_resume, discards1_sha256, all_discards_sha256)
 
     def test_postcopy_success(self):
@@ -186,7 +200,6 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
         downtime = event_dist(event_stop, event_resume)
         postcopy_time = event_dist(event_resume, event_complete)
 
-        assert downtime * 10 < postcopy_time
         if debug:
             print('downtime:', downtime)
             print('postcopy_time:', postcopy_time)
Signature · 1 line
2.53.0
Reviewed-by on v2 3/3
#6 lore raw

Re: [PATCH v2 3/3] iotests/migrate-bitmaps-postcopy-test: replace the timing assertion with a content check

Quoted 10 lines · Denis V. Lunev

downtime * 10 < postcopy_time was an unnormalized wall-clock heuristic (commit e80a4150a5) that fails on fast hosts, where the bitmap payload now transfers in under a second.

Check the actual invariant instead: right after RESUME, bitmap0's content hash on the destination must not yet match the fully migrated value. Throttle max-bandwidth first, since all-zero chunks skip the payload write and would otherwise let a fast host finish the transfer before the check runs.

Quoted 4 lines
CC Kevin Wolf<[email protected]>
CC Hanna Reitz<[email protected]>
CC Vladimir Sementsov-Ogievskiy<[email protected]>
Signature · 2 lines
Best regards,
Vladimir
#7 lore raw
In reply to Denis V. Lunev
Quoted 24 lines · Denis V. Lunev

This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Three independent fixes found while chasing a clean './check -qcow2'
run on a Python 3.14 host. Unrelated to each other, sent as one
series since they came out of the same session.

v2:
- patch 1: drop the now-unused 'Pool' import (Kevin)
- patch 2: rename _notrun_on_no_fuse to _notrun_on_fuse_error (Kevin)
- patch 3: replace the dropped assertion with a check that bitmap0's content on the destination has not reached its final value right after RESUME, throttling migration bandwidth

CC Kevin Wolf <[email protected]>
CC Hanna Reitz <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>

Denis V. Lunev (3):
iotests: run the test pool with the 'fork' start method
iotests: skip FUSE tests when FUSE is not usable iotests/migrate-bitmaps-postcopy-test: replace the timing assertion
with a content check

Quoted 8 lines
 tests/qemu-iotests/common.rc                      | 14 ++++++++++++++
 tests/qemu-iotests/testrunner.py                  |  4 ++--
 tests/qemu-iotests/tests/file-io-error            |  4 +---
 tests/qemu-iotests/tests/fuse-allow-other         |  2 ++
 tests/qemu-iotests/tests/fuse-mmap-shared         |  9 ++++++---
 .../tests/migrate-bitmaps-postcopy-test           | 15 ++++++++++++++-
 6 files changed, 39 insertions(+), 9 deletions(-)

ping

Applied
#8 lore raw
In reply to Denis V. Lunev

Am 15.07.2026 um 12:34 hat Denis V. Lunev geschrieben:

Quoted 10 lines

Three independent fixes found while chasing a clean './check -qcow2' run on a Python 3.14 host. Unrelated to each other, sent as one series since they came out of the same session.

v2:
- patch 1: drop the now-unused 'Pool' import (Kevin)
- patch 2: rename _notrun_on_no_fuse to _notrun_on_fuse_error (Kevin)
- patch 3: replace the dropped assertion with a check that bitmap0's content on the destination has not reached its final value right after RESUME, throttling migration bandwidth

Thanks, applied to the block branch.

Kevin