Skip to contentSkip to conversation

monitor: only run coroutine commands in qemu_aio_context

PATCH Superseded Stefan Hajnoczi 7 participants 17 messages Jan 16, 2024 – Feb 13, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover monitor: only run coroutine commands in qemu_aio_context

Several bugs have been reported related to how QMP commands are rescheduled in qemu_aio_context:
- https://gitlab.com/qemu-project/qemu/-/issues/1933
- https://issues.redhat.com/browse/RHEL-17369
- https://bugzilla.redhat.com/show_bug.cgi?id=2215192
- https://bugzilla.redhat.com/show_bug.cgi?id=2214985

The first instance of the bug interacted with drain_call_rcu() temporarily dropping the BQL and resulted in vCPU threads entering device emulation code simultaneously (something that should never happen). I set out to make drain_call_rcu() safe to use in this environment, but Paolo and Kevin discussed the possibility of avoiding rescheduling the monitor_qmp_dispatcher_co() coroutine for non-coroutine commands. This would prevent monitor commands from running during vCPU thread aio_poll() entirely and addresses the root cause.

This patch series implements this idea. qemu-iotests is sensitive to the exact order in which QMP events and responses are emitted. Running QMP handlers in the iohandler AioContext causes some QMP events to be ordered differently than before. It is therefore necessary to adjust the reference output in many test cases. The actual QMP code change is small and everything else is just to make qemu-iotests happy.

If you have bugs related to the same issue, please retest them with these patches. Thanks!

Stefan Hajnoczi (3):
iotests: add filter_qmp_generated_node_ids()
iotests: port 141 to Python for reliable QMP testing
monitor: only run coroutine commands in qemu_aio_context

 monitor/qmp.c                                 |  17 -
 qapi/qmp-dispatch.c                           |  24 +-
 tests/qemu-iotests/060.out                    |   4 +-
 tests/qemu-iotests/071.out                    |   4 +-
 tests/qemu-iotests/081.out                    |  16 +-
 tests/qemu-iotests/087.out                    |  12 +-
 tests/qemu-iotests/108.out                    |   2 +-
 tests/qemu-iotests/109                        |   4 +-
 tests/qemu-iotests/109.out                    |  78 ++---
 tests/qemu-iotests/117.out                    |   2 +-
 tests/qemu-iotests/120.out                    |   2 +-
 tests/qemu-iotests/127.out                    |   2 +-
 tests/qemu-iotests/140.out                    |   2 +-
 tests/qemu-iotests/141                        | 297 +++++++-----------
 tests/qemu-iotests/141.out                    | 190 +++--------
 tests/qemu-iotests/143.out                    |   2 +-
 tests/qemu-iotests/156.out                    |   2 +-
 tests/qemu-iotests/176.out                    |  16 +-
 tests/qemu-iotests/182.out                    |   2 +-
 tests/qemu-iotests/183.out                    |   4 +-
 tests/qemu-iotests/184.out                    |  32 +-
 tests/qemu-iotests/185                        |   6 +-
 tests/qemu-iotests/185.out                    |  45 ++-
 tests/qemu-iotests/191.out                    |  16 +-
 tests/qemu-iotests/195.out                    |  16 +-
 tests/qemu-iotests/223.out                    |  16 +-
 tests/qemu-iotests/227.out                    |  32 +-
 tests/qemu-iotests/247.out                    |   2 +-
 tests/qemu-iotests/273.out                    |   8 +-
 tests/qemu-iotests/308                        |   4 +-
 tests/qemu-iotests/308.out                    |   4 +-
 tests/qemu-iotests/iotests.py                 |   7 +
 tests/qemu-iotests/tests/file-io-error        |   5 +-
 tests/qemu-iotests/tests/iothreads-resize.out |   2 +-
 tests/qemu-iotests/tests/qsd-jobs.out         |   4 +-
 35 files changed, 375 insertions(+), 506 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/3 iotests: add filter_qmp_generated_node_ids()

Add a filter function for QMP responses that contain QEMU's automatically generated node ids. The ids change between runs and must be masked in the reference output.

The next commit will use this new function.

---
 tests/qemu-iotests/iotests.py | 7 +++++++
 1 file changed, 7 insertions(+)
tests/qemu-iotests/iotests.py +7 -0
index e5c5798c71..ea48af4a7b 100644
@@ -651,6 +651,13 @@ def _filter(_key, value):
 def filter_generated_node_ids(msg):
     return re.sub("#block[0-9]+", "NODE_NAME", msg)
 
+def filter_qmp_generated_node_ids(qmsg):
+    def _filter(_key, value):
+        if is_str(value):
+            return filter_generated_node_ids(value)
+        return value
+    return filter_qmp(qmsg, _filter)
+
 def filter_img_info(output: str, filename: str,
                     drop_child_info: bool = True) -> str:
     lines = []
Signature · 1 line
2.43.0
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] iotests: add filter_qmp_generated_node_ids()

Am 16.01.2024 um 20:00 hat Stefan Hajnoczi geschrieben:

Quoted 7 lines

Add a filter function for QMP responses that contain QEMU's automatically generated node ids. The ids change between runs and must be masked in the reference output.

The next commit will use this new function.

#3 lore raw

PATCH 2/3 iotests: port 141 to Python for reliable QMP testing

In reply to Stefan Hajnoczi

The common.qemu bash functions allow tests to interact with the QMP monitor of a QEMU process. I spent two days trying to update 141 when the order of the test output changed, but found it would still fail occassionally because printf() and QMP events race with synchronous QMP communication.

I gave up and ported 141 to the existing Python API for QMP tests. The Python API is less affected by the order in which QEMU prints output because it does not print all QMP traffic by default.

The next commit changes the order in which QMP messages are received. Make 141 reliable first.

---
 tests/qemu-iotests/141     | 297 +++++++++++++++----------------------
 tests/qemu-iotests/141.out | 190 +++++-------------------
 2 files changed, 161 insertions(+), 326 deletions(-)
tests/qemu-iotests/141 +121 -176
index a37030ee17..4248d7c696 100755
@@ -1,9 +1,12 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
 # group: rw auto quick
 #
 # Test case for ejecting BDSs with block jobs still running on them
 #
-# Copyright (C) 2016 Red Hat, Inc.
+# Originally written in bash by Hanna Czenczek, ported to Python by Stefan
+# Hajnoczi.
+#
+# Copyright Red Hat
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -19,177 +22,119 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-# creator
[email protected]
-
-seq="$(basename $0)"
-echo "QA output created by $seq"
-
-status=1	# failure is the default!
-
-_cleanup()
-{
-    _cleanup_qemu
-    _cleanup_test_img
-    for img in "$TEST_DIR"/{b,m,o}.$IMGFMT; do
-        _rm_test_img "$img"
-    done
-}
-trap "_cleanup; exit \$status" 0 1 2 3 15
-
-# get standard environment, filters and checks
-. ./common.rc
-. ./common.filter
-. ./common.qemu
-
-# Needs backing file and backing format support
-_supported_fmt qcow2 qed
-_supported_proto file
-_supported_os Linux
-
-
-test_blockjob()
-{
-    _send_qemu_cmd $QEMU_HANDLE \
-        "{'execute': 'blockdev-add',
-          'arguments': {
-              'node-name': 'drv0',
-              'driver': '$IMGFMT',
-              'file': {
-                  'driver': 'file',
-                  'filename': '$TEST_IMG'
-              }}}" \
-        'return'
-
-    # If "$2" is an event, we may or may not see it before the
-    # {"return": {}}.  Therefore, filter the {"return": {}} out both
-    # here and in the next command.  (Naturally, if we do not see it
-    # here, we will see it before the next command can be executed,
-    # so it will appear in the next _send_qemu_cmd's output.)
-    _send_qemu_cmd $QEMU_HANDLE \
-        "$1" \
-        "$2" \
-        | _filter_img_create | _filter_qmp_empty_return
-
-    # We want this to return an error because the block job is still running
-    _send_qemu_cmd $QEMU_HANDLE \
-        "{'execute': 'blockdev-del',
-          'arguments': {'node-name': 'drv0'}}" \
-        'error' | _filter_generated_node_ids | _filter_qmp_empty_return
-
-    _send_qemu_cmd $QEMU_HANDLE \
-        "{'execute': 'block-job-cancel',
-          'arguments': {'device': 'job0'}}" \
-        "$3"
-
-    _send_qemu_cmd $QEMU_HANDLE \
-        "{'execute': 'blockdev-del',
-          'arguments': {'node-name': 'drv0'}}" \
-        'return'
-}
-
-
-TEST_IMG="$TEST_DIR/b.$IMGFMT" _make_test_img 1M
-TEST_IMG="$TEST_DIR/m.$IMGFMT" _make_test_img -b "$TEST_DIR/b.$IMGFMT" -F $IMGFMT 1M
-_make_test_img -b "$TEST_DIR/m.$IMGFMT" 1M -F $IMGFMT
-
-_launch_qemu -nodefaults
-
-_send_qemu_cmd $QEMU_HANDLE \
-    "{'execute': 'qmp_capabilities'}" \
-    'return'
-
-echo
-echo '=== Testing drive-backup ==='
-echo
-
-# drive-backup will not send BLOCK_JOB_READY by itself, and cancelling the job
-# will consequently result in BLOCK_JOB_CANCELLED being emitted.
-
-test_blockjob \
-    "{'execute': 'drive-backup',
-      'arguments': {'job-id': 'job0',
-                    'device': 'drv0',
-                    'target': '$TEST_DIR/o.$IMGFMT',
-                    'format': '$IMGFMT',
-                    'sync': 'none'}}" \
-    'return' \
-    '"status": "null"'
-
-echo
-echo '=== Testing drive-mirror ==='
-echo
-
-# drive-mirror will send BLOCK_JOB_READY basically immediately, and cancelling
-# the job will consequently result in BLOCK_JOB_COMPLETED being emitted.
-
-test_blockjob \
-    "{'execute': 'drive-mirror',
-      'arguments': {'job-id': 'job0',
-                    'device': 'drv0',
-                    'target': '$TEST_DIR/o.$IMGFMT',
-                    'format': '$IMGFMT',
-                    'sync': 'none'}}" \
-    'BLOCK_JOB_READY' \
-    '"status": "null"'
-
-echo
-echo '=== Testing active block-commit ==='
-echo
-
-# An active block-commit will send BLOCK_JOB_READY basically immediately, and
-# cancelling the job will consequently result in BLOCK_JOB_COMPLETED being
-# emitted.
-
-test_blockjob \
-    "{'execute': 'block-commit',
-      'arguments': {'job-id': 'job0', 'device': 'drv0'}}" \
-    'BLOCK_JOB_READY' \
-    '"status": "null"'
-
-echo
-echo '=== Testing non-active block-commit ==='
-echo
-
-# Give block-commit something to work on, otherwise it would be done
-# immediately, send a BLOCK_JOB_COMPLETED and ejecting the BDS would work just
-# fine without the block job still running.
-
-$QEMU_IO -c 'write 0 1M' "$TEST_DIR/m.$IMGFMT" | _filter_qemu_io
-
-test_blockjob \
-    "{'execute': 'block-commit',
-      'arguments': {'job-id': 'job0',
-                    'device': 'drv0',
-                    'top':    '$TEST_DIR/m.$IMGFMT',
-                    'speed':  1}}" \
-    'return' \
-    '"status": "null"'
-
-echo
-echo '=== Testing block-stream ==='
-echo
-
-# Give block-stream something to work on, otherwise it would be done
-# immediately, send a BLOCK_JOB_COMPLETED and ejecting the BDS would work just
-# fine without the block job still running.
-
-$QEMU_IO -c 'write 0 1M' "$TEST_DIR/b.$IMGFMT" | _filter_qemu_io
-
-# With some data to stream (and @speed set to 1), block-stream will not complete
-# until we send the block-job-cancel command.
-
-test_blockjob \
-    "{'execute': 'block-stream',
-      'arguments': {'job-id': 'job0',
-                    'device': 'drv0',
-                    'speed': 1}}" \
-    'return' \
-    '"status": "null"'
-
-_cleanup_qemu
-
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+import iotests
+
+
+class TestCase:
+    def __init__(self, name, vm, image_path, cancel_event):
+        self.name = name
+        self.vm = vm
+        self.image_path = image_path
+        self.cancel_event = cancel_event
+
+    def __enter__(self):
+        iotests.log(f'=== Testing {self.name} ===')
+        self.vm.qmp_log('blockdev-add', \
+                        node_name='drv0', \
+                        driver=iotests.imgfmt, \
+                        file={'driver': 'file', 'filename': self.image_path}, \
+                        filters=[iotests.filter_qmp_testfiles])
+
+    def __exit__(self, *exc_details):
+        # This is expected to fail because the job still exists
+        self.vm.qmp_log('blockdev-del', node_name='drv0', \
+                        filters=[iotests.filter_qmp_generated_node_ids])
+
+        self.vm.qmp_log('block-job-cancel', device='job0')
+        event = self.vm.event_wait(self.cancel_event)
+        iotests.log(event, filters=[iotests.filter_qmp_event])
+
+        # This time it succeeds
+        self.vm.qmp_log('blockdev-del', node_name='drv0')
+
+        # Separate test cases in output
+        iotests.log('')
+
+
+def main() -> None:
+    with iotests.FilePath('bottom', 'middle', 'top', 'target') as \
+            (bottom_path, middle_path, top_path, target_path), \
+         iotests.VM() as vm:
+
+        iotests.log('Creating bottom <- middle <- top backing file chain...')
+        IMAGE_SIZE='1M'
+        iotests.qemu_img_create('-f', iotests.imgfmt, bottom_path, IMAGE_SIZE)
+        iotests.qemu_img_create('-f', iotests.imgfmt, \
+                                '-F', iotests.imgfmt, \
+                                '-b', bottom_path, \
+                                middle_path, \
+                                IMAGE_SIZE)
+        iotests.qemu_img_create('-f', iotests.imgfmt, \
+                                '-F', iotests.imgfmt, \
+                                '-b', middle_path, \
+                                top_path, \
+                                IMAGE_SIZE)
+
+        iotests.log('Starting VM...')
+        vm.add_args('-nodefaults')
+        vm.launch()
+
+        # drive-backup will not send BLOCK_JOB_READY by itself, and cancelling the job
+        # will consequently result in BLOCK_JOB_CANCELLED being emitted.
+        with TestCase('drive-backup', vm, top_path, 'BLOCK_JOB_CANCELLED'):
+            vm.qmp_log('drive-backup', \
+                       job_id='job0', \
+                       device='drv0', \
+                       target=target_path, \
+                       format=iotests.imgfmt, \
+                       sync='none', \
+                       filters=[iotests.filter_qmp_testfiles])
+
+        # drive-mirror will send BLOCK_JOB_READY basically immediately, and cancelling
+        # the job will consequently result in BLOCK_JOB_COMPLETED being emitted.
+        with TestCase('drive-mirror', vm, top_path, 'BLOCK_JOB_COMPLETED'):
+            vm.qmp_log('drive-mirror', \
+                       job_id='job0', \
+                       device='drv0', \
+                       target=target_path, \
+                       format=iotests.imgfmt, \
+                       sync='none', \
+                       filters=[iotests.filter_qmp_testfiles])
+            event = vm.event_wait('BLOCK_JOB_READY')
+            iotests.log(event, filters=[iotests.filter_qmp_event])
+
+        # An active block-commit will send BLOCK_JOB_READY basically immediately, and
+        # cancelling the job will consequently result in BLOCK_JOB_COMPLETED being
+        # emitted.
+        with TestCase('active block-commit', vm, top_path, 'BLOCK_JOB_COMPLETED'):
+            vm.qmp_log('block-commit', \
+                       job_id='job0', \
+                       device='drv0')
+            event = vm.event_wait('BLOCK_JOB_READY')
+            iotests.log(event, filters=[iotests.filter_qmp_event])
+
+        # Give block-commit something to work on, otherwise it would be done
+        # immediately, send a BLOCK_JOB_COMPLETED and ejecting the BDS would work just
+        # fine without the block job still running.
+        iotests.qemu_io(middle_path, '-c', f'write 0 {IMAGE_SIZE}')
+        with TestCase('non-active block-commit', vm, top_path, 'BLOCK_JOB_CANCELLED'):
+            vm.qmp_log('block-commit', \
+                       job_id='job0', \
+                       device='drv0', \
+                       top=middle_path, \
+                       speed=1, \
+                       filters=[iotests.filter_qmp_testfiles])
+
+        # Give block-stream something to work on, otherwise it would be done
+        # immediately, send a BLOCK_JOB_COMPLETED and ejecting the BDS would work just
+        # fine without the block job still running.
+        iotests.qemu_io(bottom_path, '-c', f'write 0 {IMAGE_SIZE}')
+        with TestCase('block-stream', vm, top_path, 'BLOCK_JOB_CANCELLED'):
+            vm.qmp_log('block-stream', \
+                       job_id='job0', \
+                       device='drv0', \
+                       speed=1)
+
+if __name__ == '__main__':
+    iotests.script_main(main, supported_fmts=['qcow2', 'qed'],
+                        supported_protocols=['file'])
tests/qemu-iotests/141.out +40 -150

219 lines not on this page · raw message

Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH 2/3] iotests: port 141 to Python for reliable QMP testing

Am 16.01.2024 um 20:00 hat Stefan Hajnoczi geschrieben:

Quoted 31 lines

The common.qemu bash functions allow tests to interact with the QMP monitor of a QEMU process. I spent two days trying to update 141 when the order of the test output changed, but found it would still fail occassionally because printf() and QMP events race with synchronous QMP communication.

I gave up and ported 141 to the existing Python API for QMP tests. The Python API is less affected by the order in which QEMU prints output because it does not print all QMP traffic by default.

The next commit changes the order in which QMP messages are received. Make 141 reliable first.

tests/qemu-iotests/141.out +2 -7
index 63203d9944..986f9f7598 100644
@@ -1,179 +1,69 @@
-QA output created by 141
-Formatting 'TEST_DIR/b.IMGFMT', fmt=IMGFMT size=1048576
-Formatting 'TEST_DIR/m.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/b.IMGFMT backing_fmt=IMGFMT
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/m.IMGFMT backing_fmt=IMGFMT
-{'execute': 'qmp_capabilities'}
-{"return": {}}
-
+Creating bottom <- middle <- top backing file chain...
+Starting VM...
 === Testing drive-backup ===

tests/qemu-iotests/141.out line 4

Quoted 10 lines
tests/qemu-iotests/141.out +1 -9
-
-{'execute': 'blockdev-add',
-          'arguments': {
-              'node-name': 'drv0',
-              'driver': 'IMGFMT',
-              'file': {
-                  'driver': 'file',
-                  'filename': 'TEST_DIR/t.IMGFMT'
-              }}}
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-top"}, "node-name": "drv0"}}

This and other lines in the test output need image format filtering. You have a literal "qcow2" in there, but the test case also runs (and fails because of this) for qed.

Apart from this, the patch looks good to me.

Kevin

#5 lore raw

Re: [PATCH 2/3] iotests: port 141 to Python for reliable QMP testing

In reply to Stefan Hajnoczi

Am 16.01.2024 um 20:00 hat Stefan Hajnoczi geschrieben:

Quoted 10 lines

The common.qemu bash functions allow tests to interact with the QMP monitor of a QEMU process. I spent two days trying to update 141 when the order of the test output changed, but found it would still fail occassionally because printf() and QMP events race with synchronous QMP communication.

I gave up and ported 141 to the existing Python API for QMP tests. The Python API is less affected by the order in which QEMU prints output because it does not print all QMP traffic by default.

Quoted 5 lines

The next commit changes the order in which QMP messages are received. Make 141 reliable first.

iotests 297 finds problems with the new Python code:

=== pylint ===
+************* Module 141
+141:82:0: C0301: Line too long (86/79) (line-too-long) +141:93:0: C0301: Line too long (86/79) (line-too-long) +141:94:0: C0301: Line too long (80/79) (line-too-long) +141:106:0: C0301: Line too long (85/79) (line-too-long) +141:107:0: C0301: Line too long (82/79) (line-too-long) +141:109:0: C0301: Line too long (82/79) (line-too-long) +141:117:0: C0301: Line too long (86/79) (line-too-long) +141:120:0: C0301: Line too long (86/79) (line-too-long) +141:129:0: C0301: Line too long (86/79) (line-too-long)
=== mypy ===

+141:104: error: Value of type variable "Msg" of "log" cannot be "dict[str, Any] | None"  [type-var]
+141:114: error: Value of type variable "Msg" of "log" cannot be "dict[str, Any] | None"  [type-var]

+Found 2 errors in 1 file (checked 37 source files)

Kevin

#6 lore raw

Re: [PATCH 2/3] iotests: port 141 to Python for reliable QMP testing

Quoted 24 lines · Kevin Wolf

Am 16.01.2024 um 20:00 hat Stefan Hajnoczi geschrieben:

Quoted 10 lines

The common.qemu bash functions allow tests to interact with the QMP monitor of a QEMU process. I spent two days trying to update 141 when the order of the test output changed, but found it would still fail occassionally because printf() and QMP events race with synchronous QMP communication.

I gave up and ported 141 to the existing Python API for QMP tests. The Python API is less affected by the order in which QEMU prints output because it does not print all QMP traffic by default.

Quoted 5 lines

The next commit changes the order in which QMP messages are received. Make 141 reliable first.

iotests 297 finds problems with the new Python code:

=== pylint ===

(quoted diff) +4 -0
+************* Module 141
+141:82:0: C0301: Line too long (86/79) (line-too-long)
+141:93:0: C0301: Line too long (86/79) (line-too-long)
+141:94:0: C0301: Line too long (80/79) (line-too-long)

Review

Quoted 10 lines
(quoted diff) +9 -0
+141:106:0: C0301: Line too long (85/79) (line-too-long)
+141:107:0: C0301: Line too long (82/79) (line-too-long)
+141:109:0: C0301: Line too long (82/79) (line-too-long)
+141:117:0: C0301: Line too long (86/79) (line-too-long)
+141:120:0: C0301: Line too long (86/79) (line-too-long)
+141:129:0: C0301: Line too long (86/79) (line-too-long)
 === mypy ===
+141:104: error: Value of type variable "Msg" of "log" cannot be "dict[str, Any] | None"  [type-var]
+141:114: error: Value of type variable "Msg" of "log" cannot be "dict[str, Any] | None"  [type-var]
+Found 2 errors in 1 file (checked 37 source files)

I'm not sure how to run this but I manually ran pylint and mypy and fixed those errors. Can you share how you ran this?

The mypy errors are because the type system cannot express that event_wait() only returns None when timeout=0. That's not the case in 141 but mypy complains anyway, so I added an assertion that event is not None to silence mypy.

Thanks,
Stefan

#7 lore raw

Re: [PATCH 2/3] iotests: port 141 to Python for reliable QMP testing

Am 18.01.2024 um 15:55 hat Stefan Hajnoczi geschrieben:

Quoted 30 lines · Kevin Wolf
Quoted 29 lines

Am 16.01.2024 um 20:00 hat Stefan Hajnoczi geschrieben:

Quoted 10 lines

The common.qemu bash functions allow tests to interact with the QMP monitor of a QEMU process. I spent two days trying to update 141 when the order of the test output changed, but found it would still fail occassionally because printf() and QMP events race with synchronous QMP communication.

I gave up and ported 141 to the existing Python API for QMP tests. The Python API is less affected by the order in which QEMU prints output because it does not print all QMP traffic by default.

Quoted 5 lines

The next commit changes the order in which QMP messages are received. Make 141 reliable first.

iotests 297 finds problems with the new Python code:

=== pylint ===
+************* Module 141
+141:82:0: C0301: Line too long (86/79) (line-too-long) +141:93:0: C0301: Line too long (86/79) (line-too-long) +141:94:0: C0301: Line too long (80/79) (line-too-long) +141:106:0: C0301: Line too long (85/79) (line-too-long) +141:107:0: C0301: Line too long (82/79) (line-too-long) +141:109:0: C0301: Line too long (82/79) (line-too-long) +141:117:0: C0301: Line too long (86/79) (line-too-long) +141:120:0: C0301: Line too long (86/79) (line-too-long)

Quoted 8 lines
Quoted 5 lines

+141:129:0: C0301: Line too long (86/79) (line-too-long)
=== mypy ===

+141:104: error: Value of type variable "Msg" of "log" cannot be "dict[str, Any] | None"  [type-var]
+141:114: error: Value of type variable "Msg" of "log" cannot be "dict[str, Any] | None"  [type-var]

+Found 2 errors in 1 file (checked 37 source files)

I'm not sure how to run this but I manually ran pylint and mypy and fixed those errors. Can you share how you ran this?

I already answered this in IRC, but for the record: This is from running qemu-iotests 297, which is kind of a meta-test that runs the linters on Python files in qemu-iotests.

Quoted 4 lines

The mypy errors are because the type system cannot express that event_wait() only returns None when timeout=0. That's not the case in 141 but mypy complains anyway, so I added an assertion that event is not None to silence mypy.

Sounds like the right solution to me.

Kevin

#8 lore raw

PATCH 3/3 monitor: only run coroutine commands in qemu_aio_context

In reply to Stefan Hajnoczi

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not polled during nested event loops. The coroutine currently reschedules itself in the main loop's qemu_aio_context AioContext, which is polled during nested event loops. One known problem is that QMP device-add calls drain_call_rcu(), which temporarily drops the BQL, leading to all sorts of havoc like other vCPU threads re-entering device emulation code while another vCPU thread is waiting in device emulation code with aio_poll().

Paolo Bonzini suggested running non-coroutine QMP handlers in the iohandler AioContext. This avoids trouble with nested event loops. His original idea was to move coroutine rescheduling to monitor_qmp_dispatch(), but I resorted to moving it to qmp_dispatch() because we don't know if the QMP handler needs to run in coroutine context in monitor_qmp_dispatch(). monitor_qmp_dispatch() would have been nicer since it's associated with the monitor implementation and not as general as qmp_dispatch(), which is also used by qemu-ga.

A number of qemu-iotests need updated .out files because the order of QMP events vs QMP responses has changed.

Solves Issue #1933.

Fixes 7bed89958bfbf40df9ca681cefbdca63abdde39d ("device_core: use drain_call_rcu in in qmp_device_add")
---
 monitor/qmp.c                                 | 17 ----
 qapi/qmp-dispatch.c                           | 24 +++++-
 tests/qemu-iotests/060.out                    |  4 +-
 tests/qemu-iotests/071.out                    |  4 +-
 tests/qemu-iotests/081.out                    | 16 ++--
 tests/qemu-iotests/087.out                    | 12 +--
 tests/qemu-iotests/108.out                    |  2 +-
 tests/qemu-iotests/109                        |  4 +-
 tests/qemu-iotests/109.out                    | 78 ++++++++-----------
 tests/qemu-iotests/117.out                    |  2 +-
 tests/qemu-iotests/120.out                    |  2 +-
 tests/qemu-iotests/127.out                    |  2 +-
 tests/qemu-iotests/140.out                    |  2 +-
 tests/qemu-iotests/143.out                    |  2 +-
 tests/qemu-iotests/156.out                    |  2 +-
 tests/qemu-iotests/176.out                    | 16 ++--
 tests/qemu-iotests/182.out                    |  2 +-
 tests/qemu-iotests/183.out                    |  4 +-
 tests/qemu-iotests/184.out                    | 32 ++++----
 tests/qemu-iotests/185                        |  6 +-
 tests/qemu-iotests/185.out                    | 45 +++++++++--
 tests/qemu-iotests/191.out                    | 16 ++--
 tests/qemu-iotests/195.out                    | 16 ++--
 tests/qemu-iotests/223.out                    | 16 ++--
 tests/qemu-iotests/227.out                    | 32 ++++----
 tests/qemu-iotests/247.out                    |  2 +-
 tests/qemu-iotests/273.out                    |  8 +-
 tests/qemu-iotests/308                        |  4 +-
 tests/qemu-iotests/308.out                    |  4 +-
 tests/qemu-iotests/tests/file-io-error        |  5 +-
 tests/qemu-iotests/tests/iothreads-resize.out |  2 +-
 tests/qemu-iotests/tests/qsd-jobs.out         |  4 +-
 32 files changed, 207 insertions(+), 180 deletions(-)
monitor/qmp.c +0 -17
index 6eee450fe4..a239945e8d 100644
@@ -321,14 +321,6 @@ void coroutine_fn monitor_qmp_dispatcher_co(void *data)
             qemu_coroutine_yield();
         }
 
-        /*
-         * Move the coroutine from iohandler_ctx to qemu_aio_context for
-         * executing the command handler so that it can make progress if it
-         * involves an AIO_WAIT_WHILE().
-         */
-        aio_co_schedule(qemu_get_aio_context(), qmp_dispatcher_co);
-        qemu_coroutine_yield();
-
         /* Process request */
         if (req_obj->req) {
             if (trace_event_get_state(TRACE_MONITOR_QMP_CMD_IN_BAND)) {
@@ -355,15 +347,6 @@ void coroutine_fn monitor_qmp_dispatcher_co(void *data)
         }
 
         qmp_request_free(req_obj);
-
-        /*
-         * Yield and reschedule so the main loop stays responsive.
-         *
-         * Move back to iohandler_ctx so that nested event loops for
-         * qemu_aio_context don't start new monitor commands.
-         */
-        aio_co_schedule(iohandler_get_aio_context(), qmp_dispatcher_co);
-        qemu_coroutine_yield();
     }
     qatomic_set(&qmp_dispatcher_co, NULL);
 }
qapi/qmp-dispatch.c +23 -1
index 555528b6bb..176b549473 100644
@@ -206,9 +206,31 @@ QDict *coroutine_mixed_fn qmp_dispatch(const QmpCommandList *cmds, QObject *requ
     assert(!(oob && qemu_in_coroutine()));
     assert(monitor_cur() == NULL);
     if (!!(cmd->options & QCO_COROUTINE) == qemu_in_coroutine()) {
+        if (qemu_in_coroutine()) {
+            /*
+             * Move the coroutine from iohandler_ctx to qemu_aio_context for
+             * executing the command handler so that it can make progress if it
+             * involves an AIO_WAIT_WHILE().
+             */
+            aio_co_schedule(qemu_get_aio_context(), qemu_coroutine_self());
+            qemu_coroutine_yield();
+        }
+
         monitor_set_cur(qemu_coroutine_self(), cur_mon);
         cmd->fn(args, &ret, &err);
         monitor_set_cur(qemu_coroutine_self(), NULL);
+
+        if (qemu_in_coroutine()) {
+            /*
+             * Yield and reschedule so the main loop stays responsive.
+             *
+             * Move back to iohandler_ctx so that nested event loops for
+             * qemu_aio_context don't start new monitor commands.
+             */
+            aio_co_schedule(iohandler_get_aio_context(),
+                            qemu_coroutine_self());
+            qemu_coroutine_yield();
+        }
     } else {
        /*
         * Actual context doesn't match the one the command needs.
@@ -232,7 +254,7 @@ QDict *coroutine_mixed_fn qmp_dispatch(const QmpCommandList *cmds, QObject *requ
             .errp       = &err,
             .co         = qemu_coroutine_self(),
         };
-        aio_bh_schedule_oneshot(qemu_get_aio_context(), do_qmp_dispatch_bh,
+        aio_bh_schedule_oneshot(iohandler_get_aio_context(), do_qmp_dispatch_bh,
                                 &data);
         qemu_coroutine_yield();
     }
tests/qemu-iotests/060.out +2 -2
index 329977d9b9..a37bf446e9 100644
@@ -421,8 +421,8 @@ QMP_VERSION
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_IMAGE_CORRUPTED", "data": {"device": "none0", "msg": "Preventing invalid write on metadata (overlaps with refcount table)", "offset": 65536, "node-name": "drive", "fatal": true, "size": 65536}}
 write failed: Input/output error
 {"return": ""}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 === Testing incoming inactive corrupted image ===
 
@@ -432,8 +432,8 @@ QMP_VERSION
 qcow2: Image is corrupt: L2 table offset 0x2a2a2a00 unaligned (L1 index: 0); further non-fatal corruption events will be suppressed
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_IMAGE_CORRUPTED", "data": {"device": "", "msg": "L2 table offset 0x2a2a2a00 unaligned (L1 index: 0)", "node-name": "drive", "fatal": false}}
 {"return": ""}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
     corrupt: false
 *** done
tests/qemu-iotests/071.out +2 -2
index bca0c02f5c..a2923b05c2 100644
@@ -45,8 +45,8 @@ QMP_VERSION
 {"return": {}}
 read failed: Input/output error
 {"return": ""}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 === Testing blkverify on existing block device ===
@@ -84,9 +84,9 @@ wrote 512/512 bytes at offset 0
 {"return": ""}
 read failed: Input/output error
 {"return": ""}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
 QEMU_PROG: Failed to flush the L2 table cache: Input/output error
 QEMU_PROG: Failed to flush the refcount block cache: Input/output error
+{"return": {}}
 
 *** done
tests/qemu-iotests/081.out +8 -8
index 615c083549..aba85ea564 100644
@@ -35,8 +35,8 @@ QMP_VERSION
 read 10485760/10485760 bytes at offset 0
 10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {"return": ""}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 == using quorum rewrite corrupted mode ==
@@ -67,8 +67,8 @@ QMP_VERSION
 read 10485760/10485760 bytes at offset 0
 10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {"return": ""}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 -- checking that the image has been corrected --
 read 10485760/10485760 bytes at offset 0
@@ -106,8 +106,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 Testing:
 QMP_VERSION
@@ -115,8 +115,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "Cannot add a child to a quorum in blkverify mode"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 == dynamically removing a child from a quorum ==
@@ -125,31 +125,31 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 Testing:
 QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "The number of children cannot be lower than the vote threshold 2"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 Testing:
 QMP_VERSION
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "blkverify=on can only be set if there are exactly two files and vote-threshold is 2"}}
 {"error": {"class": "GenericError", "desc": "Cannot find device='drive0-quorum' nor node-name='drive0-quorum'"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 Testing:
 QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "The number of children cannot be lower than the vote threshold 2"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 *** done
tests/qemu-iotests/087.out +6 -6
index e1c23a6983..97b6d8036d 100644
@@ -7,8 +7,8 @@ Testing:
 QMP_VERSION
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "'node-name' must be specified for the root node"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 === Duplicate ID ===
@@ -18,8 +18,8 @@ QMP_VERSION
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "node-name=disk is conflicting with a device id"}}
 {"error": {"class": "GenericError", "desc": "Duplicate nodes with node-name='test-node'"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 === aio=native without O_DIRECT ===
@@ -28,8 +28,8 @@ Testing:
 QMP_VERSION
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "aio=native was specified, but it requires cache.direct=on, which was not specified."}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 === Encrypted image QCow ===
@@ -40,8 +40,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "Use of AES-CBC encrypted IMGFMT images is no longer supported in system emulators"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 === Encrypted image LUKS ===
@@ -52,8 +52,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 === Missing driver ===
@@ -63,7 +63,7 @@ Testing: -S
 QMP_VERSION
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "Parameter 'driver' is missing"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 *** done
tests/qemu-iotests/108.out +1 -1
index b5401d788d..b9c876b394 100644
@@ -173,8 +173,8 @@ OK: Reftable is where we expect it
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "create"}}
 {"return": {}}
 { "execute": "quit" }
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 wrote 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/109 +2 -2
index e207a555f3..0fb580f9a5 100755
@@ -57,13 +57,13 @@ run_qemu()
     _launch_qemu -drive file="${source_img}",format=raw,cache=${CACHEMODE},aio=${AIOMODE},id=src
     _send_qemu_cmd $QEMU_HANDLE "{ 'execute': 'qmp_capabilities' }" "return"
 
-    _send_qemu_cmd $QEMU_HANDLE \
+    capture_events="$qmp_event" _send_qemu_cmd $QEMU_HANDLE \
         "{'execute':'drive-mirror', 'arguments':{
             'device': 'src', 'target': '$raw_img', $qmp_format
             'mode': 'existing', 'sync': 'full'}}" \
         "return"
 
-    _send_qemu_cmd $QEMU_HANDLE '' "$qmp_event"
+    capture_events="$qmp_event JOB_STATUS_CHANGE" _wait_event $QEMU_HANDLE "$qmp_event"
     if test "$qmp_event" = BLOCK_JOB_ERROR; then
         _send_qemu_cmd $QEMU_HANDLE '' '"status": "null"'
     fi
tests/qemu-iotests/109.out +33 -45

405 lines not on this page · raw message

tests/qemu-iotests/117.out +1 -1
index 735ffd25c6..1cea9e0217 100644
@@ -18,8 +18,8 @@ wrote 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {"return": ""}
 { 'execute': 'quit' }
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 No errors were found on the image.
 read 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/120.out +1 -1
index 0744c1f136..35d84a5bc5 100644
@@ -5,8 +5,8 @@ QMP_VERSION
 wrote 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {"return": ""}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 read 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 read 65536/65536 bytes at offset 0
tests/qemu-iotests/127.out +1 -1
index 1685c4850a..dd8c4a8aa9 100644
@@ -28,6 +28,6 @@ wrote 42/42 bytes at offset 0
 { 'execute': 'quit' }
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "mirror"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "mirror"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 *** done
tests/qemu-iotests/140.out +1 -1
index 312f76d5da..32866440ae 100644
@@ -19,6 +19,6 @@ read 65536/65536 bytes at offset 0
 qemu-io: can't open device nbd+unix:///drv?socket=SOCK_DIR/nbd: Requested export not available
 server reported: export 'drv' not present
 { 'execute': 'quit' }
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 *** done
tests/qemu-iotests/143.out +1 -1
index 9ec5888e0e..d6afa32abc 100644
@@ -10,6 +10,6 @@ server reported: export 'no_such_export' not present
 qemu-io: can't open device nbd+unix:///aa--aa1?socket=SOCK_DIR/nbd: Requested export not available
 server reported: export 'aa--aa...' not present
 { 'execute': 'quit' }
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 *** done
tests/qemu-iotests/156.out +1 -1
index 4a22f0c41a..07e5e83f5d 100644
@@ -72,8 +72,8 @@ read 65536/65536 bytes at offset 196608
 {"return": ""}
 
 { 'execute': 'quit' }
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 read 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/176.out +8 -8
index 9d09b60452..45e9153ef3 100644
@@ -169,8 +169,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 wrote 196608/196608 bytes at offset 2147287040
 192 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 131072/131072 bytes at offset 2147352576
@@ -206,8 +206,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {"sha256": HASH}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 === Test pass bitmap.1 ===
 
@@ -218,8 +218,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 wrote 196608/196608 bytes at offset 2147287040
 192 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 131072/131072 bytes at offset 2147352576
@@ -256,8 +256,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {"sha256": HASH}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 === Test pass bitmap.2 ===
 
@@ -268,8 +268,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 wrote 196608/196608 bytes at offset 2147287040
 192 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 131072/131072 bytes at offset 2147352576
@@ -306,8 +306,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {"sha256": HASH}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 === Test pass bitmap.3 ===
 
@@ -318,8 +318,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 wrote 196608/196608 bytes at offset 2147287040
 192 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 131072/131072 bytes at offset 2147352576
@@ -353,6 +353,6 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {"sha256": HASH}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 *** done
tests/qemu-iotests/182.out +1 -1
index 57f7265458..83fc1a4797 100644
@@ -53,6 +53,6 @@ Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 cluster_size=65536 extended_l2=
 {'execute': 'qmp_capabilities'}
 {"return": {}}
 {'execute': 'quit'}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 *** done
tests/qemu-iotests/183.out +2 -2
index fd9c2e52a5..51aa41c888 100644
@@ -53,11 +53,11 @@ wrote 65536/65536 bytes at offset 1048576
 === Shut down and check image ===
 
 {"execute":"quit"}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
 {"return": {}}
 {"execute":"quit"}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
 {"return": {}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
 No errors were found on the image.
 No errors were found on the image.
 wrote 65536/65536 bytes at offset 1048576
tests/qemu-iotests/184.out +16 -16
index 77e5489d65..e8f631f853 100644
@@ -89,10 +89,6 @@ Testing:
     "return": [
     ]
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -104,6 +100,10 @@ Testing:
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 
 == property changes in ThrottleGroup ==
@@ -169,10 +169,6 @@ Testing:
         "iops-total-max": 0
     }
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -184,6 +180,10 @@ Testing:
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 
 == object creation/set errors  ==
@@ -211,10 +211,6 @@ Testing:
         "desc": "bps/iops/max total values and read/write values cannot be used at the same time"
     }
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -226,6 +222,10 @@ Testing:
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 
 == don't specify group ==
@@ -247,10 +247,6 @@ Testing:
         "desc": "Parameter 'throttle-group' is missing"
     }
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -262,6 +258,10 @@ Testing:
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 
 *** done
tests/qemu-iotests/185 +3 -3
index 2ae0a85bbf..17489fb91c 100755
@@ -344,14 +344,14 @@ wait_for_job_and_quit() {
 
     sleep 1
 
+    # List of expected events
+    capture_events='BLOCK_JOB_CANCELLED JOB_STATUS_CHANGE SHUTDOWN'
+
     _send_qemu_cmd $h \
         '{"execute": "quit"}' \
         'return'
 
-    # List of expected events
-    capture_events='BLOCK_JOB_CANCELLED JOB_STATUS_CHANGE SHUTDOWN'
     _wait_event $h 'SHUTDOWN'
-    QEMU_EVENTS= # Ignore all JOB_STATUS_CHANGE events that came before SHUTDOWN
     _wait_event $h 'JOB_STATUS_CHANGE' # standby
     _wait_event $h 'JOB_STATUS_CHANGE' # ready
     _wait_event $h 'JOB_STATUS_CHANGE' # standby
tests/qemu-iotests/185.out +40 -5

85 lines not on this page · raw message

tests/qemu-iotests/191.out +8 -8
index ea88777374..c3309e4bc6 100644
@@ -378,10 +378,6 @@ wrote 65536/65536 bytes at offset 1048576
     ]
 }
 { 'execute': 'quit' }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -393,6 +389,10 @@ wrote 65536/65536 bytes at offset 1048576
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 64 MiB (67108864 bytes)
@@ -796,10 +796,6 @@ wrote 65536/65536 bytes at offset 1048576
     ]
 }
 { 'execute': 'quit' }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -811,6 +807,10 @@ wrote 65536/65536 bytes at offset 1048576
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 64 MiB (67108864 bytes)
tests/qemu-iotests/195.out +8 -8
index ec84df5012..91717d302e 100644
@@ -17,10 +17,6 @@ Testing: -drive if=none,file=TEST_DIR/t.IMGFMT,backing.node-name=mid
     "return": {
     }
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -32,6 +28,10 @@ Testing: -drive if=none,file=TEST_DIR/t.IMGFMT,backing.node-name=mid
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 image: TEST_DIR/t.IMGFMT.mid
 file format: IMGFMT
@@ -55,10 +55,6 @@ Testing: -drive if=none,file=TEST_DIR/t.IMGFMT,node-name=top
     "return": {
     }
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -70,6 +66,10 @@ Testing: -drive if=none,file=TEST_DIR/t.IMGFMT,node-name=top
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
tests/qemu-iotests/223.out +8 -8
index e5e7f42caa..5f5b42e2dc 100644
@@ -11,8 +11,8 @@ QMP_VERSION
 {"return": {}}
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 
 === Write part of the file under active bitmap ===
@@ -145,14 +145,14 @@ read 2097152/2097152 bytes at offset 2097152
 
 {"execute":"nbd-server-remove",
   "arguments":{"name":"n"}}
-{"return": {}}
-{"execute":"nbd-server-remove",
-  "arguments":{"name":"n2"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "n"}}
 {"return": {}}
 {"execute":"nbd-server-remove",
   "arguments":{"name":"n2"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "n2"}}
+{"return": {}}
+{"execute":"nbd-server-remove",
+  "arguments":{"name":"n2"}}
 {"error": {"class": "GenericError", "desc": "Export 'n2' is not found"}}
 {"execute":"nbd-server-stop"}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "n3"}}
@@ -267,14 +267,14 @@ read 2097152/2097152 bytes at offset 2097152
 
 {"execute":"nbd-server-remove",
   "arguments":{"name":"n"}}
-{"return": {}}
-{"execute":"nbd-server-remove",
-  "arguments":{"name":"n2"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "n"}}
 {"return": {}}
 {"execute":"nbd-server-remove",
   "arguments":{"name":"n2"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "n2"}}
+{"return": {}}
+{"execute":"nbd-server-remove",
+  "arguments":{"name":"n2"}}
 {"error": {"class": "GenericError", "desc": "Export 'n2' is not found"}}
 {"execute":"nbd-server-stop"}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "n3"}}
@@ -282,8 +282,8 @@ read 2097152/2097152 bytes at offset 2097152
 {"execute":"nbd-server-stop"}
 {"error": {"class": "GenericError", "desc": "NBD server not running"}}
 {"execute":"quit"}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 
 === Use qemu-nbd as server ===
 
tests/qemu-iotests/227.out +16 -16
index a947b1a87d..d6a1d4ecb6 100644
@@ -54,10 +54,6 @@ Testing: -drive driver=null-co,read-zeroes=on,if=virtio
         }
     ]
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -69,6 +65,10 @@ Testing: -drive driver=null-co,read-zeroes=on,if=virtio
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 
 === blockstats with -drive if=none ===
@@ -124,10 +124,6 @@ Testing: -drive driver=null-co,if=none
         }
     ]
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -139,6 +135,10 @@ Testing: -drive driver=null-co,if=none
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 
 === blockstats with -blockdev ===
@@ -155,10 +155,6 @@ Testing: -blockdev driver=null-co,node-name=null
     "return": [
     ]
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -170,6 +166,10 @@ Testing: -blockdev driver=null-co,node-name=null
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 
 === blockstats with -blockdev and -device ===
@@ -226,10 +226,6 @@ Testing: -blockdev driver=null-co,read-zeroes=on,node-name=null -device virtio-b
         }
     ]
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -241,5 +237,9 @@ Testing: -blockdev driver=null-co,read-zeroes=on,node-name=null -device virtio-b
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 *** done
tests/qemu-iotests/247.out +1 -1
index e909e83994..7d252e7fe4 100644
@@ -17,6 +17,6 @@ QMP_VERSION
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 134217728, "offset": 134217728, "speed": 0, "type": "commit"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "job0"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "job0"}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 *** done
tests/qemu-iotests/273.out +4 -4
index 6a74a8138b..71843f02de 100644
@@ -282,10 +282,6 @@ Testing: -blockdev file,node-name=base,filename=TEST_DIR/t.IMGFMT.base -blockdev
         ]
     }
 }
-{
-    "return": {
-    }
-}
 {
     "timestamp": {
         "seconds":  TIMESTAMP,
@@ -297,5 +293,9 @@ Testing: -blockdev file,node-name=base,filename=TEST_DIR/t.IMGFMT.base -blockdev
         "reason": "host-qmp-quit"
     }
 }
+{
+    "return": {
+    }
+}
 
 *** done
tests/qemu-iotests/308 +2 -2
index de12b2b1b9..ea81dc496a 100755
@@ -77,6 +77,7 @@ fuse_export_add()
 # $1: Export ID
 fuse_export_del()
 {
+    capture_events="BLOCK_EXPORT_DELETED" \
     _send_qemu_cmd $QEMU_HANDLE \
         "{'execute': 'block-export-del',
           'arguments': {
@@ -84,8 +85,7 @@ fuse_export_del()
           } }" \
         'return'
 
-    _send_qemu_cmd $QEMU_HANDLE \
-        '' \
+    _wait_event $QEMU_HANDLE \
         'BLOCK_EXPORT_DELETED'
 }
 
tests/qemu-iotests/308.out +2 -2
index d5767133b1..e5e233691d 100644
@@ -165,9 +165,9 @@ OK: Post-truncate image size is as expected
 
 === Tear down ===
 {'execute': 'quit'}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "export-mp"}}
+{"return": {}}
 
 === Compare copy with original ===
 Images are identical.
@@ -201,9 +201,9 @@ wrote 67108864/67108864 bytes at offset 0
 read 67108864/67108864 bytes at offset 0
 64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {'execute': 'quit'}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "export"}}
+{"return": {}}
 read 67108864/67108864 bytes at offset 0
 64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 *** done
tests/qemu-iotests/tests/file-io-error +2 -3
index 88ee5f670c..fb8db73b31 100755
@@ -99,13 +99,12 @@ echo
 $QEMU_IO -f file -c 'write 0 64M' "$TEST_DIR/fuse-export" | _filter_qemu_io
 echo
 
-_send_qemu_cmd $QEMU_HANDLE \
+capture_events=BLOCK_EXPORT_DELETED _send_qemu_cmd $QEMU_HANDLE \
     "{'execute': 'block-export-del',
       'arguments': {'id': 'exp0'}}" \
     'return'
 
-_send_qemu_cmd $QEMU_HANDLE \
-    '' \
+_wait_event $QEMU_HANDLE \
     'BLOCK_EXPORT_DELETED'
 
 _send_qemu_cmd $QEMU_HANDLE \
tests/qemu-iotests/tests/iothreads-resize.out +1 -1
index 2ca5a9d964..2967ac8f0d 100644
@@ -3,8 +3,8 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 QMP_VERSION
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"return": {}}
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
tests/qemu-iotests/tests/qsd-jobs.out +2 -2
index c1bc9b8356..aa6b6d1aef 100644
@@ -7,8 +7,8 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
 QMP_VERSION
 {"return": {}}
 {"return": {}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
+{"return": {}}
 
 === Streaming can't get permission on base node ===
 
@@ -17,6 +17,6 @@ QMP_VERSION
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "job0"}}
 {"error": {"class": "GenericError", "desc": "Permission conflict on node 'fmt_base': permissions 'write' are both required by an unnamed block device (uses node 'fmt_base' as 'root' child) and unshared by stream job 'job0' (uses node 'fmt_base' as 'intermediate node' child)."}}
-{"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_EXPORT_DELETED", "data": {"id": "export1"}}
+{"return": {}}
 *** done
Signature · 1 line
2.43.0
Reviewed-by on v1 3/3
#9 lore raw

Am 16.01.2024 um 20:00 hat Stefan Hajnoczi geschrieben:

Quoted 21 lines

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not polled during nested event loops. The coroutine currently reschedules itself in the main loop's qemu_aio_context AioContext, which is polled during nested event loops. One known problem is that QMP device-add calls drain_call_rcu(), which temporarily drops the BQL, leading to all sorts of havoc like other vCPU threads re-entering device emulation code while another vCPU thread is waiting in device emulation code with aio_poll().

Paolo Bonzini suggested running non-coroutine QMP handlers in the iohandler AioContext. This avoids trouble with nested event loops. His original idea was to move coroutine rescheduling to monitor_qmp_dispatch(), but I resorted to moving it to qmp_dispatch() because we don't know if the QMP handler needs to run in coroutine context in monitor_qmp_dispatch(). monitor_qmp_dispatch() would have been nicer since it's associated with the monitor implementation and not as general as qmp_dispatch(), which is also used by qemu-ga.

A number of qemu-iotests need updated .out files because the order of QMP events vs QMP responses has changed.

Quoted 7 lines

Solves Issue #1933.

Fixes 7bed89958bfbf40df9ca681cefbdca63abdde39d ("device_core: use drain_call_rcu in in qmp_device_add")
Tested-by on v1 3/3
#10 lore raw
In reply to Stefan Hajnoczi

Am 16.01.24 um 20:00 schrieb Stefan Hajnoczi:

Quoted 21 lines

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not polled during nested event loops. The coroutine currently reschedules itself in the main loop's qemu_aio_context AioContext, which is polled during nested event loops. One known problem is that QMP device-add calls drain_call_rcu(), which temporarily drops the BQL, leading to all sorts of havoc like other vCPU threads re-entering device emulation code while another vCPU thread is waiting in device emulation code with aio_poll().

Paolo Bonzini suggested running non-coroutine QMP handlers in the iohandler AioContext. This avoids trouble with nested event loops. His original idea was to move coroutine rescheduling to monitor_qmp_dispatch(), but I resorted to moving it to qmp_dispatch() because we don't know if the QMP handler needs to run in coroutine context in monitor_qmp_dispatch(). monitor_qmp_dispatch() would have been nicer since it's associated with the monitor implementation and not as general as qmp_dispatch(), which is also used by qemu-ga.

A number of qemu-iotests need updated .out files because the order of QMP events vs QMP responses has changed.

Quoted 7 lines

Solves Issue #1933.

Fixes 7bed89958bfbf40df9ca681cefbdca63abdde39d ("device_core: use drain_call_rcu in in qmp_device_add")

With the patch I can no longer see any do_qmp_dispatch_bh() calls run in vCPU threads.

I also did a bit of smoke testing with some other QMP and QGA commands and didn't find any obvious breakage, so:

P.S.

Unfortunately, the patch does not solve the other issue I came across back then [0] with snapshot_save_job_bh() being executed during a vCPU thread's aio_poll(). See also [1].

I suppose this would need some other mechanism to solve or could it also be scheduled to the iohandler AioContext? It's not directly related to your patch of course, just mentioning it, because it's a similar theme.

[0]:
https://lore.kernel.org/qemu-devel/[email protected]/
[1]: https://gitlab.com/qemu-project/qemu/-/issues/2111

Best Regards,
Fiona

#11 lore raw
In reply to Stefan Hajnoczi

16.01.2024 22:00, Stefan Hajnoczi пишет:

Quoted 21 lines

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not polled during nested event loops. The coroutine currently reschedules itself in the main loop's qemu_aio_context AioContext, which is polled during nested event loops. One known problem is that QMP device-add calls drain_call_rcu(), which temporarily drops the BQL, leading to all sorts of havoc like other vCPU threads re-entering device emulation code while another vCPU thread is waiting in device emulation code with aio_poll().

Paolo Bonzini suggested running non-coroutine QMP handlers in the iohandler AioContext. This avoids trouble with nested event loops. His original idea was to move coroutine rescheduling to monitor_qmp_dispatch(), but I resorted to moving it to qmp_dispatch() because we don't know if the QMP handler needs to run in coroutine context in monitor_qmp_dispatch(). monitor_qmp_dispatch() would have been nicer since it's associated with the monitor implementation and not as general as qmp_dispatch(), which is also used by qemu-ga.

A number of qemu-iotests need updated .out files because the order of QMP events vs QMP responses has changed.

Quoted 7 lines

Solves Issue #1933.

Fixes 7bed89958bfbf40df9ca681cefbdca63abdde39d ("device_core: use drain_call_rcu in in qmp_device_add")

This change broke something in 7.2. I'm still debugging it, will come with a follow-up once some more details are found, I'll also check current master with and without this commit.

The prob happens with multiple suspend-resume cycles, - with this change applied, guest does not work as expected after *second* suspend-resume.

JFYI for now.

Thanks,

/mjt

#12 lore raw

03.02.2024 12:01, Michael Tokarev wrote:
...

Quoted 7 lines

This change broke something in 7.2. I'm still debugging it, will come with a follow-up once some more details are found, I'll also check current master with and without this commit.

The prob happens with multiple suspend-resume cycles, - with this change applied, guest does not work as expected after *second* suspend-resume.

So, it turned out the prob here exists on master too, and manifests itself the same way on 7.2.9 or on 8.2.1, - in all cases where we have this change applied it works (or breaks) equally.

A (simple) reproducer so far is a hibernate test, - it fails *only* after suspend-to-ram, but works fine after just hibernate.

I used just an initrd (with a drive image used for swap - for hibernation space).

qemu-img create s.img 256M
mkswap s.img
qemu-system-x86_64 \
-serial stdio -vga none -display none -parallel none -net none \
-machine q35 \
-drive file=s.img,if=ide,format=raw \
-m 256 \
-monitor unix:ttyS0,server,nowait \
-kernel /boot/vmlinuz-6.1.0-15-amd64 \
-initrd /boot/initrd.img-6.1.0-15-amd64 \
-append "shell=/bin/sh console=ttyS0 root=none"

There, in the guest (it has busybox only here):

 # swapon /dev/sda
 # echo mem > /sys/power/state
 (system_wakeup on the monitor)
 # echo disk > /sys/power/state

The system will hibernate but *not* turn off power, qemu will continue running, while all console messages are the same as when it works fine. qemu process is spinning up with 100% cpu usage at this stage.

Without the intermediate suspend-to-ram or without the commit in question, qemu process will exit normally at this stage.

This is a somewhat patalogical test case, but I see it as an indicator of something else being wrong, like we aren't saving or restoring some state now which we should do.

The tight loop also suggests we're not having success in there.

Thanks,

/mjt

#13 lore raw
Quoted 48 lines · Michael Tokarev

03.02.2024 12:01, Michael Tokarev wrote:
...

Quoted 7 lines

This change broke something in 7.2. I'm still debugging it, will come with a follow-up once some more details are found, I'll also check current master with and without this commit.

The prob happens with multiple suspend-resume cycles, - with this change applied, guest does not work as expected after *second* suspend-resume.

So, it turned out the prob here exists on master too, and manifests itself the same way on 7.2.9 or on 8.2.1, - in all cases where we have this change applied it works (or breaks) equally.

A (simple) reproducer so far is a hibernate test, - it fails *only* after suspend-to-ram, but works fine after just hibernate.

I used just an initrd (with a drive image used for swap - for hibernation space).

  qemu-img create s.img 256M
  mkswap s.img
  qemu-system-x86_64 \
   -serial stdio -vga none -display none -parallel none -net none \
   -machine q35 \
   -drive file=s.img,if=ide,format=raw \
   -m 256 \
   -monitor unix:ttyS0,server,nowait \
   -kernel /boot/vmlinuz-6.1.0-15-amd64 \
   -initrd /boot/initrd.img-6.1.0-15-amd64 \
   -append "shell=/bin/sh console=ttyS0 root=none"

There, in the guest (it has busybox only here):

  # swapon /dev/sda
  # echo mem > /sys/power/state
  (system_wakeup on the monitor)
  # echo disk > /sys/power/state

The system will hibernate but *not* turn off power, qemu will continue running, while all console messages are the same as when it works fine. qemu process is spinning up with 100% cpu usage at this stage.

Without the intermediate suspend-to-ram or without the commit in question, qemu process will exit normally at this stage.

Quoted 5 lines

This is a somewhat patalogical test case, but I see it as an indicator of something else being wrong, like we aren't saving or restoring some state now which we should do.

The tight loop also suggests we're not having success in there.

I'm unable to reproduce this. QEMU v8.2.0, v8.1.0, and even v7.2.0 all spin with 100% CPU usage on my machine. It looks to me like this behavior is not a regression caused by this commit.

Do v7.2.0, v8.1.0, and v8.2.0 terminate QEMU on your machine?

Stefan

#14 lore raw
In reply to Stefan Hajnoczi
Quoted 15 lines · Stefan Hajnoczi

Several bugs have been reported related to how QMP commands are rescheduled in qemu_aio_context:
- https://gitlab.com/qemu-project/qemu/-/issues/1933
- https://issues.redhat.com/browse/RHEL-17369
- https://bugzilla.redhat.com/show_bug.cgi?id=2215192
- https://bugzilla.redhat.com/show_bug.cgi?id=2214985

The first instance of the bug interacted with drain_call_rcu() temporarily dropping the BQL and resulted in vCPU threads entering device emulation code simultaneously (something that should never happen). I set out to make drain_call_rcu() safe to use in this environment, but Paolo and Kevin discussed the possibility of avoiding rescheduling the monitor_qmp_dispatcher_co() coroutine for non-coroutine commands. This would prevent monitor commands from running during vCPU thread aio_poll() entirely and addresses the root cause.

Quoted 6 lines

This patch series implements this idea. qemu-iotests is sensitive to the exact order in which QMP events and responses are emitted. Running QMP handlers in the iohandler AioContext causes some QMP events to be ordered differently than before. It is therefore necessary to adjust the reference output in many test cases. The actual QMP code change is small and everything else is just to make qemu-iotests happy.

This seems to be -stable material too, once it is applied to master. It's difficult for me to catch the issue (#1933 @gitlab) locally, but I did have some success there, and it always work after this patch is applied. It also seem to work fine on 7.2, not only on 8.2, fwiw.

Thanks,

/mjt

#15 lore raw
In reply to Stefan Hajnoczi
Quoted 16 lines · Stefan Hajnoczi

Several bugs have been reported related to how QMP commands are rescheduled in qemu_aio_context:
- https://gitlab.com/qemu-project/qemu/-/issues/1933
- https://issues.redhat.com/browse/RHEL-17369
- https://bugzilla.redhat.com/show_bug.cgi?id=2215192
- https://bugzilla.redhat.com/show_bug.cgi?id=2214985

The first instance of the bug interacted with drain_call_rcu() temporarily dropping the BQL and resulted in vCPU threads entering device emulation code simultaneously (something that should never happen). I set out to make drain_call_rcu() safe to use in this environment, but Paolo and Kevin discussed the possibility of avoiding rescheduling the monitor_qmp_dispatcher_co() coroutine for non-coroutine commands. This would prevent monitor commands from running during vCPU thread aio_poll() entirely and addresses the root cause.

Quoted 6 lines

This patch series implements this idea. qemu-iotests is sensitive to the exact order in which QMP events and responses are emitted. Running QMP handlers in the iohandler AioContext causes some QMP events to be ordered differently than before. It is therefore necessary to adjust the reference output in many test cases. The actual QMP code change is small and everything else is just to make qemu-iotests happy.

Hi; we have a suspicion that this change has resulted in a flaky-CI
test: iotest-144 sometimes fails, apparently because a "return" result from QMP isn't always returned at the same place in relation to other QMP events. Could you have a look at it?

https://gitlab.com/qemu-project/qemu/-/issues/2126

thanks
-- PMM

Tested-by
#16 lore raw

It's easily for me to encounter " ../block/qcow2.c:5263:
ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *, Error **): Assertion `false' failed" issue during 1Q vhost-user interface + RT VM + post-copy migration

After applying this patch, the issue is still not reproduced even if I repeat the same migration test for 60 times.

Best Regards,
YangHang Liu

Quoted 35 lines · Peter Maydell
Quoted 16 lines · Stefan Hajnoczi

Several bugs have been reported related to how QMP commands are rescheduled in qemu_aio_context:
- https://gitlab.com/qemu-project/qemu/-/issues/1933
- https://issues.redhat.com/browse/RHEL-17369
- https://bugzilla.redhat.com/show_bug.cgi?id=2215192
- https://bugzilla.redhat.com/show_bug.cgi?id=2214985

The first instance of the bug interacted with drain_call_rcu() temporarily dropping the BQL and resulted in vCPU threads entering device emulation code simultaneously (something that should never happen). I set out to make drain_call_rcu() safe to use in this environment, but Paolo and Kevin discussed the possibility of avoiding rescheduling the monitor_qmp_dispatcher_co() coroutine for non-coroutine commands. This would prevent monitor commands from running during vCPU thread aio_poll() entirely and addresses the root cause.

Quoted 6 lines

This patch series implements this idea. qemu-iotests is sensitive to the exact order in which QMP events and responses are emitted. Running QMP handlers in the iohandler AioContext causes some QMP events to be ordered differently than before. It is therefore necessary to adjust the reference output in many test cases. The actual QMP code change is small and everything else is just to make qemu-iotests happy.

Hi; we have a suspicion that this change has resulted in a flaky-CI
test: iotest-144 sometimes fails, apparently because a "return" result from QMP isn't always returned at the same place in relation to other QMP events. Could you have a look at it?

https://gitlab.com/qemu-project/qemu/-/issues/2126

thanks
-- PMM