Misc patches for iotests, qom, crypt & io
Subsystems (2)
The following changes since commit 3f129ea545f16e82b5e43fcc3866b134b7cc4dfc:
Merge tag 'pull-vfio-20260520' of https://github.com/legoater/qemu into staging (2026-05-20 16:53:28 -0400)
are available in the Git repository at:
https://gitlab.com/berrange/qemu tags/misc-next-pull-request
for you to fetch changes up to 7b0376d24ce34893a9512587d1d82a6e07680e6d:
qom: drop user_creatable_add_type method (2026-05-21 12:39:54 +0100)
----------------------------------------------------------------
Misc patches for iotests, qom, crypt & io
* Fix client side anoymous TLS credentials
* Fix return value semantics for qio_channel_flush
* Add ID validation of internal QOM constructor
* Fix ability to create internal QOM objects without a parent
* Merge user creatable object constructor into main QOM file
* Print reason for skipping I/O tests
* Remove redundant meson suits for I/O tests
* Add optional suites for many block drivers I/O tests
* Run I/O tests for 10 block drivers in GitLab CI
* Fix sudo check for LUKS I/O test
* Mark I/O test 151, 181, 185, 308 as flaky
----------------------------------------------------------------
Daniel P. Berrangé (27):
crypto: fix client side anonymous TLS credentials
gitlab: ensure all meson jobs capture build/meson-logs by default
iotests: print reason when I/O test is skipped in TAP mode
iotests: remove redundant meson suite for iotests
iotests: ensure all qcow2 I/O tests are able to be run via make scripts/mtest2make: ensure output has stable sorting scripts/mtest2make: support optional tests grouping
iotests: add a meson suite / make target per block I/O tests format docs/devel/testing: expand documentation for 'make check-block'
iotests: add nbd and luks to the I/O test suites
iotests: use 'driver' as collective term for either format or protocol
iotests: validate dmsetup result in test 128
iotests: fix check for sudo access in LUKS I/O test
iotests: mark 151, 181, 185 & 308 as flaky tests
gitlab: add jobs for thorough block tests
gitlab: remove I/O tests from build-tcg-disabled job
qom: add trace events for object/property lifecycle hw/vfio-user: use a valid object ID for iothread
qom: validate ID format when creating objects
qom: make errp last param in methods taking va_list
qom: shorten name of object_set_properties_from_keyval
qom: have object_set_props_keyval return bool
qom: move object_set_prop_keyval into object.c
qom: add object_new_with_props_from_qdict
qom: fix ability to create objects without a parent
qom: allow object_new_with_prop* to trigger module loading
qom: drop user_creatable_add_type method
John Snow (1):
python: bump qemu.qmp to v0.0.6
Tejus GK (1):
io: invert the return semantics of qio_channel_flush
.gitlab-ci.d/buildtest-template.yml | 30 ++-
.gitlab-ci.d/buildtest.yml | 27 +-
authz/listfile.c | 4 +-
crypto/tlscredsanon.c | 2 +
docs/devel/testing/main.rst | 26 +-
hw/vfio-user/proxy.c | 2 +-
include/io/channel-socket.h | 6 +-
include/io/channel.h | 4 +-
include/qom/object.h | 121 +++++++--
include/qom/object_interfaces.h | 18 --
io/channel-socket.c | 16 +-
python/scripts/vendor.py | 4 +-
...ny.whl => qemu_qmp-0.0.6-py3-none-any.whl} | Bin 72263 -> 72379 bytes
pythondeps.toml | 2 +-
qom/object.c | 230 ++++++++++++++++--
qom/object_interfaces.c | 101 +-------
qom/trace-events | 12 +-
scripts/mtest2make.py | 30 ++-
system/qdev-monitor.c | 4 +-
system/vl.c | 7 +-
tests/Makefile.include | 3 +-
tests/qemu-iotests/128 | 6 +
tests/qemu-iotests/149 | 13 +-
tests/qemu-iotests/151 | 1 +
tests/qemu-iotests/181 | 2 +
tests/qemu-iotests/185 | 1 +
tests/qemu-iotests/308 | 2 +
tests/qemu-iotests/common.rc | 16 ++
tests/qemu-iotests/iotests.py | 16 ++
tests/qemu-iotests/meson.build | 65 ++++-
tests/qemu-iotests/testrunner.py | 2 +-
tests/unit/check-qom-proplist.c | 97 ++++++--
tests/unit/test-crypto-tlssession.c | 120 ++++++++-
33 files changed, 734 insertions(+), 256 deletions(-)
rename python/wheels/{qemu_qmp-0.0.5-py3-none-any.whl => qemu_qmp-0.0.6-py3-none-any.whl} (64%)Signature · 1 line
2.54.0
[PULL 04/29] gitlab: ensure all meson jobs capture build/meson-logs by default
The build-without-defaults and build-tci jobs do not capture any artifacts, despite running 'make check'. This has proved a repeated bug with CI jobs, so introduce a new '.meson_job_template' rule which always captures 'build/meson-logs'. Jobs can still provide their own 'artifacts:' config which will override this default behaviour.
--- .gitlab-ci.d/buildtest-template.yml | 30 ++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-)
.gitlab-ci.d/buildtest-template.yml +19 -11
@@ -1,5 +1,20 @@ -.native_build_job_template: + +# Any job running meson should capture meson logs +# by default. Some jobs might override the artifacts +# to capture further files +.meson_job_template: extends: .base_job_template + artifacts: + name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" + when: always + expire_in: 7 days + paths: + - build/meson-logs + reports: + junit: build/meson-logs/*.junit.xml + +.native_build_job_template: + extends: .meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG cache:
@@ -60,7 +75,7 @@ - build/**/*.c.o.d .common_test_job_template: - extends: .base_job_template + extends: .meson_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG script:
@@ -87,14 +102,7 @@ # Prevent logs from the build job that run earlier # from being duplicated in the test job artifacts - rm -f build/meson-logs/* - artifacts: - name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" - when: always - expire_in: 7 days - paths: - - build/meson-logs - reports: - junit: build/meson-logs/*.junit.xml + .functional_test_job_template: extends: .common_test_job_template
@@ -125,7 +133,7 @@ QEMU_JOB_FUNCTIONAL: 1 .wasm_build_job_template: - extends: .base_job_template + extends: .meson_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG before_script:
Signature · 1 line
2.54.0
[PULL 07/29] iotests: ensure all qcow2 I/O tests are able to be run via make
In reply to Daniel P. BerrangéFor block formats marked as 'quick', only tests in the 'auto' group are added to the meson test suite.
The result of this is that qcow2 tests not in the 'auto' group cannot be run at all, even if passing SPEED=slow or SPEED=thorough.
To fix this we need todo two passes over the I/O test list. First add all tests from 'auto' group into the 'block' suite, so they are run by default. Then on the second pass add any tests which were not in 'auto' into the 'block-slow' suite, so they get run when SPEED=slow or SPEED=thorough.
--- tests/qemu-iotests/meson.build | 48 ++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 5 deletions(-)
tests/qemu-iotests/meson.build +43 -5
@@ -30,16 +30,48 @@ endforeach qemu_iotests_check_cmd = files('check') foreach format, speed: qemu_iotests_formats + # Formats tagged 'quick' get the subset of tests in the 'auto' + # group, run by default with 'make check' / 'make check-block' + seen = [] if speed == 'quick' - suites = 'block' + args = ['-tap', '-' + format, '-g', 'auto'] + suites = ['block'] + + rc = run_command( + [python, qemu_iotests_check_cmd] + args + ['-n'], + check: true, + ) + + foreach item: rc.stdout().strip().split() + seen += item + args = [qemu_iotests_check_cmd, + '-tap', '-' + format, item, + '--source-dir', meson.current_source_dir(), + '--build-dir', meson.current_build_dir()] + # Some individual tests take as long as 45 seconds + # Bump the timeout to 3 minutes for some headroom + # on slow machines to minimize spurious failures + test('io-' + format + '-' + item, + python, + args: args, + depends: qemu_iotests_binaries, + env: qemu_iotests_env, + protocol: 'tap', + timeout: 180, + suite: suites) + endforeach + endif + + suites = [] + # Any format tagged quick or slow also gets added to slow + # otherwise its tagged thorough + if speed != 'thorough' + suites += ['block-slow'] else - suites = ['block-' + speed] + suites += ['block-thorough'] endif args = ['-tap', '-' + format] - if speed == 'quick' - args += ['-g', 'auto'] - endif rc = run_command( [python, qemu_iotests_check_cmd] + args + ['-n'],
@@ -47,6 +79,12 @@ foreach format, speed: qemu_iotests_formats ) foreach item: rc.stdout().strip().split() + # Skip any tests already added from the 'auto' group + # as they're run in the 'quick' suite already + if item in seen + continue + endif + args = [qemu_iotests_check_cmd, '-tap', '-' + format, item, '--source-dir', meson.current_source_dir(),
Signature · 1 line
2.54.0
[PULL 03/29] python: bump qemu.qmp to v0.0.6
In reply to Daniel P. BerrangéFrom: John Snow <[email protected]>
This release removes some deprecated warnings for our use of the old sendmsg API on older python versions.
---
python/scripts/vendor.py | 4 ++--
...ny.whl => qemu_qmp-0.0.6-py3-none-any.whl} | Bin 72263 -> 72379 bytes
pythondeps.toml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
rename python/wheels/{qemu_qmp-0.0.5-py3-none-any.whl => qemu_qmp-0.0.6-py3-none-any.whl} (64%)python/scripts/vendor.py +2 -2
@@ -43,8 +43,8 @@ def main() -> int: packages = { "meson==1.11.1": "9b3a023657e393dbc5335b95c561337d49b7a458f5541e47ec44f2cc566e0d80", - "qemu.qmp==0.0.5": - "e05782d6df5844b34e0d2f7c68693525da074deef7b641c1401dda6e4e3d6303", + "qemu.qmp==0.0.6": + "5d7c5af0e9de427696e3bf72e333965c3a697929f77f6b7ddc30c989fc7b539b", "pycotap==1.3.1": "1c3a25b3ff89e48f4e00f1f71dbbc1642b4f65c65d416524d07e73492fff25ea", }
python/wheels/qemu_qmp-0.0.6-py3-none-any.whl renamed from python/wheels/qemu_qmp-0.0.5-py3-none-any.whl binary +0 -0
pythondeps.toml +1 -1
@@ -36,7 +36,7 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.2.2" } # only include dependencies that can be guaranteed via configure from # system packages, or python packages we vendor. [tooling] -"qemu.qmp" = { accepted = ">=0.0.5", installed = "0.0.5" } +"qemu.qmp" = { accepted = ">=0.0.5", installed = "0.0.6" } "qemu" = { path = "python/" } # NB: The following dependencies should be a little bit more modern than # the versions listed here, but we are still using Debian 11 for several
Signature · 1 line
2.54.0
[PULL 02/29] crypto: fix client side anonymous TLS credentials
In reply to Daniel P. BerrangéThe previous refactoring of credential creation failed to allocate storage fo the anonymous TLS credentials on the client endpoint.
--- crypto/tlscredsanon.c | 2 + tests/unit/test-crypto-tlssession.c | 120 +++++++++++++++++++++++++++- 2 files changed, 121 insertions(+), 1 deletion(-)
crypto/tlscredsanon.c +2 -0
@@ -73,6 +73,8 @@ qcrypto_tls_creds_anon_load(QCryptoTLSCredsAnon *creds, box->dh_params); } }else { + box =qcrypto_tls_creds_box_new_client (GNUTLS_CRD_ANON ); + ret =gnutls_anon_allocate_client_credentials (&box->data.anonclient);if (ret <0 ) { error_setg(errp, "Cannot allocate credentials: %s",
tests/unit/test-crypto-tlssession.c +119 -1
140 lines not on this page · raw message
Signature · 1 line
2.54.0
[PULL 01/29] io: invert the return semantics of qio_channel_flush
In reply to Daniel P. BerrangéFrom: Tejus GK <[email protected]>
With the kernel's zerocopy notification mechanism, the caller can determine whether
* All syscalls successfully used zero copy
* At least one syscall failed to use zero copy
But, as of now QEMU's IO channel flush function semantics are like
* 1 => all syscalls failed to use zero copy
* 0 => at least one syscall successfully used zero copy
This is not aligned with what the kernel reports, and ends up reporting false negatives for cases like when there's just a single successful zerocopy amongst a collection of deferred zero-copies during a flush.
Fix this by inverting the return semantics of the IO flush function.
--- include/io/channel-socket.h | 6 +----- include/io/channel.h | 4 ++-- io/channel-socket.c | 16 ++++++++-------- 3 files changed, 11 insertions(+), 15 deletions(-)
include/io/channel-socket.h +1 -5
@@ -50,11 +50,7 @@ struct QIOChannelSocket {ssize_t zero_copy_queued;ssize_t zero_copy_sent;bool blocking; -/** -* This flag indicates whether any new data was successfully sent with -* zerocopy since the last qio_channel_socket_flush() call. -*/ -bool new_zero_copy_sent_success; +bool zero_copy_fallback; };
include/io/channel.h +2 -2
@@ -1147,8 +1147,8 @@ int coroutine_mixed_fn qio_channel_writev_full_all(QIOChannel *ioc,* If not implemented, acts as a no-op, and returns 0. * * Returns -1 if any error is found, -* 1 if every send failed to use zero copy. -* 0 otherwise. +* 1 if at least one send failed to use zero copy. +* 0 if every send successfully used zero copy. */ int qio_channel_flush(QIOChannel *ioc,
io/channel-socket.c +8 -8
@@ -72,7 +72,7 @@ qio_channel_socket_new(void) sioc->zero_copy_queued =0 ; sioc->zero_copy_sent =0 ; sioc->blocking = false; - sioc->new_zero_copy_sent_success = false; + sioc->zero_copy_fallback = false; ioc =QIO_CHANNEL (sioc);qio_channel_set_feature (ioc,QIO_CHANNEL_FEATURE_SHUTDOWN );
@@ -880,9 +880,9 @@ static int qio_channel_socket_flush_internal(QIOChannel *ioc,/* No errors, count successfully finished sendmsg()*/ sioc->zero_copy_sent += serr->ee_data - serr->ee_info +1 ; -/* If any sendmsg() succeeded using zero copy, mark zerocopy success */ -if (serr->ee_code !=SO_EE_CODE_ZEROCOPY_COPIED ) { - sioc->new_zero_copy_sent_success = true; +if (serr->ee_code ==SO_EE_CODE_ZEROCOPY_COPIED ) { +/* If any sendmsg() fell back to a copy, mark fallback as true */ + sioc->zero_copy_fallback = true; } }
@@ -900,12 +900,12 @@ static int qio_channel_socket_flush(QIOChannel *ioc, return ret; } -if (sioc->new_zero_copy_sent_success) { - sioc->new_zero_copy_sent_success = false; -return 0 ; +if (sioc->zero_copy_fallback) { + sioc->zero_copy_fallback = false; +return 1 ; } -return 1 ; +return 0 ; }#endif /* QEMU_MSG_ZEROCOPY */
Signature · 1 line
2.54.0
[PULL 06/29] iotests: remove redundant meson suite for iotests
In reply to Daniel P. BerrangéIf a test is in the 'block-slow' or 'block-thorough' suite, there is no need to also add it to the 'slow' or 'thorough' suites.
--- tests/qemu-iotests/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/meson.build +1 -1
@@ -33,7 +33,7 @@ foreach format, speed: qemu_iotests_formats if speed == 'quick' suites = 'block' else - suites = ['block-' + speed, speed] + suites = ['block-' + speed] endif args = ['-tap', '-' + format]
Signature · 1 line
2.54.0
[PULL 15/29] iotests: fix check for sudo access in LUKS I/O test
In reply to Daniel P. BerrangéThe test did not cope with the possibility that 'sudo' was not installed at all, merely that it was not configured. This broke tests in any CI env which lacks 'sudo'.
--- tests/qemu-iotests/149 | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
tests/qemu-iotests/149 +8 -5
@@ -95,11 +95,14 @@ def verify_passwordless_sudo(): args = ["sudo", "-n", "/bin/true"] - proc = subprocess.Popen(args, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - universal_newlines=True) + try: + proc = subprocess.Popen(args, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + universal_newlines=True) + except FileNotFoundError as e: + iotests.notrun('requires sudo binary: %s' % e) msg = proc.communicate()[0]
Signature · 1 line
2.54.0
[PULL 05/29] iotests: print reason when I/O test is skipped in TAP mode
In reply to Daniel P. BerrangéThe TAP output on a skipped test:
ok raw 181 # SKIP
is not informative.
The test program included a reason, and that should be displayed in TAP mode (it is already shown in non-TAP mode):
ok raw 181 # SKIP Postcopy is not supported
--- tests/qemu-iotests/testrunner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/testrunner.py +1 -1
@@ -174,7 +174,7 @@ def test_print_one_line(self, test: str, elif status == 'fail': print(f'not ok {self.env.imgfmt} {test}') elif status == 'not run': - print(f'ok {self.env.imgfmt} {test} # SKIP') + print(f'ok {self.env.imgfmt} {test} # SKIP {description}') return if lasttime:
Signature · 1 line
2.54.0
[PULL 13/29] iotests: use 'driver' as collective term for either format or protocol
In reply to Daniel P. BerrangéThe I/O tests integration previously exclusively tested block formats and now also covers the NBD protocol. Replace references to 'format' with 'driver', as a generic term to collectively apply to either a format or protocol.
--- docs/devel/testing/main.rst | 13 +++++++------ tests/Makefile.include | 4 ++-- tests/qemu-iotests/meson.build | 24 ++++++++++++------------ 3 files changed, 21 insertions(+), 20 deletions(-)
docs/devel/testing/main.rst +7 -6
@@ -237,7 +237,8 @@ check-block ~~~~~~~~~~~ There are a variety of ways to exercise the block layer I/O tests -via make targets. +via make targets for a selection of formats / protocols (collectively +referred to as ``drivers`` below). A default ``make check`` or ``make check-block`` command will exercise the ``qcow2`` format, using the tests tagged into the ``auto`` group
@@ -246,14 +247,14 @@ only. These targets accept the ``SPEED`` variable to augment the set of tests to run. A slightly more comprehensive test plan can be run by defining ``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw`` -formats. The most comprehensive test plan can be run by defining -``SPEED=thorough``, which enables all available tests for the formats +drivers. The most comprehensive test plan can be run by defining +``SPEED=thorough``, which enables all available tests for the drivers ``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, ``vhdx``, ``vmdk``, and ``vpc``. -Each of formats also has its own dedicated make target, named -``make check-block-$FORMAT`` which will run all available tests for -the designated format and does not require the ``SPEED`` variable +Each of drivers also has its own dedicated make target, named +``make check-block-$DRIVER`` which will run all available tests for +the designated driver and does not require the ``SPEED`` variable to be set. See the "QEMU iotests" section below for more information on the
tests/Makefile.include +2 -2
@@ -14,8 +14,8 @@ check-help: @echo " $(MAKE) check-unit Run qobject tests" @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" @echo " $(MAKE) check-tracetool Run tracetool generator tests" - @echo " $(MAKE) check-block Run block tests (all formats)" - @echo " $(MAKE) check-block-FORMAT Run block tests (only for FORMAT)" + @echo " $(MAKE) check-block Run block tests (all formats/protocols)" + @echo " $(MAKE) check-block-DRIVER Run block tests (only for format/protocol 'DRIVER')" ifneq ($(filter $(all-check-targets), check-softfloat),) @echo " $(MAKE) check-softfloat Run FPU emulation tests" @echo " $(MAKE) check-tcg Run TCG tests"
tests/qemu-iotests/meson.build +12 -12
@@ -12,7 +12,7 @@ qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd] qemu_iotests_env = {'PYTHON': python.full_path()} # If altering this definition, also update docs/devel/testing/main.rst # section on 'check-block' targets to reflect the changes -qemu_iotests_formats = { +qemu_iotests_drivers = { 'qcow2': 'quick', 'raw': 'slow', 'luks': 'thorough',
@@ -33,12 +33,12 @@ endforeach qemu_iotests_check_cmd = files('check') -foreach format, speed: qemu_iotests_formats - # Formats tagged 'quick' get the subset of tests in the 'auto' +foreach driver, speed: qemu_iotests_drivers + # Drivers tagged 'quick' get the subset of tests in the 'auto' # group, run by default with 'make check' / 'make check-block' seen = [] if speed == 'quick' - args = ['-tap', '-' + format, '-g', 'auto'] + args = ['-tap', '-' + driver, '-g', 'auto'] suites = ['block'] rc = run_command(
@@ -49,13 +49,13 @@ foreach format, speed: qemu_iotests_formats foreach item: rc.stdout().strip().split() seen += item args = [qemu_iotests_check_cmd, - '-tap', '-' + format, item, + '-tap', '-' + driver, item, '--source-dir', meson.current_source_dir(), '--build-dir', meson.current_build_dir()] # Some individual tests take as long as 45 seconds # Bump the timeout to 3 minutes for some headroom # on slow machines to minimize spurious failures - test('io-' + format + '-' + item, + test('io-' + driver + '-' + item, python, args: args, depends: qemu_iotests_binaries,
@@ -66,9 +66,9 @@ foreach format, speed: qemu_iotests_formats endforeach endif - # Every format gets put in the format specific suite - suites = ['block-' + format + '-optional'] - # Any format tagged quick or slow also gets added to slow + # Every driver gets put in the driver specific suite + suites = ['block-' + driver + '-optional'] + # Any driver tagged quick or slow also gets added to slow # otherwise its tagged thorough if speed != 'thorough' suites += ['block-slow']
@@ -76,7 +76,7 @@ foreach format, speed: qemu_iotests_formats suites += ['block-thorough'] endif - args = ['-tap', '-' + format] + args = ['-tap', '-' + driver] rc = run_command( [python, qemu_iotests_check_cmd] + args + ['-n'],
@@ -91,13 +91,13 @@ foreach format, speed: qemu_iotests_formats endif args = [qemu_iotests_check_cmd, - '-tap', '-' + format, item, + '-tap', '-' + driver, item, '--source-dir', meson.current_source_dir(), '--build-dir', meson.current_build_dir()] # Some individual tests take as long as 45 seconds # Bump the timeout to 3 minutes for some headroom # on slow machines to minimize spurious failures - test('io-' + format + '-' + item, + test('io-' + driver + '-' + item, python, args: args, depends: qemu_iotests_binaries,
Signature · 1 line
2.54.0
[PULL 09/29] scripts/mtest2make: support optional tests grouping
In reply to Daniel P. BerrangéCurrently tests can be classified into three speed groups depending on whether the meson suite name ends in '-slow' or '-thorough' or neither.
This gets turned into make targets that match the name of the meson suite, with the speed suffix stripped. e.g.
* suite=block -> 'make check-block'
* suite=block-slow -> 'make check-block SPEED=slow'
* suite=block-thorough -> 'make check-block SPEED=thorough'
The set of tests under the "thorough" speed, however, can get rather large and it would be useful to have a way to expose further make targets for directly running a particular subset of tests.
This needs a way to run a target without requiring the SPEED variable, while also not having them enabled by default as if they were 'quick' tests.
This modifies mtest2make.py to support this idea by allowing for a new suffix '-optional' on a suite. When this is present, a correspondingly named make target will be created without the '-optional' suffix which will never be run automatically.
This is intended to be combined with use of other suites. For example, a single NBD test might be added to two suites, 'block-thorough' and 'block-nbd-optional'.
This would allow running it as part of all the block tests with 'make check-block SPEED=thorough', and as part of a standalone target 'make check-block-nbd'.
--- scripts/mtest2make.py | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-)
scripts/mtest2make.py +15 -7
@@ -22,7 +22,7 @@ def names(self, base): print(r''' SPEED = quick -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1)) +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1)) .speed.slow = $(sort $(filter-out %-thorough, $1)) .speed.thorough = $(sort $1)
@@ -66,10 +66,15 @@ def process_tests(test, targets, suites): s = s[:-9] suites[s].speeds.add('thorough') +def target_name(suite): + if suite.endswith('-optional'): + return suite[0:-9] + return suite + def emit_prolog(suites, prefix): - all_targets = ' '.join((f'{prefix}-{k}' + all_targets = ' '.join((f'{prefix}-{target_name(k)}' for k in sorted(suites.keys()))) - all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml' + all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml' for k in sorted(suites.keys()))) print() print(f'all-{prefix}-targets = {all_targets}')
@@ -83,14 +88,17 @@ def emit_prolog(suites, prefix): print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .') def emit_suite(name, suite, prefix): + tgtname = target_name(name) deps = ' '.join(sorted(suite.deps)) print() - print(f'.{prefix}-{name}.deps = {deps}') - print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)') + print(f'.{prefix}-{tgtname}.deps = {deps}') + print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)') names = ' '.join(sorted(suite.names(name))) - targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml' - if not name.endswith('-slow') and not name.endswith('-thorough'): + targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml' + if not name.endswith('-slow') and \ + not name.endswith('-thorough') and \ + not name.endswith('-optional'): targets += f' {prefix} {prefix}-report.junit.xml' print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)') # for the "base" suite possibly add FOO-slow and FOO-thorough
Signature · 1 line
2.54.0
[PULL 24/29] qom: have object_set_props_keyval return bool
In reply to Daniel P. BerrangéThis matches the convention established by the object_set_props and object_set_propv methods.
--- include/qom/object.h | 4 +++- qom/object_interfaces.c | 13 +++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-)
include/qom/object.h +3 -1
@@ -924,8 +924,10 @@ bool type_print_class_properties(const char *type);* * For each key in the dictionary, parse the value string if needed, * then set the corresponding property in @obj. +* +* Returns: %true on success, %false on error. */ -void object_set_props_from_keyval (Object *obj,const QDict *qdict, +bool object_set_props_from_keyval (Object *obj,const QDict *qdict,bool from_json,Error **errp);/**
qom/object_interfaces.c +9 -4
@@ -44,13 +44,14 @@ bool user_creatable_can_be_deleted(UserCreatable *uc) } } -static void object_set_props_from_qdict (Object *obj,const QDict *qdict, +static bool object_set_props_from_qdict (Object *obj,const QDict *qdict,Visitor *v,Error **errp) { +ERRP_GUARD ();const QDictEntry *e;if (!visit_start_struct (v,NULL ,NULL ,0 , errp)) { -return ; +return false; }for (e =qdict_first (qdict); e; e =qdict_next (qdict, e)) {if (!object_property_set (obj, e->key, v, errp)) {
@@ -60,19 +61,23 @@ static void object_set_props_from_qdict(Object *obj, const QDict *qdict,visit_check_struct (v, errp); out:visit_end_struct (v,NULL ); + +return *errp ==NULL ; } -void object_set_props_from_keyval (Object *obj,const QDict *qdict, +bool object_set_props_from_keyval (Object *obj,const QDict *qdict,bool from_json,Error **errp) { +bool ret;Visitor *v;if (from_json) { v =qobject_input_visitor_new (QOBJECT (qdict)); }else { v =qobject_input_visitor_new_keyval (QOBJECT (qdict)); } -object_set_props_from_qdict (obj, qdict, v, errp); + ret =object_set_props_from_qdict (obj, qdict, v, errp);visit_free (v); +return ret; } Object *user_creatable_add_type(const char *type, const char *id,
Signature · 1 line
2.54.0
[PULL 28/29] qom: allow object_new_with_prop* to trigger module loading
In reply to Daniel P. BerrangéThe object_new_with_prop* methods will shortly be replacing the user_creatable_add_type method. In order to do that, the object_new_with_prop* methods must allow module loading to be triggered for any types.
--- qom/object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
qom/object.c +1 -1
@@ -774,7 +774,7 @@ object_new_with_props_helper(const char *typename,return NULL ; } - klass =object_class_by_name (typename); + klass =module_object_class_by_name (typename);if (!klass) {error_setg (errp,"invalid object type: %s" , typename);return NULL ;
Signature · 1 line
2.54.0
[PULL 25/29] qom: move object_set_prop_keyval into object.c
In reply to Daniel P. BerrangéThis matches the location of the object_set_props and object_set_propv methods, since this method is not inherently tied to the user creatable interface. As part of this, object_set_props_from_qdict is also exposed as a public API since it is still called from object_interfaces.c.
--- include/qom/object.h | 47 +++++++++++++++++++++++++++-------------- qom/object.c | 36 +++++++++++++++++++++++++++++++ qom/object_interfaces.c | 36 ------------------------------- 3 files changed, 67 insertions(+), 52 deletions(-)
include/qom/object.h +31 -16
@@ -748,6 +748,37 @@ bool object_set_props(Object *obj, Error **errp, ...) G_GNUC_NULL_TERMINATED;*/ bool object_set_propv (Object *obj,va_list vargs,Error **errp); +/** +* object_set_props_from_qdict: +* @obj: a QOM object +* @qdict: a dictionary with the properties to be set +* @v: a visitor to iterate over @dict +* @errp: pointer to error object +* +* For each key in the dictionary, set the corresponding +* property in @obj. +* +* Returns: %true on success, %false on error. +*/ +bool object_set_props_from_qdict (Object *obj,const QDict *qdict, +Visitor *v,Error **errp); + +/** +* object_set_props_from_keyval: +* @obj: a QOM object +* @qdict: a dictionary with the properties to be set +* @from_json: true if leaf values of @qdict are typed, false if they +* are strings +* @errp: pointer to error object +* +* For each key in the dictionary, parse the value string if needed, +* then set the corresponding property in @obj. +* +* Returns: %true on success, %false on error. +*/ +bool object_set_props_from_keyval (Object *obj,const QDict *qdict, +bool from_json,Error **errp); +/** * object_initialize: * @obj: A pointer to the memory to be used for the object.
@@ -914,22 +945,6 @@ type_init(do_qemu_init_ ## type_array)*/ bool type_print_class_properties (const char *type); -/** -* object_set_props_from_keyval: -* @obj: a QOM object -* @qdict: a dictionary with the properties to be set -* @from_json: true if leaf values of @qdict are typed, false if they -* are strings -* @errp: pointer to error object -* -* For each key in the dictionary, parse the value string if needed, -* then set the corresponding property in @obj. -* -* Returns: %true on success, %false on error. -*/ -bool object_set_props_from_keyval (Object *obj,const QDict *qdict, -bool from_json,Error **errp); -/** * object_class_dynamic_cast_assert: * @klass: The #ObjectClass to attempt to cast.
qom/object.c +36 -0
@@ -23,6 +23,7 @@#include "qapi/qobject-input-visitor.h" #include "qapi/forward-visitor.h" #include "qapi/qapi-builtin-visit.h" +#include "qobject/qdict.h" #include "qobject/qjson.h" #include "qemu/id.h" #include "qapi/qmp/qerror.h"
@@ -838,6 +839,41 @@ bool object_set_propv(Object *obj,return true; } +bool object_set_props_from_qdict (Object *obj,const QDict *qdict, +Visitor *v,Error **errp) +{ +ERRP_GUARD (); +const QDictEntry *e; + +if (!visit_start_struct (v,NULL ,NULL ,0 , errp)) { +return false; + } +for (e =qdict_first (qdict); e; e =qdict_next (qdict, e)) { +if (!object_property_set (obj, e->key, v, errp)) { + goto out; + } + } +visit_check_struct (v, errp); +out: +visit_end_struct (v,NULL ); + +return *errp ==NULL ; +} + +bool object_set_props_from_keyval (Object *obj,const QDict *qdict, +bool from_json,Error **errp) +{ +bool ret; +Visitor *v; +if (from_json) { + v =qobject_input_visitor_new (QOBJECT (qdict)); + }else { + v =qobject_input_visitor_new_keyval (QOBJECT (qdict)); + } + ret =object_set_props_from_qdict (obj, qdict, v, errp); +visit_free (v); +return ret; +}Object *object_dynamic_cast (Object *obj,const char *typename) {
qom/object_interfaces.c +0 -36
@@ -44,42 +44,6 @@ bool user_creatable_can_be_deleted(UserCreatable *uc) } } -static bool object_set_props_from_qdict (Object *obj,const QDict *qdict, -Visitor *v,Error **errp) -{ -ERRP_GUARD (); -const QDictEntry *e; - -if (!visit_start_struct (v,NULL ,NULL ,0 , errp)) { -return false; - } -for (e =qdict_first (qdict); e; e =qdict_next (qdict, e)) { -if (!object_property_set (obj, e->key, v, errp)) { - goto out; - } - } -visit_check_struct (v, errp); -out: -visit_end_struct (v,NULL ); - -return *errp ==NULL ; -} - -bool object_set_props_from_keyval (Object *obj,const QDict *qdict, -bool from_json,Error **errp) -{ -bool ret; -Visitor *v; -if (from_json) { - v =qobject_input_visitor_new (QOBJECT (qdict)); - }else { - v =qobject_input_visitor_new_keyval (QOBJECT (qdict)); - } - ret =object_set_props_from_qdict (obj, qdict, v, errp); -visit_free (v); -return ret; -} -Object *user_creatable_add_type (const char *type,const char *id,const QDict *qdict,Visitor *v,Error **errp)
Signature · 1 line
2.54.0
[PULL 11/29] docs/devel/testing: expand documentation for 'make check-block'
In reply to Daniel P. BerrangéExplain in greater detail what 'check-block' will run for each format, and also document the new format specific targets.
--- docs/devel/testing/main.rst | 25 ++++++++++++++++++++++--- tests/qemu-iotests/meson.build | 2 ++ 2 files changed, 24 insertions(+), 3 deletions(-)
docs/devel/testing/main.rst +22 -3
@@ -236,9 +236,28 @@ same commit that alters the generator code. check-block ~~~~~~~~~~~ -``make check-block`` runs a subset of the block layer iotests (the tests that -are in the "auto" group). -See the "QEMU iotests" section below for more information. +There are a variety of ways to exercise the block layer I/O tests +via make targets. + +A default ``make check`` or ``make check-block`` command will exercise +the ``qcow2`` format, using the tests tagged into the ``auto`` group +only. + +These targets accept the ``SPEED`` variable to augment the set of tests +to run. A slightly more comprehensive test plan can be run by defining +``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw`` +formats. The most comprehensive test plan can be run by defining +``SPEED=thorough``, which enables all available tests for the formats +``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, ``vhdx``, +``vmdk``, and ``vpc``. + +Each of formats also has its own dedicated make target, named +``make check-block-$FORMAT`` which will run all available tests for +the designated format and does not require the ``SPEED`` variable +to be set. + +See the "QEMU iotests" section below for more information on the +block I/O test framework that is leveraged by these ``make`` targets. .. _qemu-iotests:
tests/qemu-iotests/meson.build +2 -0
@@ -10,6 +10,8 @@ endif qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd] qemu_iotests_env = {'PYTHON': python.full_path()} +# If altering this definition, also update docs/devel/testing/main.rst +# section on 'check-block' targets to reflect the changes qemu_iotests_formats = { 'qcow2': 'quick', 'raw': 'slow',
Signature · 1 line
2.54.0
[PULL 10/29] iotests: add a meson suite / make target per block I/O tests format
In reply to Daniel P. BerrangéCurrently each block format is classified as either 'quick', 'slow' or 'thorough' and this controls whether its I/O tests are added to the meson suites 'block-quick', 'block-slow' or 'block-thorough'.
This suites are exposed the 'check-block' make target, accepting the optional SPEED variable.
As we add more formats to the 'thorough' group, however, it becomes increasingly large and time consuming to run. What is needed is a make target that can exercise all tests for an individual format, regardless of speed classification.
This makes use of the previous enhancement to mtest2make.py to introduce new meson suites 'block-$FORMAT-optional', which translate to new top level make targets 'check-block-$FORMAT'. These new targets always run all tests and as such do not need the SPEED variable to be set, but are not triggered by 'make check' or 'make check-block'.
--- tests/Makefile.include | 3 ++- tests/qemu-iotests/meson.build | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
tests/Makefile.include +2 -1
@@ -14,7 +14,8 @@ check-help: @echo " $(MAKE) check-unit Run qobject tests" @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" @echo " $(MAKE) check-tracetool Run tracetool generator tests" - @echo " $(MAKE) check-block Run block tests" + @echo " $(MAKE) check-block Run block tests (all formats)" + @echo " $(MAKE) check-block-FORMAT Run block tests (only for FORMAT)" ifneq ($(filter $(all-check-targets), check-softfloat),) @echo " $(MAKE) check-softfloat Run FPU emulation tests" @echo " $(MAKE) check-tcg Run TCG tests"
tests/qemu-iotests/meson.build +2 -1
@@ -62,7 +62,8 @@ foreach format, speed: qemu_iotests_formats endforeach endif - suites = [] + # Every format gets put in the format specific suite + suites = ['block-' + format + '-optional'] # Any format tagged quick or slow also gets added to slow # otherwise its tagged thorough if speed != 'thorough'
Signature · 1 line
2.54.0
[PULL 14/29] iotests: validate dmsetup result in test 128
In reply to Daniel P. BerrangéThe I/O test 128 uses 'dmsetup create' to create a device, optionally using sudo to elevate privileges.
This dmsetup command works in GitLab CI, however, the test then fails with a missing device name:
1..1
# running raw 128 not ok raw 128 ----------------------------------- stderr ----------------------------------- --- /builds/berrange/qemu/tests/qemu-iotests/128.out +++ /builds/berrange/qemu/build/scratch/raw-file-128/128.out.bad @@ -1,5 +1,5 @@ QA output created by 128
== reading from error device ==
-read failed: Input/output error
+qemu-io: can't open device /dev/mapper/eiodev16546: Could not open '/dev/mapper/eiodev16546': No such file or directory
*** done
(test program exited with status code 1)
It is believed that this is due to the build env using a manually populated /dev, such that the device mapper node won't ever appear. It is not a race, since a test adding a sleep did not result in the device appearing.
--- tests/qemu-iotests/128 | 6 ++++++ 1 file changed, 6 insertions(+)
tests/qemu-iotests/128 +6 -0
@@ -42,6 +42,12 @@ _setup_eiodev() echo "0 $((1024 * 1024 * 1024 / 512)) error" | \ $cmd dmsetup create "$devname" 2>/dev/null if [ "$?" -eq 0 ]; then + DEV="/dev/mapper/$devname" + if ! -e $DEV + then + _notrun "Device $DEV not appearing" + fi + sudo="$cmd" return fi
Signature · 1 line
2.54.0
[PULL 18/29] gitlab: remove I/O tests from build-tcg-disabled job
In reply to Daniel P. BerrangéNow that we have dedicated CI jobs for running I/O tests on each supported format/protocol, we no longer need to special case a run of a hand picked set of tests in the build-tcg-disabled job.
--- .gitlab-ci.d/buildtest.yml | 9 --------- 1 file changed, 9 deletions(-)
.gitlab-ci.d/buildtest.yml +0 -9
@@ -366,15 +366,6 @@ build-tcg-disabled: - make -j"$JOBS" - make check-unit - make check-qapi-schema - - ./run tests/qemu-iotests/check -raw 001 002 003 004 005 008 009 - 010 011 012 021 025 032 033 048 052 063 077 086 101 104 106 - 113 148 150 151 152 157 159 160 163 170 171 184 192 194 208 - 221 226 227 236 253 277 image-fleecing - - ./run tests/qemu-iotests/check -qcow2 028 051 056 057 058 065 068 - 082 085 091 095 096 102 122 124 132 139 142 144 145 151 152 - 155 157 165 194 196 200 202 208 209 216 218 227 234 246 247 - 248 250 254 255 257 258 260 261 262 263 264 270 272 273 277 - 279 image-fleecing - make distclean build-user:
Signature · 1 line
2.54.0
[PULL 23/29] qom: shorten name of object_set_properties_from_keyval
In reply to Daniel P. BerrangéThis matches the convention established by the object_set_props and object_set_propv methods.
--- include/qom/object.h | 6 +++--- qom/object_interfaces.c | 12 ++++++------ system/qdev-monitor.c | 4 ++-- system/vl.c | 7 ++++--- 4 files changed, 15 insertions(+), 14 deletions(-)
include/qom/object.h +3 -3
@@ -915,7 +915,7 @@ type_init(do_qemu_init_ ## type_array)bool type_print_class_properties (const char *type);/** -* object_set_properties_from_keyval: +* object_set_props_from_keyval: * @obj: a QOM object * @qdict: a dictionary with the properties to be set * @from_json: true if leaf values of @qdict are typed, false if they
@@ -925,8 +925,8 @@ bool type_print_class_properties(const char *type);* For each key in the dictionary, parse the value string if needed, * then set the corresponding property in @obj. */ -void object_set_properties_from_keyval (Object *obj,const QDict *qdict, -bool from_json,Error **errp); +void object_set_props_from_keyval (Object *obj,const QDict *qdict, +bool from_json,Error **errp);/** * object_class_dynamic_cast_assert:
qom/object_interfaces.c +6 -6
@@ -44,8 +44,8 @@ bool user_creatable_can_be_deleted(UserCreatable *uc) } } -static void object_set_properties_from_qdict (Object *obj,const QDict *qdict, -Visitor *v,Error **errp) +static void object_set_props_from_qdict (Object *obj,const QDict *qdict, +Visitor *v,Error **errp) {const QDictEntry *e;
@@ -62,8 +62,8 @@ out:visit_end_struct (v,NULL ); } -void object_set_properties_from_keyval (Object *obj,const QDict *qdict, -bool from_json,Error **errp) +void object_set_props_from_keyval (Object *obj,const QDict *qdict, +bool from_json,Error **errp) {Visitor *v;if (from_json) {
@@ -71,7 +71,7 @@ void object_set_properties_from_keyval(Object *obj, const QDict *qdict, }else { v =qobject_input_visitor_new_keyval (QOBJECT (qdict)); } -object_set_properties_from_qdict (obj, qdict, v, errp); +object_set_props_from_qdict (obj, qdict, v, errp);visit_free (v); }
@@ -110,7 +110,7 @@ Object *user_creatable_add_type(const char *type, const char *id,assert (qdict); obj =object_new_with_class (klass); -object_set_properties_from_qdict (obj, qdict, v, &local_err); +object_set_props_from_qdict (obj, qdict, v, &local_err);if (local_err) { goto out; }
system/qdev-monitor.c +2 -2
@@ -730,8 +730,8 @@ DeviceState *qdev_device_add_from_qdict(const QDict *opts,qdict_del (properties,"bus" );qdict_del (properties,"id" ); -object_set_properties_from_keyval (&dev->parent_obj, properties, from_json, - errp); +object_set_props_from_keyval (&dev->parent_obj, properties, from_json, + errp);qobject_unref (properties);if (*errp) { goto err_del_dev;
system/vl.c +4 -3
@@ -2010,7 +2010,8 @@ static bool object_create_early(const char *type)static void qemu_apply_machine_options (QDict *qdict) { -object_set_properties_from_keyval (OBJECT (current_machine), qdict, false, &error_fatal); +object_set_props_from_keyval (OBJECT (current_machine), qdict, + false, &error_fatal);if (semihosting_enabled (false) && !semihosting_get_argc ()) {/* fall back to the -kernel/-append */
@@ -2225,8 +2226,8 @@ static void qemu_create_machine(QDict *qdict)keyval_parse (machine_class->default_machine_opts,NULL ,NULL , &error_abort);qemu_apply_legacy_machine_options (default_opts); -object_set_properties_from_keyval (OBJECT (current_machine), default_opts, - false, &error_abort); +object_set_props_from_keyval (OBJECT (current_machine), default_opts, + false, &error_abort);qobject_unref (default_opts); } }
Signature · 1 line
2.54.0
[PULL 20/29] hw/vfio-user: use a valid object ID for iothread
In reply to Daniel P. BerrangéThe objectg_new_with_propv/props methods failed to validate that the QOM "id" was well formed. This allowed the vfio-user code to use an invalid ID of "VFIO user" (space is not permitted) in its internal code.
--- hw/vfio-user/proxy.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/vfio-user/proxy.c +1 -1
@@ -933,7 +933,7 @@ VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp)qemu_cond_init (&proxy->close_cv);if (vfio_user_iothread ==NULL ) { - vfio_user_iothread =iothread_create ("VFIO user" , errp); + vfio_user_iothread =iothread_create ("vfio-user" , errp); } proxy->ctx =iothread_get_aio_context (vfio_user_iothread);
Signature · 1 line
2.54.0
[PULL 17/29] gitlab: add jobs for thorough block tests
In reply to Daniel P. BerrangéCI is only exercising the qcow2 'auto' tests currently. As a result we get no exposure of changes which cause regressions in other block format drivers.
This adds new CI jobs for each block format, that will run the target 'make check-block-$FORMAT'. The jobs are separate so that we have the ability to make each format gating or not, depending on their level of reliability.
The 'centos' image is used to run the I/O tests since several tests have an implicit dependency on x86_64-softmmu, and thus break with other architecture targets. The 'centos' build job is the only one that creates the x86_64-softmmu target in CI. Ideally this target portability in I/O tests would be fixed to avoid this limitation.
--- .gitlab-ci.d/buildtest.yml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
.gitlab-ci.d/buildtest.yml +18 -0
@@ -174,6 +174,24 @@ build-system-centos: x86_64-softmmu rx-softmmu sh4-softmmu MAKE_CHECK_ARGS: check-build + +# NB: block-XXX jobs use 'centos' since that is the build +# job that provides the x86_64-softmmu. Some I/O tests +# are currently buggy and blindly assume characteristics +# of x86 (such as PCIe) causing failures with other arches + +block: + extends: .native_test_job_template + needs: + - job: build-system-centos + artifacts: true + variables: + IMAGE: centos9 + MAKE_CHECK_ARGS: "check-block-$FORMAT" + parallel: + matrix: + - FORMAT: [luks, nbd, parallels, qcow2, qed, raw, vdi, vhdx, vmdk, vpc] + # Previous QEMU release. Used for cross-version migration tests. build-previous-qemu: extends: .native_build_job_template
Signature · 1 line
2.54.0
[PULL 08/29] scripts/mtest2make: ensure output has stable sorting
In reply to Daniel P. BerrangéWhen debugging mtest2make.py changes it is important to be able to compare the old and new output. This requires that any lists in the output have stable sort ordering.
--- scripts/mtest2make.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
scripts/mtest2make.py +6 -4
@@ -67,8 +67,10 @@ def process_tests(test, targets, suites): suites[s].speeds.add('thorough') def emit_prolog(suites, prefix): - all_targets = ' '.join((f'{prefix}-{k}' for k in suites.keys())) - all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml' for k in suites.keys())) + all_targets = ' '.join((f'{prefix}-{k}' + for k in sorted(suites.keys()))) + all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml' + for k in sorted(suites.keys()))) print() print(f'all-{prefix}-targets = {all_targets}') print(f'all-{prefix}-xml = {all_xml}')
@@ -81,12 +83,12 @@ def emit_prolog(suites, prefix): print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .') def emit_suite(name, suite, prefix): - deps = ' '.join(suite.deps) + deps = ' '.join(sorted(suite.deps)) print() print(f'.{prefix}-{name}.deps = {deps}') print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)') - names = ' '.join(suite.names(name)) + names = ' '.join(sorted(suite.names(name))) targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml' if not name.endswith('-slow') and not name.endswith('-thorough'): targets += f' {prefix} {prefix}-report.junit.xml'
Signature · 1 line
2.54.0
[PULL 12/29] iotests: add nbd and luks to the I/O test suites
In reply to Daniel P. BerrangéThis introduces new suites for running I/O tests on NBD and LUKS drivers, giving new make targets
* make check-block-luks
* make check-block-nbd
as well as adding their tests to 'make check-block SPEED=thorough'
--- docs/devel/testing/main.rst | 4 ++-- tests/qemu-iotests/meson.build | 4 +++- 2 files changed, 5 insertions(+), 3 deletions(-)
docs/devel/testing/main.rst +2 -2
@@ -248,8 +248,8 @@ to run. A slightly more comprehensive test plan can be run by defining ``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw`` formats. The most comprehensive test plan can be run by defining ``SPEED=thorough``, which enables all available tests for the formats -``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, ``vhdx``, -``vmdk``, and ``vpc``. +``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, +``vhdx``, ``vmdk``, and ``vpc``. Each of formats also has its own dedicated make target, named ``make check-block-$FORMAT`` which will run all available tests for
tests/qemu-iotests/meson.build +3 -1
@@ -15,12 +15,14 @@ qemu_iotests_env = {'PYTHON': python.full_path()} qemu_iotests_formats = { 'qcow2': 'quick', 'raw': 'slow', + 'luks': 'thorough', + 'nbd': 'thorough', 'parallels': 'thorough', 'qed': 'thorough', 'vdi': 'thorough', 'vhdx': 'thorough', 'vmdk': 'thorough', - 'vpc': 'thorough' + 'vpc': 'thorough', } foreach k, v : emulators
Signature · 1 line
2.54.0
[PULL 16/29] iotests: mark 151, 181, 185 & 308 as flaky tests
In reply to Daniel P. BerrangéIntroduce a "_flaky_test" function for shell based I/O tests which accepts a GitLab issue URL, and causes the I/O test to be skipped unless the $QEMU_TEST_FLAKY_TESTS environment variable is set.
The equivalent "skip_flaky" test decorator is added for python based I/O tests with the same behaviour.
This is used by:
* Test 151 which fails in QEMU private AWS runners due to failure to make progress in time
* Test 181 which fails with a non-responsive QEMU on AWS runners with the QED format
* Test 185 which fails in GitLab shared runners
* Test 308 which fails to see disk usage increase after fallocate
--- tests/qemu-iotests/151 | 1 + tests/qemu-iotests/181 | 2 ++ tests/qemu-iotests/185 | 1 + tests/qemu-iotests/308 | 2 ++ tests/qemu-iotests/common.rc | 16 ++++++++++++++++ tests/qemu-iotests/iotests.py | 16 ++++++++++++++++ 6 files changed, 38 insertions(+)
tests/qemu-iotests/151 +1 -0
@@ -308,6 +308,7 @@ class TestThrottledWithNbdExportBase(iotests.QMPTestCase): class TestLowThrottledWithNbdExport(TestThrottledWithNbdExportBase): iops = 16 + @iotests.skip_flaky("https://gitlab.com/qemu-project/qemu/-/work_items/3513") def testUnderLoad(self): ''' Throttle the source node, then issue a whole bunch of external requests
tests/qemu-iotests/181 +2 -0
@@ -48,6 +48,8 @@ _unsupported_fmt qcow vdi vhdx vmdk vpc vvfat parallels _supported_proto generic _supported_os Linux +_flaky_test https://gitlab.com/qemu-project/qemu/-/work_items/3515 + size=64M _make_test_img $size
tests/qemu-iotests/185 +1 -0
@@ -50,6 +50,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +_flaky_test https://gitlab.com/qemu-project/qemu/-/issues/3270 size=$((64 * 1048576)) TEST_IMG="${TEST_IMG}.base" _make_test_img $size
tests/qemu-iotests/308 +2 -0
@@ -53,6 +53,8 @@ _supported_proto file # We create the FUSE export manually _supported_os Linux # We need /dev/urandom _require_disk_usage +_flaky_test https://gitlab.com/qemu-project/qemu/-/work_items/3514 + # $1: Export ID # $2: Options (beyond the node-name and ID) # $3: Expected return value (defaults to 'return')
tests/qemu-iotests/common.rc +16 -0
@@ -1088,5 +1088,21 @@ _qcow2_dump_header() fi } +# This must be referenced after any _require_ lines, so that +# test filtering happens first +_flaky_test() +{ + if test -z "$1" + then + echo "A GitLab issue URL must be provided for a flaky test" + exit 1 + fi + + if test -z "$QEMU_TEST_FLAKY_TESTS" + then + _notrun "Test is flaky (see $1) and \$QEMU_TEST_FLAKY_TESTS is not set" + fi +} + # make sure this script returns success true
tests/qemu-iotests/iotests.py +16 -0
@@ -1603,6 +1603,22 @@ def func_wrapper(*args, **kwargs): return func(*args, **kwargs) return func_wrapper +def skip_flaky(bugurl): + '''Skip Test Decorator + Always skips test due to unreliable design. + Requires a bug report URL for historical record.''' + def skip_test_decorator(func): + def func_wrapper(*args, **kwargs): + if os.environ.get("QEMU_TEST_FLAKY_TESTS", None) is None: + case_notrun( + ('{}: test is flaky (see {}) and $QEMU_TEST_FLAKY_TESTS ' + + 'is not set').format(args[0], bugurl)) + return None + else: + return func(*args, **kwargs) + return func_wrapper + return skip_test_decorator + # We need to filter out the time taken from the output so that # qemu-iotest can reliably diff the results against master output, # and hide skipped tests from the reference output.
Signature · 1 line
2.54.0
[PULL 27/29] qom: fix ability to create objects without a parent
In reply to Daniel P. Berrangéobject_new_with_propv allowed id/parent to be optional, in which case the caller was expected to own the returned object. Unfortunately a trailing object_unref() meant that the returned object was already freed.
It is confusing to have a single method with two different ownership scenarios for the returned object.
Make id/parent mandatory in object_new_with_propv once more, and add a new object_new_with_propv_parentless that does not accept id/parent at all and lets the caller own the returned reference.
The helper method has abstracted the way properties are represented and set in order to facilitate the subsequent commit.
Unit tests are added to address the root cause that allowed the bug to slip through in commit 6134d752.
--- include/qom/object.h | 47 ++++++++++++++++++ qom/object.c | 74 +++++++++++++++++++++++---- tests/unit/check-qom-proplist.c | 88 ++++++++++++++++++++++++++++----- 3 files changed, 187 insertions(+), 22 deletions(-)
include/qom/object.h +47 -0
@@ -719,6 +719,53 @@ Object *object_new_with_props_from_qdict(const char *typename, Visitor *v, Error **errp); +/** +* object_new_with_props_parentless: +* @typename: The name of the type of the object to instantiate. +* @errp: pointer to error object +* @...: list of property names and values +* +* Behaviour as object_new_with_props(), except the object +* will not be added to any parent and thus the caller will +* own the returned instance. The caller must call +* object_unref when it is no longer required. +*/ +Object *object_new_with_props_parentless (const char *typename, +Error **errp, + ...)G_GNUC_NULL_TERMINATED ; + +/** +* object_new_with_propv_parentless: +* @typename: The name of the type of the object to instantiate. +* @vargs: list of property names and values +* @errp: pointer to error object +* +* Behaviour as object_new_with_propv(), except the object +* will not be added to any parent and thus the caller will +* own the returned instance. The caller must call +* object_unref when it is no longer required. +*/ +Object *object_new_with_propv_parentless (const char *typename, +va_list vargs, +Error **errp); + +/** +* object_new_with_props_from_qdict_parentless: +* @typename: The name of the type of the object to instantiate. +* @props: dictionary of property names and values +* @v: visitor to iterate over @props +* @errp: pointer to error object +* +* Behaviour as object_new_with_props_from_qdict(), except the +* object will not be added to any parent and thus the caller +* will own the returned instance. The caller must call +* object_unref when it is no longer required. +*/ +Object *object_new_with_props_from_qdict_parentless (const char *typename, +const QDict *props, +Visitor *v, +Error **errp); +/** * object_set_props: * @obj: the object instance to set properties on
qom/object.c +65 -9
120 lines not on this page · raw message
tests/unit/check-qom-proplist.c +75 -13
130 lines not on this page · raw message
Signature · 1 line
2.54.0
[PULL 21/29] qom: validate ID format when creating objects
In reply to Daniel P. BerrangéThe object_new_with_props/propv methods failed to validate the ID string format, thus diverging from user_creatable_add_type.
--- qom/object.c | 9 +++++++++ 1 file changed, 9 insertions(+)
qom/object.c +9 -0
@@ -24,6 +24,8 @@#include "qapi/forward-visitor.h" #include "qapi/qapi-builtin-visit.h" #include "qobject/qjson.h" +#include "qemu/id.h" +#include "qapi/qmp/qerror.h" #include "trace.h" /* TODO: replace QObject with a simpler visitor to avoid a dependency
@@ -755,6 +757,13 @@ Object *object_new_with_propv(const char *typename,ObjectClass *klass;UserCreatable *uc; +if (id !=NULL && !id_wellformed (id)) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"id" ,"an identifier" ); +error_append_hint (errp,"Identifiers consist of letters, digits, " +"'-', '.', '_', starting with a letter.\n" ); +return NULL ; + } + klass =object_class_by_name (typename);if (!klass) {error_setg (errp,"invalid object type: %s" , typename);
Signature · 1 line
2.54.0
[PULL 26/29] qom: add object_new_with_props_from_qdict
In reply to Daniel P. BerrangéThis will be used to replace user_creatable_add_type with an impl that shares most code with object_new_with_props, and is not tied to the user creatable interface.
--- include/qom/object.h | 19 +++++++++++ qom/object.c | 75 ++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 88 insertions(+), 6 deletions(-)
include/qom/object.h +19 -0
@@ -700,6 +700,25 @@ Object *object_new_with_propv(const char *typename,va_list vargs, Error **errp); +/** +* object_new_with_props_from_qdict: +* @typename: The name of the type of the object to instantiate. +* @parent: the parent object +* @id: The unique ID of the object +* @props: dictionary of property names and values +* @v: visitor to iterate over @props +* @errp: pointer to error object +* +* A variant of object_new_with_props() which accepts the +* properties in a QDict. +*/ +Object *object_new_with_props_from_qdict (const char *typename, +Object *parent, +const char *id, +const QDict *props, +Visitor *v, +Error **errp); +/** * object_set_props: * @obj: the object instance to set properties on
qom/object.c +69 -6
93 lines not on this page · raw message
Signature · 1 line
2.54.0
[PULL 29/29] qom: drop user_creatable_add_type method
In reply to Daniel P. BerrangéThis can be replaced by object_new_with_props_from_qdict, which does functionally the same job, but the caller does not own the returned reference, instead the parent object owns it.
In one case we can use object_new_with_props_from_qdict_owned instead since the object is not intended to have any parent.
--- authz/listfile.c | 4 +- include/qom/object_interfaces.h | 18 --------- qom/object_interfaces.c | 70 ++------------------------------- tests/unit/check-qom-proplist.c | 5 +-- 4 files changed, 7 insertions(+), 90 deletions(-)
authz/listfile.c +2 -2
@@ -79,8 +79,8 @@ qauthz_list_file_load(QAuthZListFile *fauthz, Error **errp) v =qobject_input_visitor_new (obj); - ret = (QAuthZ *)user_creatable_add_type (TYPE_QAUTHZ_LIST , -NULL , pdict, v, errp); + ret =QAUTHZ (object_new_with_props_from_qdict_parentless ( +TYPE_QAUTHZ_LIST , pdict, v, errp)); cleanup:visit_free (v);
include/qom/object_interfaces.h +0 -18
@@ -69,24 +69,6 @@ bool user_creatable_complete(UserCreatable *uc, Error **errp);*/ bool user_creatable_can_be_deleted (UserCreatable *uc); -/** -* user_creatable_add_type: -* @type: the object type name -* @id: the unique ID for the object -* @qdict: the object properties -* @v: the visitor -* @errp: if an error occurs, a pointer to an area to store the error -* -* Create an instance of the user creatable object @type, placing -* it in the object composition tree with name @id, initializing -* it with properties from @qdict -* -* Returns: the newly created object or NULL on error -*/ -Object *user_creatable_add_type (const char *type,const char *id, -const QDict *qdict, -Visitor *v,Error **errp); -/** * user_creatable_add_qapi: * @options: the object definition
qom/object_interfaces.c +3 -67
87 lines not on this page · raw message
tests/unit/check-qom-proplist.c +2 -3
@@ -461,10 +461,9 @@ static void test_dummy_createlist_parentless(void)static bool test_create_obj (QDict *qdict,Error **errp) {Visitor *v =qobject_input_visitor_new_keyval (QOBJECT (qdict)); -Object *obj =user_creatable_add_type (TYPE_DUMMY ,"dev0" , qdict, v, errp); - +Object *obj =object_new_with_props_from_qdict ( +TYPE_DUMMY ,object_get_objects_root (),"dev0" , qdict, v, errp);visit_free (v); -object_unref (obj);return !!obj; }
Signature · 1 line
2.54.0
[PULL 22/29] qom: make errp last param in methods taking va_list
In reply to Daniel P. Berrangéobject_new_with_props can't put 'errp' last due to the use of variadic arguments. That constraint does not apply to the use of va_list with object_new_with_propv, so follow normal practice with 'errp' placement.
The same rationale applies to object_set_propv.
--- include/qom/object.h | 10 +++++----- qom/object.c | 20 ++++++++++---------- tests/unit/check-qom-proplist.c | 4 ++-- 3 files changed, 17 insertions(+), 17 deletions(-)
include/qom/object.h +5 -5
@@ -689,16 +689,16 @@ Object *object_new_with_props(const char *typename,* @typename: The name of the type of the object to instantiate. * @parent: the parent object * @id: The unique ID of the object -* @errp: pointer to error object * @vargs: list of property names and values +* @errp: pointer to error object * * See object_new_with_props() for documentation. */ Object *object_new_with_propv (const char *typename,Object *parent,const char *id, -Error **errp, -va_list vargs); +va_list vargs, +Error **errp);/** * object_set_props:
@@ -739,14 +739,14 @@ bool object_set_props(Object *obj, Error **errp, ...) G_GNUC_NULL_TERMINATED;/** * object_set_propv: * @obj: the object instance to set properties on -* @errp: pointer to error object * @vargs: list of property names and values +* @errp: pointer to error object * * See object_set_props() for documentation. * * Returns: %true on success, %false on error. */ -bool object_set_propv (Object *obj,Error **errp,va_list vargs); +bool object_set_propv (Object *obj,va_list vargs,Error **errp);/** * object_initialize:
qom/object.c +10 -10
@@ -541,7 +541,7 @@ bool object_initialize_child_with_propsv(Object *parentobj,object_initialize (childobj, size, type); obj =OBJECT (childobj); -if (!object_set_propv (obj, errp, vargs)) { +if (!object_set_propv (obj, vargs, errp)) { goto out; }
@@ -740,7 +740,7 @@ Object *object_new_with_props(const char *typename,Object *obj;va_start (vargs, errp); - obj =object_new_with_propv (typename, parent, id, errp, vargs); + obj =object_new_with_propv (typename, parent, id, vargs, errp);va_end (vargs);return obj;
@@ -750,8 +750,8 @@ Object *object_new_with_props(const char *typename,Object *object_new_with_propv (const char *typename,Object *parent,const char *id, -Error **errp, -va_list vargs) +va_list vargs, +Error **errp) {Object *obj;ObjectClass *klass;
@@ -776,7 +776,7 @@ Object *object_new_with_propv(const char *typename, } obj =object_new_with_type (klass->type); -if (!object_set_propv (obj, errp, vargs)) { +if (!object_set_propv (obj, vargs, errp)) { goto error; }
@@ -804,14 +804,14 @@ Object *object_new_with_propv(const char *typename,bool object_set_props (Object *obj, -Error **errp, - ...) +Error **errp, + ...) {va_list vargs;bool ret;va_start (vargs, errp); - ret =object_set_propv (obj, errp, vargs); + ret =object_set_propv (obj, vargs, errp);va_end (vargs);return ret;
@@ -819,8 +819,8 @@ bool object_set_props(Object *obj,bool object_set_propv (Object *obj, -Error **errp, -va_list vargs) +va_list vargs, +Error **errp) {const char *propname;
tests/unit/check-qom-proplist.c +2 -2
@@ -373,8 +373,8 @@ static Object *new_helper(Error **errp, obj =object_new_with_propv (TYPE_DUMMY , parent,"dummy0" , - errp, - vargs); + vargs, + errp);va_end (vargs);return obj; }
Signature · 1 line
2.54.0
[PULL 19/29] qom: add trace events for object/property lifecycle
In reply to Daniel P. BerrangéThis adds tracing around object allocation & finalization, the addition & deletion of properties, and the addition & deletion of children.
--- qom/object.c | 34 +++++++++++++++++++++++++++------- qom/trace-events | 12 ++++++++++-- 2 files changed, 37 insertions(+), 9 deletions(-)
qom/object.c +27 -7
@@ -594,6 +594,8 @@ static void object_property_del_all(Object *obj)object_property_iter_init (&iter, obj);while ((prop =object_property_iter_next (&iter)) !=NULL ) {if (g_hash_table_add (done, prop)) { +trace_object_property_del (obj, obj->class->type->name, + prop->name, prop->opaque);if (prop->release) { prop->release (obj, prop->name, prop->opaque); released = true;
@@ -612,10 +614,14 @@ static void object_property_del_child(Object *obj, Object *child)GHashTableIter iter;gpointer key, value; +trace_object_property_del_child (obj, obj->class->type->name, + child, child->class->type->name);g_hash_table_iter_init (&iter, obj->properties);while (g_hash_table_iter_next (&iter, &key, &value)) { prop = value;if (object_property_is_child (prop) && prop->opaque == child) { +trace_object_property_del (obj, obj->class->type->name, + prop->name, prop->opaque);if (prop->release) { prop->release (obj, prop->name, prop->opaque); prop->release =NULL ;
@@ -655,7 +661,7 @@ static void object_finalize(void *data) {Object *obj = data;TypeImpl *ti = obj->class->type; - +trace_object_finalize (obj, obj->class->type->name);object_property_del_all (obj);object_deinit (obj, ti);
@@ -705,6 +711,7 @@ static Object *object_new_with_type(Type type)object_initialize_with_type (obj, size, type); obj->free = obj_free; +trace_object_new (obj, obj->class->type->name);return obj; }
@@ -835,8 +842,9 @@ Object *object_dynamic_cast(Object *obj, const char *typename)Object *object_dynamic_cast_assert (Object *obj,const char *typename,const char *file,int line,const char *func) { -trace_object_dynamic_cast_assert (obj ? obj->class->type->name :"(null)" , - typename, file, line, func); +trace_object_dynamic_cast_assert ( + obj, obj ? obj->class->type->name :"(null)" , + typename, file, line, func);#ifdef CONFIG_QOM_CAST_DEBUG int i;
@@ -926,8 +934,9 @@ ObjectClass *object_class_dynamic_cast_assert(ObjectClass *class, {ObjectClass *ret; -trace_object_class_dynamic_cast_assert (class ? class->type->name :"(null)" , - typename, file, line, func); +trace_object_class_dynamic_cast_assert ( + class ? class->type->name :"(null)" , + typename, file, line, func);#ifdef CONFIG_QOM_CAST_DEBUG int i;
@@ -1211,6 +1220,8 @@ object_property_try_add(Object *obj, const char *name, const char *type, prop->release = release; prop->opaque = opaque; +trace_object_property_add (obj, obj->class->type->name, + prop->name, prop->opaque);g_hash_table_insert (obj->properties, prop->name, prop);return prop; }
@@ -1249,6 +1260,8 @@ object_class_property_add(ObjectClass *klass, prop->release = release; prop->opaque = opaque; +trace_object_class_property_add (klass->type->name, prop->name, + prop->opaque);g_hash_table_insert (klass->properties, prop->name, prop);return prop;
@@ -1337,6 +1350,8 @@ void object_property_del(Object *obj, const char *name) {ObjectProperty *prop =g_hash_table_lookup (obj->properties, name); +trace_object_property_del (obj, obj->class->type->name, prop->name, + prop->opaque);if (prop->release) { prop->release (obj, name, prop->opaque); }
@@ -1625,8 +1640,11 @@ int object_property_get_enum(Object *obj, const char *name,bool object_property_parse (Object *obj,const char *name,const char *string,Error **errp) { -Visitor *v =string_input_visitor_new (string); -bool ok =object_property_set (obj, name, v, errp); +Visitor *v; +bool ok; +trace_object_property_parse (obj, obj->class->type->name, name, string); + v =string_input_visitor_new (string); + ok =object_property_set (obj, name, v, errp);visit_free (v);return ok;
@@ -1757,6 +1775,8 @@ object_property_try_add_child(Object *obj, const char *name, g_autofree char *type = NULL;ObjectProperty *op; +trace_object_property_add_child (obj, obj->class->type->name, name, + child, child->class->type->name);assert (!child->parent); type =g_strdup_printf ("child<%s>" ,object_get_typename (child));
qom/trace-events +10 -2
@@ -1,5 +1,13 @@ # See docs/devel/tracing.rst for syntax documentation. # object.c -object_dynamic_cast_assert(const char *type, const char *target, const char *file, int line, const char *func) "%s->%s (%s:%d:%s)" -object_class_dynamic_cast_assert(const char *type, const char *target, const char *file, int line, const char *func) "%s->%s (%s:%d:%s)" +object_dynamic_cast_assert(void *obj, const char *type, const char *target, const char *file, int line, const char *func) "obj=%p type=%s->%s (%s:%d:%s)" +object_finalize(void *obj, const char *type) "obj=%p type=%s" +object_new(void *obj, const char *type) "obj=%p type=%s" +object_property_add(void *obj, const char *type, const char *name, void *value) "obj=%p type=%s name=%s value=%p" +object_property_add_child(void *obj, const char *type, const char *name, void *child, const char *childtype) "obj=%p type=%s name=%s child=%p child-type=%s" +object_property_del(void *obj, const char *type, const char *name, void *value) "obj=%p type=%s name=%s value=%p" +object_property_del_child(void *obj, const char *type, void *child, const char *childtype) "obj=%p type=%s child=%p child-type=%s" +object_property_parse(void *obj, const char *type, const char *name, const char *value) "obj=%p type=%s prop=%s value=%s" +object_class_dynamic_cast_assert(const char *type, const char *target, const char *file, int line, const char *func) "type=%s->%s (%s:%d:%s)" +object_class_property_add(const char *type, const char *name, void *value) "type=%s name=%s value=%p"
Signature · 1 line
2.54.0
Stefan Hajnoczi
AppliedApplied, thanks.
Please update the changelog at https://wiki.qemu.org/ChangeLog/11.1 for any user-visible changes.