Skip to contentSkip to conversation

tests: Do not treat the iotests as separate meson test target anymore

PATCH In review Thomas Huth 2 participants 7 messages Mar 10, 2022 – Mar 21, 2022 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 tests: Do not treat the iotests as separate meson test target anymore

If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now.

---
v4: updated commit description

 meson.build            | 6 +++---
 scripts/mtest2make.py  | 4 ----
 tests/Makefile.include | 9 +--------
 3 files changed, 4 insertions(+), 15 deletions(-)
meson.build +3 -3
index 2d6601467f..4f5b9f5ec5 100644
@@ -3,9 +3,9 @@ project('qemu', ['c'], meson_version: '>=0.59.3',
                           'b_staticpic=false', 'stdsplit=false'],
         version: files('VERSION'))
 
-add_test_setup('quick', exclude_suites: ['block', 'slow', 'thorough'], is_default: true)
-add_test_setup('slow', exclude_suites: ['block', 'thorough'], env: ['G_TEST_SLOW=1', 'SPEED=slow'])
-add_test_setup('thorough', exclude_suites: ['block'], env: ['G_TEST_SLOW=1', 'SPEED=thorough'])
+add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true)
+add_test_setup('slow', exclude_suites: ['thorough'], env: ['G_TEST_SLOW=1', 'SPEED=slow'])
+add_test_setup('thorough', env: ['G_TEST_SLOW=1', 'SPEED=thorough'])
 
 not_found = dependency('', required: false)
 keyval = import('keyval')
scripts/mtest2make.py +0 -4
index 4d542e8aaa..304634b71e 100644
@@ -101,10 +101,6 @@ def emit_suite(name, suite, prefix):
 testsuites = defaultdict(Suite)
 for test in introspect['tests']:
     process_tests(test, targets, testsuites)
-# HACK: check-block is a separate target so that it runs with --verbose;
-# only write the dependencies
-emit_suite_deps('block', testsuites['block'], 'check')
-del testsuites['block']
 emit_prolog(testsuites, 'check')
 for name, suite in testsuites.items():
     emit_suite(name, suite, 'check')
tests/Makefile.include +1 -8
index e7153c8e91..b89018cdcc 100644
@@ -147,16 +147,9 @@ check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
 # Consolidated targets
 
-.PHONY: check-block check check-clean get-vm-images
+.PHONY: check check-clean get-vm-images
 check:
 
-ifneq ($(.check-block.deps),)
-check: check-block
-check-block: run-ninja
-	$(if $(MAKE.n),,+)$(MESON) test $(MTESTARGS) $(.mtestargs) --verbose \
-		--logbase iotestslog $(call .speed.$(SPEED), block block-slow block-thorough)
-endif
-
 check-build: run-ninja
 
 check-clean:
Signature · 1 line
2.27.0
Tested-by on v4 1/1
#1 lore raw
Quoted 13 lines · Thomas Huth

If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now.

Quoted 8 lines

---
v4: updated commit description

  meson.build            | 6 +++---
  scripts/mtest2make.py  | 4 ----
  tests/Makefile.include | 9 +--------
  3 files changed, 4 insertions(+), 15 deletions(-)

I can’t really say I understand what’s going on in this patch and around it, but I can confirm that it before this patch, fail diffs aren’t printed; but afterwards, they are.  So I’m afraid all I can give is a

If noone else steps up and you need a tree for this to go in, I’d be up for it.

Hanna

#2 lore raw
Quoted 18 lines · Hanna Reitz
Quoted 17 lines · Thomas Huth

If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now.

---
  v4: updated commit description

Quoted 6 lines
Quoted 4 lines

  meson.build            | 6 +++---
  scripts/mtest2make.py  | 4 ----
  tests/Makefile.include | 9 +--------

  3 files changed, 4 insertions(+), 15 deletions(-)

I can’t really say I understand what’s going on in this patch and around it, but I can confirm that it before this patch, fail diffs aren’t printed; but afterwards, they are

It's a bug in Meson. It will be fixed in 0.61.3 and later (so this patch won't be needed there anymore), but the update to meson 0.61.3 caused other problems so we also can't do that right now... so I'm not sure whether we now want to have this patch here included, wait for a better version of meson, or even rather want to revert the TAP support / meson integration again for 7.0 ... ?

Thomas

#3 lore raw
Quoted 19 lines · Thomas Huth
Quoted 18 lines · Hanna Reitz
Quoted 17 lines · Thomas Huth

If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now.

---
  v4: updated commit description

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

  meson.build            | 6 +++---
  scripts/mtest2make.py  | 4 ----
  tests/Makefile.include | 9 +--------

  3 files changed, 4 insertions(+), 15 deletions(-)

I can’t really say I understand what’s going on in this patch and around it, but I can confirm that it before this patch, fail diffs aren’t printed; but afterwards, they are

It's a bug in Meson. It will be fixed in 0.61.3 and later (so this patch won't be needed there anymore), but the update to meson 0.61.3 caused other problems so we also can't do that right now... so I'm not sure whether we now want to have this patch here included, wait for a better version of meson, or even rather want to revert the TAP support / meson integration again for 7.0 ... ?

I don’t have anything against this patch, I just don’t fully understand what it does, and how it works.

So as far as I understand, check-block was its own target and used --verbose so that the progress indication would work (with -j1). Now that causes problems because of a bug in meson, and so this patch drops that special-casing again.  The only disadvantage is that the progress indication (which only worked with -j1) no longer ever works.

(Is that right?)

I personally don’t mind that disadvantage, because on CI systems it doesn’t really matter anyway; and on developers’ systems, I would assume make check to always be run with -jX anyway.

Hanna

#4 lore raw
Quoted 27 lines · Hanna Reitz
Quoted 26 lines · Thomas Huth
Quoted 24 lines · Hanna Reitz
Quoted 13 lines · Thomas Huth

If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now.

Quoted 8 lines

---
  v4: updated commit description

  meson.build            | 6 +++---
  scripts/mtest2make.py  | 4 ----
  tests/Makefile.include | 9 +--------

  3 files changed, 4 insertions(+), 15 deletions(-)

I can’t really say I understand what’s going on in this patch and around it, but I can confirm that it before this patch, fail diffs aren’t printed; but afterwards, they are

Quoted 7 lines
Quoted 1 line

It's a bug in Meson. It will be fixed in 0.61.3 and later (so this patch won't be needed there anymore), but the update to meson 0.61.3 caused other problems so we also can't do that right now... so I'm not sure whether we now want to have this patch here included, wait for a better version of meson, or even rather want to revert the TAP support / meson integration again for 7.0 ... ?

I don’t have anything against this patch, I just don’t fully understand what it does, and how it works.

So as far as I understand, check-block was its own target and used --verbose so that the progress indication would work (with -j1). Now that causes problems because of a bug in meson, and so this patch drops that special-casing again.  The only disadvantage is that the progress indication (which only worked with -j1) no longer ever works.

(Is that right?)

Right!

Quoted 1 line

I personally don’t mind that disadvantage, because on CI systems it doesn’t really matter anyway; and on developers’ systems, I would assume make check to always be run with -jX anyway.

Right again. So currently the only question is: Do we want to see a nice progress output with -j1 and do not care about the error logs, or do we rather want to see the error logs with -j1 and do not care about the nice progress output? For -jX with X > 1, the patch does not change much, and we'd need a newer version of meson to fix that.

Thomas

#5 lore raw
Quoted 36 lines · Thomas Huth
Quoted 35 lines · Hanna Reitz
Quoted 19 lines · Thomas Huth
Quoted 18 lines · Hanna Reitz
Quoted 17 lines · Thomas Huth

If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now.

---
  v4: updated commit description

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

  meson.build            | 6 +++---
  scripts/mtest2make.py  | 4 ----
  tests/Makefile.include | 9 +--------

  3 files changed, 4 insertions(+), 15 deletions(-)

I can’t really say I understand what’s going on in this patch and around it, but I can confirm that it before this patch, fail diffs aren’t printed; but afterwards, they are

It's a bug in Meson. It will be fixed in 0.61.3 and later (so this patch won't be needed there anymore), but the update to meson 0.61.3 caused other problems so we also can't do that right now... so I'm not sure whether we now want to have this patch here included, wait for a better version of meson, or even rather want to revert the TAP support / meson integration again for 7.0 ... ?

I don’t have anything against this patch, I just don’t fully understand what it does, and how it works.

So as far as I understand, check-block was its own target and used
--verbose so that the progress indication would work (with -j1). Now
that causes problems because of a bug in meson, and so this patch drops that special-casing again.  The only disadvantage is that the progress indication (which only worked with -j1) no longer ever works.

Quoted 7 lines
Quoted 1 line

(Is that right?)

Right!

Quoted 1 line

I personally don’t mind that disadvantage, because on CI systems it doesn’t really matter anyway; and on developers’ systems, I would assume make check to always be run with -jX anyway.

Right again. So currently the only question is: Do we want to see a nice progress output with -j1 and do not care about the error logs, or do we rather want to see the error logs with -j1 and do not care about the nice progress output? For -jX with X > 1, the patch does not change much, and we'd need a newer version of meson to fix that.

OK, to me the answer sounds obvious.  We absolutely need error logs, nice output is secondary to it.

Waiting for a new usable version of meson is not really an option, because when it comes around, we can just revert this patch (or take any other course of action that seems best then).

I guess we could revert TAP and/or the meson integration, I suppose that’d mean we’d get some progress output again, but it’s just the plain one from the iotests’ check script, right?  I’m hard-pressed to find good arguments against that, but I don’t really like that idea either.

Having this patch as a workaround until the functionality can be restored (which seems in sight) seems absolutely fine to me.  I guess I’ll just take it to my tree, then.  Won’t stop others from being able to protest, after all. :)

(I.e.: Thanks, applied to my block branch: https://gitlab.com/hreitz/qemu/-/commits/block)

Hanna

#6 lore raw
Quoted 43 lines · Hanna Reitz
Quoted 42 lines · Thomas Huth
Quoted 29 lines · Hanna Reitz
Quoted 19 lines · Thomas Huth
Quoted 18 lines · Hanna Reitz
Quoted 17 lines · Thomas Huth

If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now.

---
  v4: updated commit description

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

  meson.build            | 6 +++---
  scripts/mtest2make.py  | 4 ----
  tests/Makefile.include | 9 +--------

  3 files changed, 4 insertions(+), 15 deletions(-)

I can’t really say I understand what’s going on in this patch and around it, but I can confirm that it before this patch, fail diffs aren’t printed; but afterwards, they are

It's a bug in Meson. It will be fixed in 0.61.3 and later (so this patch won't be needed there anymore), but the update to meson 0.61.3 caused other problems so we also can't do that right now... so I'm not sure whether we now want to have this patch here included, wait for a better version of meson, or even rather want to revert the TAP support / meson integration again for 7.0 ... ?

Quoted 7 lines

I don’t have anything against this patch, I just don’t fully understand what it does, and how it works.

So as far as I understand, check-block was its own target and used
--verbose so that the progress indication would work (with -j1). Now that
causes problems because of a bug in meson, and so this patch drops that special-casing again.  The only disadvantage is that the progress indication (which only worked with -j1) no longer ever works.

(Is that right?)

Right!

Quoted 1 line

I personally don’t mind that disadvantage, because on CI systems it doesn’t really matter anyway; and on developers’ systems, I would assume make check to always be run with -jX anyway.

Quoted 7 lines
Quoted 1 line

Right again. So currently the only question is: Do we want to see a nice progress output with -j1 and do not care about the error logs, or do we rather want to see the error logs with -j1 and do not care about the nice progress output? For -jX with X > 1, the patch does not change much, and we'd need a newer version of meson to fix that.

OK, to me the answer sounds obvious.  We absolutely need error logs, nice output is secondary to it.

Waiting for a new usable version of meson is not really an option, because when it comes around, we can just revert this patch (or take any other course of action that seems best then).

I guess we could revert TAP and/or the meson integration, I suppose that’d mean we’d get some progress output again, but it’s just the plain one from the iotests’ check script, right?

Right, reverting means to get the old progress output again.

Quoted 1 line

  I’m hard-pressed to find good arguments against that, but I don’t really like that idea either.

Me neither, since it feels like a big step backwards, but if someone insists on having both, progress indication *and* error logs, it's currently the only feasible way, I think.

Quoted 3 lines

Having this patch as a workaround until the functionality can be restored (which seems in sight) seems absolutely fine to me.  I guess I’ll just take it to my tree, then.  Won’t stop others from being able to protest, after all. :)

(I.e.: Thanks, applied to my block branch: https://gitlab.com/hreitz/qemu/-/commits/block)

Thank you!

Thomas