Skip to contentSkip to conversation

tests/qtest/qom-test: Do not print tested properties by default

PATCH Applied Thomas Huth 5 participants 14 messages Dec 15, 2022 – Jan 18, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 tests/qtest/qom-test: Do not print tested properties by default

We're still running into the problem that some logs are cut in the gitlab-CI since they got too big. The biggest part of the log is still the output of the qom-test. Let's stop printing the properties by default to get to a saner size here. The full output can still be enabled by setting V=2 (or higher) in the environment.

---
v2: Use atoi() to do proper checking of the verbosity level

 tests/qtest/qom-test.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/qtest/qom-test.c +11 -1
index 13510bc349..d380261f8f 100644
@@ -14,6 +14,8 @@
 #include "qemu/cutils.h"
 #include "libqtest.h"
 
+static bool verbose;
+
 static void test_properties(QTestState *qts, const char *path, bool recurse)
 {
     char *child_path;
@@ -49,7 +51,9 @@ static void test_properties(QTestState *qts, const char *path, bool recurse)
             }
         } else {
             const char *prop = qdict_get_str(tuple, "name");
-            g_test_message("-> %s", prop);
+            if (verbose) {
+                g_test_message("-> %s", prop);
+            }
             tmp = qtest_qmp(qts,
                             "{ 'execute': 'qom-get',"
                             "  'arguments': { 'path': %s, 'property': %s } }",
@@ -103,6 +107,12 @@ static void add_machine_test_case(const char *mname)
 
 int main(int argc, char **argv)
 {
+    char *v_env = getenv("V");
+
+    if (v_env && atoi(v_env) >= 2) {
+        verbose = true;
+    }
+
     g_test_init(&argc, &argv, NULL);
 
     qtest_cb_for_every_machine(add_machine_test_case, g_test_quick());
Signature · 1 line
2.31.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 6 lines · Thomas Huth

We're still running into the problem that some logs are cut in the gitlab-CI since they got too big. The biggest part of the log is still the output of the qom-test. Let's stop printing the properties by default to get to a saner size here. The full output can still be enabled by setting V=2 (or higher) in the environment.

Quoted 6 lines

---
v2: Use atoi() to do proper checking of the verbosity level

  tests/qtest/qom-test.c | 12 +++++++++++-
  1 file changed, 11 insertions(+), 1 deletion(-)
#2 lore raw
In reply to Thomas Huth
Quoted 10 lines · Thomas Huth

We're still running into the problem that some logs are cut in the gitlab-CI since they got too big. The biggest part of the log is still the output of the qom-test. Let's stop printing the properties by default to get to a saner size here. The full output can still be enabled by setting V=2 (or higher) in the environment.

---
v2: Use atoi() to do proper checking of the verbosity level

Applied to master in the hope of improving the CI logs; thanks.

-- PMM

#3 lore raw
Quoted 9 lines · Peter Maydell
Quoted 7 lines · Thomas Huth

We're still running into the problem that some logs are cut in the gitlab-CI since they got too big. The biggest part of the log is still the output of the qom-test. Let's stop printing the properties by default to get to a saner size here. The full output can still be enabled by setting V=2 (or higher) in the environment.

Quoted 5 lines
Quoted 3 lines

---
v2: Use atoi() to do proper checking of the verbosity level

Applied to master in the hope of improving the CI logs; thanks.

Looks like we can still hit the gitlab log limit in some cases, even with this patch; here's an example job:
https://gitlab.com/qemu-project/qemu/-/jobs/3610621992

thanks
-- PMM

#4 lore raw
Quoted 14 lines · Peter Maydell
Quoted 13 lines · Peter Maydell
Quoted 10 lines · Thomas Huth

We're still running into the problem that some logs are cut in the gitlab-CI since they got too big. The biggest part of the log is still the output of the qom-test. Let's stop printing the properties by default to get to a saner size here. The full output can still be enabled by setting V=2 (or higher) in the environment.

---
v2: Use atoi() to do proper checking of the verbosity level

Quoted 5 lines
Quoted 1 line

Applied to master in the hope of improving the CI logs; thanks.

Looks like we can still hit the gitlab log limit in some cases, even with this patch; here's an example job:
https://gitlab.com/qemu-project/qemu/-/jobs/3610621992

Ok, too bad ... three ideas to improve this situation further:

- We could shut up those "Obtaining properties of" lines by default

- Stop running the tests with "V=1"

- Instead of doing "cat meson-logs/testlog.txt" in the CI scripts, we switch to "tail -n 1000 meson-logs/testlog.txt" instead

What do you think?

Thomas

#5 lore raw
Quoted 17 lines · Thomas Huth
Quoted 16 lines · Peter Maydell
Quoted 14 lines · Peter Maydell
Quoted 10 lines · Thomas Huth

We're still running into the problem that some logs are cut in the gitlab-CI since they got too big. The biggest part of the log is still the output of the qom-test. Let's stop printing the properties by default to get to a saner size here. The full output can still be enabled by setting V=2 (or higher) in the environment.

---
v2: Use atoi() to do proper checking of the verbosity level

Applied to master in the hope of improving the CI logs; thanks.

Quoted 8 lines
Quoted 3 lines

Looks like we can still hit the gitlab log limit in some cases, even with this patch; here's an example job:
https://gitlab.com/qemu-project/qemu/-/jobs/3610621992

Ok, too bad ... three ideas to improve this situation further:

- We could shut up those "Obtaining properties of" lines by default

Yes, they add little value in debugging test results from CI.

There's plenty of other largely useless junk printed too

Pages and pages of:

Looking for expected file 'tests/data/acpi/q35/FACP.memhp' Looking for expected file 'tests/data/acpi/q35/FACP' Using expected file 'tests/data/acpi/q35/FACP' Looking for expected file 'tests/data/acpi/q35/APIC.memhp' Looking for expected file 'tests/data/acpi/q35/APIC'
...

Something random that looks like bad text data

▶ 166/619 /ptimer/oneshot policy=no_immediate_����������������������������������������������������������������...snip....�������������

And what i think is test float being overly verbose

Quoted 1 line
Quoted 1 line

Testing f16_le_quiet

46464 tests total.

    10000
    20000
    30000
    40000

46464 tests performed.

Could be written as

Quoted 1 line
Quoted 1 line

Testing f16_le_quiet: 46464 tests total .... OK

(one '.' for each 10,000 tests run, before final 'OK' is printed)

Quoted 1 line

- Stop running the tests with "V=1"

Definitely not, the info about individual test cases being executed is absolutely critical when a test hangs, so we can narrow down what was running when it hung.

I think the key is that when we pass 'V=1' to tests, we want to see lists of individual test cases being run, but we don't want to see adhoc debugging output. The latter should either be deleted if useless, or hidden by a getenv("QEMU_TESTS_DEBUG") check

Quoted 3 lines

- Instead of doing "cat meson-logs/testlog.txt" in the CI scripts, we switch to "tail -n 1000 meson-logs/testlog.txt" instead

Tail requires that it has consumed the entire doc before it prints anything. This in turn implies that the test suite has finished. IOW, we'll get zero output if it hangs IIUC.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw
Quoted 3 lines · Daniel P. Berrangé

Something random that looks like bad text data

▶ 166/619 /ptimer/oneshot policy=no_immediate_����������������������������������������������������������������...snip....�������������

I don't understand this one. The string comes from tests/unit/ptimer-test.c:add_ptimer_tests() but that code creates the test string using g_strlcat() and g_strdup_printf(), so I don't see how it could end up being junk, especially not at that point in the string. I guess it's possible something else in the test is managing to corrupt the g_test test name strings?

Quoted 1 line

And what i think is test float being overly verbose

Yes, the float tests are massively over-verbose. This is a problem not just in the length of the test log but also in finding the actual cause of a test failure, because often the float tests were running in parallel with the failed test and print all their very verbose output after the failed test output, so you have to scroll back up for miles to find it.

thanks
-- PMM

#7 lore raw
Quoted 5 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrangé

Something random that looks like bad text data

▶ 166/619 /ptimer/oneshot policy=no_immediate_����������������������������������������������������������������...snip....�������������

Quoted 8 lines

I don't understand this one. The string comes from tests/unit/ptimer-test.c:add_ptimer_tests() but that code creates the test string using g_strlcat() and g_strdup_printf(), so I don't see how it could end up being junk, especially not at that point in the string. I guess it's possible something else in the test is managing to corrupt the g_test test name strings?

Actually it looks like something else spewing ���������� a to stderr, and this getting intermixed with stdout from the test suite. The test case name isn't corrupt, as I see the remaining "reload,..." bit that jouins with 'policy=no_immediate_"

Wonder what's responsible for this garbage on stderr...

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw
Quoted 16 lines · Daniel P. Berrangé
Quoted 5 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrangé

Something random that looks like bad text data

▶ 166/619 /ptimer/oneshot policy=no_immediate_����������������������������������������������������������������...snip....�������������

Quoted 8 lines

I don't understand this one. The string comes from tests/unit/ptimer-test.c:add_ptimer_tests() but that code creates the test string using g_strlcat() and g_strdup_printf(), so I don't see how it could end up being junk, especially not at that point in the string. I guess it's possible something else in the test is managing to corrupt the g_test test name strings?

Quoted 6 lines

Actually it looks like something else spewing ���������� a to stderr, and this getting intermixed with stdout from the test suite. The test case name isn't corrupt, as I see the remaining "reload,..." bit that jouins with 'policy=no_immediate_"

Wonder what's responsible for this garbage on stderr...

Mmm.

Also I notice that we could save some output if we can fix whatever it is that's producing out-of-spec TAP format and causing this to be printed for every test:

Unknown TAP version. The first line MUST be TAP version <int>. Assuming version 12.

-- PMM

#9 lore raw
Quoted 24 lines · Peter Maydell
Quoted 16 lines · Daniel P. Berrangé
Quoted 5 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrangé

Something random that looks like bad text data

▶ 166/619 /ptimer/oneshot policy=no_immediate_����������������������������������������������������������������...snip....�������������

Quoted 8 lines

I don't understand this one. The string comes from tests/unit/ptimer-test.c:add_ptimer_tests() but that code creates the test string using g_strlcat() and g_strdup_printf(), so I don't see how it could end up being junk, especially not at that point in the string. I guess it's possible something else in the test is managing to corrupt the g_test test name strings?

Quoted 6 lines

Actually it looks like something else spewing ���������� a to stderr, and this getting intermixed with stdout from the test suite. The test case name isn't corrupt, as I see the remaining "reload,..." bit that jouins with 'policy=no_immediate_"

Wonder what's responsible for this garbage on stderr...

Quoted 8 lines

Mmm.

Also I notice that we could save some output if we can fix whatever it is that's producing out-of-spec TAP format and causing this to be printed for every test:

Unknown TAP version. The first line MUST be TAP version <int>. Assuming version 12.

That's a meson bug caused by this misguided change

https://github.com/mesonbuild/meson/pull/11186

and suggested to revert

https://github.com/mesonbuild/meson/pull/11242

Though it seems the meson maintainer prefers their only patch:

https://github.com/eli-schwartz/meson/commit/f0928378d2246bcf8a81be35c44b1350a549d647

but that's not submitted for merge that I've found.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#10 lore raw
In reply to Daniel P. Berrangé
Quoted 1 line · Daniel P. Berrangé

On Tue, Jan 17, 2023 at 11:32:42AM +0100, Thomas Huth wrote:

...

Quoted 7 lines
Quoted 3 lines

- Instead of doing "cat meson-logs/testlog.txt" in the CI scripts, we switch to "tail -n 1000 meson-logs/testlog.txt" instead

Tail requires that it has consumed the entire doc before it prints anything. This in turn implies that the test suite has finished. IOW, we'll get zero output if it hangs IIUC.

I was referring to the places where we are dumping the testlog.txt like this:

make --output-sync -jnproc check
|| { cat meson-logs/testlog.txt; exit 1; }

See .gitlab-ci.d/custom-runners/* or .gitlab-ci.d/windows.yml for example. I assume it should be ok to switch to "tail" instead of "cat" there?

Thomas

#11 lore raw
Quoted 11 lines · Thomas Huth
Quoted 1 line · Daniel P. Berrangé

On Tue, Jan 17, 2023 at 11:32:42AM +0100, Thomas Huth wrote:

...

Quoted 7 lines
Quoted 3 lines

- Instead of doing "cat meson-logs/testlog.txt" in the CI scripts, we switch to "tail -n 1000 meson-logs/testlog.txt" instead

Tail requires that it has consumed the entire doc before it prints anything. This in turn implies that the test suite has finished. IOW, we'll get zero output if it hangs IIUC.

Quoted 7 lines

I was referring to the places where we are dumping the testlog.txt like this:

make --output-sync -jnproc check
|| { cat meson-logs/testlog.txt; exit 1; }

See .gitlab-ci.d/custom-runners/* or .gitlab-ci.d/windows.yml for example. I assume it should be ok to switch to "tail" instead of "cat" there?

Yes, it would be OK, bug lets make sure meson-logs/testlog.txt is included as an artifact published by the job, so we can access the full set if needed

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#12 lore raw

float tests are too verbose (was: [PATCH v2] tests/qtest/qom-test: Do not print tested properties by default)

In reply to Daniel P. Berrangé

On 17/01/2023 11.44, Daniel P. Berrangé wrote:
...

Quoted 12 lines

And what i think is test float being overly verbose

Quoted 1 line
Quoted 1 line

Testing f16_le_quiet

   46464 tests total.

     10000
     20000
     30000
     40000
   46464 tests performed.
Quoted 5 lines

Could be written as

Quoted 1 line
Quoted 1 line

Testing f16_le_quiet: 46464 tests total .... OK

(one '.' for each 10,000 tests run, before final 'OK' is printed)

Unfortunately, the float tests are included via a git submodule, so the source code is not under our direct control here ... has anybody a good idea how to get this tackled best?

Thomas

#13 lore raw

Re: float tests are too verbose (was: [PATCH v2] tests/qtest/qom-test: Do not print tested properties by default)

Quoted 14 lines · Thomas Huth

On 17/01/2023 11.44, Daniel P. Berrangé wrote:
...

Quoted 11 lines

And what i think is test float being overly verbose

Quoted 1 line
Quoted 1 line

Testing f16_le_quiet

46464 tests total.

     10000
     20000
     30000
     40000

46464 tests performed.
Could be written as

Quoted 1 line
Quoted 1 line

Testing f16_le_quiet: 46464 tests total .... OK

(one '.' for each 10,000 tests run, before final 'OK' is printed)

Quoted 3 lines

Unfortunately, the float tests are included via a git submodule, so the source code is not under our direct control here ... has anybody a good idea how to get this tackled best?

It is under our control. It is a lightly patched version of the downstream testfloat package so we can patch it some more.

That said reducing 5 lines to one is hardly a massive reduction. If we just stopped passing V=1 you would get:

➜ make check-softfloat
GIT ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc roms/SLOF
[1/1] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
/usr/bin/python3 -B /home/alex/lsrc/qemu.git/meson/meson.py test --no-rebuild -t 0 --num-processes 1 --print-errorlogs --suite softfloat

 1/17 qemu:softfloat+softfloat-conv / fp-test-float-to-float           OK              0.02s
 2/17 qemu:softfloat+softfloat-conv / fp-test-int-to-float             OK              0.01s
 3/17 qemu:softfloat+softfloat-conv / fp-test-uint-to-float            OK              0.01s
 4/17 qemu:softfloat+softfloat-conv / fp-test-float-to-int             OK              0.02s
 5/17 qemu:softfloat+softfloat-conv / fp-test-float-to-uint            OK              0.01s
 6/17 qemu:softfloat+softfloat-conv / fp-test-round-to-integer         OK              0.01s
 7/17 qemu:softfloat+softfloat-compare / fp-test-eq_signaling          OK              0.05s
 8/17 qemu:softfloat+softfloat-compare / fp-test-le                    OK              0.05s
 9/17 qemu:softfloat+softfloat-compare / fp-test-le_quiet              OK              0.05s
10/17 qemu:softfloat+softfloat-compare / fp-test-lt_quiet              OK              0.05s
11/17 qemu:softfloat+softfloat-ops / fp-test-add                       OK              0.58s
12/17 qemu:softfloat+softfloat-ops / fp-test-sub                       OK              0.59s
13/17 qemu:softfloat+softfloat-ops / fp-test-mul                       OK              2.83s
14/17 qemu:softfloat+softfloat-ops / fp-test-div                       OK              2.11s
15/17 qemu:softfloat+softfloat-ops / fp-test-rem                       OK              1.27s
16/17 qemu:softfloat+softfloat-ops / fp-test-sqrt                      OK              0.03s
17/17 qemu:softfloat+softfloat-ops / fp-test-log2                      OK              0.02s
Ok:                 17  
Expected Fail:      0   
Fail:               0   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/alex/lsrc/qemu.git/builds/all/meson-logs/testlog.txt

with the testlog being:

wc -l meson-logs/testlog.txt
2553 meson-logs/testlog.txt

Quoted 2 lines

Thomas

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro