Skip to contentSkip to conversation

Use same test names in the default and the TAP13 output format

PATCH Applied Marc Hartmayer 4 participants 13 messages Aug 25, 2020 – Sep 22, 2020 KVM

lore
Share & tools
lore

Marc Hartmayer

author
#0 lore raw

v2 cover Use same test names in the default and the TAP13 output format

For everybody's convenience there is a branch:
https://gitlab.com/mhartmay/kvm-unit-tests/-/tree/tap_v2

Changelog:
v1 -> v2:
+ added r-b's to patch 1
+ patch 2:
- I've not added Andrew's r-b since I've worked in the comment from Janosch (don't drop the first prefix)

Marc Hartmayer (2):
runtime.bash: remove outdated comment
Use same test names in the default and the TAP13 output format

 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  9 +++------
 2 files changed, 12 insertions(+), 12 deletions(-)
Signature · 1 line
2.25.4
author
#1 lore raw

PATCH v2 1/2 runtime.bash: remove outdated comment

Since commit 6e1d3752d7ca ("tap13: list testcases individually") the comment is no longer valid. Therefore let's remove it.

Reviewed-by Andrew Jones <[email protected]>
Reviewed-by Cornelia Huck <[email protected]>
Signed-off-by Marc Hartmayer <[email protected]>
---
 scripts/runtime.bash | 3 ---
 1 file changed, 3 deletions(-)
scripts/runtime.bash +0 -3
index c88e246245a6..caa4c5ba18cc 100644
@@ -53,9 +53,6 @@ skip_nodefault()
 
 function print_result()
 {
-    # output test results in a TAP format
-    # https://testanything.org/tap-version-13-specification.html
-
     local status="$1"
     local testname="$2"
     local summary="$3"
Signature · 1 line
2.25.4
Reviewed-by on v2 1/2
#2 lore raw

Re: [kvm-unit-tests PATCH v2 1/2] runtime.bash: remove outdated comment

Quoted 16 lines · Marc Hartmayer

Since commit 6e1d3752d7ca ("tap13: list testcases individually") the comment is no longer valid. Therefore let's remove it.

Reviewed-by Andrew Jones <[email protected]>
Reviewed-by Cornelia Huck <[email protected]>
Signed-off-by Marc Hartmayer <[email protected]>
---
 scripts/runtime.bash | 3 ---
 1 file changed, 3 deletions(-)
scripts/runtime.bash +0 -0
index c88e246245a6..caa4c5ba18cc 100644

scripts/runtime.bash skip_nodefault() · line 58

Quoted 9 lines
scripts/runtime.bash +0 -3
@@ -53,9 +53,6 @@ skip_nodefault()
 
 function print_result()
 {
-    # output test results in a TAP format
-    # https://testanything.org/tap-version-13-specification.html
-
     local status="$1"
     local testname="$2"
     local summary="$3"
Reviewed-by Thomas Huth <[email protected]>
author
#3 lore raw

PATCH v2 2/2 Use same test names in the default and the TAP13 output format

In reply to Marc Hartmayer

Use the same test names in the TAP13 output as in the default output format. This makes the output more consistent. To achieve this, we need to pass the test name as an argument to the function process_test_output.

Before this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest: true
ok 2 - selftest: argc == 3
...

After this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest-setup: selftest: true
ok 2 - selftest-setup: selftest: argc == 3
...

While at it, introduce a local variable kernel in RUNTIME_log_stdout since this makes the function easier to read.

Signed-off-by Marc Hartmayer <[email protected]>
---
 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  6 +++---
 2 files changed, 12 insertions(+), 9 deletions(-)
run_tests.sh +9 -6
index 01e36dcfa06e..f49dd864524e 100755
@@ -81,18 +81,19 @@ if [[ $tap_output == "no" ]]; then
     postprocess_suite_output() { cat; }
 else
     process_test_output() {
+        local testname="$1"
         CR=$'\r'
         while read -r line; do
             line="${line%$CR}"
             case "${line:0:4}" in
                 PASS)
-                    echo "ok TEST_NUMBER - ${line#??????}" >&3
+                    echo "ok TEST_NUMBER - ${testname}: ${line#??????}" >&3
                     ;;
                 FAIL)
-                    echo "not ok TEST_NUMBER - ${line#??????}" >&3
+                    echo "not ok TEST_NUMBER - ${testname}: ${line#??????}" >&3
                     ;;
                 SKIP)
-                    echo "ok TEST_NUMBER - ${line#??????} # skip" >&3
+                    echo "ok TEST_NUMBER - ${testname}: ${line#??????} # skip" >&3
                     ;;
                 *)
                     ;;

@@ -114,12 +115,14 @@ else
     }
 fi
 
-RUNTIME_log_stderr () { process_test_output; }
+RUNTIME_log_stderr () { process_test_output "$1"; }
 RUNTIME_log_stdout () {
+    local testname="$1"
     if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
-        ./scripts/pretty_print_stacks.py $1 | process_test_output
+        local kernel="$2"
+        ./scripts/pretty_print_stacks.py "$kernel" | process_test_output "$testname"
     else
-        process_test_output
+        process_test_output "$testname"
     fi
 }
scripts/runtime.bash +3 -3
index caa4c5ba18cc..294e6b15a5e2 100644
@@ -140,10 +140,10 @@ function run()
     # extra_params in the config file may contain backticks that need to be
     # expanded, so use eval to start qemu.  Use "> >(foo)" instead of a pipe to
     # preserve the exit status.
-    summary=$(eval $cmdline 2> >(RUNTIME_log_stderr) \
-                             > >(tee >(RUNTIME_log_stdout $kernel) | extract_summary))
+    summary=$(eval $cmdline 2> >(RUNTIME_log_stderr $testname) \
+                             > >(tee >(RUNTIME_log_stdout $testname $kernel) | extract_summary))
     ret=$?
-    [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $kernel)
+    [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $testname $kernel)
 
     if [ $ret -eq 0 ]; then
         print_result "PASS" $testname "$summary"
Signature · 1 line
2.25.4
Reviewed-by on v2 2/2
#4 lore raw
Quoted 31 lines · On Tue, 25 Aug 2020 12:20:36 +0200 Marc Hartmayer <[email protected]>…

Use the same test names in the TAP13 output as in the default output format. This makes the output more consistent. To achieve this, we need to pass the test name as an argument to the function process_test_output.

Before this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest: true
ok 2 - selftest: argc == 3
...

After this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest-setup: selftest: true
ok 2 - selftest-setup: selftest: argc == 3
...
Quoted 8 lines

While at it, introduce a local variable kernel in RUNTIME_log_stdout since this makes the function easier to read.

Signed-off-by Marc Hartmayer <[email protected]>
---
 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  6 +++---
 2 files changed, 12 insertions(+), 9 deletions(-)
Reviewed-by Cornelia Huck <[email protected]>
author
#5 lore raw
Quoted 37 lines · Cornelia Huck
Quoted 34 lines · On Tue, 25 Aug 2020 12:20:36 +0200 Marc Hartmayer <[email protected]>…

Use the same test names in the TAP13 output as in the default output format. This makes the output more consistent. To achieve this, we need to pass the test name as an argument to the function process_test_output.

Before this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest: true
ok 2 - selftest: argc == 3
...

After this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest-setup: selftest: true
ok 2 - selftest-setup: selftest: argc == 3
...

While at it, introduce a local variable kernel in RUNTIME_log_stdout since this makes the function easier to read.

Quoted 8 lines
Quoted 5 lines
Signed-off-by Marc Hartmayer <[email protected]>
---
 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  6 +++---
 2 files changed, 12 insertions(+), 9 deletions(-)
Reviewed-by Cornelia Huck <[email protected]>

Thanks.

Signature · 8 lines
Kind regards / Beste Grüße
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen 
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
Reviewed-by on v2 2/2
#6 lore raw
In reply to Marc Hartmayer
Quoted 31 lines · Marc Hartmayer

Use the same test names in the TAP13 output as in the default output format. This makes the output more consistent. To achieve this, we need to pass the test name as an argument to the function process_test_output.

Before this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest: true
ok 2 - selftest: argc == 3
...

After this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest-setup: selftest: true
ok 2 - selftest-setup: selftest: argc == 3
...
Quoted 8 lines

While at it, introduce a local variable kernel in RUNTIME_log_stdout since this makes the function easier to read.

Signed-off-by Marc Hartmayer <[email protected]>
---
 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  6 +++---
 2 files changed, 12 insertions(+), 9 deletions(-)
Reviewed-by Thomas Huth <[email protected]>
author
#7 lore raw
Quoted 35 lines · Thomas Huth
Quoted 34 lines · Marc Hartmayer

Use the same test names in the TAP13 output as in the default output format. This makes the output more consistent. To achieve this, we need to pass the test name as an argument to the function process_test_output.

Before this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest: true
ok 2 - selftest: argc == 3
...

After this change:

$ ./run_tests.sh
PASS selftest-setup (14 tests)
...

vs.

$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest-setup: selftest: true
ok 2 - selftest-setup: selftest: argc == 3
...

While at it, introduce a local variable kernel in RUNTIME_log_stdout since this makes the function easier to read.

Quoted 8 lines
Quoted 5 lines
Signed-off-by Marc Hartmayer <[email protected]>
---
 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  6 +++---
 2 files changed, 12 insertions(+), 9 deletions(-)
Reviewed-by Thomas Huth <[email protected]>

Thanks!

Signature · 8 lines
Kind regards / Beste Grüße
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen 
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
author
#8 lore raw
In reply to Marc Hartmayer
Quoted 10 lines · Marc Hartmayer

For everybody's convenience there is a branch:
https://gitlab.com/mhartmay/kvm-unit-tests/-/tree/tap_v2

Changelog:
v1 -> v2:
+ added r-b's to patch 1
+ patch 2:
- I've not added Andrew's r-b since I've worked in the comment from Janosch (don't drop the first prefix)

Quoted 11 lines

Marc Hartmayer (2):
runtime.bash: remove outdated comment
Use same test names in the default and the TAP13 output format

 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  9 +++------
 2 files changed, 12 insertions(+), 12 deletions(-)
Signature · 1 line
2.25.4

Polite ping :) How should we proceed further?

Signature · 8 lines
Kind regards / Beste Grüße
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen 
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
#9 lore raw
Quoted 19 lines · Marc Hartmayer
Quoted 18 lines · Marc Hartmayer

For everybody's convenience there is a branch:
https://gitlab.com/mhartmay/kvm-unit-tests/-/tree/tap_v2

Changelog:
v1 -> v2:
+ added r-b's to patch 1
+ patch 2:
- I've not added Andrew's r-b since I've worked in the comment from Janosch (don't drop the first prefix)

Marc Hartmayer (2):
runtime.bash: remove outdated comment
Use same test names in the default and the TAP13 output format

 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  9 +++------
 2 files changed, 12 insertions(+), 12 deletions(-)
Quoted 5 lines
Quoted 3 lines
Signature · 1 line
2.25.4

Polite ping :) How should we proceed further?

Sorry, it's been some quite busy weeks ... I'll try to collect some pending kvm-unit-tests patches in the next days and then send a pull request to Paolo...

Thomas

author
#10 lore raw
Quoted 24 lines · Thomas Huth
Quoted 23 lines · Marc Hartmayer
Quoted 21 lines · Marc Hartmayer

For everybody's convenience there is a branch:
https://gitlab.com/mhartmay/kvm-unit-tests/-/tree/tap_v2

Changelog:
v1 -> v2:
+ added r-b's to patch 1
+ patch 2:
- I've not added Andrew's r-b since I've worked in the comment from Janosch (don't drop the first prefix)

Marc Hartmayer (2):
runtime.bash: remove outdated comment
Use same test names in the default and the TAP13 output format

 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  9 +++------
 2 files changed, 12 insertions(+), 12 deletions(-)
Signature · 1 line
2.25.4
Quoted 5 lines
Quoted 1 line

Polite ping :) How should we proceed further?

Sorry, it's been some quite busy weeks ... I'll try to collect some pending kvm-unit-tests patches in the next days and then send a pull request to Paolo...

No problem :) Thanks for taking care!

Quoted 3 lines

Thomas

Signature · 8 lines
Kind regards / Beste Grüße
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen 
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
maintainer
#11 lore raw
In reply to Thomas Huth
Quoted 24 lines · Thomas Huth
Quoted 23 lines · Marc Hartmayer
Quoted 21 lines · Marc Hartmayer

For everybody's convenience there is a branch:
https://gitlab.com/mhartmay/kvm-unit-tests/-/tree/tap_v2

Changelog:
v1 -> v2:
+ added r-b's to patch 1
+ patch 2:
- I've not added Andrew's r-b since I've worked in the comment from Janosch (don't drop the first prefix)

Marc Hartmayer (2):
runtime.bash: remove outdated comment
Use same test names in the default and the TAP13 output format

 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  9 +++------
 2 files changed, 12 insertions(+), 12 deletions(-)
Signature · 1 line
2.25.4
Quoted 8 lines
Quoted 1 line

Polite ping :) How should we proceed further?

Sorry, it's been some quite busy weeks ... I'll try to collect some pending kvm-unit-tests patches in the next days and then send a pull request to Paolo...

Thomas

Don't worry, I'm finally back and learning again how to do code review with "lighter" series. :)

Paolo

maintainer Applied
#12 lore raw
In reply to Marc Hartmayer
Quoted 10 lines · Marc Hartmayer

For everybody's convenience there is a branch:
https://gitlab.com/mhartmay/kvm-unit-tests/-/tree/tap_v2

Changelog:
v1 -> v2:
+ added r-b's to patch 1
+ patch 2:
- I've not added Andrew's r-b since I've worked in the comment from Janosch (don't drop the first prefix)

Quoted 8 lines

Marc Hartmayer (2):
runtime.bash: remove outdated comment
Use same test names in the default and the TAP13 output format

 run_tests.sh         | 15 +++++++++------
 scripts/runtime.bash |  9 +++------
 2 files changed, 12 insertions(+), 12 deletions(-)

Queued, thanks.

Paolo