test-manual/ptest: Add info about fail state of ptest-runner/ptests
PATCH 1/1 test-manual/ptest: Add info about fail state of ptest-runner/ptests
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.
https://lists.yoctoproject.org/g/yocto-patches/message/1622
--- documentation/test-manual/ptest.rst | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
documentation/test-manual/ptest.rst +12 -5
@@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of the ``run-ptest`` +scripts fails. + +Consequently, you might want to add this package to your image. + Getting Your Package Ready ========================== -In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on target hardware, you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: - *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it + as succesfully executed. + Here is an example that starts a test for ``dbus``:: #!/bin/sh cd test
Signature · 1 line
2.39.5
Hello,
Le mar. 10 juin 2025 à 00:21, Anibal Limon <[email protected]> a écrit :
Quoted 7 lines
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.
https://lists.yoctoproject.org/g/yocto-patches/message/1622
Signed-off-by Anibal Limon <[email protected]>
Thank you for working on this! :)
Can you reference the bug
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832?
With "[YOCTO #15832]" somewhere in your commit message.
Quoted 19 lines
--- documentation/test-manual/ptest.rst | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)documentation/test-manual/ptest.rst +4
-2@@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of the
Quoted 3 lines
``run-ptest``
+scripts fails.
Counting failed tests might lead to weird overflows.
For example, there is a special error code for "ptest-runner itself failed" (127 IIRC).
If you have 127 failing tests, the calling code may reach a wrong conclusion.
I'd recommend returning a static non-zero error code on error (e.g 1).
Quoted 14 lines
documentation/test-manual/ptest.rst +4
-1+ +Consequently, you might want to add this package to your image. + Getting Your Package Ready ========================== -In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on targethardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:
documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 87
Quoted 7 lines
documentation/test-manual/ptest.rst +1
-2@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: - *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it
Use ``run-ptest`` to stay consistent with the rest of the doc?
+ as succesfully executed.
Quoted 7 lines
documentation/test-manual/ptest.rst +1
-1+ Here is an example that starts a test for ``dbus``:: #!/bin/sh cd test --2.39.5
Signature · 2 lines
Yoann Congal Smile ECS
Quoted 49 lines · Yoann Congal
Hello,
Le mar. 10 juin 2025 à 00:21, Anibal Limon <[email protected]> a écrit :
Quoted 7 lines
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.
https://lists.yoctoproject.org/g/yocto-patches/message/1622
Signed-off-by Anibal Limon <[email protected]>Thank you for working on this! :)
Can you reference the bug
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832?
With "[YOCTO #15832]" somewhere in your commit message.Quoted 19 lines
--- documentation/test-manual/ptest.rst | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)documentation/test-manual/ptest.rst +4
-2@@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of theQuoted 3 lines
``run-ptest``
+scripts fails.Counting failed tests might lead to weird overflows.
For example, there is a special error code for "ptest-runner itself failed" (127 IIRC).
If you have 127 failing tests, the calling code may reach a wrong conclusion.
Quoted 2 lines
I'd recommend returning a static non-zero error code on error (e.g 1).
I will look into the bug description, and see how we can fix the behaviour you are experiencing.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c0
Quoted 17 lines
Quoted 17 lines
documentation/test-manual/ptest.rst +4
-1+ +Consequently, you might want to add this package to your image. + Getting Your Package Ready ========================== -In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on targethardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:documentation/test-manual/ptest.rst +0
-0@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
Quoted 9 lines
documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 89
Quoted 6 lines
documentation/test-manual/ptest.rst +1
-2- *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark itUse ``run-ptest`` to stay consistent with the rest of the doc?
I sent a v2.
Cheers!,
Anibal
Quoted 15 lines
+ as succesfully executed.
Quoted 7 lines
documentation/test-manual/ptest.rst +1
-1+ Here is an example that starts a test for ``dbus``:: #!/bin/sh cd test --2.39.5
--
Yoann Congal
Smile ECS