Skip to contentSkip to conversation

test-manual/ptest: Add info about fail state of ptest-runner/ptests

PATCH Superseded Anibal Limon 2 participants 3 messages Jun 10, 2025 – Jun 11, 2025

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 test-manual/ptest: Add info about fail state of ptest-runner/ptests

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

[YOCTO #15832]

---
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
documentation/test-manual/ptest.rst +12 -5
index 2c021af51..d01735c08 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the ``run-ptest``
+scripts fails.
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target hardware,
 you need to prepare the recipes that build the packages you want to
 test. Here is what you have to do for each recipe:
 
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it
+   as succesfully executed.
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test
Signature · 1 line
2.39.5
#1 lore raw
Quoted 40 lines · Anibal Limon

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

[YOCTO #15832]

---
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
documentation/test-manual/ptest.rst +9 -3
index 2c021af51..d01735c08 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the ``run-ptest``
+scripts fails.
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target hardware,
 you need to prepare the recipes that build the packages you want to
 test. Here is what you have to do for each recipe:
 

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 86

Quoted 6 lines
documentation/test-manual/ptest.rst +1 -2
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it

I think Yoann suggested that the run-ptest script should return 1 in case of failure. Can you also mention that? We need to be clear that the run-ptest script must return a static code and not any non-zero code, as it could clash with ptest-runner's own exit code (e.g. 127), if I understood correctly.

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 87

Quoted 1 line
documentation/test-manual/ptest.rst +1 -0
+   as succesfully executed.

s/succesfully/sucessfully/

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 91

Quoted 4 lines
documentation/test-manual/ptest.rst +1 -0
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test

Thanks,
Antonin

Signature · 3 lines
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw
Quoted 45 lines · Antonin Godard
Quoted 5 lines · Anibal Limon

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

Review

Quoted 8 lines

[YOCTO #15832]

---
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
(quoted diff) +0 -0

b/documentation/test-manual/ptest.rst

documentation/test-manual/ptest.rst Running ptest · line 52

Quoted 13 lines

index 2c021af51..d01735c08 100644

documentation/test-manual/ptest.rst +4 -2
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the

``run-ptest``

documentation/test-manual/ptest.rst Running ptest

Quoted 10 lines
documentation/test-manual/ptest.rst +5 -1
+scripts fails.
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target

hardware,

Quoted 5 lines
documentation/test-manual/ptest.rst +0 -0
 you need to prepare the recipes that build the packages you want to
 test. Here is what you have to do for each recipe:
 
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
Quoted 13 lines

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 86

Quoted 5 lines
documentation/test-manual/ptest.rst +1 -2
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it

I think Yoann suggested that the run-ptest script should return 1 in case of
failure. Can you also mention that? We need to be clear that the run-ptest script must return a static code and not any non-zero code, as it could clash
with ptest-runner's own exit code (e.g. 127), if I understood correctly.

Current logic is any non-zero is taken as a failure, so at level of the ptest_runner only increments the counter + 1 when
tests fail, at this stage the PATCHv3 matches the current behaviour.

Quoted 5 lines

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 87

Quoted 2 lines
documentation/test-manual/ptest.rst +1 -0
+   as succesfully executed.

s/succesfully/sucessfully/

Fixed on v3.

Regards!,
Anibal

Quoted 15 lines

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 91

Quoted 5 lines
documentation/test-manual/ptest.rst +1 -0
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test

Thanks,
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com