Skip to contentSkip to conversation

test-manual/ptest: Add info about fail state of ptest-runner/ptests

PATCH Superseded Anibal Limon 2 participants 3 messages Jun 9, 2025 – Jun 10, 2025

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 test-manual/ptest: Add info about fail state of ptest-runner/ptests

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

---
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
documentation/test-manual/ptest.rst +12 -5
index 2c021af51..3b50e354a 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the ``run-ptest``
+scripts fails.
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target hardware,
 you need to prepare the recipes that build the packages you want to
 test. Here is what you have to do for each recipe:
 
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it
+   as succesfully executed.
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test
Signature · 1 line
2.39.5
#1 lore raw

Hello,

Le mar. 10 juin 2025 à 00:21, Anibal Limon <[email protected]> a écrit :

Quoted 7 lines

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

Thank you for working on this! :)

Can you reference the bug
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832?
With "[YOCTO #15832]" somewhere in your commit message.

Quoted 19 lines
---
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
documentation/test-manual/ptest.rst +4 -2
index 2c021af51..3b50e354a 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the
Quoted 3 lines

``run-ptest``
+scripts fails.

Counting failed tests might lead to weird overflows.
For example, there is a special error code for "ptest-runner itself failed" (127 IIRC).
If you have 127 failing tests, the calling code may reach a wrong conclusion.

I'd recommend returning a static non-zero error code on error (e.g 1).

Quoted 14 lines
documentation/test-manual/ptest.rst +4 -1
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target

hardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 87

Quoted 7 lines
documentation/test-manual/ptest.rst +1 -2
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it
 

Use ``run-ptest`` to stay consistent with the rest of the doc?

+ as succesfully executed.

Quoted 7 lines
documentation/test-manual/ptest.rst +1 -1
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test
--

2.39.5

Signature · 2 lines
Yoann Congal
Smile ECS
#2 lore raw
Quoted 49 lines · Yoann Congal

Hello,

Le mar. 10 juin 2025 à 00:21, Anibal Limon <[email protected]> a écrit :

Quoted 7 lines

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

Thank you for working on this! :)

Can you reference the bug
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832?
With "[YOCTO #15832]" somewhere in your commit message.

Quoted 19 lines
---
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
documentation/test-manual/ptest.rst +4 -2
index 2c021af51..3b50e354a 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the
Quoted 3 lines

``run-ptest``
+scripts fails.

Counting failed tests might lead to weird overflows.
For example, there is a special error code for "ptest-runner itself failed" (127 IIRC).
If you have 127 failing tests, the calling code may reach a wrong conclusion.

Quoted 2 lines

I'd recommend returning a static non-zero error code on error (e.g 1).

I will look into the bug description, and see how we can fix the behaviour you are experiencing.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c0

Quoted 17 lines
Quoted 17 lines
documentation/test-manual/ptest.rst +4 -1
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target

hardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:

documentation/test-manual/ptest.rst +0 -0
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
Quoted 9 lines

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 89

Quoted 6 lines
documentation/test-manual/ptest.rst +1 -2
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it
 

Use ``run-ptest`` to stay consistent with the rest of the doc?

I sent a v2.

Cheers!,
Anibal

Quoted 15 lines

+ as succesfully executed.

Quoted 7 lines
documentation/test-manual/ptest.rst +1 -1
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test
--

2.39.5

--
Yoann Congal
Smile ECS