Skip to contentSkip to conversation

test-manual/ptest: Add info about fail state of ptest-runner/ptests

PATCH Superseded Anibal Limon 2 participants 3 messages Jun 11, 2025

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 test-manual/ptest: Add info about fail state of ptest-runner/ptests

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

[YOCTO #15832]

---
 documentation/Pipfile               |  2 ++
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 2 files changed, 14 insertions(+), 5 deletions(-)
documentation/Pipfile +2 -0
index 7ee1d2290..c99d09c75 100644
@@ -9,6 +9,8 @@ verify_ssl = true
 sphinx = "*"
 sphinx-rtd-theme = "*"
 pyyaml = "*"
+vale = "*"
+sphinx-lint = "*"
 
 [requires]
 python_version = "3"
documentation/test-manual/ptest.rst +12 -5
index 2c021af51..e064419d7 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the ``run-ptest``
+scripts fails.
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target hardware,
 you need to prepare the recipes that build the packages you want to
 test. Here is what you have to do for each recipe:
 
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it
+   as sucessfully executed.
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test
Signature · 1 line
2.39.5
#1 lore raw

Hello,

Le mer. 11 juin 2025 à 03:54, Anibal Limon <[email protected]> a écrit :

Quoted 18 lines

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

[YOCTO #15832]

---
 documentation/Pipfile               |  2 ++
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 2 files changed, 14 insertions(+), 5 deletions(-)
documentation/Pipfile +0 -0
index 7ee1d2290..c99d09c75 100644

documentation/Pipfile verify_ssl = true · line 14

Quoted 6 lines
documentation/Pipfile +2 -0
@@ -9,6 +9,8 @@ verify_ssl = true
 sphinx = "*"
 sphinx-rtd-theme = "*"
 pyyaml = "*"
+vale = "*"
+sphinx-lint = "*"
 

^ this change is unrelated and should be split into a separate commit (or removed if that was an error)

Quoted 19 lines
documentation/Pipfile +0 -0
 [requires]
 python_version = "3"
documentation/test-manual/ptest.rst +4 -2
index 2c021af51..e064419d7 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the

``run-ptest``

documentation/test-manual/ptest.rst Running ptest · line 56

Quoted 4 lines
documentation/test-manual/ptest.rst +4 -0
+scripts fails.
+
+Consequently, you might want to add this package to your image.
+

"keeps counter of failed ``ptests`` and use it as exit value" is not entirely true.
I went to look in the code of ptest-runner2, and found this commit:
https://git.yoctoproject.org/ptest-runner2/commit/main.c?id=c10e7475353f4cfb5afb30ab1012c32c2efde85a
(It was committed by you?) => ptest-runner does keep a count of failed ptests but returns 1 if the count is > 0.

Getting Your Package Ready

Quoted 9 lines

==========================

documentation/test-manual/ptest.rst +1 -1
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target

hardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 88

Quoted 8 lines
documentation/test-manual/ptest.rst +2 -2
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it
+   as sucessfully executed.
 

Can you clarify here that ptest-runner expects 0 for a successful test or !=0 for a test failure.

Also (but that may be tackled in another patch), oe-selftest's testimage seems to expect at least a "PASS: ...", "FAIL: ..." line somewhere. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c1.

Quoted 7 lines
documentation/test-manual/ptest.rst +1 -0
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh

cd test
--
2.39.5

Regards,

Signature · 2 lines
Yoann Congal
Smile ECS
#2 lore raw
Quoted 18 lines · Yoann Congal

Hello,

Le mer. 11 juin 2025 à 03:54, Anibal Limon <[email protected]> a écrit :

Quoted 13 lines

On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

[YOCTO #15832]

---
 documentation/Pipfile               |  2 ++
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 2 files changed, 14 insertions(+), 5 deletions(-)
Quoted 14 lines

documentation/Pipfile verify_ssl = true · line 14

Quoted 11 lines
documentation/Pipfile +2 -0
index 7ee1d2290..c99d09c75 100644
@@ -9,6 +9,8 @@ verify_ssl = true
 sphinx = "*"
 sphinx-rtd-theme = "*"
 pyyaml = "*"
+vale = "*"
+sphinx-lint = "*"
 

^ this change is unrelated and should be split into a separate commit (or removed if that was an error)

Nice catch,

Quoted 23 lines
Quoted 23 lines
documentation/Pipfile +0 -0
 [requires]
 python_version = "3"
documentation/test-manual/ptest.rst +4 -2
index 2c021af51..e064419d7 100644
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the

``run-ptest``

documentation/test-manual/ptest.rst +3 -0
+scripts fails.
+
+Consequently, you might want to add this package to your image.
Quoted 9 lines
Quoted 2 lines

+

"keeps counter of failed ``ptests`` and use it as exit value" is not entirely true.
I went to look in the code of ptest-runner2, and found this commit:
https://git.yoctoproject.org/ptest-runner2/commit/main.c?id=c10e7475353f4cfb5afb30ab1012c32c2efde85a
(It was committed by you?) => ptest-runner does keep a count of failed ptests but returns 1 if the count is > 0.

Yeah, I remembered, fixing doc.

Quoted 12 lines

Getting Your Package Ready

Quoted 10 lines

==========================

documentation/test-manual/ptest.rst +1 -1
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target

hardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:

documentation/test-manual/ptest.rst +0 -0
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
Quoted 11 lines

documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 89

Quoted 7 lines
documentation/test-manual/ptest.rst +2 -2
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it
+   as sucessfully executed.
 

Can you clarify here that ptest-runner expects 0 for a successful test or !=0 for a test failure.

Let me rephrase.

Quoted 19 lines

Also (but that may be tackled in another patch), oe-selftest's testimage seems to expect at least a "PASS: ...", "FAIL: ..." line somewhere. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c1.

Quoted 7 lines
documentation/test-manual/ptest.rst +1 -0
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh

cd test
--
2.39.5

Regards,
--
Yoann Congal
Smile ECS