test-manual/ptest: Add info about fail state of ptest-runner/ptests
PATCH v3 1/1 test-manual/ptest: Add info about fail state of ptest-runner/ptests
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.
https://lists.yoctoproject.org/g/yocto-patches/message/1622
[YOCTO #15832]
--- documentation/Pipfile | 2 ++ documentation/test-manual/ptest.rst | 17 ++++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-)
documentation/Pipfile +2 -0
@@ -9,6 +9,8 @@ verify_ssl = true sphinx = "*" sphinx-rtd-theme = "*" pyyaml = "*" +vale = "*" +sphinx-lint = "*" [requires] python_version = "3"
documentation/test-manual/ptest.rst +12 -5
@@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of the ``run-ptest`` +scripts fails. + +Consequently, you might want to add this package to your image. + Getting Your Package Ready ========================== -In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on target hardware, you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: - *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it + as sucessfully executed. + Here is an example that starts a test for ``dbus``:: #!/bin/sh cd test
Signature · 1 line
2.39.5
Hello,
Le mer. 11 juin 2025 à 03:54, Anibal Limon <[email protected]> a écrit :
Quoted 18 lines
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.
https://lists.yoctoproject.org/g/yocto-patches/message/1622
[YOCTO #15832]
Signed-off-by Anibal Limon <[email protected]>--- documentation/Pipfile | 2 ++ documentation/test-manual/ptest.rst | 17 ++++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-)documentation/Pipfile +0
-0
documentation/Pipfile verify_ssl = true · line 14
Quoted 6 lines
documentation/Pipfile +2
-0@@ -9,6 +9,8 @@ verify_ssl = true sphinx = "*" sphinx-rtd-theme = "*" pyyaml = "*" +vale = "*" +sphinx-lint = "*"
^ this change is unrelated and should be split into a separate commit (or removed if that was an error)
Quoted 19 lines
documentation/Pipfile +0
-0[requires] python_version = "3"documentation/test-manual/ptest.rst +4
-2@@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of the``run-ptest``
documentation/test-manual/ptest.rst Running ptest · line 56
Quoted 4 lines
documentation/test-manual/ptest.rst +4
-0+scripts fails. + +Consequently, you might want to add this package to your image. +
"keeps counter of failed ``ptests`` and use it as exit value" is not entirely true.
I went to look in the code of ptest-runner2, and found this commit:
https://git.yoctoproject.org/ptest-runner2/commit/main.c?id=c10e7475353f4cfb5afb30ab1012c32c2efde85a
(It was committed by you?) => ptest-runner does keep a count of failed ptests but returns 1 if the count is > 0.
Getting Your Package Ready
Quoted 9 lines
==========================
documentation/test-manual/ptest.rst +1
-1-In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on targethardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:
documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 88
Quoted 8 lines
documentation/test-manual/ptest.rst +2
-2@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: - *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it + as sucessfully executed.
Can you clarify here that ptest-runner expects 0 for a successful test or !=0 for a test failure.
Also (but that may be tackled in another patch), oe-selftest's testimage seems to expect at least a "PASS: ...", "FAIL: ..." line somewhere. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c1.
Quoted 7 lines
documentation/test-manual/ptest.rst +1
-0+ Here is an example that starts a test for ``dbus``:: #!/bin/shcd test
--
2.39.5
Regards,
Signature · 2 lines
Yoann Congal Smile ECS
Quoted 18 lines · Yoann Congal
Hello,
Le mer. 11 juin 2025 à 03:54, Anibal Limon <[email protected]> a écrit :
Quoted 13 lines
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around.
https://lists.yoctoproject.org/g/yocto-patches/message/1622
[YOCTO #15832]
Signed-off-by Anibal Limon <[email protected]>--- documentation/Pipfile | 2 ++ documentation/test-manual/ptest.rst | 17 ++++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-)
Quoted 14 lines
documentation/Pipfile verify_ssl = true · line 14
Quoted 11 lines
documentation/Pipfile +2
-0@@ -9,6 +9,8 @@ verify_ssl = true sphinx = "*" sphinx-rtd-theme = "*" pyyaml = "*" +vale = "*" +sphinx-lint = "*"^ this change is unrelated and should be split into a separate commit (or removed if that was an error)
Nice catch,
Quoted 23 lines
Quoted 23 lines
documentation/Pipfile +0
-0[requires] python_version = "3"documentation/test-manual/ptest.rst +4
-2@@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of the``run-ptest``
documentation/test-manual/ptest.rst +3
-0+scripts fails. + +Consequently, you might want to add this package to your image.
Quoted 9 lines
Quoted 2 lines
+
"keeps counter of failed ``ptests`` and use it as exit value" is not entirely true.
I went to look in the code of ptest-runner2, and found this commit:
https://git.yoctoproject.org/ptest-runner2/commit/main.c?id=c10e7475353f4cfb5afb30ab1012c32c2efde85a
(It was committed by you?) => ptest-runner does keep a count of failed ptests but returns 1 if the count is > 0.
Yeah, I remembered, fixing doc.
Quoted 12 lines
Getting Your Package Ready
Quoted 10 lines
==========================
documentation/test-manual/ptest.rst +1
-1-In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on targethardware,
you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe:documentation/test-manual/ptest.rst +0
-0@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
Quoted 11 lines
documentation/test-manual/ptest.rst test. Here is what you have to do for each recipe: · line 89
Quoted 7 lines
documentation/test-manual/ptest.rst +2
-2- *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it + as sucessfully executed.Can you clarify here that ptest-runner expects 0 for a successful test or !=0 for a test failure.
Let me rephrase.
Quoted 19 lines
Also (but that may be tackled in another patch), oe-selftest's testimage seems to expect at least a "PASS: ...", "FAIL: ..." line somewhere. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c1.
Quoted 7 lines
documentation/test-manual/ptest.rst +1
-0+ Here is an example that starts a test for ``dbus``:: #!/bin/shcd test
--
2.39.5Regards,
--
Yoann Congal
Smile ECS