Skip to contentSkip to conversation

Refactor meta-oe, meta-perl, meta-python ptests with BBCLASSEXTEND parallel execution

PATCH In review Tim Orling 2 participants 14 messages Apr 2, 2023 – Apr 3, 2023

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

cover Refactor meta-oe, meta-perl, meta-python ptests with BBCLASSEXTEND parallel execution

This series implements parallel ptest execution, similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-*-ptest-image.bb for an image per ptest in the form of meta-*-ptest-image-XXX using class extensions.

* Add meta-*-ptest-all-image.bb depends on ALL meta-*-ptest-image-XXX in meta-*
* Add meta-*-ptest-fast-image.bb depends on meta-* ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating this number of class extensions is slow but making it easier to execute and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-*-ptest-image-XXX can be built with:

$ bitbake meta-*-ptest-all-image

Subsequently, al the ptest images can be run with:

$ bitbake -c testimage meta-*-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresults.json

Call to arms:

1. When upgrading recipes that have ptest enabled, run testimage

   $ bitbake meta-*-ptest-image-<pn>
   $ bitbake -c testimage meta-*-ptest-image-<pn>
2. Help move recipes in PTEST_PROBLEMS_* to a state where all ptests pass
3. Test on architectures you care about and report problems.
4. Enable ptest in more recipes:
   * inherit ptest
   * provide a run-ptest script
   * add to conf/include/ptest-packagelists-meta-*.inc
5. Migrate python recipes which use pytest to the new "pytest --automake" methodology
6. Enable this methodology in more meta-openembedded layers.

The following changes since commit 17243e70c8309751a1129a9214899ec476ab121a:

python3-betamax: fix ptest failture of fixture and record modes (2023-03-28 10:44:00 -0700)

are available in the Git repository at:

https://git.openembedded.org/meta-openembedded-contrib timo/ptest-refactoring
http://cgit.openembedded.org/meta-openembedded-contrib/log/?h=timo/ptest-refactoring

Tim Orling (11):
cli11: do not inherit ptest
span-lite: do not inherit ptest ptest-packagelists-meta-oe.inc: add ptest recipes meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution ptest-packagelists-meta-python.inc: add ptest recipes meta-python-ptest*-image: enable BBCLASSEXTEND parallel execution
python3-aspectlib: fix ptest ptest-packagelists-meta-perl.inc: add ptest recipes recipes-perl/packagegroups: move to recipes-core/ recipes-perl/images: move to recipes-core meta-perl-ptest*-image: enable BBCLASSEXTEND parallel execution

 .../include/ptest-packagelists-meta-oe.inc    | 60 ++++++++++++++
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 ++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++-
 meta-oe/recipes-support/cli11/cli11_2.3.2.bb  |  3 +-
 .../span-lite/span-lite_0.10.3.bb             |  1 -
 .../include/ptest-packagelists-meta-perl.inc  | 50 +++++++++++
 .../images/meta-perl-base.bb                  |  0
 .../images/meta-perl-image.bb                 |  0
 .../images/meta-perl-ptest-all-image.bb       | 25 ++++++
 .../images/meta-perl-ptest-fast-image.bb      |  6 ++
 .../images/meta-perl-ptest-image.bb           | 41 ++++++++++
 .../packagegroups/packagegroup-meta-perl.bb   |  0
 .../images/meta-perl-ptest-image.bb           |  9 --
 .../ptest-packagelists-meta-python.inc        | 82 +++++++++++++++++++
 .../images/meta-python-ptest-all-image.bb     | 25 ++++++
 .../images/meta-python-ptest-fast-image.bb    |  6 ++
 .../images/meta-python-ptest-image.bb         | 38 ++++++++-
 .../python/python3-aspectlib/run-ptest        |  2 +-
 .../python/python3-aspectlib_2.0.0.bb         |  4 +-
 20 files changed, 405 insertions(+), 16 deletions(-)
 create mode 100644 meta-oe/conf/include/ptest-packagelists-meta-oe.inc
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
 create mode 100644 meta-perl/conf/include/ptest-packagelists-meta-perl.inc
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-base.bb (100%)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-image.bb (100%)
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-image.bb
 rename meta-perl/{recipes-perl => recipes-core}/packagegroups/packagegroup-meta-perl.bb (100%)
 delete mode 100644 meta-perl/recipes-perl/images/meta-perl-ptest-image.bb
 create mode 100644 meta-python/conf/include/ptest-packagelists-meta-python.inc
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-all-image.bb
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-fast-image.bb
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/11 cli11: do not inherit ptest

This recipe provides no run-ptest script.

---
 meta-oe/recipes-support/cli11/cli11_2.3.2.bb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta-oe/recipes-support/cli11/cli11_2.3.2.bb +1 -2
index b9ff914da..7c6b2547c 100644
@@ -15,11 +15,10 @@ S = "${WORKDIR}/git"
 
 do_configure:prepend() {
     mkdir -p ${S}/tests/catch2
-    cp ${DL_DIR}/catch.hpp ${S}/tests/catch2/catch.hpp 
+    cp ${DL_DIR}/catch.hpp ${S}/tests/catch2/catch.hpp
 }
 
 inherit cmake
-inherit ptest
 
 # cli11 is a header only C++ library, so the main package will be empty.
 RDEPENDS:${PN}-dev = ""
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/11 span-lite: do not inherit ptest

In reply to Tim Orling

This recipe provides no run-ptest script.

---
 meta-oe/recipes-support/span-lite/span-lite_0.10.3.bb | 1 -
 1 file changed, 1 deletion(-)
meta-oe/recipes-support/span-lite/span-lite_0.10.3.bb +0 -1
index 0d9a8c93d..15380f5d7 100644
@@ -9,4 +9,3 @@ SRCREV = "cbb9c3c5162a0d7018c7b3e053153a04d4fbbbb9"
 S = "${WORKDIR}/git"
 
 inherit cmake
-inherit ptest
Signature · 1 line
2.34.1
author
#4 lore raw

PATCH 4/11 meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution

In reply to Tim Orling

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-oe-ptest-image.bb for an image per ptest in the form of meta-oe-ptest-image-XXX using class extensions.

* Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX in meta-oe
* Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating this number of class extensions is slow but making it easier to execute and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-oe-ptest-image-XXX can be built with:

$ bitbake meta-oe-ptest-all-image

Subsequently, all the ptest images can be run with:

$ bitbake -c testimage meta-oe-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresult.json
---
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 +++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb new file +25 -0
new file mode 100644
index 000000000..0ea03dd2e
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
+# Include the full set of ptests
+PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_OE').split()])}"
+
+do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_OE').split()])}"
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb new file +6 -0
new file mode 100644
index 000000000..f56e841ff
@@ -0,0 +1,6 @@
+require meta-oe-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest images."
+
+PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
+
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +37 -1
index 0166b9c41..951d91f82 100644
@@ -1,5 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
 require  meta-oe-image-base.bb
 
 SUMMARY = "meta-oe ptest test image"
 
-IMAGE_INSTALL += "packagegroup-meta-oe"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_OE').split()])}"
+
+# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memroy, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
+
Signature · 1 line
2.34.1
author
#7 lore raw

PATCH 5/11 ptest-packagelists-meta-python.inc: add ptest recipes

In reply to Tim Orling

Similar to ptest-packagelists.inc in oe-core, this list enables per image ptest (isolated ptests for a given recipe)

* Add recipes to PTEST_FAST_META_PYTHON for ptests which take less than ~30s
* Add recipes to PTEST_SLOW_META_PYTHON for ptests which take longer
* Add recipes to PTEST_PROBLEMS_META_PYTHON for ptests which have failing tests or other issues with ptest package

The initial list was taken from:

meta_python_ptest_recipes=$(bitbake-layers show-recipes --recipes-only \
      --layer meta-python --inherits ptest --bare | tr '\n' ' ' | \
      pcregrep -o1 '^NOTE:.+===(.+)$')
---
 .../ptest-packagelists-meta-python.inc        | 82 +++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100644 meta-python/conf/include/ptest-packagelists-meta-python.inc
meta-python/conf/include/ptest-packagelists-meta-python.inc new file +82 -0
new file mode 100644
index 000000000..ef5d20bcd
@@ -0,0 +1,82 @@
+#
+# Lists of the ptest in meta-python, sorted into two sets by the time they take
+# Please keep these sorted in alphabetical order
+#
+# A first pass at getting all meta-python recipes which inherit ptest
+# meta_python_ptest_recipes=$(bitbake-layers show-recipes --recipes-only --layer meta-python --inherits ptest --bare | tr '\n' ' ' | pcregrep -o1 '^NOTE:.+===(.+)$')
+#
+# ptests which take less than ~30s each
+PTESTS_FAST_META_PYTHON = "\
+    python3-ansicolors \
+    python3-asgiref \
+    python3-aspectlib \
+    python3-blinker \
+    python3-cachetools \
+    python3-cbor2 \
+    python3-click \
+    python3-dominate \
+    python3-execnet \
+    python3-geojson \
+    python3-html2text \
+    python3-inflection \
+    python3-intervals \
+    python3-ipy \
+    python3-iso3166 \
+    python3-parse-type \
+    python3-polyline \
+    python3-precise-runner \
+    python3-prettytable \
+    python3-ptyprocess \
+    python3-pyasn1-modules \
+    python3-pyroute2 \
+    python3-pyserial \
+    python3-pytoml \
+    python3-requests-file \
+    python3-requests-toolbelt \
+    python3-semver \
+    python3-serpent \
+    python3-simpleeval \
+    python3-smpplib \
+    python3-soupsieve \
+    python3-sqlparse \
+    python3-u-msgpack-python \
+    python3-unidiff \
+    python3-uritemplate \
+    python3-xmltodict \
+    python3-xxhash \
+"
+
+PTESTS_SLOW_META_PYTHON = "\
+    python3-lz4 \
+"
+
+PTESTS_PROBLEMS_META_PYTHON ="\
+    python3-appdirs \
+    python3-betamax \
+    python3-dnspython \
+    python3-fastjsonschema \
+    python3-gpiod \
+    python3-gunicorn \
+    python3-inotify \
+    python3-jdcal \
+    python3-jsmin \
+    python3-license-expression \
+    python3-msgpack \
+    python3-multidict \
+    python3-ordered-set \
+    python3-parse \
+    python3-pillow \
+    python3-pint \
+    python3-py-cpuinfo \
+    python3-pytest-lazy-fixture \
+    python3-pyzmq \
+    python3-scrypt \
+    python3-service-identity \
+    python3-trustme \
+    python3-typeguard \
+    python3-whoosh \
+    python3-ujson \
+    python3-xlrd \
+    python3-yappi \
+    python3-yarl \
+"
Signature · 1 line
2.34.1
author
#8 lore raw

PATCH 6/11 meta-python-ptest*-image: enable BBCLASSEXTEND parallel execution

In reply to Tim Orling

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-python-ptest-image.bb for an image per ptest in the form of meta-python-ptest-image-XXX using class extensions.

* Add meta-python-ptest-all-image.bb depends on ALL meta-python-ptest-image-XXX in meta-python
* Add meta-python-ptest-fast-image.bb depends on meta-python ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating this number of class extensions is slow but making it easier to execute and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-python-ptest-image-XXX can be built with:

$ bitbake meta-python-ptest-all-image

Subsequently, all the ptest images can be run with:

$ bitbake -c testimage meta-python-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresults.json
---
 .../images/meta-python-ptest-all-image.bb     | 25 ++++++++++++
 .../images/meta-python-ptest-fast-image.bb    |  6 +++
 .../images/meta-python-ptest-image.bb         | 38 ++++++++++++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-all-image.bb
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-fast-image.bb
meta-python/recipes-core/images/meta-python-ptest-all-image.bb new file +25 -0
new file mode 100644
index 000000000..fd4dc4248
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-python ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-python.inc
+
+# Include the full set of ptests
+PTESTS_META_PYTHON = "${PTESTS_FAST_META_PYTHON} ${PTESTS_SLOW_META_PYTHON}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@' '.join(['meta-python-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_PYTHON').split()])}"
+
+do_build[depends] = "${@' '.join(['meta-python-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_PYTHON').split()])}"
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
meta-python/recipes-core/images/meta-python-ptest-fast-image.bb new file +6 -0
new file mode 100644
index 000000000..4f93a1595
@@ -0,0 +1,6 @@
+require meta-python-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-python ptest images."
+
+PTESTS_META_PYTHON = "${PTESTS_FAST_META_PYTHON}"
+
meta-python/recipes-core/images/meta-python-ptest-image.bb +37 -1
index d497016d4..a649cbb94 100644
@@ -1,5 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-python.inc
+
 require  meta-python-image-base.bb
 
 SUMMARY = "meta-python ptest test image"
 
-IMAGE_INSTALL += "packagegroup-meta-python3-ptest"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_PYTHON = "${PTESTS_SLOW_META_PYTHON} ${PTESTS_FAST_META_PYTHON}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_PYTHON').split()])}"
+
+# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memory, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
+
Signature · 1 line
2.34.1
author
#9 lore raw

PATCH 7/11 python3-aspectlib: fix ptest

In reply to Tim Orling

* ptest RDEPENDS on pytest
* Use python3-unittest-automake-output instead of using sed to reformat the output use this new module.

---
 .../recipes-devtools/python/python3-aspectlib/run-ptest       | 2 +-
 .../recipes-devtools/python/python3-aspectlib_2.0.0.bb        | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)
meta-python/recipes-devtools/python/python3-aspectlib/run-ptest +1 -1
index b63c4de0d..8d2017d39 100644
@@ -1,3 +1,3 @@
 #!/bin/sh
 
-pytest -o log_cli=true -o log_cli_level=INFO | sed -e 's/\[...%\]//g'| sed -e 's/PASSED/PASS/g'| sed -e 's/FAILED/FAIL/g'|sed -e 's/SKIPPED/SKIP/g'| awk '{if ($NF=="PASS" || $NF=="FAIL" || $NF=="SKIP" || $NF=="XFAIL" || $NF=="XPASS"){printf "%s: %s\n", $NF, $0}else{print}}'| awk '{if ($NF=="PASS" || $NF=="FAIL" || $NF=="SKIP" || $NF=="XFAIL" || $NF=="XPASS") {$NF="";print $0}else{print}}'
+pytest --automake
meta-python/recipes-devtools/python/python3-aspectlib_2.0.0.bb +3 -1
index 416b031c5..d2b0864ca 100644
@@ -21,8 +21,10 @@ do_install_ptest() {
 }
 
 RDEPENDS:${PN}-ptest += "\
-    python3-tornado \
     python3-process-tests \
+    python3-pytest \
+    python3-tornado \
+    python3-unittest-automake-output \
 "
 
 RDEPENDS:${PN} += "python3-core python3-fields"
Signature · 1 line
2.34.1
author
#10 lore raw

PATCH 8/11 ptest-packagelists-meta-perl.inc: add ptest recipes

In reply to Tim Orling

Similar to ptest-packagelists.inc in oe-core, this list enables per image ptest (isolated ptests for a given recipe)

* Add recipes to PTEST_FAST_META_PERL for ptests which take less than ~30s
* Add recipes to PTEST_SLOW_META_PERL for ptests which take longer
* Add recipes to PTEST_PROBLEMS_META_PERL for ptests which have failing tests or other issues with ptest package

The initial list was taken from:

meta_perl_ptest_recipes=$(bitbake-layers show-recipes --recipes-only \
    --layer meta-perl --inherits ptest --bare | tr '\n' ' ' | \
    pcregrep -o1 '^NOTE:.+===(.+)$')
---
 .../include/ptest-packagelists-meta-perl.inc  | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 meta-perl/conf/include/ptest-packagelists-meta-perl.inc
meta-perl/conf/include/ptest-packagelists-meta-perl.inc new file +50 -0
new file mode 100644
index 000000000..1a8c759b1
@@ -0,0 +1,50 @@
+#
+# Lists of the ptest in meta-perl, sorted into two sets by the time they take
+# Please keep these sorted in alphabetical order
+#
+# A first pass at getting all meta-perl recipes which inherit ptest
+# meta_perl_ptest_recipes=$(bitbake-layers show-recipes --recipes-only --layer meta-perl --inherits ptest --bare | tr '\n' ' ' | pcregrep -o1 '^NOTE:.+===(.+)$')
+#
+# ptests which take less than ~30s each
+PTESTS_FAST_META_PERL = "\
+    libcapture-tiny-perl \
+    libcgi-perl \
+    libcrypt-openssl-random-perl \
+    libextutils-config-perl \
+    libhtml-tagset-perl \
+    libmozilla-ca-perl \
+    libstrictures-perl \
+    libterm-readkey-perl \
+    libtest-nowarnings-perl \
+"
+
+PTESTS_SLOW_META_PERL = "\
+    libio-socket-ssl-perl \
+"
+
+PTESTS_PROBLEMS_META_PERL ="\
+    libauthen-sasl-perl \
+    libconfig-autoconf-perl \
+    libcrypt-openssl-guess-perl \
+    libcrypt-openssl-rsa-perl \
+    libdata-hexdump-perl \
+    libdbd-sqlite-perl \
+    libdigest-hmac-perl \
+    libdigest-sha1-perl \
+    libnet-dns-perl \
+    libnet-dns-sec-perl \
+    libnet-ldap-perl \
+    libencode-perl \
+    libextutils-installpaths-perl \
+    libfile-slurper-perl \
+    libhtml-parser-perl \
+    libmime-types-perl \
+    libnet-ssleay-perl \
+    libsub-uplevel-perl \
+    libtest-deep-perl \
+    libtest-harness-perl \
+    libtest-warn-perl \
+    libunicode-linebreak-perl \
+    libunix-statgrab \
+    libxml-libxml-perl \
+"
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 3/11 ptest-packagelists-meta-oe.inc: add ptest recipes

In reply to Tim Orling

Similar to ptest-packagelists.inc in oe-core, this list enables per image ptest (isolated ptests for a given recipe)

* Add recipes to PTEST_FAST_META_OE for ptests which take less than ~30s
* Add recipes to PTEST_SLOW_META_OE for ptests which take longer
* Add recipes to PTEST_PROBLEMS_META_OE for ptests which have failing tests or other issues with ptest package

The initial list was taken from:

meta_oe_ptest_recipes=$(bitbake-layers show-recipes --recipes-only \
--layer meta-oe --inherits ptest --bare | tr '\n' ' ' | \
pcregrep -o1 '^NOTE:.+===(.+)$')

---
 .../include/ptest-packagelists-meta-oe.inc    | 60 +++++++++++++++++++
 1 file changed, 60 insertions(+)
 create mode 100644 meta-oe/conf/include/ptest-packagelists-meta-oe.inc
meta-oe/conf/include/ptest-packagelists-meta-oe.inc new file +60 -0
new file mode 100644
index 000000000..e8898ce78
@@ -0,0 +1,60 @@
+#
+# Lists of the ptest in meta-oe, sorted into two sets by the time they take
+# Please keep these sorted in alphabetical order
+#
+# A first pass at getting all meta-oe recipes which inherit ptest
+# meta_oe_ptest_recipes=$(bitbake-layers show-recipes --recipes-only --layer meta-oe --inherits ptest --bare | tr '\n' ' ' | pcregrep -o1 '^NOTE:.+===(.+)$')
+#
+# ptests which take less than ~30s each
+PTESTS_FAST_META_OE = "\
+    cmocka \
+    cunit \
+    duktape \
+    fmt \
+    function2 \
+    fwupd \
+    gcab \
+    jq \
+    leveldb \
+    libcereal \
+    libee \
+    libio-pty-perl \
+    libmanette \
+    libssh \
+    libxml++-5.0 \
+    libxmlb \
+    libyang \
+    lmdb \
+    minicoredumper \
+    neon \
+    nodejs \
+    onig \
+    poco \
+    protobuf \
+    sdbus-c++ \
+    uthash \
+    xmlsec1 \
+    zeromq \
+"
+
+PTESTS_SLOW_META_OE = "\
+    fftw \
+    libusb-compat \
+    rsyslog \
+"
+
+PTESTS_PROBLEMS_META_OE ="\
+    jemalloc \
+    kernel-selftest \
+    keyutils \
+    libdbi-perl \
+    libgpiod \
+    libjcat \
+    libteam \
+    libxml++ \
+    mcelog \
+    oprofile \
+    ostree \
+    psqlodbc \
+    pv \
+"
Signature · 1 line
2.34.1
author
#11 lore raw

PATCH 9/11 recipes-perl/packagegroups: move to recipes-core/

In reply to Tim Orling

Be consistent with other layers, like oe-core, meta-oe and meta-python

---
 .../packagegroups/packagegroup-meta-perl.bb                       | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename meta-perl/{recipes-perl => recipes-core}/packagegroups/packagegroup-meta-perl.bb (100%)
meta-perl/recipes-core/packagegroups/packagegroup-meta-perl.bb renamed from meta-perl/recipes-perl/packagegroups/packagegroup-meta-perl.bb +0 -0
similarity index 100%
rename from meta-perl/recipes-perl/packagegroups/packagegroup-meta-perl.bb
rename to meta-perl/recipes-core/packagegroups/packagegroup-meta-perl.bb
Signature · 1 line
2.34.1
author
#12 lore raw

PATCH 10/11 recipes-perl/images: move to recipes-core

In reply to Tim Orling

Be consistent with other layers like oe-core, meta-oe and meta-python

---
 meta-perl/{recipes-perl => recipes-core}/images/meta-perl-base.bb | 0
 .../{recipes-perl => recipes-core}/images/meta-perl-image.bb      | 0
 .../images/meta-perl-ptest-image.bb                               | 0
 3 files changed, 0 insertions(+), 0 deletions(-)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-base.bb (100%)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-image.bb (100%)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-ptest-image.bb (100%)
meta-perl/recipes-core/images/meta-perl-base.bb renamed from meta-perl/recipes-perl/images/meta-perl-base.bb +0 -0
similarity index 100%
rename from meta-perl/recipes-perl/images/meta-perl-base.bb
rename to meta-perl/recipes-core/images/meta-perl-base.bb
meta-perl/recipes-core/images/meta-perl-image.bb renamed from meta-perl/recipes-perl/images/meta-perl-image.bb +0 -0
similarity index 100%
rename from meta-perl/recipes-perl/images/meta-perl-image.bb
rename to meta-perl/recipes-core/images/meta-perl-image.bb
meta-perl/recipes-core/images/meta-perl-ptest-image.bb renamed from meta-perl/recipes-perl/images/meta-perl-ptest-image.bb +0 -0
similarity index 100%
rename from meta-perl/recipes-perl/images/meta-perl-ptest-image.bb
rename to meta-perl/recipes-core/images/meta-perl-ptest-image.bb
Signature · 1 line
2.34.1
author
#13 lore raw

PATCH 11/11 meta-perl-ptest*-image: enable BBCLASSEXTEND parallel execution

In reply to Tim Orling

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-perl-ptest-image.bb for an image per ptest in the form of meta-perl-ptest-image-XXX using class extensions.

* Add meta-perl-ptest-all-image.bb depends on ALL meta-perl-ptest-image-XXX in meta-perl
* Add meta-perl-ptest-fast-image.bb depends on meta-perl ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating this number of class extensions is slow but making it easier to execute and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-perl-ptest-image-XXX can be built with:

$ bitbake meta-perl-ptest-all-image

Subsequently, all the ptest images can be run with:

$ bitbake -c testimage meta-perl-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresults.json
---
 .../images/meta-perl-ptest-all-image.bb       | 25 ++++++++++++
 .../images/meta-perl-ptest-fast-image.bb      |  6 +++
 .../images/meta-perl-ptest-image.bb           | 38 +++++++++++++++++--
 3 files changed, 66 insertions(+), 3 deletions(-)
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb
meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb new file +25 -0
new file mode 100644
index 000000000..236043c22
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-perl ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-perl.inc
+
+# Include the full set of ptests
+PTESTS_META_PERL = "${PTESTS_FAST_META_PERL} ${PTESTS_SLOW_META_PERL}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@' '.join(['meta-perl-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_PERL').split()])}"
+
+do_build[depends] = "${@' '.join(['meta-perl-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_PERL').split()])}"
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb new file +6 -0
new file mode 100644
index 000000000..5d8afa08e
@@ -0,0 +1,6 @@
+require meta-perl-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-perl ptest images."
+
+PTESTS_META_PERL = "${PTESTS_FAST_META_PERL}"
+
meta-perl/recipes-core/images/meta-perl-ptest-image.bb +35 -3
index a9b4cc3c7..7cca6402b 100644
@@ -1,9 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES += "ptest"
+
 require  meta-perl-base.bb
 
 SUMMARY = "meta-perl build ptest image"
 
-inherit features_check
+require conf/include/ptest-packagelists-meta-perl.inc
 
-REQUIRED_DISTRO_FEATURES += "ptest"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_PERL = "${PTESTS_SLOW_META_PERL} ${PTESTS_FAST_META_PERL}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_PERL').split()])}"
+
+# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memory, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
 
-IMAGE_INSTALL += "packagegroup-meta-perl-ptest"
Signature · 1 line
2.34.1
#5 lore raw

Re: [oe] [meta-oe][PATCH 04/11] meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution

In reply to Tim Orling

Thanks for this patchset Tim, really appreciate it. I have few comments.

Quoted 38 lines · Tim Orling

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-oe-ptest-image.bb for an image per ptest in the form of meta-oe-ptest-image-XXX using class extensions.

* Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX in meta-oe
* Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating this number of class extensions is slow but making it easier to execute and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-oe-ptest-image-XXX can be built with:

$ bitbake meta-oe-ptest-all-image

Subsequently, all the ptest images can be run with:

$ bitbake -c testimage meta-oe-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresult.json
Quoted 8 lines
---
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 +++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb

I think these images also need ( and also the ones in meta-python and meta-perl too I think )

IMAGE_FEATURES += "debug-tweaks"

since some of them add ssh and it will be helpful to be able to do passwordless access. and I was also wondering
if meta-oe-image-base.bb is really needed. We could simply include core-image-base.bb here. This change also makes meta-oe-image.bb and packagegroup-meta-oe.bb also questionable. Since build part can be taken care by bitbake world.
but we can address that later.

Quoted 103 lines
meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb new file +25 -0
new file mode 100644
index 000000000..0ea03dd2e
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
+# Include the full set of ptests
+PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_OE').split()])}"
+
+do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_OE').split()])}"
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb new file +6 -0
new file mode 100644
index 000000000..f56e841ff
@@ -0,0 +1,6 @@
+require meta-oe-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest images."
+
+PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
+
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +37 -2
index 0166b9c41..951d91f82 100644
@@ -1,5 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
 require  meta-oe-image-base.bb
 
 SUMMARY = "meta-oe ptest test image"
 
-IMAGE_INSTALL += "packagegroup-meta-oe"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_OE').split()])}"
+
+# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memroy, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
+
--

2.34.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#101882): https://lists.openembedded.org/g/openembedded-devel/message/101882
Mute This Topic: https://lists.openembedded.org/mt/98023314/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

author
#6 lore raw

Re: [oe] [meta-oe][PATCH 04/11] meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution

Quoted 55 lines · Khem Raj

Thanks for this patchset Tim, really appreciate it. I have few comments.

Quoted 7 lines · Tim Orling

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-oe-ptest-image.bb for an image per ptest in the form of meta-oe-ptest-image-XXX using class extensions.

* Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX

in meta-oe

Quoted 1 line

* Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take

less than ~30s

Quoted 16 lines

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating this number of class extensions is slow but making it easier to execute and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-oe-ptest-image-XXX can be built with:

$ bitbake meta-oe-ptest-all-image
Quoted 8 lines

Subsequently, all the ptest images can be run with:

$ bitbake -c testimage meta-oe-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of

simultaneous

Quoted 6 lines

QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresult.json
Quoted 7 lines
---
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 +++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 meta-oe/recipes-core/images/

meta-oe-ptest-all-image.bb

Quoted 1 line
 create mode 100644 meta-oe/recipes-core/images/

meta-oe-ptest-fast-image.bb

Quoted 7 lines

I think these images also need ( and also the ones in meta-python and meta-perl too I think )

IMAGE_FEATURES += "debug-tweaks"

since some of them add ssh and it will be helpful to be able to do passwordless access.

I was mirroring the setup in oe-core. The default local.conf.sample in poky enables
debug-tweaks and that is what I used as well as what the AutoBuilder uses.

The question is do we want to enforce a hard requirement on 'debug-tweaks' when
some users may want to test in a different environment with a defined user/password or
with ssh keys added to known_hosts, etc. Food for thought and I am open to the change.

Quoted 3 lines

and I was also wondering
if meta-oe-image-base.bb is really needed. We could simply include core-image-base.bb here.

I was trying to do minimal change to what was already there. Open to the change.

Quoted 4 lines

This change also makes
meta-oe-image.bb and packagegroup-meta-oe.bb also questionable. Since build part can be taken care by bitbake world.
but we can address that later.

Again, I was trying to make only the changes to enable this new "per image ptest" methodology. I personally
never used meta-oe-image.bb, meta-python-image.bb nor meta-perl-image.bb. I am open to dropping them if
we decide to drop the packagegroups (which are TBH a bit of a challenge since they are not necessarily always
up to date). In case others were using these for their workflows, I did not want to cause breakage without some
discussion.

I will note that meta-oe-ptest-image.bb, meta-python-ptest-image.bb and meta-perl-ptest-image.bb are all currently
needed in order to have layer isolation of the ptests. In core-image-ptest.bb, the PTESTS variable is set and
I did not want to clobber that. Hence the new PTESTS_META_OE and friends variables.

I'll also note that I tried to enable the "missing-ptest" QA check from ptest.bbclass, but this is a per recipe check (not
an image level check) and is currently hard coded to only look at meta/conf/distro/include/ptest-packagelists.inc and the PTESTS_FAST and PTESTS_SLOW variables defined there. I didn't immediately come up with a way to modularize that
behavior.

Quoted 118 lines

Review

Quoted 3 lines
(quoted diff) +0 -0

b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb

Quoted 10 lines

new file mode 100644
index 000000000..0ea03dd2e

meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb new file +5 -0
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+

meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb line 15

Quoted 10 lines
meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb new file +10 -0
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
+# Include the full set of ptests
+PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@'

'.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_OE').split()])}"

meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb line 17

Quoted 2 lines
meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb new file +2 -0
+
+do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build'

for x in d.getVar('PTESTS_META_OE').split()])}"

Review

Quoted 9 lines
meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb new file +8 -0
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
(quoted diff) +0 -0

b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb

meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb line 3

Quoted 8 lines

new file mode 100644
index 000000000..f56e841ff

meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb new file +3 -0
@@ -0,0 +1,6 @@
+require meta-oe-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest

images."

Review

Quoted 4 lines
meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb new file +3 -0
+
+PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
+
(quoted diff) +0 -0

b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb

Quoted 12 lines

index 0166b9c41..951d91f82 100644

meta-oe/recipes-core/images/meta-oe-ptest-image.bb +5 -0
@@ -1,5 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
 require  meta-oe-image-base.bb
 
 SUMMARY = "meta-oe ptest test image"

meta-oe/recipes-core/images/meta-oe-ptest-image.bb line 17

Quoted 10 lines
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +8 -1
 
-IMAGE_INSTALL += "packagegroup-meta-oe"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in

d.getVar('PTESTS_META_OE').split()])}"

meta-oe/recipes-core/images/meta-oe-ptest-image.bb line 19

Quoted 2 lines
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +2 -0
+
+# The image can be sufficiently large (~1.8GB) that we need to be

careful that it fits in a live

meta-oe/recipes-core/images/meta-oe-ptest-image.bb line 20

Quoted 1 line
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +1 -0
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x

out of the

Quoted 4 lines
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +4 -0
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:

meta-oe/recipes-core/images/meta-oe-ptest-image.bb

Quoted 10 lines
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +10 -0
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memroy, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and

sporadically fails in random places

Quoted 11 lines
meta-oe/recipes-core/images/meta-oe-ptest-image.bb +7 -1
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
+
--

2.34.1

Quoted 3 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#101882):

https://lists.openembedded.org/g/openembedded-devel/message/101882

Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/98023314/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [

[email protected]]

Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-