coreutils: add ptest support
v5 cover coreutils: add ptest support
Note that this patch set does not include Richard's "coreutils hack" patch, so that will have to be added on top (as discussed in #yocto).
Sample test results:
core-image-minimal:
MACHINE | PASS | FAIL | SKIP | TOTAL | TIME (m) | qemux86-64 | 470 | 0 | 144 | 614 | 2.5 | qemuarm64 | 470 | 0 | 144 | 614 | 51 |
core-image-sato:
MACHINE | PASS | FAIL | SKIP | TOTAL | Time (m) | qemux86-64 | 472 | 0 | 142 | 614 | 2.4 | qemuarm64 | 472 | 0 | 142 | 614 | 52 |
Trevor Gamblin (2):
coreutils: add ptest
ptest-packagelists.inc: add coreutils to SLOW
.../distro/include/ptest-packagelists.inc | 1 + .../coreutils/coreutils/run-ptest | 17 +++++++ meta/recipes-core/coreutils/coreutils_8.31.bb | 44 +++++++++++++++++++ 3 files changed, 62 insertions(+) create mode 100755 meta/recipes-core/coreutils/coreutils/run-ptest
Signature · 1 line
2.17.1
PATCH v5 1/2 coreutils: add ptest
coreutils has a large number of tests, including some added by the Makefile flags RUN_EXPENSIVE_TESTS and RUN_VERY_EXPENSIVE_TESTS that significantly increase runtime (and that have been disabled). Note that the coreutils ptest directory is given blanket permissions at runtime with chmod -R 777 to ensure that the user created for the tests will be able to run the test scripts and create the necessary files in the process.
There is still room to improve the results of this ptest without the aforementioned additions. Of the tests marked SKIP, there are 30 tests that are currently counted as SKIP because they require sudo permissions, and another 21 that require membership in multiple user groups. It is important to know that coreutils has tests for both root and non-root users. Testing showed that 42 tests are skipped when running as root versus 30 when running as a non-root user, so the decision was made to run the suite as the latter. Additionally, gdb, valgrind, and strace could be included in the RDEPENDS list to increase pass rate, but their total contribution is 13 tests, so they were omitted to reduce image size.
Finally, note that at least one ptest (misc/head-write-error.sh) is prone to ERROR on builds of core-image-minimal if extra space is not provided with IMAGE_ROOTFS_EXTRA_SPACE.
--- .../coreutils/coreutils/run-ptest | 17 ++++++++ meta/recipes-core/coreutils/coreutils_8.31.bb | 40 +++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100755 meta/recipes-core/coreutils/coreutils/run-ptest
meta/recipes-core/coreutils/coreutils/run-ptest new file +17 -0
@@ -0,0 +1,17 @@ +#!/bin/sh + +# remove any stale lock files so that the calls to groupadd/useradd don't stop +# the ptest if re-using the same image +rm -rf /etc/passwd.lock /etc/group.lock /etc/gshadow.lock + +COREUTILSLIB=@libdir@/coreutils +LOG="${COREUTILSLIB}/ptest/coreutils_ptest_$(date +%Y%m%d-%H%M%S).log" +USERNAME="tester" +groupadd ugroup1 +groupadd ugroup2 +useradd -G ugroup1,ugroup2 $USERNAME || echo "user $USERNAME already exists" + +su tester -c "cd ${COREUTILSLIB}/ptest && make check-TESTS top_srcdir=. srcdir=." 2>&1 | tee -a ${LOG} +userdel $USERNAME +groupdel ugroup1 +groupdel ugroup2
meta/recipes-core/coreutils/coreutils_8.31.bb +40 -0
@@ -18,6 +18,7 @@ SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.xz \ file://0001-uname-report-processor-and-hardware-correctly.patch \ file://disable-ls-output-quoting.patch \ file://0001-local.mk-fix-cross-compiling-problem.patch \ + file://run-ptest \ " SRC_URI_append_libc-musl = "file://strtod_fix_clash_with_strtold.patch"
@@ -143,3 +144,42 @@ python __anonymous() { } BBCLASSEXTEND = "native nativesdk" + +inherit ptest + +RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow" + +do_install_ptest () { + install -d ${D}${PTEST_PATH}/tests + cp -r ${S}/tests/* ${D}${PTEST_PATH}/tests + sed -i 's/ginstall/install/g' `grep -R ginstall ${D}${PTEST_PATH}/tests | awk -F: '{print $1}' | uniq` + install -d ${D}${PTEST_PATH}/build-aux + install ${S}/build-aux/test-driver ${D}${PTEST_PATH}/build-aux/ + cp ${B}/Makefile ${D}${PTEST_PATH}/ + cp ${S}/init.cfg ${D}${PTEST_PATH}/ + cp -r ${B}/src ${D}${PTEST_PATH}/ + cp -r ${S}/src/*.c ${D}${PTEST_PATH}/src + sed -i '/^VPATH/s/= .*$/= ./g' ${D}${PTEST_PATH}/Makefile + sed -i '/^PROGRAMS/s/^/#/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^Makefile: /s/^.*$/Makefile:/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^abs_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^abs_top_builddir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^abs_top_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^built_programs/s/ginstall/install/g' ${D}${PTEST_PATH}/Makefile + chmod -R 777 ${D}${PTEST_PATH} + + # Disable subcase stty-pairs.sh, it will cause test framework hang + sed -i '/stty-pairs.sh/d' ${D}${PTEST_PATH}/Makefile + + # Disable subcase tail-2/assert.sh as it has issues on 32-bit systems + sed -i '/assert.sh/d' ${D}${PTEST_PATH}/Makefile + + # Tweak test d_type-check to use python3 instead of python + sed -i "1s@.*@#!/usr/bin/python3@" ${D}${PTEST_PATH}/tests/d_type-check + install ${B}/src/getlimits ${D}/${bindir} + + # handle multilib + sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest +} + +FILES_${PN}-ptest += "${bindir}/getlimits"
Signature · 1 line
2.17.1
Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest
Quoted 68 lines · Trevor Gamblin
coreutils has a large number of tests, including some added by the
MakefileflagsRUN_EXPENSIVE_TESTSandRUN_VERY_EXPENSIVE_TESTSthat significantly increase runtime (and that have been disabled). Note that the coreutils ptest directory is given blanket permissions at runtime with chmod -R 777 to ensure that the user created for the tests will be able to run the test scripts and create the necessary files in the process.There is still room to improve the results of this ptest without the aforementioned additions. Of the tests marked SKIP, there are 30 tests that are currently counted as SKIP because they require sudo permissions, and another 21 that require membership in multiple user groups. It is important to know that coreutils has tests for both root and non-root users. Testing showed that 42 tests are skipped when running as root versus 30 when running as a non-root user, so the decision was made to run the suite as the latter. Additionally, gdb, valgrind, and strace could be included in the RDEPENDS list to increase pass rate, but their total contribution is 13 tests, so they were omitted to reduce image size.
Finally, note that at least one ptest (
misc/head-write-error.sh) is prone to ERROR on builds of core-image-minimal if extra space is not provided withIMAGE_ROOTFS_EXTRA_SPACE.Signed-off-by Trevor Gamblin <[email protected]>--- .../coreutils/coreutils/run-ptest | 17 ++++++++ meta/recipes-core/coreutils/coreutils_8.31.bb | 40 +++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100755 meta/recipes-core/coreutils/coreutils/run-ptestmeta/recipes-core/coreutils/coreutils/run-ptest new file +17
-0@@ -0,0 +1,17 @@ +#!/bin/sh + +# remove any stale lock files so that the calls to groupadd/useradd don't stop +# the ptest if re-using the same image +rm -rf /etc/passwd.lock /etc/group.lock /etc/gshadow.lock + +COREUTILSLIB=@libdir@/coreutils +LOG="${COREUTILSLIB}/ptest/coreutils_ptest_$(date +%Y%m%d-%H%M%S).log" +USERNAME="tester" +groupadd ugroup1 +groupadd ugroup2 +useradd -G ugroup1,ugroup2 $USERNAME || echo "user $USERNAME already exists" + +su tester -c "cd ${COREUTILSLIB}/ptest && make check-TESTS top_srcdir=. srcdir=." 2>&1 | tee -a ${LOG} +userdel $USERNAME +groupdel ugroup1 +groupdel ugroup2meta/recipes-core/coreutils/coreutils_8.31.bb +1
-0@@ -18,6 +18,7 @@ SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.xz \ file://0001-uname-report-processor-and-hardware-correctly.patch \ file://disable-ls-output-quoting.patch \ file://0001-local.mk-fix-cross-compiling-problem.patch \ + file://run-ptest \ " SRC_URI_append_libc-musl = "file://strtod_fix_clash_with_strtold.patch"
meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151
Quoted 8 lines
meta/recipes-core/coreutils/coreutils_8.31.bb +5
-0@@ -143,3 +144,42 @@ python __anonymous() { } BBCLASSEXTEND = "native nativesdk" + +inherit ptest + +RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow" +
Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+
"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"
IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.
Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.
Quoted 42 lines
meta/recipes-core/coreutils/coreutils_8.31.bb +34
-1+do_install_ptest () { + install -d ${D}${PTEST_PATH}/tests + cp -r ${S}/tests/* ${D}${PTEST_PATH}/tests + sed -i 's/ginstall/install/g' `grep -R ginstall ${D}${PTEST_PATH}/tests | awk -F: '{print $1}' | uniq` + install -d ${D}${PTEST_PATH}/build-aux + install ${S}/build-aux/test-driver ${D}${PTEST_PATH}/build-aux/ + cp ${B}/Makefile ${D}${PTEST_PATH}/ + cp ${S}/init.cfg ${D}${PTEST_PATH}/ + cp -r ${B}/src ${D}${PTEST_PATH}/ + cp -r ${S}/src/*.c ${D}${PTEST_PATH}/src + sed -i '/^VPATH/s/= .*$/= ./g' ${D}${PTEST_PATH}/Makefile + sed -i '/^PROGRAMS/s/^/#/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^Makefile: /s/^.*$/Makefile:/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^abs_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^abs_top_builddir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^abs_top_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile + sed -i '/^built_programs/s/ginstall/install/g' ${D}${PTEST_PATH}/Makefile + chmod -R 777 ${D}${PTEST_PATH} + + # Disable subcase stty-pairs.sh, it will cause test framework hang + sed -i '/stty-pairs.sh/d' ${D}${PTEST_PATH}/Makefile + + # Disable subcase tail-2/assert.sh as it has issues on 32-bit systems + sed -i '/assert.sh/d' ${D}${PTEST_PATH}/Makefile + + # Tweak test d_type-check to use python3 instead of python + sed -i "1s@.*@#!/usr/bin/python3@" ${D}${PTEST_PATH}/tests/d_type-check + install ${B}/src/getlimits ${D}/${bindir} + + # handle multilib + sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest +} + +FILES_${PN}-ptest += "${bindir}/getlimits" --2.17.1
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest
Quoted 15 lines · Khem Raj
meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151
Quoted 4 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +4
-0+inherit ptest + +RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow" +Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"
Quoted 8 lines
IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.
We were aware of this when it merged. This should already only happen if ptest is enabled?
(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).
I wish we had a better way of handling these dependencies but we sadly don't.
Cheers,
Richard
Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest
Quoted 25 lines · Richard Purdie
Quoted 5 lines · Khem Raj
meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() {
Quoted 3 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +3
-0+inherit ptest + +RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perllibmodule-build-perl make perl perl-module-file-stat python3-core sed shadow"
Quoted 5 lines
Quoted 1 line
+
Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal the dependency chain looks like, total task building for this image are
3000+
Quoted 11 lines
"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"
IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.
Quoted 6 lines
Quoted 2 lines
Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.
We were aware of this when it merged. This should already only happen if ptest is enabled?
I am seeing it with a fresh checkout of poky with no other changes
Quoted 5 lines
(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).
I wish we had a better way of handling these dependencies but we sadly don't.
Such changes should be left to users to enable imo may be a packageconfig or so
People have reputation of long build times and large disk space for OE this won’t help that cause
Quoted 8 lines
Cheers,
Richard
Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest
Quoted 23 lines · Khem Raj
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Khem Raj
meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151
Quoted 4 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +4
-0+inherit ptest + +RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow" +Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"
IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.
We were aware of this when it merged. This should already only happen if ptest is enabled?
I am seeing it with a fresh checkout of poky with no other changes
Poky enables ptest by default:
meta-poky/conf/distro/poky.conf:POKY_DEFAULT_DISTRO_FEATURES = "largefile opengl ptest multiarch wayland vulkan"
so that is expected.
Quoted 9 lines
Quoted 5 lines
(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).
I wish we had a better way of handling these dependencies but we sadly don't.
Such changes should be left to users to enable imo may be a packageconfig or so
People have reputation of long build times and large disk space for OE this won’t help that cause
Well, I tried to have this conversation when the patch was being reviewed and tested. Nobody seemed bothered then. I get tired of talking to myself in a vacuum and having a reputation about being difficult over patches when nobody seems to care (as no other voices came into the conversation). I've tried to minimise the impact which was the best we can do. We are trying to encourage people to add tests.
Starting to revert things just before release is a really bad idea so we're not doing that.
We can disable ptest in coreutils specifically if we wanted. Ultimately we need to find a better way in general of handling some of these runtime dependencies but at this point that is a 3.2 issue.
Sorry, but if we were to have this discussion it should have been months ago :(.
I'd also note that 3.1 rc1 is building now.
Cheers,
Richard
Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest
Quoted 31 lines · Richard Purdie
Quoted 30 lines · Khem Raj
Quoted 28 lines · Richard Purdie
Quoted 15 lines · Khem Raj
meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151
Quoted 4 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +4
-0+inherit ptest + +RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow" +Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"
Quoted 8 lines
IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.
We were aware of this when it merged. This should already only happen if ptest is enabled?
Quoted 8 lines
Quoted 1 line
I am seeing it with a fresh checkout of poky with no other changes
Poky enables ptest by default:
meta-poky/conf/distro/poky.conf:POKY_DEFAULT_DISTRO_FEATURES= "largefile opengl ptest multiarch wayland vulkan"so that is expected.
Makes sense to me and I'm glad it's not affecting oe-core by default. FYI poky has had ptest enabled since at least 2016.
Since poky builds with coreutils will take a bit longer, we should mention that in the release notes. Paul E added since I hear he writes much of the release notes.
../Randy
Quoted 37 lines
Quoted 10 lines
Quoted 6 lines
(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).
I wish we had a better way of handling these dependencies but we sadly don't.
Such changes should be left to users to enable imo may be a packageconfig or so
People have reputation of long build times and large disk space for OE this won’t help that cause
Well, I tried to have this conversation when the patch was being reviewed and tested. Nobody seemed bothered then. I get tired of talking to myself in a vacuum and having a reputation about being difficult over patches when nobody seems to care (as no other voices came into the conversation). I've tried to minimise the impact which was the best we can do. We are trying to encourage people to add tests.
Starting to revert things just before release is a really bad idea so we're not doing that.
We can disable ptest in coreutils specifically if we wanted. Ultimately we need to find a better way in general of handling some of these runtime dependencies but at this point that is a 3.2 issue.
Sorry, but if we were to have this discussion it should have been months ago :(.
I'd also note that 3.1 rc1 is building now.
Cheers,
Richard
Signature · 2 lines
# Randy MacLeod # Wind River Linux
PATCH v5 2/2 ptest-packagelists.inc: add coreutils to SLOW
In reply to Trevor Gamblin--- meta/conf/distro/include/ptest-packagelists.inc | 1 + 1 file changed, 1 insertion(+)
meta/conf/distro/include/ptest-packagelists.inc +1 -0
@@ -66,6 +66,7 @@ PTESTS_SLOW = "\ babeltrace-ptest \ babeltrace2-ptest \ busybox-ptest \ + coreutils-ptest \ dbus-test-ptest \ e2fsprogs-ptest \ glib-2.0-ptest \
Signature · 1 line
2.17.1