Skip to contentSkip to conversation

coreutils: add ptest support

PATCH In review Trevor Gamblin 4 participants 8 messages Mar 11, 2020 – Apr 7, 2020

lore
Share & tools
lore
#0 lore raw

v5 cover coreutils: add ptest support

Note that this patch set does not include Richard's "coreutils hack" patch, so that will have to be added on top (as discussed in #yocto).

Sample test results:

core-image-minimal:

MACHINE    | PASS | FAIL | SKIP | TOTAL | TIME (m) |
qemux86-64 |  470 |    0 |  144 |   614 |      2.5 |
qemuarm64  |  470 |    0 |  144 |   614 |       51 |

core-image-sato:

MACHINE    | PASS | FAIL | SKIP | TOTAL | Time (m) |
qemux86-64 |  472 |    0 |  142 |   614 |      2.4 |
qemuarm64  |  472 |    0 |  142 |   614 |       52 |

Trevor Gamblin (2):
coreutils: add ptest
ptest-packagelists.inc: add coreutils to SLOW

 .../distro/include/ptest-packagelists.inc     |  1 +
 .../coreutils/coreutils/run-ptest             | 17 +++++++
 meta/recipes-core/coreutils/coreutils_8.31.bb | 44 +++++++++++++++++++
 3 files changed, 62 insertions(+)
 create mode 100755 meta/recipes-core/coreutils/coreutils/run-ptest
Signature · 1 line
2.17.1
#1 lore raw

PATCH v5 1/2 coreutils: add ptest

coreutils has a large number of tests, including some added by the Makefile flags RUN_EXPENSIVE_TESTS and RUN_VERY_EXPENSIVE_TESTS that significantly increase runtime (and that have been disabled). Note that the coreutils ptest directory is given blanket permissions at runtime with chmod -R 777 to ensure that the user created for the tests will be able to run the test scripts and create the necessary files in the process.

There is still room to improve the results of this ptest without the aforementioned additions. Of the tests marked SKIP, there are 30 tests that are currently counted as SKIP because they require sudo permissions, and another 21 that require membership in multiple user groups. It is important to know that coreutils has tests for both root and non-root users. Testing showed that 42 tests are skipped when running as root versus 30 when running as a non-root user, so the decision was made to run the suite as the latter. Additionally, gdb, valgrind, and strace could be included in the RDEPENDS list to increase pass rate, but their total contribution is 13 tests, so they were omitted to reduce image size.

Finally, note that at least one ptest (misc/head-write-error.sh) is prone to ERROR on builds of core-image-minimal if extra space is not provided with IMAGE_ROOTFS_EXTRA_SPACE.

---
 .../coreutils/coreutils/run-ptest             | 17 ++++++++
 meta/recipes-core/coreutils/coreutils_8.31.bb | 40 +++++++++++++++++++
 2 files changed, 57 insertions(+)
 create mode 100755 meta/recipes-core/coreutils/coreutils/run-ptest
meta/recipes-core/coreutils/coreutils/run-ptest new file +17 -0
new file mode 100755
index 0000000000..6d4a7b365d
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+# remove any stale lock files so that the calls to groupadd/useradd don't stop
+# the ptest if re-using the same image
+rm -rf /etc/passwd.lock /etc/group.lock /etc/gshadow.lock
+
+COREUTILSLIB=@libdir@/coreutils
+LOG="${COREUTILSLIB}/ptest/coreutils_ptest_$(date +%Y%m%d-%H%M%S).log"
+USERNAME="tester"
+groupadd ugroup1
+groupadd ugroup2
+useradd -G ugroup1,ugroup2 $USERNAME || echo "user $USERNAME already exists"
+
+su tester -c "cd ${COREUTILSLIB}/ptest && make check-TESTS top_srcdir=. srcdir=." 2>&1 | tee -a ${LOG}
+userdel $USERNAME 
+groupdel ugroup1
+groupdel ugroup2
meta/recipes-core/coreutils/coreutils_8.31.bb +40 -0
index 57b2c1bdba..cba0bfe15c 100644
@@ -18,6 +18,7 @@ SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.xz \
            file://0001-uname-report-processor-and-hardware-correctly.patch \
            file://disable-ls-output-quoting.patch \
            file://0001-local.mk-fix-cross-compiling-problem.patch \
+           file://run-ptest \
           "
 
 SRC_URI_append_libc-musl = "file://strtod_fix_clash_with_strtold.patch"
@@ -143,3 +144,42 @@ python __anonymous() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+inherit ptest
+
+RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow"
+
+do_install_ptest () {
+    install -d ${D}${PTEST_PATH}/tests
+    cp -r ${S}/tests/* ${D}${PTEST_PATH}/tests
+    sed -i 's/ginstall/install/g'  `grep -R ginstall ${D}${PTEST_PATH}/tests | awk -F: '{print $1}' | uniq`
+    install -d ${D}${PTEST_PATH}/build-aux
+    install ${S}/build-aux/test-driver ${D}${PTEST_PATH}/build-aux/
+    cp ${B}/Makefile ${D}${PTEST_PATH}/
+    cp ${S}/init.cfg ${D}${PTEST_PATH}/
+    cp -r ${B}/src ${D}${PTEST_PATH}/
+    cp -r ${S}/src/*.c ${D}${PTEST_PATH}/src
+    sed -i '/^VPATH/s/= .*$/= ./g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^PROGRAMS/s/^/#/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^Makefile: /s/^.*$/Makefile:/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^abs_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^abs_top_builddir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^abs_top_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^built_programs/s/ginstall/install/g' ${D}${PTEST_PATH}/Makefile
+    chmod -R 777 ${D}${PTEST_PATH}
+
+    # Disable subcase stty-pairs.sh, it will cause test framework hang
+    sed -i '/stty-pairs.sh/d' ${D}${PTEST_PATH}/Makefile
+
+    # Disable subcase tail-2/assert.sh as it has issues on 32-bit systems
+    sed -i '/assert.sh/d' ${D}${PTEST_PATH}/Makefile
+
+    # Tweak test d_type-check to use python3 instead of python
+    sed -i "1s@.*@#!/usr/bin/python3@" ${D}${PTEST_PATH}/tests/d_type-check
+    install ${B}/src/getlimits ${D}/${bindir}
+    
+    # handle multilib
+    sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
+}
+
+FILES_${PN}-ptest += "${bindir}/getlimits"
Signature · 1 line
2.17.1
#2 lore raw

Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest

Quoted 68 lines · Trevor Gamblin

coreutils has a large number of tests, including some added by the Makefile flags RUN_EXPENSIVE_TESTS and RUN_VERY_EXPENSIVE_TESTS that significantly increase runtime (and that have been disabled). Note that the coreutils ptest directory is given blanket permissions at runtime with chmod -R 777 to ensure that the user created for the tests will be able to run the test scripts and create the necessary files in the process.

There is still room to improve the results of this ptest without the aforementioned additions. Of the tests marked SKIP, there are 30 tests that are currently counted as SKIP because they require sudo permissions, and another 21 that require membership in multiple user groups. It is important to know that coreutils has tests for both root and non-root users. Testing showed that 42 tests are skipped when running as root versus 30 when running as a non-root user, so the decision was made to run the suite as the latter. Additionally, gdb, valgrind, and strace could be included in the RDEPENDS list to increase pass rate, but their total contribution is 13 tests, so they were omitted to reduce image size.

Finally, note that at least one ptest (misc/head-write-error.sh) is prone to ERROR on builds of core-image-minimal if extra space is not provided with IMAGE_ROOTFS_EXTRA_SPACE.

---
 .../coreutils/coreutils/run-ptest             | 17 ++++++++
 meta/recipes-core/coreutils/coreutils_8.31.bb | 40 +++++++++++++++++++
 2 files changed, 57 insertions(+)
 create mode 100755 meta/recipes-core/coreutils/coreutils/run-ptest
meta/recipes-core/coreutils/coreutils/run-ptest new file +17 -0
new file mode 100755
index 0000000000..6d4a7b365d
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+# remove any stale lock files so that the calls to groupadd/useradd don't stop
+# the ptest if re-using the same image
+rm -rf /etc/passwd.lock /etc/group.lock /etc/gshadow.lock
+
+COREUTILSLIB=@libdir@/coreutils
+LOG="${COREUTILSLIB}/ptest/coreutils_ptest_$(date +%Y%m%d-%H%M%S).log"
+USERNAME="tester"
+groupadd ugroup1
+groupadd ugroup2
+useradd -G ugroup1,ugroup2 $USERNAME || echo "user $USERNAME already exists"
+
+su tester -c "cd ${COREUTILSLIB}/ptest && make check-TESTS top_srcdir=. srcdir=." 2>&1 | tee -a ${LOG}
+userdel $USERNAME
+groupdel ugroup1
+groupdel ugroup2
meta/recipes-core/coreutils/coreutils_8.31.bb +1 -0
index 57b2c1bdba..cba0bfe15c 100644
@@ -18,6 +18,7 @@ SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.xz \
            file://0001-uname-report-processor-and-hardware-correctly.patch \
            file://disable-ls-output-quoting.patch \
            file://0001-local.mk-fix-cross-compiling-problem.patch \
+           file://run-ptest \
           "
 
 SRC_URI_append_libc-musl = "file://strtod_fix_clash_with_strtold.patch"

meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151

Quoted 8 lines
meta/recipes-core/coreutils/coreutils_8.31.bb +5 -0
@@ -143,3 +144,42 @@ python __anonymous() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+inherit ptest
+
+RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow"
+

Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+

"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"

IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.

Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.

Quoted 42 lines
meta/recipes-core/coreutils/coreutils_8.31.bb +34 -1
+do_install_ptest () {
+    install -d ${D}${PTEST_PATH}/tests
+    cp -r ${S}/tests/* ${D}${PTEST_PATH}/tests
+    sed -i 's/ginstall/install/g'  `grep -R ginstall ${D}${PTEST_PATH}/tests | awk -F: '{print $1}' | uniq`
+    install -d ${D}${PTEST_PATH}/build-aux
+    install ${S}/build-aux/test-driver ${D}${PTEST_PATH}/build-aux/
+    cp ${B}/Makefile ${D}${PTEST_PATH}/
+    cp ${S}/init.cfg ${D}${PTEST_PATH}/
+    cp -r ${B}/src ${D}${PTEST_PATH}/
+    cp -r ${S}/src/*.c ${D}${PTEST_PATH}/src
+    sed -i '/^VPATH/s/= .*$/= ./g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^PROGRAMS/s/^/#/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^Makefile: /s/^.*$/Makefile:/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^abs_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^abs_top_builddir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^abs_top_srcdir/s/= .*$/= \$\{PWD\}/g' ${D}${PTEST_PATH}/Makefile
+    sed -i '/^built_programs/s/ginstall/install/g' ${D}${PTEST_PATH}/Makefile
+    chmod -R 777 ${D}${PTEST_PATH}
+
+    # Disable subcase stty-pairs.sh, it will cause test framework hang
+    sed -i '/stty-pairs.sh/d' ${D}${PTEST_PATH}/Makefile
+
+    # Disable subcase tail-2/assert.sh as it has issues on 32-bit systems
+    sed -i '/assert.sh/d' ${D}${PTEST_PATH}/Makefile
+
+    # Tweak test d_type-check to use python3 instead of python
+    sed -i "1s@.*@#!/usr/bin/python3@" ${D}${PTEST_PATH}/tests/d_type-check
+    install ${B}/src/getlimits ${D}/${bindir}
+
+    # handle multilib
+    sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
+}
+
+FILES_${PN}-ptest += "${bindir}/getlimits"
--

2.17.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#3 lore raw

Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest

Quoted 15 lines · Khem Raj

meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151

Quoted 4 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +4 -0
+inherit ptest
+
+RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow"
+

Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+

"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"

Quoted 8 lines

IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.

Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.

We were aware of this when it merged. This should already only happen if ptest is enabled?

(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).

I wish we had a better way of handling these dependencies but we sadly don't.

Cheers,

Richard

#4 lore raw

Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest

Quoted 25 lines · Richard Purdie
Quoted 5 lines · Khem Raj

meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() {

Quoted 3 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +3 -0
+inherit ptest
+
+RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl

libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow"

Quoted 5 lines
Quoted 1 line

+

Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal the dependency chain looks like, total task building for this image are

3000+

Quoted 11 lines

"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"

IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.

Quoted 6 lines
Quoted 2 lines

Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.

We were aware of this when it merged. This should already only happen if ptest is enabled?

I am seeing it with a fresh checkout of poky with no other changes

Quoted 5 lines

(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).

I wish we had a better way of handling these dependencies but we sadly don't.

Such changes should be left to users to enable imo may be a packageconfig or so

People have reputation of long build times and large disk space for OE this won’t help that cause

Quoted 8 lines

Cheers,

Richard

#5 lore raw

Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest

Quoted 23 lines · Khem Raj
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Khem Raj

meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151

Quoted 4 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +4 -0
+inherit ptest
+
+RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow"
+

Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+

"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"

IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.

We were aware of this when it merged. This should already only happen if ptest is enabled?

I am seeing it with a fresh checkout of poky with no other changes

Poky enables ptest by default:

meta-poky/conf/distro/poky.conf:
POKY_DEFAULT_DISTRO_FEATURES = "largefile opengl ptest multiarch wayland vulkan"

so that is expected.

Quoted 9 lines
Quoted 5 lines

(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).

I wish we had a better way of handling these dependencies but we sadly don't.

Such changes should be left to users to enable imo may be a packageconfig or so

People have reputation of long build times and large disk space for OE this won’t help that cause

Well, I tried to have this conversation when the patch was being reviewed and tested. Nobody seemed bothered then. I get tired of talking to myself in a vacuum and having a reputation about being difficult over patches when nobody seems to care (as no other voices came into the conversation). I've tried to minimise the impact which was the best we can do. We are trying to encourage people to add tests.

Starting to revert things just before release is a really bad idea so we're not doing that.

We can disable ptest in coreutils specifically if we wanted. Ultimately we need to find a better way in general of handling some of these runtime dependencies but at this point that is a 3.2 issue.

Sorry, but if we were to have this discussion it should have been months ago :(.

I'd also note that 3.1 rc1 is building now.

Cheers,

Richard

#6 lore raw

Re: [OE-core] [PATCH 1/2 v5] coreutils: add ptest

Quoted 31 lines · Richard Purdie
Quoted 30 lines · Khem Raj
Quoted 28 lines · Richard Purdie
Quoted 15 lines · Khem Raj

meta/recipes-core/coreutils/coreutils_8.31.bb python __anonymous() { · line 151

Quoted 4 lines · Trevor Gamblin
meta/recipes-core/coreutils/coreutils_8.31.bb +4 -0
+inherit ptest
+
+RDEPENDS_${PN}-ptest += "bash findutils gawk liberror-perl libmodule-build-perl make perl perl-module-file-stat python3-core sed shadow"
+

Adding libmodule-build-perl to rdeps here is pulling in whole target toolchain components now while building core-image-minimal
the dependency chain looks like, total task building for this image are 3000+

"coreutils.do_package_qa" -> "libmodule-build-perl.do_packagedata" -> "packagegroup-core-buildessential.do_packagedata" -> "gcc.do_packagedata"

Quoted 8 lines

IMO, it's a bad experience for the user when building a simple minimal image and it ends up building whole toolchain for target that eventually won't even
be part of the image. perhaps make it dependent on ptest or some such knob thats not commonly used.

Please revert this patch series until we can find a way to not insert these build-time dependencies in normal flow.

We were aware of this when it merged. This should already only happen if ptest is enabled?

Quoted 8 lines
Quoted 1 line

I am seeing it with a fresh checkout of poky with no other changes

Poky enables ptest by default:

meta-poky/conf/distro/poky.conf:
POKY_DEFAULT_DISTRO_FEATURES = "largefile opengl ptest multiarch wayland vulkan"

so that is expected.

Makes sense to me and I'm glad it's not affecting oe-core by default. FYI poky has had ptest enabled since at least 2016.

Since poky builds with coreutils will take a bit longer, we should mention that in the release notes. Paul E added since I hear he writes much of the release notes.

../Randy

Quoted 37 lines
Quoted 10 lines
Quoted 6 lines

(else ${PN}-ptest isn't in PACKAGES and hence the RDEPENDS isn't seen).

I wish we had a better way of handling these dependencies but we sadly don't.

Such changes should be left to users to enable imo may be a packageconfig or so

People have reputation of long build times and large disk space for OE this won’t help that cause

Well, I tried to have this conversation when the patch was being reviewed and tested. Nobody seemed bothered then. I get tired of talking to myself in a vacuum and having a reputation about being difficult over patches when nobody seems to care (as no other voices came into the conversation). I've tried to minimise the impact which was the best we can do. We are trying to encourage people to add tests.

Starting to revert things just before release is a really bad idea so we're not doing that.

We can disable ptest in coreutils specifically if we wanted. Ultimately we need to find a better way in general of handling some of these runtime dependencies but at this point that is a 3.2 issue.

Sorry, but if we were to have this discussion it should have been months ago :(.

I'd also note that 3.1 rc1 is building now.

Cheers,

Richard

Signature · 2 lines
# Randy MacLeod
# Wind River Linux
#7 lore raw

PATCH v5 2/2 ptest-packagelists.inc: add coreutils to SLOW

In reply to Trevor Gamblin
---
 meta/conf/distro/include/ptest-packagelists.inc | 1 +
 1 file changed, 1 insertion(+)
meta/conf/distro/include/ptest-packagelists.inc +1 -0
index d6f3aafc7f..c13ff724b1 100644
@@ -66,6 +66,7 @@ PTESTS_SLOW = "\
     babeltrace-ptest \
     babeltrace2-ptest \
     busybox-ptest \
+    coreutils-ptest \
     dbus-test-ptest \
     e2fsprogs-ptest \
     glib-2.0-ptest \
Signature · 1 line
2.17.1