Skip to contentSkip to conversation

glibc: Exclude common code to build tests to glibc-tests.inc

PATCH Superseded ?ukasz Majewski 4 participants 18 messages Aug 23, 2021 – Aug 27, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 glibc: Exclude common code to build tests to glibc-tests.inc

The common code to build tests has been excluded from glibc-testsuite recipe to glibc-tests.inc

This code will be reused in the recipe necessary for providing glibc tests executed with ptest framework.

---
 meta/recipes-core/glibc/glibc-tests.inc       | 32 ++++++++++++++++
 .../glibc/glibc-testsuite_2.34.bb             | 37 ++-----------------
 2 files changed, 36 insertions(+), 33 deletions(-)
 create mode 100644 meta/recipes-core/glibc/glibc-tests.inc
meta/recipes-core/glibc/glibc-tests.inc new file +32 -0
new file mode 100644
index 0000000000..a8fe160a54
@@ -0,0 +1,32 @@
+EXCLUDE_FROM_WORLD = "1"
+
+# handle PN differences
+FILESEXTRAPATHS:prepend := "${THISDIR}/glibc:"
+
+# setup depends
+INHIBIT_DEFAULT_DEPS = ""
+
+python () {
+    libc = d.getVar("PREFERRED_PROVIDER_virtual/libc")
+    libclocale = d.getVar("PREFERRED_PROVIDER_virtual/libc-locale")
+    if libc != "glibc" or libclocale != "glibc-locale":
+        raise bb.parse.SkipRecipe("glibc-testsuite requires that virtual/libc is glibc")
+}
+
+DEPENDS += "glibc-locale libgcc gcc-runtime"
+
+# remove the initial depends
+DEPENDS:remove = "libgcc-initial"
+
+do_check[dirs] += "${B}"
+do_check () {
+    # clean out previous test results
+    oe_runmake tests-clean
+    # makefiles don't clean entirely (and also sometimes fails due to too many args)
+    find ${B} -type f -name "*.out" -delete
+    find ${B} -type f -name "*.test-result" -delete
+    find ${B}/catgets -name "*.cat" -delete
+    find ${B}/conform -name "symlist-*" -delete
+    [ ! -e ${B}/timezone/testdata ] || rm -rf ${B}/timezone/testdata
+}
+addtask do_check after do_compile
meta/recipes-core/glibc/glibc-testsuite_2.34.bb +4 -33
index 72aa332171..e8ad2a938b 100644
@@ -1,30 +1,12 @@
 require glibc_${PV}.bb
+require glibc-tests.inc
 
-EXCLUDE_FROM_WORLD = "1"
+inherit qemu
 
-# handle PN differences
-FILESEXTRAPATHS:prepend := "${THISDIR}/glibc:"
+SRC_URI += "file://check-test-wrapper"
 
 # strip provides
 PROVIDES = ""
-# setup depends
-INHIBIT_DEFAULT_DEPS = ""
-
-python () {
-    libc = d.getVar("PREFERRED_PROVIDER_virtual/libc")
-    libclocale = d.getVar("PREFERRED_PROVIDER_virtual/libc-locale")
-    if libc != "glibc" or libclocale != "glibc-locale":
-        raise bb.parse.SkipRecipe("glibc-testsuite requires that virtual/libc is glibc")
-}
-
-DEPENDS += "glibc-locale libgcc gcc-runtime"
-
-# remove the initial depends
-DEPENDS:remove = "libgcc-initial"
-
-inherit qemu
-
-SRC_URI += "file://check-test-wrapper"
 
 DEPENDS += "${@'qemu-native' if d.getVar('TOOLCHAIN_TEST_TARGET') == 'user' else ''}"
 
@@ -33,20 +15,10 @@ TOOLCHAIN_TEST_HOST ??= "localhost"
 TOOLCHAIN_TEST_HOST_USER ??= "root"
 TOOLCHAIN_TEST_HOST_PORT ??= "2222"
 
-do_check[dirs] += "${B}"
 do_check[nostamp] = "1"
-do_check () {
+do_check:append () {
     chmod 0755 ${WORKDIR}/check-test-wrapper
 
-    # clean out previous test results
-    oe_runmake tests-clean
-    # makefiles don't clean entirely (and also sometimes fails due to too many args)
-    find ${B} -type f -name "*.out" -delete
-    find ${B} -type f -name "*.test-result" -delete
-    find ${B}/catgets -name "*.cat" -delete
-    find ${B}/conform -name "symlist-*" -delete
-    [ ! -e ${B}/timezone/testdata ] || rm -rf ${B}/timezone/testdata
-
     oe_runmake -i \
         QEMU_SYSROOT="${RECIPE_SYSROOT}" \
         QEMU_OPTIONS="${@qemu_target_binary(d)} ${QEMU_OPTIONS}" \
@@ -56,7 +28,6 @@ do_check () {
         test-wrapper="${WORKDIR}/check-test-wrapper ${TOOLCHAIN_TEST_TARGET}" \
         check
 }
-addtask do_check after do_compile
 
 inherit nopackages
 deltask do_stash_locale
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/2 glibc: ptest: Add support for running glibc test suite with ptest

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call:
cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

---
 .../distro/include/ptest-packagelists.inc     |   1 +
 meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101 ++++++++++++++++++
 meta/recipes-core/glibc/glibc/run-ptest-all   |  25 +++++
 meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
 4 files changed, 157 insertions(+)
 create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb
 create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all
 create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time
meta/conf/distro/include/ptest-packagelists.inc +1 -0
index 3872bdc942..e460ad1c8b 100644
@@ -59,6 +59,7 @@ PTESTS_FAST = "\
     slang-ptest \
     wayland-ptest \
     zlib-ptest \
+    glibc-tests-ptest \
 "
 PTESTS_FAST:remove:mips64 = "qemu-ptest"
 PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
meta/recipes-core/glibc/glibc-tests_2.34.bb new file +101 -0

101 lines not on this page · raw message

meta/recipes-core/glibc/glibc/run-ptest-all new file +25 -0
new file mode 100755
index 0000000000..47bfe5238c
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+	$i >/dev/null 2>&1
+	output
+done
meta/recipes-core/glibc/glibc/run-ptest-time new file +30 -0
new file mode 100755
index 0000000000..24dedc6e32
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
+	$i >/dev/null 2>&1
+	output
+done
Signature · 1 line
2.20.1
#2 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 17 lines · ?ukasz Majewski

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call:
cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 4 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

Quoted 203 lines
---
  .../distro/include/ptest-packagelists.inc     |   1 +
  meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101 ++++++++++++++++++
  meta/recipes-core/glibc/glibc/run-ptest-all   |  25 +++++
  meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
  4 files changed, 157 insertions(+)

create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time

meta/conf/distro/include/ptest-packagelists.inc +1 -0
index 3872bdc942..e460ad1c8b 100644
@@ -59,6 +59,7 @@ PTESTS_FAST = "\
      slang-ptest \
      wayland-ptest \
      zlib-ptest \
+    glibc-tests-ptest \
  "
  PTESTS_FAST:remove:mips64 = "qemu-ptest"
  PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
meta/recipes-core/glibc/glibc-tests_2.34.bb new file +101 -0
new file mode 100644
index 0000000000..896809f700
@@ -0,0 +1,101 @@
+require glibc_${PV}.bb
+require glibc-tests.inc
+
+inherit ptest
+
+SRC_URI:append = " \
+	file://run-ptest-all file://run-ptest-time \
+"
+
+SUMMARY = "glibc tests to be run with ptest"
+
+# Erase some variables already set by glibc_${PV}
+python __anonymous() {
+       # Remove packages provided by glibc build, we only need a subset of them
+       d.setVar("PACKAGES", "${PN}")
+
+       d.setVar("PROVIDES", "${PN}")
+       d.setVar("RPROVIDES", "${PN} glibc-ptest")
+
+       d.setVar("RRECOMMENDS", "")
+}
+
+# Remove any lefovers from original glibc recipe
+RPROVIDES:${PN} = "${PN}"
+RRECOMMENDS:${PN} = ""
+RDEPENDS:${PN} = "sed"
+
+# Just build tests for target - do not run them
+do_check:append () {
+	oe_runmake -i check run-built-tests=no
+}
+addtask do_check after do_compile before do_install_ptest_base
+
+glibc_strip_build_directory () {
+	# Delete all non executable files from build directory
+	find ${B} ! -executable -type f -delete
+
+	# Remove build dynamic libraries and links to them as
+	# those are already installed in the target device
+	find ${B} -type f -name "*.so" -delete
+	find ${B} -type l -name "*.so*" -delete
+
+	# Remove headers (installed with glibc)
+	find ${B} -type f -name "*.h" -delete
+
+	find ${B} -type f -name "isomac" -delete
+	find ${B} -type f -name "annexc" -delete
+}
+
+do_install_ptest_base () {
+	glibc_strip_build_directory
+
+	# Install build test programs to the image
+	install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
+	cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
+
+	install -d ${D}${PTEST_PATH}
+	cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
+
+	# By default run all glibc tests
+	cd ${D}${PTEST_PATH}/
+	ln -s run-ptest-all run-ptest
+}
+
+# The datadir directory is required to allow core (and reused)
+# glibc cleanup function to finish correctly, as this directory
+# is not created for ptests
+stash_locale_package_cleanup:prepend () {
+	mkdir -p ${PKGD}${datadir}
+}
+
+stash_locale_sysroot_cleanup:prepend () {
+	mkdir -p ${SYSROOT_DESTDIR}${datadir}
+}
+
+# Prevent the do_package() task to set 'libc6' prefix
+# for glibc tests related packages
+python populate_packages:prepend () {
+    if d.getVar('DEBIAN_NAMES'):
+        d.setVar('DEBIAN_NAMES', '')
+}
+
+FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
+
+EXCLUDE_FROM_SHLIBS = "1"
+
+# Install debug data in .debug and sources in /usr/src/debug
+# It is more handy to have _all_ the sources and symbols in one
+# place (package) as this recipe will be used for validation and
+# debugging.
+PACKAGE_DEBUG_SPLIT_STYLE = "debug"
+
+# glibc test cases violate by default some Yocto/OE checks (staticdev,
+# textrel)
+# 'debug-files' - add everything (including debug) into one package
+#                 (no need to install/build *-src package)
+INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths"
+
+deltask do_stash_locale
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
meta/recipes-core/glibc/glibc/run-ptest-all new file +25 -0
new file mode 100755
index 0000000000..47bfe5238c
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+	$i >/dev/null 2>&1
+	output
+done
meta/recipes-core/glibc/glibc/run-ptest-time new file +30 -0
new file mode 100755
index 0000000000..24dedc6e32
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
+	$i >/dev/null 2>&1
+	output
+done
#3 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Hi Khem,

Quoted 18 lines · ?ukasz Majewski
Quoted 17 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call:
cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 8 lines
Quoted 4 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

Quoted 197 lines
Quoted 197 lines
---
  .../distro/include/ptest-packagelists.inc     |   1 +
  meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all   |
25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
  4 files changed, 157 insertions(+)

create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time

(quoted diff) +0 -0

b/meta/conf/distro/include/ptest-packagelists.inc index 3872bdc942..e460ad1c8b 100644 --- a/meta/conf/distro/include/ptest-packagelists.inc +++ b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6 +59,7 @@

PTESTS_FAST = "\ slang-ptest \
      wayland-ptest \
      zlib-ptest \
meta/conf/distro/include/ptest-packagelists.inc +1 -0
+    glibc-tests-ptest \
  "
  PTESTS_FAST:remove:mips64 = "qemu-ptest"
  PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode 100644 index 0000000000..896809f700

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +14 -0
@@ -0,0 +1,101 @@
+require glibc_${PV}.bb
+require glibc-tests.inc
+
+inherit ptest
+
+SRC_URI:append = " \
+	file://run-ptest-all file://run-ptest-time \
+"
+
+SUMMARY = "glibc tests to be run with ptest"
+
+# Erase some variables already set by glibc_${PV}
+python __anonymous() {
+       # Remove packages provided by glibc build, we only need a

subset of them

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +79 -0
+       d.setVar("PACKAGES", "${PN}")
+
+       d.setVar("PROVIDES", "${PN}")
+       d.setVar("RPROVIDES", "${PN} glibc-ptest")
+
+       d.setVar("RRECOMMENDS", "")
+}
+
+# Remove any lefovers from original glibc recipe
+RPROVIDES:${PN} = "${PN}"
+RRECOMMENDS:${PN} = ""
+RDEPENDS:${PN} = "sed"
+
+# Just build tests for target - do not run them
+do_check:append () {
+	oe_runmake -i check run-built-tests=no
+}
+addtask do_check after do_compile before do_install_ptest_base
+
+glibc_strip_build_directory () {
+	# Delete all non executable files from build directory
+	find ${B} ! -executable -type f -delete
+
+	# Remove build dynamic libraries and links to them as
+	# those are already installed in the target device
+	find ${B} -type f -name "*.so" -delete
+	find ${B} -type l -name "*.so*" -delete
+
+	# Remove headers (installed with glibc)
+	find ${B} -type f -name "*.h" -delete
+
+	find ${B} -type f -name "isomac" -delete
+	find ${B} -type f -name "annexc" -delete
+}
+
+do_install_ptest_base () {
+	glibc_strip_build_directory
+
+	# Install build test programs to the image
+	install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
+	cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
+
+	install -d ${D}${PTEST_PATH}
+	cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
+
+	# By default run all glibc tests
+	cd ${D}${PTEST_PATH}/
+	ln -s run-ptest-all run-ptest
+}
+
+# The datadir directory is required to allow core (and reused)
+# glibc cleanup function to finish correctly, as this directory
+# is not created for ptests
+stash_locale_package_cleanup:prepend () {
+	mkdir -p ${PKGD}${datadir}
+}
+
+stash_locale_sysroot_cleanup:prepend () {
+	mkdir -p ${SYSROOT_DESTDIR}${datadir}
+}
+
+# Prevent the do_package() task to set 'libc6' prefix
+# for glibc tests related packages
+python populate_packages:prepend () {
+    if d.getVar('DEBIAN_NAMES'):
+        d.setVar('DEBIAN_NAMES', '')
+}
+
+FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
+
+EXCLUDE_FROM_SHLIBS = "1"
+
+# Install debug data in .debug and sources in /usr/src/debug
+# It is more handy to have _all_ the sources and symbols in one
+# place (package) as this recipe will be used for validation and
+# debugging.
+PACKAGE_DEBUG_SPLIT_STYLE = "debug"
+
+# glibc test cases violate by default some Yocto/OE checks

(staticdev, +# textrel)

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +7 -0
+# 'debug-files' - add everything (including debug) into one package
+#                 (no need to install/build *-src package)
+INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths"
+
+deltask do_stash_locale
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode 100755 index 0000000000..47bfe5238c

meta/recipes-core/glibc/glibc/run-ptest-all new file +25 -0
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+	$i >/dev/null 2>&1
+	output
+done
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode 100755 index 0000000000..24dedc6e32

meta/recipes-core/glibc/glibc/run-ptest-time new file +27 -0
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
Quoted 8 lines

meta/recipes-core/glibc/glibc/run-ptest-time line 30

Quoted 8 lines
meta/recipes-core/glibc/glibc/run-ptest-time new file +3 -0
+	$i >/dev/null 2>&1
+	output
+done

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#4 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 33 lines · Lukasz Majewski

Hi Khem,

Quoted 18 lines · ?ukasz Majewski
Quoted 17 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call:
cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 8 lines
Quoted 4 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 220 lines
Quoted 198 lines
Quoted 198 lines
---
  .../distro/include/ptest-packagelists.inc     |   1 +
  meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all   |
25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
  4 files changed, 157 insertions(+)

create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time

(quoted diff) +0 -0

b/meta/conf/distro/include/ptest-packagelists.inc index 3872bdc942..e460ad1c8b 100644 --- a/meta/conf/distro/include/ptest-packagelists.inc +++ b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6 +59,7 @@

PTESTS_FAST = "\ slang-ptest \
      wayland-ptest \
      zlib-ptest \
meta/conf/distro/include/ptest-packagelists.inc +1 -0
+    glibc-tests-ptest \
  "
  PTESTS_FAST:remove:mips64 = "qemu-ptest"
  PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode 100644 index 0000000000..896809f700

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +14 -0
@@ -0,0 +1,101 @@
+require glibc_${PV}.bb
+require glibc-tests.inc
+
+inherit ptest
+
+SRC_URI:append = " \
+   file://run-ptest-all file://run-ptest-time \
+"
+
+SUMMARY = "glibc tests to be run with ptest"
+
+# Erase some variables already set by glibc_${PV}
+python __anonymous() {
+       # Remove packages provided by glibc build, we only need a

subset of them

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +79 -0
+       d.setVar("PACKAGES", "${PN}")
+
+       d.setVar("PROVIDES", "${PN}")
+       d.setVar("RPROVIDES", "${PN} glibc-ptest")
+
+       d.setVar("RRECOMMENDS", "")
+}
+
+# Remove any lefovers from original glibc recipe
+RPROVIDES:${PN} = "${PN}"
+RRECOMMENDS:${PN} = ""
+RDEPENDS:${PN} = "sed"
+
+# Just build tests for target - do not run them
+do_check:append () {
+   oe_runmake -i check run-built-tests=no
+}
+addtask do_check after do_compile before do_install_ptest_base
+
+glibc_strip_build_directory () {
+   # Delete all non executable files from build directory
+   find ${B} ! -executable -type f -delete
+
+   # Remove build dynamic libraries and links to them as
+   # those are already installed in the target device
+   find ${B} -type f -name "*.so" -delete
+   find ${B} -type l -name "*.so*" -delete
+
+   # Remove headers (installed with glibc)
+   find ${B} -type f -name "*.h" -delete
+
+   find ${B} -type f -name "isomac" -delete
+   find ${B} -type f -name "annexc" -delete
+}
+
+do_install_ptest_base () {
+   glibc_strip_build_directory
+
+   # Install build test programs to the image
+   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
+   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
+
+   install -d ${D}${PTEST_PATH}
+   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
+
+   # By default run all glibc tests
+   cd ${D}${PTEST_PATH}/
+   ln -s run-ptest-all run-ptest
+}
+
+# The datadir directory is required to allow core (and reused)
+# glibc cleanup function to finish correctly, as this directory
+# is not created for ptests
+stash_locale_package_cleanup:prepend () {
+   mkdir -p ${PKGD}${datadir}
+}
+
+stash_locale_sysroot_cleanup:prepend () {
+   mkdir -p ${SYSROOT_DESTDIR}${datadir}
+}
+
+# Prevent the do_package() task to set 'libc6' prefix
+# for glibc tests related packages
+python populate_packages:prepend () {
+    if d.getVar('DEBIAN_NAMES'):
+        d.setVar('DEBIAN_NAMES', '')
+}
+
+FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
+
+EXCLUDE_FROM_SHLIBS = "1"
+
+# Install debug data in .debug and sources in /usr/src/debug
+# It is more handy to have _all_ the sources and symbols in one
+# place (package) as this recipe will be used for validation and
+# debugging.
+PACKAGE_DEBUG_SPLIT_STYLE = "debug"
+
+# glibc test cases violate by default some Yocto/OE checks

(staticdev, +# textrel)

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +7 -0
+# 'debug-files' - add everything (including debug) into one package
+#                 (no need to install/build *-src package)
+INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths"
+
+deltask do_stash_locale
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode 100755 index 0000000000..47bfe5238c

meta/recipes-core/glibc/glibc/run-ptest-all new file +25 -0
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+   $i >/dev/null 2>&1
+   output
+done
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode 100755 index 0000000000..24dedc6e32

meta/recipes-core/glibc/glibc/run-ptest-time new file +27 -0
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
Quoted 8 lines

meta/recipes-core/glibc/glibc/run-ptest-time line 30

Quoted 8 lines
meta/recipes-core/glibc/glibc/run-ptest-time new file +3 -0
+   $i >/dev/null 2>&1
+   output
+done

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#5 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 48 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 36 lines · Lukasz Majewski

Hi Khem,

Quoted 27 lines · ?ukasz Majewski
Quoted 19 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to
call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 2 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

Quoted 5 lines

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

Then, glibc-tests_2.34.bb could be left as it is now, as inheriting ptest requires it to be added as 'glibc-tests-ptest' in meta/conf/distro/include/ptest-packagelists.inc

(I've tried to rename it to glibc-tests-ptest_2.34.bb, but then it was required to add glibc-tests-ptest-ptest to the ptest-packagelist.inc file, which is a bit clumsy IMHO).

Quoted 214 lines
Quoted 214 lines
Quoted 200 lines
Quoted 200 lines
---
  .../distro/include/ptest-packagelists.inc     |   1 +
  meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all
 | 25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30

++++++ 4 files changed, 157 insertions(+)
create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all create mode 100755
meta/recipes-core/glibc/glibc/run-ptest-time

(quoted diff) +0 -0

b/meta/conf/distro/include/ptest-packagelists.inc index 3872bdc942..e460ad1c8b 100644 --- a/meta/conf/distro/include/ptest-packagelists.inc +++ b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6

+59,7 @@ PTESTS_FAST = "\ slang-ptest \
      wayland-ptest \
      zlib-ptest \
meta/conf/distro/include/ptest-packagelists.inc +1 -0
+    glibc-tests-ptest \
  "
  PTESTS_FAST:remove:mips64 = "qemu-ptest"
  PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode 100644 index 0000000000..896809f700

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +14 -0
@@ -0,0 +1,101 @@
+require glibc_${PV}.bb
+require glibc-tests.inc
+
+inherit ptest
+
+SRC_URI:append = " \
+   file://run-ptest-all file://run-ptest-time \
+"
+
+SUMMARY = "glibc tests to be run with ptest"
+
+# Erase some variables already set by glibc_${PV}
+python __anonymous() {
+       # Remove packages provided by glibc build, we only need

a subset of them

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +75 -0
+       d.setVar("PACKAGES", "${PN}")
+
+       d.setVar("PROVIDES", "${PN}")
+       d.setVar("RPROVIDES", "${PN} glibc-ptest")
+
+       d.setVar("RRECOMMENDS", "")
+}
+
+# Remove any lefovers from original glibc recipe
+RPROVIDES:${PN} = "${PN}"
+RRECOMMENDS:${PN} = ""
+RDEPENDS:${PN} = "sed"
+
+# Just build tests for target - do not run them
+do_check:append () {
+   oe_runmake -i check run-built-tests=no
+}
+addtask do_check after do_compile before do_install_ptest_base
+
+glibc_strip_build_directory () {
+   # Delete all non executable files from build directory
+   find ${B} ! -executable -type f -delete
+
+   # Remove build dynamic libraries and links to them as
+   # those are already installed in the target device
+   find ${B} -type f -name "*.so" -delete
+   find ${B} -type l -name "*.so*" -delete
+
+   # Remove headers (installed with glibc)
+   find ${B} -type f -name "*.h" -delete
+
+   find ${B} -type f -name "isomac" -delete
+   find ${B} -type f -name "annexc" -delete
+}
+
+do_install_ptest_base () {
+   glibc_strip_build_directory
+
+   # Install build test programs to the image
+   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
+   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
+
+   install -d ${D}${PTEST_PATH}
+   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
+
+   # By default run all glibc tests
+   cd ${D}${PTEST_PATH}/
+   ln -s run-ptest-all run-ptest
+}
+
+# The datadir directory is required to allow core (and reused)
+# glibc cleanup function to finish correctly, as this directory
+# is not created for ptests
+stash_locale_package_cleanup:prepend () {
+   mkdir -p ${PKGD}${datadir}
+}
+
+stash_locale_sysroot_cleanup:prepend () {
+   mkdir -p ${SYSROOT_DESTDIR}${datadir}
+}
+
+# Prevent the do_package() task to set 'libc6' prefix
+# for glibc tests related packages
+python populate_packages:prepend () {
+    if d.getVar('DEBIAN_NAMES'):
+        d.setVar('DEBIAN_NAMES', '')
+}
+
+FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
+
+EXCLUDE_FROM_SHLIBS = "1"
+
+# Install debug data in .debug and sources in /usr/src/debug
+# It is more handy to have _all_ the sources and symbols in one
+# place (package) as this recipe will be used for validation

and +# debugging.

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +3 -0
+PACKAGE_DEBUG_SPLIT_STYLE = "debug"
+
+# glibc test cases violate by default some Yocto/OE checks

(staticdev, +# textrel)
+# 'debug-files' - add everything (including debug) into one
package +# (no need to install/build *-src
package) +INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths" +

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +3 -0
+deltask do_stash_locale
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode 100755 index 0000000000..47bfe5238c

meta/recipes-core/glibc/glibc/run-ptest-all new file +25 -0
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+   $i >/dev/null 2>&1
+   output
+done
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode 100755 index 0000000000..24dedc6e32

meta/recipes-core/glibc/glibc/run-ptest-time new file +27 -0
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
Quoted 8 lines

meta/recipes-core/glibc/glibc/run-ptest-time line 30

Quoted 8 lines
meta/recipes-core/glibc/glibc/run-ptest-time new file +3 -0
+   $i >/dev/null 2>&1
+   output
+done

Best regards,

Quoted 8 lines
Quoted 8 lines

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang
Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
[email protected]

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#6 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 52 lines · Lukasz Majewski
Quoted 48 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 44 lines · Lukasz Majewski

Hi Khem,

Quoted 27 lines · ?ukasz Majewski
Quoted 19 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to
call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 2 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

Quoted 8 lines
Quoted 5 lines

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

Quoted 244 lines

Then, glibc-tests_2.34.bb could be left as it is now, as inheriting ptest requires it to be added as 'glibc-tests-ptest' in meta/conf/distro/include/ptest-packagelists.inc

(I've tried to rename it to glibc-tests-ptest_2.34.bb, but then it was required to add glibc-tests-ptest-ptest to the ptest-packagelist.inc file, which is a bit clumsy IMHO).

Quoted 214 lines
Quoted 214 lines
Quoted 200 lines
Quoted 200 lines
---
  .../distro/include/ptest-packagelists.inc     |   1 +
  meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all
 | 25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30

++++++ 4 files changed, 157 insertions(+)
create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all create mode 100755
meta/recipes-core/glibc/glibc/run-ptest-time

(quoted diff) +0 -0

b/meta/conf/distro/include/ptest-packagelists.inc index 3872bdc942..e460ad1c8b 100644 --- a/meta/conf/distro/include/ptest-packagelists.inc +++ b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6

+59,7 @@ PTESTS_FAST = "\ slang-ptest \
      wayland-ptest \
      zlib-ptest \
meta/conf/distro/include/ptest-packagelists.inc +1 -0
+    glibc-tests-ptest \
  "
  PTESTS_FAST:remove:mips64 = "qemu-ptest"
  PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode 100644 index 0000000000..896809f700

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +14 -0
@@ -0,0 +1,101 @@
+require glibc_${PV}.bb
+require glibc-tests.inc
+
+inherit ptest
+
+SRC_URI:append = " \
+   file://run-ptest-all file://run-ptest-time \
+"
+
+SUMMARY = "glibc tests to be run with ptest"
+
+# Erase some variables already set by glibc_${PV}
+python __anonymous() {
+       # Remove packages provided by glibc build, we only need

a subset of them

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +75 -0
+       d.setVar("PACKAGES", "${PN}")
+
+       d.setVar("PROVIDES", "${PN}")
+       d.setVar("RPROVIDES", "${PN} glibc-ptest")
+
+       d.setVar("RRECOMMENDS", "")
+}
+
+# Remove any lefovers from original glibc recipe
+RPROVIDES:${PN} = "${PN}"
+RRECOMMENDS:${PN} = ""
+RDEPENDS:${PN} = "sed"
+
+# Just build tests for target - do not run them
+do_check:append () {
+   oe_runmake -i check run-built-tests=no
+}
+addtask do_check after do_compile before do_install_ptest_base
+
+glibc_strip_build_directory () {
+   # Delete all non executable files from build directory
+   find ${B} ! -executable -type f -delete
+
+   # Remove build dynamic libraries and links to them as
+   # those are already installed in the target device
+   find ${B} -type f -name "*.so" -delete
+   find ${B} -type l -name "*.so*" -delete
+
+   # Remove headers (installed with glibc)
+   find ${B} -type f -name "*.h" -delete
+
+   find ${B} -type f -name "isomac" -delete
+   find ${B} -type f -name "annexc" -delete
+}
+
+do_install_ptest_base () {
+   glibc_strip_build_directory
+
+   # Install build test programs to the image
+   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
+   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
+
+   install -d ${D}${PTEST_PATH}
+   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
+
+   # By default run all glibc tests
+   cd ${D}${PTEST_PATH}/
+   ln -s run-ptest-all run-ptest
+}
+
+# The datadir directory is required to allow core (and reused)
+# glibc cleanup function to finish correctly, as this directory
+# is not created for ptests
+stash_locale_package_cleanup:prepend () {
+   mkdir -p ${PKGD}${datadir}
+}
+
+stash_locale_sysroot_cleanup:prepend () {
+   mkdir -p ${SYSROOT_DESTDIR}${datadir}
+}
+
+# Prevent the do_package() task to set 'libc6' prefix
+# for glibc tests related packages
+python populate_packages:prepend () {
+    if d.getVar('DEBIAN_NAMES'):
+        d.setVar('DEBIAN_NAMES', '')
+}
+
+FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
+
+EXCLUDE_FROM_SHLIBS = "1"
+
+# Install debug data in .debug and sources in /usr/src/debug
+# It is more handy to have _all_ the sources and symbols in one
+# place (package) as this recipe will be used for validation

and +# debugging.

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +3 -0
+PACKAGE_DEBUG_SPLIT_STYLE = "debug"
+
+# glibc test cases violate by default some Yocto/OE checks

(staticdev, +# textrel)
+# 'debug-files' - add everything (including debug) into one
package +# (no need to install/build *-src
package) +INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths" +

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +3 -0
+deltask do_stash_locale
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode 100755 index 0000000000..47bfe5238c

meta/recipes-core/glibc/glibc/run-ptest-all new file +25 -0
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+   $i >/dev/null 2>&1
+   output
+done
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode 100755 index 0000000000..24dedc6e32

meta/recipes-core/glibc/glibc/run-ptest-time new file +27 -0
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
Quoted 8 lines

meta/recipes-core/glibc/glibc/run-ptest-time line 30

Quoted 8 lines
meta/recipes-core/glibc/glibc/run-ptest-time new file +3 -0
+   $i >/dev/null 2>&1
+   output
+done

Best regards,

Quoted 8 lines
Quoted 8 lines

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang
Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
[email protected]

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#7 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Hi Khem,

Quoted 61 lines · Lukasz Majewski
Quoted 60 lines
Quoted 55 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 38 lines · Lukasz Majewski

Hi Khem,

Quoted 29 lines · ?ukasz Majewski
Quoted 21 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 2 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

It is also easier to debug the code with the real HW.

Quoted 238 lines
Quoted 238 lines

Then, glibc-tests_2.34.bb could be left as it is now, as inheriting ptest requires it to be added as 'glibc-tests-ptest' in meta/conf/distro/include/ptest-packagelists.inc

(I've tried to rename it to glibc-tests-ptest_2.34.bb, but then it was required to add glibc-tests-ptest-ptest to the ptest-packagelist.inc file, which is a bit clumsy IMHO).

Quoted 214 lines
Quoted 214 lines
Quoted 200 lines
Quoted 200 lines
---
  .../distro/include/ptest-packagelists.inc     |   1 +
  meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
++++++++++++++++++
meta/recipes-core/glibc/glibc/run-ptest-all | 25 +++++
meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
4 files changed, 157 insertions(+) create mode 100644

meta/recipes-core/glibc/glibc-tests_2.34.bb create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time

(quoted diff) +0 -0

b/meta/conf/distro/include/ptest-packagelists.inc index 3872bdc942..e460ad1c8b 100644 --- a/meta/conf/distro/include/ptest-packagelists.inc +++ b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6

+59,7 @@ PTESTS_FAST = "\ slang-ptest \
      wayland-ptest \
      zlib-ptest \
meta/conf/distro/include/ptest-packagelists.inc +1 -0
+    glibc-tests-ptest \
  "
  PTESTS_FAST:remove:mips64 = "qemu-ptest"
  PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode 100644 index 0000000000..896809f700

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +14 -0
@@ -0,0 +1,101 @@
+require glibc_${PV}.bb
+require glibc-tests.inc
+
+inherit ptest
+
+SRC_URI:append = " \
+   file://run-ptest-all file://run-ptest-time \
+"
+
+SUMMARY = "glibc tests to be run with ptest"
+
+# Erase some variables already set by glibc_${PV}
+python __anonymous() {
+       # Remove packages provided by glibc build, we only

need a subset of them

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +18 -0
+       d.setVar("PACKAGES", "${PN}")
+
+       d.setVar("PROVIDES", "${PN}")
+       d.setVar("RPROVIDES", "${PN} glibc-ptest")
+
+       d.setVar("RRECOMMENDS", "")
+}
+
+# Remove any lefovers from original glibc recipe
+RPROVIDES:${PN} = "${PN}"
+RRECOMMENDS:${PN} = ""
+RDEPENDS:${PN} = "sed"
+
+# Just build tests for target - do not run them
+do_check:append () {
+   oe_runmake -i check run-built-tests=no
+}
+addtask do_check after do_compile before

do_install_ptest_base +

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +32 -0
+glibc_strip_build_directory () {
+   # Delete all non executable files from build directory
+   find ${B} ! -executable -type f -delete
+
+   # Remove build dynamic libraries and links to them as
+   # those are already installed in the target device
+   find ${B} -type f -name "*.so" -delete
+   find ${B} -type l -name "*.so*" -delete
+
+   # Remove headers (installed with glibc)
+   find ${B} -type f -name "*.h" -delete
+
+   find ${B} -type f -name "isomac" -delete
+   find ${B} -type f -name "annexc" -delete
+}
+
+do_install_ptest_base () {
+   glibc_strip_build_directory
+
+   # Install build test programs to the image
+   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
+   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
+
+   install -d ${D}${PTEST_PATH}
+   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
+
+   # By default run all glibc tests
+   cd ${D}${PTEST_PATH}/
+   ln -s run-ptest-all run-ptest
+}
+
+# The datadir directory is required to allow core (and

reused) +# glibc cleanup function to finish correctly, as this directory +# is not created for ptests

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +16 -0
+stash_locale_package_cleanup:prepend () {
+   mkdir -p ${PKGD}${datadir}
+}
+
+stash_locale_sysroot_cleanup:prepend () {
+   mkdir -p ${SYSROOT_DESTDIR}${datadir}
+}
+
+# Prevent the do_package() task to set 'libc6' prefix
+# for glibc tests related packages
+python populate_packages:prepend () {
+    if d.getVar('DEBIAN_NAMES'):
+        d.setVar('DEBIAN_NAMES', '')
+}
+
+FILES:${PN} = "${PTEST_PATH}/*

/usr/src/debug/glibc-tests/*" +

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +3 -0
+EXCLUDE_FROM_SHLIBS = "1"
+
+# Install debug data in .debug and sources in

/usr/src/debug +# It is more handy to have _all_ the sources and symbols in one +# place (package) as this recipe will be used for validation and +# debugging.

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +3 -0
+PACKAGE_DEBUG_SPLIT_STYLE = "debug"
+
+# glibc test cases violate by default some Yocto/OE checks

(staticdev, +# textrel)
+# 'debug-files' - add everything (including debug) into one
package +# (no need to install/build *-src
package) +INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths" +

meta/recipes-core/glibc/glibc-tests_2.34.bb new file +3 -0
+deltask do_stash_locale
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode 100755 index 0000000000..47bfe5238c

meta/recipes-core/glibc/glibc/run-ptest-all new file +25 -0
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+   $i >/dev/null 2>&1
+   output
+done
(quoted diff) +0 -0

b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode 100755 index 0000000000..24dedc6e32

meta/recipes-core/glibc/glibc/run-ptest-time new file +27 -0
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
Quoted 8 lines

meta/recipes-core/glibc/glibc/run-ptest-time line 30

Quoted 8 lines
meta/recipes-core/glibc/glibc/run-ptest-time new file +3 -0
+   $i >/dev/null 2>&1
+   output
+done

Best regards,

Quoted 8 lines
Quoted 8 lines

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang
Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-59 Fax:
(+49)-8142-66989-80 Email: [email protected]

Best regards,

Quoted 8 lines
Quoted 8 lines

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang
Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
[email protected]

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#8 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 71 lines · ?ukasz Majewski

Hi Khem,

Quoted 61 lines · Lukasz Majewski
Quoted 60 lines
Quoted 55 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 38 lines · Lukasz Majewski

Hi Khem,

Quoted 29 lines · ?ukasz Majewski
Quoted 21 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 2 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

Quoted 7 lines

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code. This was done around the time we added support for the other toolchain test suites (binutils+gcc). We ended up with options to use full system qemu or user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

Has this patchset been run with all the tests in glibc or just a subset?

I'm always very nervous about having two ways to do similar things.

Cheers,

Richard

#9 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Hi Richard,

Quoted 80 lines · ?ukasz Majewski
Quoted 79 lines

Hi Khem,

Quoted 63 lines
Quoted 61 lines · Lukasz Majewski
Quoted 56 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 39 lines · Lukasz Majewski

Hi Khem,

Quoted 29 lines · ?ukasz Majewski
Quoted 21 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 3 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 6 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or
user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS T_USER=root
SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 2 lines

I'm always very nervous about having two ways to do similar things.

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

- Using ptest framework to execute tests

- Time related tests are "self contained", so could be executed without built data.

At least for Y2038 validation such approach seems more appealing.

Quoted 5 lines

Cheers,

Richard

Links:
[1] - https://www.mail-archive.com/[email protected]/msg05898.html
[2] -
https://www.mail-archive.com/[email protected]/msg2361284.html

Note:
[*] - some glibc tests rely on the built directory to provide correct data/environment for execution. Time related tests are (and should be) self contained, so those could be just executed as binaries.

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#10 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Hi Richard, Khem,

Quoted 151 lines

Hi Richard,

Quoted 80 lines · ?ukasz Majewski
Quoted 79 lines

Hi Khem,

Quoted 63 lines
Quoted 61 lines · Lukasz Majewski
Quoted 56 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 39 lines · Lukasz Majewski

Hi Khem,

Quoted 29 lines · ?ukasz Majewski
Quoted 21 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 3 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 6 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or
user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS T_USER=root
SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 3 lines

I'm always very nervous about having two ways to do similar things.

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

Quoted 7 lines

- Using ptest framework to execute tests

- Time related tests are "self contained", so could be executed without built data.

At least for Y2038 validation such approach seems more appealing.

Do you have any more feedback regarding this patch?

Quoted 19 lines
Quoted 5 lines

Cheers,

Richard

Links:
[1] -
https://www.mail-archive.com/[email protected]/msg05898.html
[2] -
https://www.mail-archive.com/[email protected]/msg2361284.html

Note:
[*] - some glibc tests rely on the built directory to provide correct data/environment for execution. Time related tests are (and should be) self contained, so those could be just executed as binaries.

Best regards,

Quoted 8 lines

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
[email protected]

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#11 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 155 lines · Lukasz Majewski

Hi Richard, Khem,

Quoted 153 lines

Hi Richard,

Quoted 80 lines · ?ukasz Majewski
Quoted 79 lines

Hi Khem,

Quoted 63 lines
Quoted 61 lines · Lukasz Majewski
Quoted 56 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 39 lines · Lukasz Majewski

Hi Khem,

Quoted 29 lines · ?ukasz Majewski
Quoted 21 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 3 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 6 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or
user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS T_USER=root
SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 3 lines

I'm always very nervous about having two ways to do similar things.

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

- Using ptest framework to execute tests

Quoted 7 lines
Quoted 5 lines

- Time related tests are "self contained", so could be executed without built data.

At least for Y2038 validation such approach seems more appealing.

Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for this release cycle, I'm struggling with the things that were planned.

I need to spend some time looking at what the other code is doing and how we're using it on the autobuilder and whether we want to change to use the ptest approach or not.

I added Nathan Rossi to cc since he worked on the other code for this and may have more insight.

Cheers,

Richard

#12 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Hi Richard,

Quoted 168 lines · Lukasz Majewski
Quoted 167 lines

Hi Richard, Khem,

Quoted 165 lines

Hi Richard,

Quoted 89 lines · ?ukasz Majewski
Quoted 88 lines

Hi Khem,

Quoted 70 lines
Quoted 68 lines · Lukasz Majewski
Quoted 63 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 48 lines · Lukasz Majewski

Hi Khem,

Quoted 33 lines · ?ukasz Majewski
Quoted 24 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 6 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 3 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Quoted 5 lines

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 6 lines
Quoted 3 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 6 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS
T_USER=root SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 3 lines

I'm always very nervous about having two ways to do similar things.

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

- Using ptest framework to execute tests

- Time related tests are "self contained", so could be executed without built data.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

At least for Y2038 validation such approach seems more appealing.

Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for this release cycle,

I see. Thanks for the update.

BTW: When one can expect the Yocto/poky would accept new patches again?

Honister is going to be released on October 2021 [1]. After this date this patch is going to be reconsidered?

Quoted 2 lines

I'm struggling with the things that were
planned.

I can imagine that after the "override" change a lot of attention is necessary.

Quoted 4 lines

I need to spend some time looking at what the other code is doing and how we're using it on the autobuilder and whether we want to change to use the ptest approach or not.

I see.

Quoted 3 lines

I added Nathan Rossi to cc since he worked on the other code for this and may have more insight.

I do hope that Nathan will provide some valuable input as well, as wrote the glibc-testsuite_%.bb recipe.

Quoted 5 lines

Cheers,

Richard

Links:

[1] - https://wiki.yoctoproject.org/wiki/Releases

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#13 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 7 lines · Lukasz Majewski
Quoted 6 lines · Lukasz Majewski
Quoted 2 lines

Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for this release cycle,

Quoted 7 lines

I see. Thanks for the update.

BTW: When one can expect the Yocto/poky would accept new patches again?

Honister is going to be released on October 2021 [1]. After this date this patch is going to be reconsidered?

I'd expect new development to start being looked at around the time of the release, yes.

Quoted 11 lines
Quoted 2 lines

I'm struggling with the things that were
planned.

I can imagine that after the "override" change a lot of attention is necessary.

Quoted 4 lines

I need to spend some time looking at what the other code is doing and how we're using it on the autobuilder and whether we want to change to use the ptest approach or not.

Quoted 8 lines

I see.

Quoted 3 lines

I added Nathan Rossi to cc since he worked on the other code for this and may have more insight.

I do hope that Nathan will provide some valuable input as well, as wrote the glibc-testsuite_%.bb recipe.

If you search the commits, e.g.:

http://git.yoctoproject.org/cgit.cgi/poky/log/?qt=grep&q=Rossi

you'll see he also worked on the gcc/binutils tests I mentioned.

Also, I'm not sure if you found them but these are run on our infrastructure here:

https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3877/steps/19/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3892/steps/20/logs/stdio

(one of the above uses user mode qemu iirc, the other uses system mode).

We then get reports like:

https://autobuilder.yocto.io/pub/non-release/20210825-18/testresults/testresult-report.txt

Cheers,

Richard

#14 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

In reply to Richard Purdie
Quoted 125 lines · Richard Purdie
Quoted 123 lines · Lukasz Majewski

Hi Richard, Khem,

Quoted 121 lines

Hi Richard,

Quoted 80 lines · ?ukasz Majewski
Quoted 79 lines

Hi Khem,

Quoted 63 lines
Quoted 61 lines · Lukasz Majewski
Quoted 56 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 39 lines · Lukasz Majewski

Hi Khem,

Quoted 29 lines · ?ukasz Majewski
Quoted 21 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 5 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 3 lines

does it have to be a separate recipe I wonder if we can have it built as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Quoted 8 lines

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 6 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or
user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS T_USER=root
SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

So just a note, the qemu user mode testing is not a complete solution. There are a number of problems with it, but in general it is still quite useful for validating general functionally, especially when implementing a new platform. A big one is due to the ability to run tests in parallel with user mode, it easily reduces test run times by an order of magnitude (compared to qemu system, which for some platforms is faster than real hardware).

The reason why qemu user mode is the default behaviour is because running "bitbake glibc-testsuite -c check" will work without any external dependencies (the remote machine, NFS/sshfs, etc.).

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 3 lines

I'm always very nervous about having two ways to do similar things.

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

So it appears the make variable "run-built-tests" excludes the building of some tests. So I am not sure if the complete test suite can be built in this way. Though I can see some value in having the glibc-testsuite recipe being able to only build the tests for development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).

When I had implemented the testsuite recipe the glibc testsuite relied quite heavily on the environment it was built and run in, in particular around configuration. That is probably still an issue.

It would be interesting to know how much of the test suite can be run/passes and how that compares to the results of existing system/user emulation with cross execution using the glibc makefile.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

I had not tested with sshfs that likely has quite low performance, the OEQA tests rely on NFS for QEMU system emulation and that is what I had been using for hardware testing. However it would be interesting to see if virtiofs could improve the performance for testing with QEMU system emulation.

Quoted 20 lines
Quoted 9 lines
Quoted 9 lines

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

- Using ptest framework to execute tests

Quoted 7 lines
Quoted 5 lines

- Time related tests are "self contained", so could be executed without built data.

At least for Y2038 validation such approach seems more appealing.

Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for this release cycle, I'm struggling with the things that were planned.

Quoted 6 lines

I need to spend some time looking at what the other code is doing and how we're using it on the autobuilder and whether we want to change to use the ptest approach or not.

I added Nathan Rossi to cc since he worked on the other code for this and may have more insight.

Thanks for cc'ing, I somehow managed to miss the thread.

Regards,
Nathan

#15 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Hi Nathan,

Quoted 142 lines
Quoted 140 lines · Richard Purdie
Quoted 138 lines · Lukasz Majewski

Hi Richard, Khem,

Quoted 136 lines

Hi Richard,

Quoted 90 lines · ?ukasz Majewski
Quoted 89 lines

Hi Khem,

Quoted 71 lines
Quoted 69 lines · Lukasz Majewski
Quoted 64 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 49 lines · Lukasz Majewski

Hi Khem,

Quoted 33 lines · ?ukasz Majewski
Quoted 24 lines

This patch introduces new recipe - namely 'glibc-tests', which builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe, which is run with 'bitbake glibc-testsuite -c check' and uses qemu to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image. Afterwards, either all of them or only time related subset, those tests can be executed on the real hardware, to facilitate validation of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'. To test only time related subset - one needs to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

Quoted 6 lines

To facilitate debugging, source files are provided by default with the unstripped debugging symbols. Such approach would reduce the already complex recipe (as it inherits base glibc one), so there is no need to also install *-dbg and *-src packages.

Quoted 3 lines

does it have to be a separate recipe I wonder if
we can have it built as part of glibc itself
controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Quoted 5 lines

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 7 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS
T_USER=root SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

So just a note, the qemu user mode testing is not a complete solution. There are a number of problems with it, but in general it is still quite useful for validating general functionally, especially when implementing a new platform.

+1

Quoted 4 lines

A big one is due to the ability to run
tests in parallel with user mode, it easily reduces test run times by an order of magnitude (compared to qemu system, which for some platforms is faster than real hardware).

+1

Quoted 4 lines

The reason why qemu user mode is the default behaviour is because running "bitbake glibc-testsuite -c check" will work without any external dependencies (the remote machine, NFS/sshfs, etc.).

+1

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 3 lines

I'm always very nervous about having two ways to do similar things.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

So it appears the make variable "run-built-tests" excludes the building of some tests.

As fair as I remember, the 'run-built-tests=no' switch tells the glibc build system to not execute built tests.

Quoted 2 lines

So I am not sure if the complete test suite
can be built in this way.

I'm pretty sure that all tests required for 'check' are build.

Quoted 7 lines

Though I can see some value in having the glibc-testsuite recipe being able to only build the tests for development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).

When I had implemented the testsuite recipe the glibc testsuite relied quite heavily on the environment it was built and run in, in particular around configuration. That is probably still an issue.

Indeed, some (a few?) tests rely on the build environment in which they were build. I do personally, regard this as a some kind of a bug. I do believe, that when you build test for a particular version of glibc (2.34 in this case), one shall expect that it can be run only by linking the proper library.

This is the case for "time" related tests. On my test system with this patch applied, all of them are executed with no errors. Those are self contained.

Quoted 4 lines

It would be interesting to know how much of the test suite can be run/passes and how that compares to the results of existing system/user emulation with cross execution using the glibc makefile.

I can provide exact numbers when I finish porting more Y2038 related patches to newest glibc -master.

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

I had not tested with sshfs that likely has quite low performance, the OEQA tests rely on NFS for QEMU system emulation and that is what I had been using for hardware testing.

I had some issues with NFS - either I couldn't place some OE/Yocto build directory in it or glibc's build directory was not working with it correctly.

That was why I had to use the sshfs instead, which is _really_ slow and often breaks.

Quoted 29 lines

However it would be interesting
to see if virtiofs could improve the performance for testing with QEMU system emulation.

Quoted 25 lines
Quoted 9 lines
Quoted 9 lines

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

- Using ptest framework to execute tests

Quoted 7 lines
Quoted 5 lines

- Time related tests are "self contained", so could be executed without built data.

At least for Y2038 validation such approach seems more appealing.

Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for this release cycle, I'm struggling with the things that were planned.

I need to spend some time looking at what the other code is doing and how we're using it on the autobuilder and whether we want to change to use the ptest approach or not.

Quoted 7 lines
Quoted 2 lines

I added Nathan Rossi to cc since he worked on the other code for this and may have more insight.

Thanks for cc'ing, I somehow managed to miss the thread.

Regards,
Nathan

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#16 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Quoted 187 lines · Lukasz Majewski

Hi Nathan,

Quoted 142 lines
Quoted 140 lines · Richard Purdie
Quoted 138 lines · Lukasz Majewski

Hi Richard, Khem,

Quoted 136 lines

Hi Richard,

Quoted 90 lines · ?ukasz Majewski
Quoted 89 lines

Hi Khem,

Quoted 71 lines
Quoted 69 lines · Lukasz Majewski
Quoted 64 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 49 lines · Lukasz Majewski

Hi Khem,

Quoted 33 lines · ?ukasz Majewski
Quoted 30 lines

This patch introduces new recipe - namely
'glibc-tests', which builds and installs glibc
test suite to OE/Yocto built image.

It reuses code from already available
'glibc-testsuite' recipe, which is run with
'bitbake glibc-testsuite -c check' and uses
qemu to execute remotely (via SSH) tests on
some emulated machine.

This recipe installs eligible tests on some
rootfs image. Afterwards, either all of them or
only time related subset, those tests can be
executed on the real hardware, to facilitate
validation of this platform with for example
Y2038 problem compliance.

By default all tests are executed, with
'ptest-runner glibc-tests'. To test only time
related subset - one needs to call: cd
/usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln
-s run-ptest-time run-ptest; cd -; ptest-runner
glibc-tests

To facilitate debugging, source files are
provided by default with the unstripped
debugging symbols. Such approach would reduce
the already complex recipe (as it inherits base
glibc one), so there is no need to also install
*-dbg and *-src packages.

Quoted 3 lines

does it have to be a separate recipe I wonder if
we can have it built as part of glibc itself
controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Quoted 5 lines

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 5 lines
Quoted 2 lines

I think that glibc-testsuite_2.34.bb shall be renamed to glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 7 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS
T_USER=root SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

So just a note, the qemu user mode testing is not a complete solution. There are a number of problems with it, but in general it is still quite useful for validating general functionally, especially when implementing a new platform.

+1

Quoted 4 lines

A big one is due to the ability to run
tests in parallel with user mode, it easily reduces test run times by an order of magnitude (compared to qemu system, which for some platforms is faster than real hardware).

+1

Quoted 4 lines

The reason why qemu user mode is the default behaviour is because running "bitbake glibc-testsuite -c check" will work without any external dependencies (the remote machine, NFS/sshfs, etc.).

+1

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 3 lines

I'm always very nervous about having two ways to do similar things.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

So it appears the make variable "run-built-tests" excludes the building of some tests.

Quoted 7 lines

As fair as I remember, the 'run-built-tests=no' switch tells the glibc build system to not execute built tests.

Quoted 2 lines

So I am not sure if the complete test suite
can be built in this way.

I'm pretty sure that all tests required for 'check' are build.

If you look at some of the Makefiles in glibc, it conditionally includes/excludes some tests from being built based on the value of run-built-tests. For example:

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l141

Quoted 20 lines
Quoted 8 lines

Though I can see some value in having the glibc-testsuite recipe being able to only build the tests for development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).

When I had implemented the testsuite recipe the glibc testsuite relied quite heavily on the environment it was built and run in, in particular around configuration. That is probably still an issue.

Indeed, some (a few?) tests rely on the build environment in which they were build. I do personally, regard this as a some kind of a bug. I do believe, that when you build test for a particular version of glibc (2.34 in this case), one shall expect that it can be run only by linking the proper library.

This is the case for "time" related tests. On my test system with this patch applied, all of them are executed with no errors. Those are self contained.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

It would be interesting to know how much of the test suite can be run/passes and how that compares to the results of existing system/user emulation with cross execution using the glibc makefile.

I can provide exact numbers when I finish porting more Y2038 related patches to newest glibc -master.

Also something to keep in mind is that sometimes the pass/fail of the test binary is not what is being validated by the test. For example this test uses mtrace, and compares the results of the output mtrace file, not the test itself.

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l462

Regards,
Nathan

Quoted 66 lines
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

I had not tested with sshfs that likely has quite low performance, the OEQA tests rely on NFS for QEMU system emulation and that is what I had been using for hardware testing.

I had some issues with NFS - either I couldn't place some OE/Yocto build directory in it or glibc's build directory was not working with it correctly.

That was why I had to use the sshfs instead, which is _really_ slow and often breaks.

Quoted 29 lines

However it would be interesting
to see if virtiofs could improve the performance for testing with QEMU system emulation.

Quoted 25 lines
Quoted 9 lines
Quoted 9 lines

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

- Using ptest framework to execute tests

Quoted 7 lines
Quoted 5 lines

- Time related tests are "self contained", so could be executed without built data.

At least for Y2038 validation such approach seems more appealing.

Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for this release cycle, I'm struggling with the things that were planned.

I need to spend some time looking at what the other code is doing and how we're using it on the autobuilder and whether we want to change to use the ptest approach or not.

Quoted 7 lines
Quoted 2 lines

I added Nathan Rossi to cc since he worked on the other code for this and may have more insight.

Thanks for cc'ing, I somehow managed to miss the thread.

Regards,
Nathan

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

#17 lore raw

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

Hi Nathan,

Quoted 199 lines · Lukasz Majewski
Quoted 198 lines

Hi Nathan,

Quoted 147 lines
Quoted 145 lines · Richard Purdie
Quoted 143 lines · Lukasz Majewski

Hi Richard, Khem,

Quoted 141 lines

Hi Richard,

Quoted 95 lines · ?ukasz Majewski
Quoted 94 lines

Hi Khem,

Quoted 73 lines
Quoted 71 lines · Lukasz Majewski
Quoted 66 lines · On Mon, 23 Aug 2021 12:52:44 -0700 Khem Raj <[email protected]> wrote:
Quoted 50 lines · Lukasz Majewski

Hi Khem,

Quoted 37 lines · ?ukasz Majewski

This patch introduces new recipe - namely
'glibc-tests', which builds and installs
glibc test suite to OE/Yocto built image.

It reuses code from already available
'glibc-testsuite' recipe, which is run with
'bitbake glibc-testsuite -c check' and uses
qemu to execute remotely (via SSH) tests on
some emulated machine.

This recipe installs eligible tests on some
rootfs image. Afterwards, either all of
them or only time related subset, those
tests can be executed on the real hardware,
to facilitate validation of this platform
with for example Y2038 problem compliance.

By default all tests are executed, with
'ptest-runner glibc-tests'. To test only
time related subset - one needs to call: cd
/usr/lib/glibc-tests/ptest/; rm run-ptest;
\ ln -s run-ptest-time run-ptest; cd -;
ptest-runner glibc-tests

To facilitate debugging, source files are
provided by default with the unstripped
debugging symbols. Such approach would
reduce the already complex recipe (as it
inherits base glibc one), so there is no
need to also install *-dbg and *-src
packages.

does it have to be a separate recipe I wonder
if we can have it built as part of glibc
itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest. Test creation is similar to it, but doesn't require QEMU run (tests are executed on the HW).

Another rationale was to have some kind of "features" separation in different file (liked glibc-mtrace), which is easier to maintain and review.

Quoted 5 lines

Last but not least - separate recipe (and built binaries) allow some kind of magic with selection of used test programs (this may be useful if one would like to have different sets of tests in different packages)

That’s seems ok I think the names are quite confusing now not that it was not so much better before but now we have glibc-tests and glibc-testsuites which do same things but in very different way maybe glibc-testsuite should be renamed to something like glibc-tests-crosd or some such

Quoted 6 lines
Quoted 3 lines

I think that glibc-testsuite_2.34.bb shall be
renamed to glibc-tests-qemu_2.34.bb as it is more
descriptive.

is it only runnable in qemu ? I thought we could use a read board as well with IP address

It looks like by default the QEMU is used in this
recipe.

This recipe provides custom check-test-wrapper, which can log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built tests on target (exact) HW without the need of SSH.

Quoted 7 lines
Quoted 1 line

It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a different form of tests a while back but I was told those didn't work well and they were replaced with the current setup which worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

Quoted 3 lines

This was done around
the time we added support for the other toolchain test suites (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the past. Could you share any reference?

Quoted 3 lines

We ended up with options to use full system qemu or user mode qemu emulation depending on the speed of the target (e.g. with kvm or not).

From what I see in the recipe - you either run user mode QEMU by default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS
T_USER=root SSH_HOST_PORT=2222 test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper user check

Or you log to the board via SSH (the 'ssh' mode).

Quoted 3 lines

Has this patchset been run with all the tests in glibc or just a subset?

This patch set allows running all tests [*], or (preferably in my use case) only a time related ones. The latter is to validate Y2038 code, which testing has two issues:

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

1. It cannot be tested with QEMU user mode [1]. Instead "linux timespaces" were suggested, but it lacks support for CLOCK_REALTIME manipulation for those tests [2].

So just a note, the qemu user mode testing is not a complete solution. There are a number of problems with it, but in general it is still quite useful for validating general functionally, especially when implementing a new platform.

+1

Quoted 4 lines

A big one is due to the ability to run
tests in parallel with user mode, it easily reduces test run times by an order of magnitude (compared to qemu system, which for some platforms is faster than real hardware).

+1

Quoted 4 lines

The reason why qemu user mode is the default behaviour is because running "bitbake glibc-testsuite -c check" will work without any external dependencies (the remote machine, NFS/sshfs, etc.).

+1

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

2. Testing with SSH is possible, but is not as robust as expected (i.e. imposes SSHD on the target board, and ETH connection).

Quoted 3 lines

I'm always very nervous about having two ways to do similar things.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Technically, I just reuse the glibc-testsuite_2.34.bb to the point where tests are _only_ built. Then I pack those binaries and install on the target's rootfs to be run with ptest framework.

So it appears the make variable "run-built-tests" excludes the building of some tests.

As fair as I remember, the 'run-built-tests=no' switch tells the glibc build system to not execute built tests.

Quoted 2 lines

So I am not sure if the complete test suite
can be built in this way.

Quoted 8 lines
Quoted 1 line

I'm pretty sure that all tests required for 'check' are build.

If you look at some of the Makefiles in glibc, it conditionally includes/excludes some tests from being built based on the value of run-built-tests. For example:

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l141

I see. Thanks for pointing this out.

Quoted 29 lines
Quoted 21 lines
Quoted 9 lines

Though I can see some value in having the glibc-testsuite recipe being able to only build the tests for development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).

When I had implemented the testsuite recipe the glibc testsuite relied quite heavily on the environment it was built and run in, in particular around configuration. That is probably still an issue.

Indeed, some (a few?) tests rely on the build environment in which they were build. I do personally, regard this as a some kind of a bug. I do believe, that when you build test for a particular version of glibc (2.34 in this case), one shall expect that it can be run only by linking the proper library.

This is the case for "time" related tests. On my test system with this patch applied, all of them are executed with no errors. Those are self contained.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

It would be interesting to know how much of the test suite can be run/passes and how that compares to the results of existing system/user emulation with cross execution using the glibc makefile.

I can provide exact numbers when I finish porting more Y2038 related patches to newest glibc -master.

Quoted 7 lines

Also something to keep in mind is that sometimes the pass/fail of the test binary is not what is being validated by the test. For example this test uses mtrace, and compares the results of the output mtrace file, not the test itself.

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l462

Ok. I see.

My main objective is to use time related programs, which rely on the interfaces provided by the kernel.

From the above POV, those tests are a "standalone" ones.

Quoted 70 lines

Regards,
Nathan

Quoted 67 lines
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for the setup)

I had not tested with sshfs that likely has quite low performance, the OEQA tests rely on NFS for QEMU system emulation and that is what I had been using for hardware testing.

I had some issues with NFS - either I couldn't place some OE/Yocto build directory in it or glibc's build directory was not working with it correctly.

That was why I had to use the sshfs instead, which is _really_ slow and often breaks.

Quoted 29 lines

However it would be interesting
to see if virtiofs could improve the performance for testing with QEMU system emulation.

Quoted 25 lines
Quoted 9 lines
Quoted 9 lines

- Run on the real HW (and environment) with the same version of glibc on the real target.

- Sources and debugging symbols available on the target, so easier GDB debugging.

- Using ptest framework to execute tests

Quoted 7 lines
Quoted 5 lines

- Time related tests are "self contained", so could be executed without built data.

At least for Y2038 validation such approach seems more appealing.

Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for this release cycle, I'm struggling with the things that were planned.

I need to spend some time looking at what the other code is doing and how we're using it on the autobuilder and whether we want to change to use the ptest approach or not.

Quoted 7 lines
Quoted 2 lines

I added Nathan Rossi to cc since he worked on the other code for this and may have more insight.

Thanks for cc'ing, I somehow managed to miss the thread.

Regards,
Nathan

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang
Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
[email protected]

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]