RFC: Package testing
cover RFC: Package testing
Hi.
This is a first set of patches implementing the package testing concept I proposed in
https://lists.yoctoproject.org/pipermail/yocto/2012-June/009443.html
In summary, the proposal creates a new package group called -ptest that contains the test suites included with many source packages, compiled to run on target.
Packaging the test suites sometimes requires patching makefiles to allow test tools to be built on host but ran on target. And it sometimes requires patching and/or translation of the test output to produce a generic output format that can be automatically parsed. I have included an example patch for bash that does just that.
I have also included "ptest-runnner", a simple script to find and run all installed ptests.
Björn Stenberg (3):
Add -ptest package group
New recipe: ptest-runner
Enabled bash-ptest
meta/classes/distutils-common-base.bbclass | 5 +++- meta/classes/image.bbclass | 6 +++- meta/classes/package.bbclass | 8 ++++- meta/classes/task.bbclass | 2 +- meta/conf/bitbake.conf | 14 ++++++++- .../bash/bash-4.2/build-tests.patch | 29 ++++++++++++++++++++ meta/recipes-extended/bash/bash-4.2/run-ptest | 2 + .../bash/bash-4.2/test-output.patch | 19 +++++++++++++ meta/recipes-extended/bash/bash.inc | 18 ++++++++++++ meta/recipes-extended/bash/bash_4.2.bb | 5 +++- .../ptest-runner/files/ptest-runner | 16 +++++++++++ .../ptest-runner/ptest-runner_1.0.bb | 21 ++++++++++++++ 12 files changed, 136 insertions(+), 9 deletions(-) create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patch create mode 100644 meta/recipes-support/ptest-runner/files/ptest-runner create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_1.0.bb
Signature · 1 line
1.7.5.4
PATCH 1/3 Add -ptest package group
This patch creates a new package group -ptest to contain the tests for each package.
--- meta/classes/distutils-common-base.bbclass | 5 ++++- meta/classes/image.bbclass | 6 ++++-- meta/classes/package.bbclass | 8 ++++++-- meta/classes/task.bbclass | 2 +- meta/conf/bitbake.conf | 14 ++++++++++++-- 5 files changed, 27 insertions(+), 8 deletions(-)
meta/classes/distutils-common-base.bbclass +4 -1
@@ -5,7 +5,7 @@ EXTRA_OEMAKE = "" export STAGING_INCDIR export STAGING_LIBDIR -PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-doc ${PN}" +PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-ptest ${PN}-doc ${PN}" FILES_${PN} = "${bindir}/* ${libdir}/* ${libdir}/${PYTHON_DIR}/*"
@@ -19,3 +19,6 @@ FILES_${PN}-dbg += "\ ${PYTHON_SITEPACKAGES_DIR}/*/.debug \ ${PYTHON_SITEPACKAGES_DIR}/*/*/.debug \ " +FILES_${PN}-ptest += "\ + ${PYTHON_SITEPACKAGES_DIR}/${PTEST_NAME} \ +"
meta/classes/image.bbclass +4 -2
@@ -29,13 +29,13 @@ ROOTFS_BOOTSTRAP_INSTALL = "${@base_contains("IMAGE_FEATURES", "package-manageme FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}" FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}" -# packages to install from features, excluding dev/dbg/doc +# packages to install from features, excluding dev/dbg/doc/ptest NORMAL_FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(normal_groups(d), d))}" NORMAL_FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(normal_groups(d), d))}" def normal_groups(d): """Return all the IMAGE_FEATURES, with the exception of our special package groups""" - extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs']) + extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs', 'ptest-pkgs']) features = set(oe.data.typed_value('IMAGE_FEATURES', d)) return features.difference(extras)
@@ -53,6 +53,8 @@ def complementary_globs(featurevar, d): globs.append('*-doc') elif feature == 'dbg-pkgs': globs.append('*-dbg') + elif feature == 'ptest-pkgs': + globs.append('*-ptest') return ' '.join(globs) IMAGE_INSTALL_COMPLEMENTARY = '${@complementary_globs("IMAGE_FEATURES", d)}'
meta/classes/package.bbclass +6 -2
@@ -1236,7 +1236,7 @@ python package_do_filedeps() { # Determine dependencies for pkg in packages.split(): - if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-'): + if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-') or pkg.endswith('-ptest'): continue provides_files = []
@@ -1330,7 +1330,7 @@ python package_do_shlibs() { combos.append("-".join(options[0:i])) return combos - if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg'): + if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg') and not pkg.endswith('-ptest'): # Drop suffix name = file.rsplit(".",1)[0] # Find all combinations
@@ -1648,6 +1648,8 @@ python package_depchains() { depend = depend.replace('-dev', '') if depend.endswith('-dbg'): depend = depend.replace('-dbg', '') + if depend.endswith('-ptest'): + depend = depend.replace('-ptest', '') pkgname = getname(depend, suffix) #bb.note("Adding %s for %s" % (pkgname, depend)) if pkgname not in rreclist and pkgname != pkg:
@@ -1669,6 +1671,8 @@ python package_depchains() { depend = depend.replace('-dev', '') if depend.endswith('-dbg'): depend = depend.replace('-dbg', '') + if depend.endswith('-ptest'): + depend = depend.replace('-ptest', '') pkgname = getname(depend, suffix) #bb.note("Adding %s for %s" % (pkgname, depend)) if pkgname not in rreclist and pkgname != pkg:
meta/classes/task.bbclass +1 -1
@@ -20,7 +20,7 @@ python () { packages = d.getVar('PACKAGES', True).split() genpackages = [] for pkg in packages: - for postfix in ['-dbg', '-dev']: + for postfix in ['-dbg', '-ptest', '-dev']: genpackages.append(pkg+postfix) d.setVar('PACKAGES', ' '.join(packages+genpackages)) }
meta/conf/bitbake.conf +12 -2
@@ -220,6 +220,10 @@ SUMMARY_${PN}-dbg ?= "${SUMMARY} - Debugging files" DESCRIPTION_${PN}-dbg ?= "${DESCRIPTION} \ This package contains ELF symbols and related sources for debugging purposes." +SUMMARY_${PN}-ptest ?= "${SUMMARY} - Package test files" +DESCRIPTION_${PN}-ptest ?= "${DESCRIPTION} \ +This package contains test directory with the name ${PTEST_NAME} for package test purposes." + SUMMARY_${PN}-dev ?= "${SUMMARY} - Development files" DESCRIPTION_${PN}-dev ?= "${DESCRIPTION} \ This package contains symbolic links, header files, and \
@@ -242,7 +246,7 @@ HOMEPAGE = "unknown" # Ensure that -dev packages recommend the corresponding -dev packages of their # deps, and the same for -dbg. DEPCHAIN_PRE = "" -DEPCHAIN_POST = "-dev -dbg" +DEPCHAIN_POST = "-dev -dbg -ptest" DEPENDS = "" RDEPENDS = ""
@@ -263,7 +267,7 @@ SOLIBSDEV_darwin8 = ".dylib" SOLIBSDEV_darwin9 = ".dylib" PACKAGE_BEFORE_PN ?= "" -PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}" +PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PN}-ptest ${PACKAGE_BEFORE_PN} ${PN}" PACKAGES_DYNAMIC = "${PN}-locale-*" FILES = ""
@@ -307,6 +311,12 @@ SECTION_${PN}-dbg = "devel" ALLOW_EMPTY_${PN}-dbg = "1" RRECOMMENDS_${PN}-dbg = "${PN} (= ${EXTENDPKGV})" +PTEST_PATH ?= "${libdir}/${PN}/ptest" +FILES_${PN}-ptest = "${PTEST_PATH}/*" +SECTION_${PN}-ptest = "devel" +ALLOW_EMPTY_${PN}-ptest = "1" +RDEPENDS_${PN}-ptest = "${PN} (= ${EXTENDPKGV})" + FILES_${PN}-locale = "${datadir}/locale" # File manifest
Signature · 1 line
1.7.5.4
Re: [PATCH 1/3] Add -ptest package group
meta/conf/bitbake.conf HOMEPAGE = "unknown" · line 249
Quoted 9 lines · On Thursday 23 August 2012 16:20:00 Björn Stenberg wrote:
This patch creates a new package group -ptest to contain the tests for each package.
...(quoted diff) +0
-0@@ -242,7 +246,7 @@ HOMEPAGE = "unknown" # Ensure that -dev packages recommend the corresponding -dev packages oftheir # deps, and the same for -dbg.
DEPCHAIN_PRE= ""meta/conf/bitbake.conf +1
-1-DEPCHAIN_POST = "-dev -dbg" +DEPCHAIN_POST = "-dev -dbg -ptest"
Is this really what you want? Surely the tests for a particular piece of software are mostly independent of those of its dependencies?
Rest of it looks good.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [PATCH 1/3] Add -ptest package group
Quoted 4 lines · Paul Eggleton
Quoted 1 line
+DEPCHAIN_POST = "-dev -dbg -ptest"
Is this really what you want? Surely the tests for a particular piece of software are mostly independent of those of its dependencies?
You're right, that change is not necessary. I misunderstood the purpose of DEPCHAIN_POST.
Signature · 1 line
Björn
Re: [PATCH 1/3] Add -ptest package group
In reply to Björn StenbergQuoted 65 lines · Björn Stenberg
This patch creates a new package group -ptest to contain the tests for each package.
--- meta/classes/distutils-common-base.bbclass | 5 ++++- meta/classes/image.bbclass | 6 ++++-- meta/classes/package.bbclass | 8 ++++++-- meta/classes/task.bbclass | 2 +- meta/conf/bitbake.conf | 14 ++++++++++++-- 5 files changed, 27 insertions(+), 8 deletions(-)meta/classes/distutils-common-base.bbclass +4
-1@@ -5,7 +5,7 @@ EXTRA_OEMAKE = "" export STAGING_INCDIR export STAGING_LIBDIR -PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-doc ${PN}" +PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-ptest ${PN}-doc ${PN}" FILES_${PN} = "${bindir}/* ${libdir}/* ${libdir}/${PYTHON_DIR}/*"@@ -19,3 +19,6 @@ FILES_${PN}-dbg += "\ ${PYTHON_SITEPACKAGES_DIR}/*/.debug \ ${PYTHON_SITEPACKAGES_DIR}/*/*/.debug \ " +FILES_${PN}-ptest += "\ + ${PYTHON_SITEPACKAGES_DIR}/${PTEST_NAME} \ +"meta/classes/image.bbclass +4
-2@@ -29,13 +29,13 @@ ROOTFS_BOOTSTRAP_INSTALL = "${@base_contains("IMAGE_FEATURES", "package-manageme FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}" FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}" -# packages to install from features, excluding dev/dbg/doc +# packages to install from features, excluding dev/dbg/doc/ptest NORMAL_FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(normal_groups(d), d))}" NORMAL_FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(normal_groups(d), d))}" def normal_groups(d): """Return all the IMAGE_FEATURES, with the exception of our special package groups""" - extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs']) + extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs', 'ptest-pkgs']) features = set(oe.data.typed_value('IMAGE_FEATURES', d)) return features.difference(extras)@@ -53,6 +53,8 @@ def complementary_globs(featurevar, d): globs.append('*-doc') elif feature == 'dbg-pkgs': globs.append('*-dbg') + elif feature == 'ptest-pkgs': + globs.append('*-ptest') return ' '.join(globs) IMAGE_INSTALL_COMPLEMENTARY = '${@complementary_globs("IMAGE_FEATURES", d)}'meta/classes/package.bbclass +0
-0
meta/classes/package.bbclass python package_do_filedeps() { · line 1240
Quoted 6 lines
meta/classes/package.bbclass +1
-1@@ -1236,7 +1236,7 @@ python package_do_filedeps() { # Determine dependencies for pkg in packages.split(): - if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-'): + if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-') or pkg.endswith('-ptest'): continue
I think filedeps could be useful in this case, autodetecting any dependencies scripts in the -ptest package could have for example so I'd not do the above.
meta/classes/package.bbclass python package_do_shlibs() { · line 1333
Quoted 8 lines
meta/classes/package.bbclass +1
-1provides_files = []@@ -1330,7 +1330,7 @@ python package_do_shlibs() { combos.append("-".join(options[0:i])) return combos - if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg'): + if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg') and not pkg.endswith('-ptest'):
ditto, we want to detect shlib dependencies.
Quoted 13 lines
meta/classes/package.bbclass +2
-0# Drop suffix name = file.rsplit(".",1)[0] # Find all combinations@@ -1648,6 +1648,8 @@ python package_depchains() { depend = depend.replace('-dev', '') if depend.endswith('-dbg'): depend = depend.replace('-dbg', '') + if depend.endswith('-ptest'): + depend = depend.replace('-ptest', '') pkgname = getname(depend, suffix) #bb.note("Adding %s for %s" % (pkgname, depend)) if pkgname not in rreclist and pkgname != pkg:
meta/classes/package.bbclass python package_depchains() { · line 1678
Quoted 8 lines
meta/classes/package.bbclass +2
-0@@ -1669,6 +1671,8 @@ python package_depchains() { depend = depend.replace('-dev', '') if depend.endswith('-dbg'): depend = depend.replace('-dbg', '') + if depend.endswith('-ptest'): + depend = depend.replace('-ptest', '') pkgname = getname(depend, suffix) #bb.note("Adding %s for %s" % (pkgname, depend)) if pkgname not in rreclist and pkgname != pkg:
I also suspect the above two hunks are not needed since ptest won't use depchains.
Quoted 49 lines
meta/classes/task.bbclass +1
-1@@ -20,7 +20,7 @@ python () { packages = d.getVar('PACKAGES', True).split() genpackages = [] for pkg in packages: - for postfix in ['-dbg', '-dev']: + for postfix in ['-dbg', '-ptest', '-dev']: genpackages.append(pkg+postfix) d.setVar('PACKAGES', ' '.join(packages+genpackages)) }meta/conf/bitbake.conf +6
-2@@ -220,6 +220,10 @@ SUMMARY_${PN}-dbg ?= "${SUMMARY} - Debugging files" DESCRIPTION_${PN}-dbg ?= "${DESCRIPTION} \ This package contains ELF symbols and related sources for debugging purposes." +SUMMARY_${PN}-ptest ?= "${SUMMARY} - Package test files" +DESCRIPTION_${PN}-ptest ?= "${DESCRIPTION} \ +This package contains test directory with the name ${PTEST_NAME} for package test purposes." + SUMMARY_${PN}-dev ?= "${SUMMARY} - Development files" DESCRIPTION_${PN}-dev ?= "${DESCRIPTION} \ This package contains symbolic links, header files, and \@@ -242,7 +246,7 @@ HOMEPAGE = "unknown" # Ensure that -dev packages recommend the corresponding -dev packages of their # deps, and the same for -dbg. DEPCHAIN_PRE = "" -DEPCHAIN_POST = "-dev -dbg" +DEPCHAIN_POST = "-dev -dbg -ptest" DEPENDS = "" RDEPENDS = ""@@ -263,7 +267,7 @@ SOLIBSDEV_darwin8 = ".dylib" SOLIBSDEV_darwin9 = ".dylib" PACKAGE_BEFORE_PN ?= "" -PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}" +PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PN}-ptest ${PACKAGE_BEFORE_PN} ${PN}" PACKAGES_DYNAMIC = "${PN}-locale-*" FILES = ""@@ -307,6 +311,12 @@ SECTION_${PN}-dbg = "devel" ALLOW_EMPTY_${PN}-dbg = "1" RRECOMMENDS_${PN}-dbg = "${PN} (= ${EXTENDPKGV})"
meta/conf/bitbake.conf SECTION_${PN}-dbg = "devel" · line 322
Quoted 10 lines
meta/conf/bitbake.conf +6
-0+PTEST_PATH ?= "${libdir}/${PN}/ptest" +FILES_${PN}-ptest = "${PTEST_PATH}/*" +SECTION_${PN}-ptest = "devel" +ALLOW_EMPTY_${PN}-ptest = "1" +RDEPENDS_${PN}-ptest = "${PN} (= ${EXTENDPKGV})" + FILES_${PN}-locale = "${datadir}/locale" # File manifest
Would a ptest.bbclass containing the above make more sense? How widespread do we think these tests will be?
Also these patches are against OE-Core so need to go to the openembedded-code mailing list.
I do like the idea of better testing of packages and this is a good start though. My biggest worry is the overhead of the patches to software to enable the tests. Are you planning to engage the upstreams and see if we can get them to accept the patches? I have to be very cautious of anything which would "strangle" us with patch overhead/overload and stop us being unable to update recipes.
Cheers,
Richard
Re: [PATCH 1/3] Add -ptest package group
Quoted 2 lines · Richard Purdie
Would a ptest.bbclass containing the above make more sense? How widespread do we think these tests will be?
Most source packages include some form of test suite that we want to make into a -ptest package. I would guesstimate the number to be somewhere between the number of -dbg and the number of -dev packages. This is why I felt it made sense to add the handling of -ptest side-by-side with the handling of -dev and -dbg.
Quoted 3 lines
My biggest worry is the overhead of the patches to
software to enable the tests. Are you planning to engage the upstreams and see if we can get them to accept the patches?
Upstreaming is absolutely the idea, both for the benefit of the entire ecosystem and of course to reduce the number of patches.
However I won't downplay the effort required to get hundreds of upstream projects to agree on a common concept for test cross compilation and result formatting.
I see two main options: Implement first or discuss first.
a) Implement first: We design and implement a good system for yocto/oe and then, once it has proven itself, engage the larger open source community with our already tested concept.
Pro: It's easier to discuss an already working system
Con: We'll have to manage patches for all packages until agreement
b) Discuss first: We invite the whole open source community to discuss and try to agree on how to handle cross-compiled test cases and test result formatting before we implement anything.
Pro: No increased patch load until agreement
Con: No result until agreement, which could potentially take a long time
My suggestion is a). I'm open for any and all suggestions that would make the patch load easier to manage.
Quoted 2 lines
Also these patches are against OE-Core so need to go to the openembedded-code mailing list.
I'll update the patches with the feedback I've got and post them to oe-core for further discussion.
Thanks.
Signature · 1 line
Björn
PATCH 2/3 New recipe: ptest-runner
In reply to Björn StenbergThis package contains the top-level script that seeks out and starts all package tests.
--- .../ptest-runner/files/ptest-runner | 16 +++++++++++++++ .../ptest-runner/ptest-runner_1.0.bb | 21 ++++++++++++++++++++ 2 files changed, 37 insertions(+), 0 deletions(-) create mode 100644 meta/recipes-support/ptest-runner/files/ptest-runner create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_1.0.bb
meta/recipes-support/ptest-runner/files/ptest-runner new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh + +echo "START: $0" +cd /usr/lib +for x in * +do + if [ -d "/usr/lib/$x/ptest" ]; then + date "+%Y-%m-%dT%H:%M" + echo "BEGIN: $x" + cd /usr/lib/$x/ptest + ./run-ptest + echo "END: $x" + date "+%Y-%m-%dT%H:%M" + fi +done +echo "STOP: $0"
meta/recipes-support/ptest-runner/ptest-runner_1.0.bb new file +17 -0
@@ -0,0 +1,21 @@ +SRC_URI += "file://ptest-runner" + +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \ + file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" + +INHIBIT_DEFAULT_DEPS = "1" + +do_install () { + mkdir -p ${D}${bindir} + install -m 0755 ${WORKDIR}/ptest-runner ${D}${bindir} +} + +do_patch[noexec] = "1" +do_configure[noexec] = "1" +do_compile[noexec] = "1" +do_build[noexec] = "1"
Signature · 1 line
1.7.5.4
PATCH 3/3 Enable bash-ptest
In reply to Björn StenbergPatch Makefile.in to allow test programs be built on host and ran on target.
Patch tests/run-all to output PASS/FAIL for each testcase. Patch recipe to build and install test programs and test suite.
--- .../bash/bash-4.2/build-tests.patch | 29 ++++++++++++++++++++ meta/recipes-extended/bash/bash-4.2/run-ptest | 2 + .../bash/bash-4.2/test-output.patch | 19 +++++++++++++ meta/recipes-extended/bash/bash.inc | 18 ++++++++++++ meta/recipes-extended/bash/bash_4.2.bb | 5 +++- 5 files changed, 72 insertions(+), 1 deletions(-) create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patch
meta/recipes-extended/bash/bash-4.2/build-tests.patch new file +29 -0
@@ -0,0 +1,29 @@ +diff -uNr a/Makefile.in b/Makefile.in +--- a/Makefile.in 2012-06-14 10:15:15.063465304 +0200 ++++ b/Makefile.in 2012-06-14 10:47:02.985115849 +0200 +@@ -827,18 +827,21 @@ + fi + + recho$(EXEEXT): $(SUPPORT_SRC)recho.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)recho.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + zecho$(EXEEXT): $(SUPPORT_SRC)zecho.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)zecho.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + printenv$(EXEEXT): $(SUPPORT_SRC)printenv.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)printenv.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + xcase$(EXEEXT): $(SUPPORT_SRC)xcase.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)xcase.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + test tests check: force $(Program) $(TESTS_SUPPORT) ++ $(MAKE) runtest ++ ++runtest: + @-test -d tests || mkdir tests + @cp $(TESTS_SUPPORT) tests + @( cd $(srcdir)/tests && \
meta/recipes-extended/bash/bash-4.2/run-ptest new file +2 -0
@@ -0,0 +1,2 @@ +#!/bin/sh +make -k THIS_SH=/bin/bash BUILD_DIR=. runtest
meta/recipes-extended/bash/bash-4.2/test-output.patch new file +19 -0
@@ -0,0 +1,19 @@ +diff -uNr a/tests/run-all b/tests/run-all +--- a/tests/run-all 2012-06-14 10:15:15.007467700 +0200 ++++ b/tests/run-all 2012-06-14 10:19:26.464678067 +0200 +@@ -22,6 +22,14 @@ + case $x in + $0|run-minimal|run-gprof) ;; + *.orig|*~) ;; +- *) echo $x ; sh $x ;; ++ *) echo $x ++ output=`sh $x` ++ if [ -n "$output" ]; then ++ echo "$output" ++ echo "FAIL: $x" ++ else ++ echo "PASS: $x" ++ fi ++ ;; + esac + done
meta/recipes-extended/bash/bash.inc +18 -0
@@ -21,18 +21,36 @@ ALTERNATIVE_PRIORITY = "100" export AUTOHEADER = "true" +RDEPENDS_${PN}-ptest += "make" +FILES_${PN}-dbg += "${PTEST_PATH}/.debug" + do_configure_prepend () { if [ ! -e acinclude.m4 ]; then cat aclocal.m4 > acinclude.m4 fi } +do_compile_append () { + oe_runmake recho zecho printenv xcase +} + do_install_append () { # Move /usr/bin/bash to /bin/bash, if need if [ "${base_bindir}" != "${bindir}" ]; then mkdir -p ${D}${base_bindir} mv ${D}${bindir}/bash ${D}${base_bindir} fi + + if [ "${PN}" = "${BPN}" ]; then + mkdir -p ${D}${PTEST_PATH} + install -m 0755 ${WORKDIR}/run-ptest ${D}${PTEST_PATH} + install -m 0755 ${B}/Makefile ${D}${PTEST_PATH} + install -m 0755 ${B}/recho ${D}${PTEST_PATH} + install -m 0755 ${B}/zecho ${D}${PTEST_PATH} + install -m 0755 ${B}/printenv ${D}${PTEST_PATH} + install -m 0755 ${B}/xcase ${D}${PTEST_PATH} + cp -r ${B}/tests ${D}${PTEST_PATH}/ + fi } pkg_postinst_${PN} () {
meta/recipes-extended/bash/bash_4.2.bb +4 -1
@@ -1,6 +1,6 @@ require bash.inc -PR = "r4" +PR = "r5" SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \ ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001 \
@@ -14,6 +14,9 @@ SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \ ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-009;apply=yes;striplevel=0;name=patch009 \ ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-010;apply=yes;striplevel=0;name=patch010 \ file://execute_cmd.patch;striplevel=0 \ + file://build-tests.patch \ + file://test-output.patch \ + file://run-ptest \ " SRC_URI[tarball.md5sum] = "3fb927c7c33022f1c327f14a81c0d4b0"
Signature · 1 line
1.7.5.4
Re: [PATCH 3/3] Enable bash-ptest
Quoted 41 lines · Björn Stenberg
Patch Makefile.in to allow test programs be built on host and ran on target.
Patch tests/run-all to output PASS/FAIL for each testcase. Patch recipe to build and install test programs and test suite.--- .../bash/bash-4.2/build-tests.patch | 29 ++++++++++++++++++++ meta/recipes-extended/bash/bash-4.2/run-ptest | 2 + .../bash/bash-4.2/test-output.patch | 19 +++++++++++++ meta/recipes-extended/bash/bash.inc | 18 ++++++++++++ meta/recipes-extended/bash/bash_4.2.bb | 5 +++- 5 files changed, 72 insertions(+), 1 deletions(-) create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patchmeta/recipes-extended/bash/bash-4.2/build-tests.patch new file +19
-0@@ -0,0 +1,29 @@ +diff -uNr a/Makefile.in b/Makefile.in +--- a/Makefile.in 2012-06-14 10:15:15.063465304 +0200 ++++ b/Makefile.in 2012-06-14 10:47:02.985115849 +0200 +@@ -827,18 +827,21 @@ + fi + + recho$(EXEEXT): $(SUPPORT_SRC)recho.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)recho.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + zecho$(EXEEXT): $(SUPPORT_SRC)zecho.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)zecho.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + printenv$(EXEEXT): $(SUPPORT_SRC)printenv.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)printenv.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + xcase$(EXEEXT): $(SUPPORT_SRC)xcase.c
meta/recipes-extended/bash/bash-4.2/build-tests.patch line 29
Quoted 10 lines
meta/recipes-extended/bash/bash-4.2/build-tests.patch new file +10
-0+- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)xcase.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + test tests check: force $(Program) $(TESTS_SUPPORT) ++ $(MAKE) runtest ++ ++runtest: + @-test -d tests || mkdir tests + @cp $(TESTS_SUPPORT) tests + @( cd $(srcdir)/tests && \
As it stands, I can bet on upstream saying "no" to this for good reason. A better concept might be to introduce $(CC_FOR_TEST) which could default to CC_FOR_BUILD. This would let us do what you're thinking here yet keep upstream happy too.
Just thinking out loud really...
Cheers,
Richard
Re: [PATCH 3/3] Enable bash-ptest
In reply to Björn Stenbergmeta/recipes-extended/bash/bash-4.2/build-tests.patch
Quoted 21 lines · Björn Stenberg
Patch Makefile.in to allow test programs be built on host and ran on target.
Patch tests/run-all to output PASS/FAIL for each testcase. Patch recipe to build and install test programs and test suite.--- .../bash/bash-4.2/build-tests.patch | 29 ++++++++++++++++++++ meta/recipes-extended/bash/bash-4.2/run-ptest | 2 + .../bash/bash-4.2/test-output.patch | 19 +++++++++++++ meta/recipes-extended/bash/bash.inc | 18 ++++++++++++ meta/recipes-extended/bash/bash_4.2.bb | 5 +++- 5 files changed, 72 insertions(+), 1 deletions(-)create mode 100644
meta/recipes-extended/bash/bash-4.2/build-tests.patchcreate mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest create mode 100644meta/recipes-extended/bash/bash-4.2/test-output.patchmeta/recipes-extended/bash/bash-4.2/build-tests.patch new file +0
-0
Thanks for proposing and getting the initial implementation started, this has been something we have been thinking about for a while, great to have the things like this become reality!
When introducing new patches, please follow the guidelines for patches and commit message:
http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
I know this is still and RFC, but your commits themselves also need Signed-off-by:
Quoted 74 lines
meta/recipes-extended/bash/bash-4.2/build-tests.patch new file +29
-0@@ -0,0 +1,29 @@ +diff -uNr a/Makefile.in b/Makefile.in +--- a/Makefile.in 2012-06-14 10:15:15.063465304 +0200 ++++ b/Makefile.in 2012-06-14 10:47:02.985115849 +0200 +@@ -827,18 +827,21 @@ + fi + + recho$(EXEEXT): $(SUPPORT_SRC)recho.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)recho.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + zecho$(EXEEXT): $(SUPPORT_SRC)zecho.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)zecho.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + printenv$(EXEEXT): $(SUPPORT_SRC)printenv.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)printenv.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + xcase$(EXEEXT): $(SUPPORT_SRC)xcase.c +- @$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)xcase.c ${LIBS_FOR_BUILD} ++ @$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $< + + test tests check: force $(Program) $(TESTS_SUPPORT) ++ $(MAKE) runtest ++ ++runtest: + @-test -d tests || mkdir tests + @cp $(TESTS_SUPPORT) tests + @( cd $(srcdir)/tests && \meta/recipes-extended/bash/bash-4.2/run-ptest new file +2
-0@@ -0,0 +1,2 @@ +#!/bin/sh +make -k THIS_SH=/bin/bash BUILD_DIR=. runtestmeta/recipes-extended/bash/bash-4.2/test-output.patch new file +19
-0@@ -0,0 +1,19 @@ +diff -uNr a/tests/run-all b/tests/run-all +--- a/tests/run-all 2012-06-14 10:15:15.007467700 +0200 ++++ b/tests/run-all 2012-06-14 10:19:26.464678067 +0200 +@@ -22,6 +22,14 @@ + case $x in + $0|run-minimal|run-gprof) ;; + *.orig|*~) ;; +- *) echo $x ; sh $x ;; ++ *) echo $x ++ output=`sh $x` ++ if [ -n "$output" ]; then ++ echo "$output" ++ echo "FAIL: $x" ++ else ++ echo "PASS: $x" ++ fi ++ ;; + esac + donemeta/recipes-extended/bash/bash.inc +3
-0@@ -21,18 +21,36 @@ ALTERNATIVE_PRIORITY = "100" export AUTOHEADER = "true" +RDEPENDS_${PN}-ptest += "make" +FILES_${PN}-dbg += "${PTEST_PATH}/.debug" +
meta/recipes-extended/bash/bash.inc ALTERNATIVE_PRIORITY = "100" · line 36
Quoted 10 lines
meta/recipes-extended/bash/bash.inc +4
-0do_configure_prepend () { if [ ! -e acinclude.m4 ]; then cat aclocal.m4 > acinclude.m4 fi } +do_compile_append () { + oe_runmake recho zecho printenv xcase +} +
One thing that we have discussed a couple of times, but no real outcome yet is to make the building of tests conditional so as more recipes are enabled with this kind of change, we don't extend the default build time too long.
What's been discussed is using DISTRO_FEATURE to test first.
Thoughts?
Quoted 34 lines
meta/recipes-extended/bash/bash.inc +11
-0do_install_append () { # Move /usr/bin/bash to /bin/bash, if need if [ "${base_bindir}" != "${bindir}" ]; then mkdir -p ${D}${base_bindir} mv ${D}${bindir}/bash ${D}${base_bindir} fi + + if [ "${PN}" = "${BPN}" ]; then + mkdir -p ${D}${PTEST_PATH} + install -m 0755 ${WORKDIR}/run-ptest ${D}${PTEST_PATH} + install -m 0755 ${B}/Makefile ${D}${PTEST_PATH} + install -m 0755 ${B}/recho ${D}${PTEST_PATH} + install -m 0755 ${B}/zecho ${D}${PTEST_PATH} + install -m 0755 ${B}/printenv ${D}${PTEST_PATH} + install -m 0755 ${B}/xcase ${D}${PTEST_PATH} + cp -r ${B}/tests ${D}${PTEST_PATH}/ + fi } pkg_postinst_${PN} () {meta/recipes-extended/bash/bash_4.2.bb +1
-1@@ -1,6 +1,6 @@ require bash.inc -PR = "r4" +PR = "r5" SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \ ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001 \
meta/recipes-extended/bash/bash_4.2.bb SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \ · line 22
Quoted 9 lines
meta/recipes-extended/bash/bash_4.2.bb +3
-0@@ -14,6 +14,9 @@ SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \ ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-009;apply=yes;striplevel=0;name=patch009 \ ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-010;apply=yes;striplevel=0;name=patch010 \ file://execute_cmd.patch;striplevel=0 \ + file://build-tests.patch \ + file://test-output.patch \ + file://run-ptest \ " SRC_URI[tarball.md5sum] = "3fb927c7c33022f1c327f14a81c0d4b0"
Thanks
Sau!
Quoted 4 lines · Björn Stenberg
And it sometimes
requires patching and/or translation of the test output to produce a generic output format that can be automatically parsed. I have included an example patch for bash that does just that.
What is this generic output format? From your patch to bash I suspect it's at least inspired by the automake "make check" output.
FWIW, this is what "make check" on librest (the first package I had to hand) gives:
...
make check-TESTS
make[2]: Entering directory `/home/ross/Programming/cvs/librest/tests'
PASS: proxy
PASS: proxy-continuous
PASS: threaded
PASS: oauth
PASS: oauth-async
PASS: oauth2
PASS: flickr
PASS: lastfm
** (process:1267): ERROR **: Generated output for parsed XML does not match:
in: <node0 a00='v00' a01='v01'><node1 a10='v10'></node1><node1 a10='v10'></node1>Cont0</node0>
out: <node0 a01='v01' a00='v00'><node1 a10='v10'></node1><node1 a10='v10'></node1>Cont0</node0>
/bin/sh: line 5: 1267 Trace/breakpoint trap (core dumped) ${dir}$tst
XFAIL: xml
PASS: custom-serialize
=====================================================
All 10 tests behaved as expected (1 expected failure)
=====================================================
make[2]: Leaving directory `/home/ross/Programming/cvs/librest/tests'
...
Would this test suite just work if packaged as ptests?
Ross
Quoted 2 lines · Ross
What is this generic output format? From your patch to bash I suspect it's at least inspired by the automake "make check" output.
Yes, I used the automake-style "(PASS|FAIL): <testcase>" format.
In addition to that, before/after each package test suite is started, "(BEGIN|END): <package>" is printed. So a typical output looks like this (paraphrased):
BEGIN: bash
PASS: test1
FAIL: test2
PASS: test3
END: bash
BEGIN: busybox
PASS: test1
FAIL: test2
END: busybox
Signature · 1 line
Björn