Script and layer for running tests
cover Script and layer for running tests
Hello,
This series adds an oe-selftest script, some modules and a new layer meta-selftest. which are meant to help in writing tests (using python unittest) for various bitbake tools/scripts as well as simple output checks or do
complete builds with different options (with the emphasis that everything checked is done outside of bitbake context, just like a human would do.) For more details, please see YOCTO #4740.
Cheers,
Stefan
The following changes since commit e15893adf9268b2920b24c52d5c2bb777c6f778e:
bitbake: serv.py: Give pr-server up to 5 seconds to commit data (2013-11-18 17:19:11 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib stefans/testme4
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/testme4
Alexandru Palalau (1):
scripts/lib/selftest: add more simple build tests
Corneliu Stoicescu (2):
meta-selftest: create a new test layer to be used by oe-selftest
script
scripts/lib/selftest: add a new module for bitbake output tests
Stefan Stanacar (1):
scripts/oe-selftest: script to run builds as unitests against bitbake
or various scripts
.gitignore | 1 + meta-selftest/COPYING.MIT | 17 +++ meta-selftest/README | 2 + meta-selftest/classes/test_events.bbclass | 16 +++ meta-selftest/conf/layer.conf | 10 ++ .../recipes-test/aspell/aspell_0.0.0.1.bb | 28 ++++ .../recipes-test/man/man/man-1.5h1-make.patch | 16 +++ meta-selftest/recipes-test/man/man_1.6g.bbappend | 2 + .../xcursor-transparent-theme_0.1.1.bbappend | 3 + scripts/lib/selftest/__init__.py | 0 scripts/lib/selftest/base.py | 54 ++++++++ scripts/lib/selftest/tests/__init__.py | 2 + scripts/lib/selftest/tests/bboptions.py | 39 ++++++ scripts/lib/selftest/tests/bboutput.py | 86 ++++++++++++ scripts/lib/selftest/utils/__init__.py | 0 scripts/lib/selftest/utils/commands.py | 146 +++++++++++++++++++++ scripts/lib/selftest/utils/ftools.py | 27 ++++ scripts/oe-selftest | 130 ++++++++++++++++++ 18 files changed, 579 insertions(+) create mode 100644 meta-selftest/COPYING.MIT create mode 100644 meta-selftest/README create mode 100644 meta-selftest/classes/test_events.bbclass create mode 100644 meta-selftest/conf/layer.conf create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend create mode 100644 scripts/lib/selftest/__init__.py create mode 100644 scripts/lib/selftest/base.py create mode 100644 scripts/lib/selftest/tests/__init__.py create mode 100644 scripts/lib/selftest/tests/bboptions.py create mode 100644 scripts/lib/selftest/tests/bboutput.py create mode 100644 scripts/lib/selftest/utils/__init__.py create mode 100644 scripts/lib/selftest/utils/commands.py create mode 100644 scripts/lib/selftest/utils/ftools.py create mode 100755 scripts/oe-selftest
Signature · 1 line
1.8.3.1
RFC PATCH 2/4 meta-selftest: create a new test layer to be used by oe-selftest script
From: Corneliu Stoicescu <[email protected]>
Everything in this layer is meant to be used by tests.
--- .gitignore | 1 + meta-selftest/COPYING.MIT | 17 +++++++++++++ meta-selftest/README | 2 ++ meta-selftest/classes/test_events.bbclass | 16 +++++++++++++ meta-selftest/conf/layer.conf | 10 ++++++++ .../recipes-test/aspell/aspell_0.0.0.1.bb | 28 ++++++++++++++++++++++ .../recipes-test/man/man/man-1.5h1-make.patch | 16 +++++++++++++ meta-selftest/recipes-test/man/man_1.6g.bbappend | 2 ++ .../xcursor-transparent-theme_0.1.1.bbappend | 3 +++ 9 files changed, 95 insertions(+) create mode 100644 meta-selftest/COPYING.MIT create mode 100644 meta-selftest/README create mode 100644 meta-selftest/classes/test_events.bbclass create mode 100644 meta-selftest/conf/layer.conf create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
.gitignore +1 -0
@@ -21,3 +21,4 @@ bitbake/doc/manual/pdf/ bitbake/doc/manual/txt/ bitbake/doc/manual/xhtml/ pull-*/ +!meta-selftest
meta-selftest/COPYING.MIT new file +17 -0
@@ -0,0 +1,17 @@ +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE.
meta-selftest/README new file +2 -0
@@ -0,0 +1,2 @@ +This layer is intended as test layer, used by scripts/oe-selftest +and it's probably a mistake to include it in your builds.
meta-selftest/classes/test_events.bbclass new file +16 -0
@@ -0,0 +1,16 @@ +python test1_eventhandler() { + bb.note("Test for bb.event.BuildStarted") +} +python test2_eventhandler() { + bb.note("Test for bb.event.BuildCompleted") +} +python test3_eventhandler() { + bb.note("Test for bb.event.InvalidEvent") +} + +addhandler test1_eventhandler +test1_eventhandler[eventmask] = "bb.event.BuildStarted" +addhandler test2_eventhandler +test2_eventhandler[eventmask] = "bb.event.BuildCompleted" +addhandler test3_eventhandler +test3_eventhandler[eventmask] = "bb.event.InvalidEvent"
meta-selftest/conf/layer.conf new file +10 -0
@@ -0,0 +1,10 @@ +# We have a conf and classes directory, add to BBPATH +BBPATH .= ":${LAYERDIR}" + +# We have recipes-* directories, add to BBFILES +BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ + ${LAYERDIR}/recipes-*/*/*.bbappend" + +BBFILE_COLLECTIONS += "selftest" +BBFILE_PATTERN_selftest = "^${LAYERDIR}/" +BBFILE_PRIORITY_selftest = "6"
meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb new file +28 -0
@@ -0,0 +1,28 @@ +# This recipe is a copy from the oe-core one. +# It has a lower and invalid version number in order not to be accidentally used by bitbake. +# It is used for tests that require duplicate recipe files. + +DESCRIPTION = "GNU Aspell spell-checker" +SECTION = "console/utils" +LICENSE="LGPLv2 | LGPLv2.1" +LIC_FILES_CHKSUM = "file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34" +PR = "r1" + +SRC_URI = "${GNU_MIRROR}/aspell/aspell-${PV}.tar.gz" +SRC_URI[md5sum] = "e66a9c9af6a60dc46134fdacf6ce97d7" +SRC_URI[sha256sum] = "f52583a83a63633701c5f71db3dc40aab87b7f76b29723aeb27941eff42df6e1" + +PACKAGECONFIG ??= "" +PACKAGECONFIG[curses] = "--enable-curses,--disable-curses,ncurses" + +PACKAGES += "libaspell libpspell libpspell-dev aspell-utils" + +FILES_${PN}-dbg += "${libdir}/aspell-0.60/.debu*" +FILES_libaspell = "${libdir}/libaspell.so.* ${libdir}/aspell*" +FILES_aspell-utils = "${bindir}/word-list-compress ${bindir}/aspell-import ${bindir}/run-with-aspell ${bindir}/pre*" +FILES_${PN} = "${bindir}/aspell" +FILES_libpspell = "${libdir}/libpspell.so.*" +FILES_libpspell-dev = "${libdir}/libpspell* ${bindir}/pspell-config ${includedir}/pspell" + +ARM_INSTRUCTION_SET = "arm" +inherit autotools gettext
meta-selftest/recipes-test/man/man/man-1.5h1-make.patch new file +16 -0
@@ -0,0 +1,16 @@ +Test patch here! +This is invalid patch used by tests in scripts/lib/selftests. + +--- man-1.5g/man/Makefile.in.mike Fri Apr 9 13:35:54 1999 ++++ man-1.5g/man/Makefile.in Fri Apr 9 13:36:45 1999 +@@ -1,8 +1,8 @@ + #MAKE THIS PATCH INVALID + MAN1 = man whatis apropos +-MAN5 = man.conf ++MAN5 = man.config + MAN8 = makewhatis +-ALL = man.1 whatis.1 apropos.1 man.conf.5 ++ALL = man.1 whatis.1 apropos.1 man.config.5 + MAYBE8 = makewhatis + + .SUFFIXES: .man .1 .5 .8
meta-selftest/recipes-test/man/man_1.6g.bbappend new file +2 -0
@@ -0,0 +1,2 @@ +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests. +include test_recipe.inc
meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend new file +3 -0
@@ -0,0 +1,3 @@ +TEST_VAR = "" +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests. +include test_recipe.inc
Signature · 1 line
1.8.3.1
RFC PATCH 3/4 scripts/lib/selftest: add a new module for bitbake output tests
In reply to Stefan StanacarFrom: Corneliu Stoicescu <[email protected]>
Tests for bitbake-layers and expected output for some bitbake options.
--- scripts/lib/selftest/tests/__init__.py | 1 + scripts/lib/selftest/tests/bboutput.py | 86 ++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100644 scripts/lib/selftest/tests/bboutput.py
scripts/lib/selftest/tests/__init__.py +1 -0
@@ -0,0 +1 @@ +from selftest.tests.bboutput import *
scripts/lib/selftest/tests/bboutput.py new file +86 -0
86 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RFC PATCH 1/4 scripts/oe-selftest: script to run builds as unitests against bitbake or various scripts
In reply to Stefan StanacarThe purpose of oe-selftest is to run unittest modules added in scripts/lib/selftest/tests, which are tests against bitbake tools.
The scripts/lib/selftest/tests path for tests is only temporary.
Tests probably will be moved to meta/lib/oeqa/<something> once we've worked out how that can work. (the reason to have them there is to allow layers to add their own tests).
Right now the script it's useful for simple tests like:
- "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
- or "bitbake-layers <...>" type scripts and yocto-bsp tools.
This commit also adds some helper methods that the tests will use (which are currently used by the script itself, and those also need to move if the tests move).
Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.
It's a simple matter or sourcing the enviroment, adding the meta-selftest layer and running: oe-selftest to get some results.
[ YOCTO #4740 ]
--- scripts/lib/selftest/__init__.py | 0 scripts/lib/selftest/base.py | 54 ++++++++++++ scripts/lib/selftest/tests/__init__.py | 0 scripts/lib/selftest/utils/__init__.py | 0 scripts/lib/selftest/utils/commands.py | 146 +++++++++++++++++++++++++++++++++ scripts/lib/selftest/utils/ftools.py | 27 ++++++ scripts/oe-selftest | 130 +++++++++++++++++++++++++++++ 7 files changed, 357 insertions(+) create mode 100644 scripts/lib/selftest/__init__.py create mode 100644 scripts/lib/selftest/base.py create mode 100644 scripts/lib/selftest/tests/__init__.py create mode 100644 scripts/lib/selftest/utils/__init__.py create mode 100644 scripts/lib/selftest/utils/commands.py create mode 100644 scripts/lib/selftest/utils/ftools.py create mode 100755 scripts/oe-selftest
scripts/lib/selftest/__init__.py new file +0 -0
scripts/lib/selftest/base.py new file +54 -0
54 lines not on this page · raw message
scripts/lib/selftest/tests/__init__.py new file +0 -0
scripts/lib/selftest/utils/__init__.py new file +0 -0
scripts/lib/selftest/utils/commands.py new file +146 -0
146 lines not on this page · raw message
scripts/lib/selftest/utils/ftools.py new file +27 -0
@@ -0,0 +1,27 @@ +import os +import re + +def write_file(path, data): + wdata = data.rstrip() + "\n" + with open(path, "w") as f: + f.write(wdata) + +def append_file(path, data): + wdata = data.rstrip() + "\n" + with open(path, "a") as f: + f.write(wdata) + +def read_file(path): + data = None + with open(path) as f: + data = f.read() + return data + +def remove_from_file(path, data): + lines = read_file(path).splitlines() + rmdata = data.strip().splitlines() + for l in rmdata: + for c in range(0, lines.count(l)): + i = lines.index(l) + del(lines[i]) + write_file(path, "\n".join(lines))
scripts/oe-selftest new file +130 -0
130 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RFC PATCH 4/4 scripts/lib/selftest: add more simple build tests
In reply to Stefan StanacarFrom: Alexandru Palalau <[email protected]>
Tests for RM_OLD_IMAGE and for WARN_QA/ERROR_QA behaviour.
--- scripts/lib/selftest/tests/__init__.py | 1 + scripts/lib/selftest/tests/bboptions.py | 39 +++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) create mode 100644 scripts/lib/selftest/tests/bboptions.py
scripts/lib/selftest/tests/__init__.py +1 -0
@@ -1 +1,2 @@ from selftest.tests.bboutput import * +from selftest.tests.bboptions import *
scripts/lib/selftest/tests/bboptions.py new file +39 -0
@@ -0,0 +1,39 @@ +import unittest +import os +import logging + +from selftest.base import oeSelfTest +from selftest.utils.commands import runCmd, bitbake, get_bb_var, get_test_layer +import selftest.utils.ftools as ftools + +class OptionsTests(oeSelfTest): + + def test_options_warnqa_errorqa_switch(self): + bitbake("xcursor-transparent-theme -ccleansstate") + + if "packages-list" not in get_bb_var("ERROR_QA"): + self.write_config("ERROR_QA_append = \" packages-list\"") + + xcursor_path = os.path.join(get_test_layer(), "recipes-test/xcursor-transparent-theme/test_recipe.inc") + ftools.write_file(xcursor_path, 'PACKAGES += \"${PN}-dbg\"') + + res = bitbake("xcursor-transparent-theme", ignore_status=True) + self.assertTrue("ERROR: QA Issue" in res.output) + self.assertEqual(res.status, 1) + + self.append_config("ERROR_QA_remove = \"packages-list\"") + self.append_config("WARN_QA_append = \" packages-list\"") + bitbake("xcursor-transparent-theme") + bitbake("xcursor-transparent-theme -ccleansstate") + os.remove(xcursor_path) + + def test_options_rm_old_image(self): + bitbake("core-image-minimal") + + deploydir = get_bb_var("DEPLOY_DIR_IMAGE", target="core-image-minimal") + imagename = get_bb_var("IMAGE_LINK_NAME", target="core-image-minimal") + oldimgpath = os.path.realpath(os.path.join(deploydir, imagename + ".ext3")) + + self.append_config("RM_OLD_IMAGE = \"1\"") + bitbake("core-image-minimal") + self.assertFalse(os.path.exists(oldimgpath), msg="Old image path still exists: %s" % oldimgpath)
Signature · 1 line
1.8.3.1
Re: [RFC PATCH 2/4] meta-selftest: create a new test layer to be used by oe-selftest script
In reply to Stefan StanacarHi Stefan,
Quoted 6 lines · On Tuesday 19 November 2013 17:43:35 Stefan Stanacar wrote:
+++ b/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
(quoted diff) +2
-0@@ -0,0 +1,28 @@ +# This recipe is a copy from the oe-core one. +# It has a lower and invalid version number in order not to be accidentallyused by bitbake.
+# It is used for tests that require duplicate recipe files.
FYI the term we use for this is "overlayed" rather than "duplicate" (where the two or more recipes exist in the same configuration).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre