Skip to contentSkip to conversation

Script and layer for running tests

RFC PATCH In review Stefan Stanacar 2 participants 6 messages Nov 19, 2013 – Nov 25, 2013

lore
Share & tools
lore
#0 lore raw

cover Script and layer for running tests

Hello,

This series adds an oe-selftest script, some modules and a new layer meta-selftest. which are meant to help in writing tests (using python unittest) for various bitbake tools/scripts as well as simple output checks or do
complete builds with different options (with the emphasis that everything checked is done outside of bitbake context, just like a human would do.) For more details, please see YOCTO #4740.

Cheers,
Stefan

The following changes since commit e15893adf9268b2920b24c52d5c2bb777c6f778e:

bitbake: serv.py: Give pr-server up to 5 seconds to commit data (2013-11-18 17:19:11 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib stefans/testme4
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/testme4

Alexandru Palalau (1):
scripts/lib/selftest: add more simple build tests

Corneliu Stoicescu (2):
meta-selftest: create a new test layer to be used by oe-selftest
script
scripts/lib/selftest: add a new module for bitbake output tests

Stefan Stanacar (1):
scripts/oe-selftest: script to run builds as unitests against bitbake
or various scripts

 .gitignore                                         |   1 +
 meta-selftest/COPYING.MIT                          |  17 +++
 meta-selftest/README                               |   2 +
 meta-selftest/classes/test_events.bbclass          |  16 +++
 meta-selftest/conf/layer.conf                      |  10 ++
 .../recipes-test/aspell/aspell_0.0.0.1.bb          |  28 ++++
 .../recipes-test/man/man/man-1.5h1-make.patch      |  16 +++
 meta-selftest/recipes-test/man/man_1.6g.bbappend   |   2 +
 .../xcursor-transparent-theme_0.1.1.bbappend       |   3 +
 scripts/lib/selftest/__init__.py                   |   0
 scripts/lib/selftest/base.py                       |  54 ++++++++
 scripts/lib/selftest/tests/__init__.py             |   2 +
 scripts/lib/selftest/tests/bboptions.py            |  39 ++++++
 scripts/lib/selftest/tests/bboutput.py             |  86 ++++++++++++
 scripts/lib/selftest/utils/__init__.py             |   0
 scripts/lib/selftest/utils/commands.py             | 146 +++++++++++++++++++++
 scripts/lib/selftest/utils/ftools.py               |  27 ++++
 scripts/oe-selftest                                | 130 ++++++++++++++++++
 18 files changed, 579 insertions(+)
 create mode 100644 meta-selftest/COPYING.MIT
 create mode 100644 meta-selftest/README
 create mode 100644 meta-selftest/classes/test_events.bbclass
 create mode 100644 meta-selftest/conf/layer.conf
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
 create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
 create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend
 create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
 create mode 100644 scripts/lib/selftest/__init__.py
 create mode 100644 scripts/lib/selftest/base.py
 create mode 100644 scripts/lib/selftest/tests/__init__.py
 create mode 100644 scripts/lib/selftest/tests/bboptions.py
 create mode 100644 scripts/lib/selftest/tests/bboutput.py
 create mode 100644 scripts/lib/selftest/utils/__init__.py
 create mode 100644 scripts/lib/selftest/utils/commands.py
 create mode 100644 scripts/lib/selftest/utils/ftools.py
 create mode 100755 scripts/oe-selftest
Signature · 1 line
1.8.3.1
#1 lore raw

RFC PATCH 1/4 scripts/oe-selftest: script to run builds as unitests against bitbake or various scripts

The purpose of oe-selftest is to run unittest modules added in scripts/lib/selftest/tests, which are tests against bitbake tools.
The scripts/lib/selftest/tests path for tests is only temporary.
Tests probably will be moved to meta/lib/oeqa/<something> once we've worked out how that can work. (the reason to have them there is to allow layers to add their own tests).

Right now the script it's useful for simple tests like:
- "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
- or "bitbake-layers <...>" type scripts and yocto-bsp tools.
This commit also adds some helper methods that the tests will use (which are currently used by the script itself, and those also need to move if the tests move).

Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.

It's a simple matter or sourcing the enviroment, adding the meta-selftest layer and running: oe-selftest to get some results.

[ YOCTO #4740 ]

---
 scripts/lib/selftest/__init__.py       |   0
 scripts/lib/selftest/base.py           |  54 ++++++++++++
 scripts/lib/selftest/tests/__init__.py |   0
 scripts/lib/selftest/utils/__init__.py |   0
 scripts/lib/selftest/utils/commands.py | 146 +++++++++++++++++++++++++++++++++
 scripts/lib/selftest/utils/ftools.py   |  27 ++++++
 scripts/oe-selftest                    | 130 +++++++++++++++++++++++++++++
 7 files changed, 357 insertions(+)
 create mode 100644 scripts/lib/selftest/__init__.py
 create mode 100644 scripts/lib/selftest/base.py
 create mode 100644 scripts/lib/selftest/tests/__init__.py
 create mode 100644 scripts/lib/selftest/utils/__init__.py
 create mode 100644 scripts/lib/selftest/utils/commands.py
 create mode 100644 scripts/lib/selftest/utils/ftools.py
 create mode 100755 scripts/oe-selftest
scripts/lib/selftest/__init__.py new file +0 -0
new file mode 100644
index 0000000..e69de29
scripts/lib/selftest/base.py new file +54 -0

54 lines not on this page · raw message

scripts/lib/selftest/tests/__init__.py new file +0 -0
new file mode 100644
index 0000000..e69de29
scripts/lib/selftest/utils/__init__.py new file +0 -0
new file mode 100644
index 0000000..e69de29
scripts/lib/selftest/utils/commands.py new file +146 -0

146 lines not on this page · raw message

scripts/lib/selftest/utils/ftools.py new file +27 -0
new file mode 100644
index 0000000..64ebe3d
@@ -0,0 +1,27 @@
+import os
+import re
+
+def write_file(path, data):
+    wdata = data.rstrip() + "\n"
+    with open(path, "w") as f:
+        f.write(wdata)
+
+def append_file(path, data):
+    wdata = data.rstrip() + "\n"
+    with open(path, "a") as f:
+            f.write(wdata)
+
+def read_file(path):
+    data = None
+    with open(path) as f:
+        data = f.read()
+    return data
+
+def remove_from_file(path, data):
+    lines = read_file(path).splitlines()
+    rmdata = data.strip().splitlines()
+    for l in rmdata:
+        for c in range(0, lines.count(l)):
+            i = lines.index(l)
+            del(lines[i])
+    write_file(path, "\n".join(lines))
scripts/oe-selftest new file +130 -0

130 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#2 lore raw

RFC PATCH 2/4 meta-selftest: create a new test layer to be used by oe-selftest script

In reply to Stefan Stanacar

From: Corneliu Stoicescu <[email protected]>

Everything in this layer is meant to be used by tests.

---
 .gitignore                                         |  1 +
 meta-selftest/COPYING.MIT                          | 17 +++++++++++++
 meta-selftest/README                               |  2 ++
 meta-selftest/classes/test_events.bbclass          | 16 +++++++++++++
 meta-selftest/conf/layer.conf                      | 10 ++++++++
 .../recipes-test/aspell/aspell_0.0.0.1.bb          | 28 ++++++++++++++++++++++
 .../recipes-test/man/man/man-1.5h1-make.patch      | 16 +++++++++++++
 meta-selftest/recipes-test/man/man_1.6g.bbappend   |  2 ++
 .../xcursor-transparent-theme_0.1.1.bbappend       |  3 +++
 9 files changed, 95 insertions(+)
 create mode 100644 meta-selftest/COPYING.MIT
 create mode 100644 meta-selftest/README
 create mode 100644 meta-selftest/classes/test_events.bbclass
 create mode 100644 meta-selftest/conf/layer.conf
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
 create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
 create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend
 create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
.gitignore +1 -0
index 0171597..b6755fe 100644
@@ -21,3 +21,4 @@ bitbake/doc/manual/pdf/
 bitbake/doc/manual/txt/
 bitbake/doc/manual/xhtml/
 pull-*/
+!meta-selftest
meta-selftest/COPYING.MIT new file +17 -0
new file mode 100644
index 0000000..89de354
@@ -0,0 +1,17 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
meta-selftest/README new file +2 -0
new file mode 100644
index 0000000..e0f803d
@@ -0,0 +1,2 @@
+This layer is intended as test layer, used by scripts/oe-selftest
+and it's probably a mistake to include it in your builds.
meta-selftest/classes/test_events.bbclass new file +16 -0
new file mode 100644
index 0000000..35324eb
@@ -0,0 +1,16 @@
+python test1_eventhandler() {
+    bb.note("Test for bb.event.BuildStarted")
+}
+python test2_eventhandler() {
+    bb.note("Test for bb.event.BuildCompleted")
+}
+python test3_eventhandler() {
+    bb.note("Test for bb.event.InvalidEvent")
+}
+
+addhandler test1_eventhandler
+test1_eventhandler[eventmask] = "bb.event.BuildStarted"
+addhandler test2_eventhandler
+test2_eventhandler[eventmask] = "bb.event.BuildCompleted"
+addhandler test3_eventhandler
+test3_eventhandler[eventmask] = "bb.event.InvalidEvent"
meta-selftest/conf/layer.conf new file +10 -0
new file mode 100644
index 0000000..e3322f6
@@ -0,0 +1,10 @@
+# We have a conf and classes directory, add to BBPATH
+BBPATH .= ":${LAYERDIR}"
+
+# We have recipes-* directories, add to BBFILES
+BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
+	${LAYERDIR}/recipes-*/*/*.bbappend"
+
+BBFILE_COLLECTIONS += "selftest"
+BBFILE_PATTERN_selftest = "^${LAYERDIR}/"
+BBFILE_PRIORITY_selftest = "6"
meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb new file +28 -0
new file mode 100644
index 0000000..79eb06b
@@ -0,0 +1,28 @@
+# This recipe is a copy from the oe-core one.
+# It has a lower and invalid version number in order not to be accidentally used by bitbake.
+# It is used for tests that require duplicate recipe files.
+
+DESCRIPTION = "GNU Aspell spell-checker"
+SECTION = "console/utils"
+LICENSE="LGPLv2 | LGPLv2.1"
+LIC_FILES_CHKSUM = "file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34"
+PR = "r1"
+
+SRC_URI = "${GNU_MIRROR}/aspell/aspell-${PV}.tar.gz"
+SRC_URI[md5sum] = "e66a9c9af6a60dc46134fdacf6ce97d7"
+SRC_URI[sha256sum] = "f52583a83a63633701c5f71db3dc40aab87b7f76b29723aeb27941eff42df6e1"
+
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[curses] = "--enable-curses,--disable-curses,ncurses"
+
+PACKAGES += "libaspell libpspell libpspell-dev aspell-utils"
+
+FILES_${PN}-dbg += "${libdir}/aspell-0.60/.debu*"
+FILES_libaspell = "${libdir}/libaspell.so.* ${libdir}/aspell*"
+FILES_aspell-utils = "${bindir}/word-list-compress ${bindir}/aspell-import ${bindir}/run-with-aspell ${bindir}/pre*"
+FILES_${PN} = "${bindir}/aspell"
+FILES_libpspell = "${libdir}/libpspell.so.*"
+FILES_libpspell-dev = "${libdir}/libpspell* ${bindir}/pspell-config ${includedir}/pspell"
+
+ARM_INSTRUCTION_SET = "arm"
+inherit autotools gettext
meta-selftest/recipes-test/man/man/man-1.5h1-make.patch new file +16 -0
new file mode 100644
index 0000000..a0d59b7
@@ -0,0 +1,16 @@
+Test patch here!
+This is invalid patch used by tests in scripts/lib/selftests.
+
+--- man-1.5g/man/Makefile.in.mike	Fri Apr  9 13:35:54 1999
++++ man-1.5g/man/Makefile.in	Fri Apr  9 13:36:45 1999
+@@ -1,8 +1,8 @@
+ #MAKE THIS PATCH INVALID
+ MAN1 = man whatis apropos
+-MAN5 = man.conf
++MAN5 = man.config
+ MAN8 = makewhatis
+-ALL = man.1 whatis.1 apropos.1 man.conf.5
++ALL = man.1 whatis.1 apropos.1 man.config.5
+ MAYBE8 = makewhatis
+ 
+ .SUFFIXES: .man .1 .5 .8
meta-selftest/recipes-test/man/man_1.6g.bbappend new file +2 -0
new file mode 100644
index 0000000..2057209
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend new file +3 -0
new file mode 100644
index 0000000..29e86f6
@@ -0,0 +1,3 @@
+TEST_VAR = ""
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
Signature · 1 line
1.8.3.1
#3 lore raw

Re: [RFC PATCH 2/4] meta-selftest: create a new test layer to be used by oe-selftest script

Hi Stefan,

Quoted 6 lines · On Tuesday 19 November 2013 17:43:35 Stefan Stanacar wrote:

+++ b/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb

(quoted diff) +2 -0
@@ -0,0 +1,28 @@
+# This recipe is a copy from the oe-core one.
+# It has a lower and invalid version number in order not to be accidentally

used by bitbake.
+# It is used for tests that require duplicate recipe files.

FYI the term we use for this is "overlayed" rather than "duplicate" (where the two or more recipes exist in the same configuration).

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#4 lore raw

RFC PATCH 3/4 scripts/lib/selftest: add a new module for bitbake output tests

In reply to Stefan Stanacar

From: Corneliu Stoicescu <[email protected]>

Tests for bitbake-layers and expected output for some bitbake options.

---
 scripts/lib/selftest/tests/__init__.py |  1 +
 scripts/lib/selftest/tests/bboutput.py | 86 ++++++++++++++++++++++++++++++++++
 2 files changed, 87 insertions(+)
 create mode 100644 scripts/lib/selftest/tests/bboutput.py
scripts/lib/selftest/tests/__init__.py +1 -0
index e69de29..3acc480 100644
@@ -0,0 +1 @@
+from selftest.tests.bboutput import *
scripts/lib/selftest/tests/bboutput.py new file +86 -0

86 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#5 lore raw

RFC PATCH 4/4 scripts/lib/selftest: add more simple build tests

In reply to Stefan Stanacar

From: Alexandru Palalau <[email protected]>

Tests for RM_OLD_IMAGE and for WARN_QA/ERROR_QA behaviour.

---
 scripts/lib/selftest/tests/__init__.py  |  1 +
 scripts/lib/selftest/tests/bboptions.py | 39 +++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 scripts/lib/selftest/tests/bboptions.py
scripts/lib/selftest/tests/__init__.py +1 -0
index 3acc480..7826054 100644
@@ -1 +1,2 @@
 from selftest.tests.bboutput import *
+from selftest.tests.bboptions import *
scripts/lib/selftest/tests/bboptions.py new file +39 -0
new file mode 100644
index 0000000..ad35386
@@ -0,0 +1,39 @@
+import unittest
+import os
+import logging
+
+from selftest.base import oeSelfTest
+from selftest.utils.commands import runCmd, bitbake, get_bb_var, get_test_layer
+import selftest.utils.ftools as ftools
+
+class OptionsTests(oeSelfTest):
+
+    def test_options_warnqa_errorqa_switch(self):
+        bitbake("xcursor-transparent-theme -ccleansstate")
+
+        if "packages-list" not in get_bb_var("ERROR_QA"):
+            self.write_config("ERROR_QA_append = \" packages-list\"")
+
+        xcursor_path = os.path.join(get_test_layer(), "recipes-test/xcursor-transparent-theme/test_recipe.inc")
+        ftools.write_file(xcursor_path, 'PACKAGES += \"${PN}-dbg\"')
+
+        res = bitbake("xcursor-transparent-theme", ignore_status=True)
+        self.assertTrue("ERROR: QA Issue" in res.output)
+        self.assertEqual(res.status, 1)
+
+        self.append_config("ERROR_QA_remove = \"packages-list\"")
+        self.append_config("WARN_QA_append = \" packages-list\"")
+        bitbake("xcursor-transparent-theme")
+        bitbake("xcursor-transparent-theme -ccleansstate")
+        os.remove(xcursor_path)
+
+    def test_options_rm_old_image(self):
+        bitbake("core-image-minimal")
+
+        deploydir = get_bb_var("DEPLOY_DIR_IMAGE", target="core-image-minimal")
+        imagename = get_bb_var("IMAGE_LINK_NAME", target="core-image-minimal")
+        oldimgpath = os.path.realpath(os.path.join(deploydir, imagename + ".ext3"))
+
+        self.append_config("RM_OLD_IMAGE = \"1\"")
+        bitbake("core-image-minimal")
+        self.assertFalse(os.path.exists(oldimgpath), msg="Old image path still exists: %s" % oldimgpath)
Signature · 1 line
1.8.3.1