testimage/oeqa: Add ability to export tests to decouple them from the build system
cover testimage/oeqa: Add ability to export tests to decouple them from the build system
Hello,
These patches add the ability to export and run tests outside of the build.
As tests used to query values from the data storage this exports parts of d
and makes them avaiable in a json file, so tests don't need to change to
make them exportable. Tests exported are the ones defined in TEST_SUITES regardless of layer.
Starting the target (in the default case qemu) is still handled by the build system, as before, so keep that in mind.
Simplest way to test this is with a qemu image that you manually start. For an already build image use this in local.conf:TEST_EXPORT_ONLY = "1"TEST_TARGET = "simpleremote"TEST_TARGET_IP = "192.168.7.2"TEST_SERVER_IP = "192.168.7.1"
Export the tests: bitbake core-image-sato -c testimage
Then: runqemu core-image-sato
And: cd tmp/testimage/core-image-sato && ./runexported.py testdata.json
In the default case (TEST_EXPORT_ONLY = "0") nothing really changes so no tests should start failing because of this.
Cheers,
Stefan
The following changes since commit 76f52a81f600fe7dfee0987db361e973b9ab71f8:
insane: Fix python function whitespace changes (2014-02-03 16:16:30 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib stefans/export
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/export
Stefan Stanacar (3):
testimage: add ability to export tests
lib/oeqa: add module for running tests outside of the build system oeqa/utils: targetbuild: don't use bb.fetch anymore
meta/classes/testimage.bbclass | 107 ++++++++++++++++++++++++---- meta/lib/oeqa/oetest.py | 17 ++--- meta/lib/oeqa/runexported.py | 140 +++++++++++++++++++++++++++++++++++++ meta/lib/oeqa/utils/targetbuild.py | 27 +++---- 4 files changed, 246 insertions(+), 45 deletions(-) create mode 100755 meta/lib/oeqa/runexported.py
Signature · 1 line
1.8.5.3
PATCH 1/3 testimage: add ability to export tests
Add the ability to export the tests so that they can run independently of the build system, as is required if you want to be able to hand the test execution off to a scheduler.
Booting/deployment of the target is still handled by the build system, as before, only the execution of the tests happens outside of the build system. Tests exported are the ones defined in TEST_SUITES.
No tests have been changed as interesting parts of the data store have been exported and tests can continue to query them as before. Small adjustments were made for a couple of oeqa modules though.
[YOCTO #5613]
--- meta/classes/testimage.bbclass | 107 +++++++++++++++++++++++++++++++++++------ meta/lib/oeqa/oetest.py | 17 +++---- 2 files changed, 98 insertions(+), 26 deletions(-)
meta/classes/testimage.bbclass +92 -15
@@ -25,15 +25,16 @@ TEST_LOG_DIR ?= "${WORKDIR}/testimage" +TEST_EXPORT_DIR ?= "${TMPDIR}/testimage/${PN}" +TEST_EXPORT_ONLY ?= "0" + DEFAULT_TEST_SUITES = "ping auto" DEFAULT_TEST_SUITES_pn-core-image-minimal = "ping" DEFAULT_TEST_SUITES_pn-core-image-sato = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg python" DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "ping ssh df connman syslog xorg scp vnc date perl ldd gcc rpm smart kernelmodule dmesg python" - TEST_SUITES ?= "${DEFAULT_TEST_SUITES}" TEST_QEMUBOOT_TIMEOUT ?= "1000" - TEST_TARGET ?= "qemu" TEST_TARGET_IP ?= "" TEST_SERVER_IP ?= ""
@@ -85,6 +86,71 @@ def get_tests_list(d): return testslist + +def exportTests(d,tc): + import json + import shutil + import pkgutil + + exportpath = d.getVar("TEST_EXPORT_DIR", True) + + savedata = {} + savedata["d"] = {} + savedata["target"] = {} + for key in tc.__dict__: + # special cases + if key != "d" and key != "target": + savedata[key] = getattr(tc, key) + savedata["target"]["ip"] = tc.target.ip or d.getVar("TEST_TARGET_IP", True) + savedata["target"]["server_ip"] = tc.target.server_ip or d.getVar("TEST_SERVER_IP", True) + + keys = [ key for key in d.keys() if not key.startswith("_") and not key.startswith("BB") \ + and not key.startswith("B_pn") and not key.startswith("do_") and not d.getVarFlag(key, "func")] + for key in keys: + try: + savedata["d"][key] = d.getVar(key, True) + except bb.data_smart.ExpansionError: + # we don't care about those anyway + pass + + with open(os.path.join(exportpath, "testdata.json"), "w") as f: + json.dump(savedata, f, skipkeys=True, indent=4, sort_keys=True) + + # now start copying files + # we'll basically copy everything under meta/lib/oeqa, with these exceptions + # - oeqa/targetcontrol.py - not needed + # - oeqa/selftest - something else + # That means: + # - all tests from oeqa/runtime defined in TEST_SUITES (including from other layers) + # - the contents of oeqa/utils and oeqa/runtime/files + # - oeqa/oetest.py and oeqa/runexport.py (this will get copied to exportpath not exportpath/oeqa) + # - __init__.py files + bb.utils.mkdirhier(os.path.join(exportpath, "oeqa/runtime/files")) + bb.utils.mkdirhier(os.path.join(exportpath, "oeqa/utils")) + # copy test modules, this should cover tests in other layers too + for t in tc.testslist: + mod = pkgutil.get_loader(t) + shutil.copy2(mod.filename, os.path.join(exportpath, "oeqa/runtime")) + # copy __init__.py files + oeqadir = pkgutil.get_loader("oeqa").filename + shutil.copy2(os.path.join(oeqadir, "__init__.py"), os.path.join(exportpath, "oeqa")) + shutil.copy2(os.path.join(oeqadir, "runtime/__init__.py"), os.path.join(exportpath, "oeqa/runtime")) + # copy oeqa/oetest.py and oeqa/runexported.py + shutil.copy2(os.path.join(oeqadir, "oetest.py"), os.path.join(exportpath, "oeqa")) + shutil.copy2(os.path.join(oeqadir, "runexported.py"), exportpath) + # copy oeqa/utils/*.py + for root, dirs, files in os.walk(os.path.join(oeqadir, "utils")): + for f in files: + if f.endswith(".py"): + shutil.copy2(os.path.join(root, f), os.path.join(exportpath, "oeqa/utils")) + # copy oeqa/runtime/files/* + for root, dirs, files in os.walk(os.path.join(oeqadir, "runtime/files")): + for f in files: + shutil.copy2(os.path.join(root, f), os.path.join(exportpath, "oeqa/runtime/files")) + + bb.plain("Exported tests to: %s" % exportpath) + + def testimage_main(d): import unittest import os
@@ -94,7 +160,11 @@ def testimage_main(d): from oeqa.targetcontrol import get_target_controller pn = d.getVar("PN", True) + export = oe.utils.conditional("TEST_EXPORT_ONLY", "1", True, False, d) bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True)) + if export: + bb.utils.remove(d.getVar("TEST_EXPORT_DIR", True), recurse=True) + bb.utils.mkdirhier(d.getVar("TEST_EXPORT_DIR", True)) # tests in TEST_SUITES become required tests # they won't be skipped even if they aren't suitable for a image (like xorg for minimal)
@@ -105,13 +175,18 @@ def testimage_main(d): # the robot dance target = get_target_controller(d) - class TestContext: + class TestContext(object): def __init__(self): self.d = d self.testslist = testslist self.testsrequired = testsrequired self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files") self.target = target + self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split() + self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split() + manifest = os.path.join(d.getVar("DEPLOY_DIR_IMAGE", True), d.getVar("IMAGE_LINK_NAME", True) + ".manifest") + with open(manifest) as f: + self.pkgmanifest = f.read() # test context tc = TestContext()
@@ -129,19 +204,21 @@ def testimage_main(d): try: target.start() - # run tests and get the results - starttime = time.time() - result = runTests(tc) - stoptime = time.time() - if result.wasSuccessful(): - bb.plain("%s - Ran %d test%s in %.3fs" % (pn, result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime)) - msg = "%s - OK - All required tests passed" % pn - skipped = len(result.skipped) - if skipped: - msg += " (skipped=%d)" % skipped - bb.plain(msg) + if export: + exportTests(d,tc) else: - raise bb.build.FuncFailed("%s - FAILED - check the task log and the ssh log" % pn ) + starttime = time.time() + result = runTests(tc) + stoptime = time.time() + if result.wasSuccessful(): + bb.plain("%s - Ran %d test%s in %.3fs" % (pn, result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime)) + msg = "%s - OK - All required tests passed" % pn + skipped = len(result.skipped) + if skipped: + msg += " (skipped=%d)" % skipped + bb.plain(msg) + else: + raise bb.build.FuncFailed("%s - FAILED - check the task log and the ssh log" % pn ) finally: target.stop()
meta/lib/oeqa/oetest.py +6 -11
@@ -10,8 +10,6 @@ import os, re, mmap import unittest import inspect -import bb -from oeqa.utils.sshcontrol import SSHControl def loadTests(tc):
@@ -31,15 +29,14 @@ def loadTests(tc): def runTests(tc): suite = loadTests(tc) - bb.note("Test modules %s" % tc.testslist) - bb.note("Found %s tests" % suite.countTestCases()) + print("Test modules %s" % tc.testslist) + print("Found %s tests" % suite.countTestCases()) runner = unittest.TextTestRunner(verbosity=2) result = runner.run(suite) return result - class oeTest(unittest.TestCase): longMessage = True
@@ -60,18 +57,16 @@ class oeTest(unittest.TestCase): @classmethod def hasPackage(self, pkg): - manifest = os.path.join(oeTest.tc.d.getVar("DEPLOY_DIR_IMAGE", True), oeTest.tc.d.getVar("IMAGE_LINK_NAME", True) + ".manifest") - with open(manifest) as f: - data = f.read() - if re.search(pkg, data): + + if re.search(pkg, oeTest.tc.pkgmanifest): return True return False @classmethod def hasFeature(self,feature): - if feature in oeTest.tc.d.getVar("IMAGE_FEATURES", True).split() or \ - feature in oeTest.tc.d.getVar("DISTRO_FEATURES", True).split(): + if feature in oeTest.tc.imagefeatures or \ + feature in oeTest.tc.distrofeatures: return True else: return False
Signature · 1 line
1.8.5.3
PATCH 2/3 lib/oeqa: add module for running tests outside of the build system
In reply to Stefan StanacarThis script will run the exported tests outside of the build system.
Simplest way to test this is with a qemu image that you manually start. For an already build image use this in local.conf:TEST_EXPORT_ONLY = "1"TEST_TARGET = "simpleremote"TEST_TARGET_IP = "192.168.7.2"TEST_SERVER_IP = "192.168.7.1"
Export the tests: bitbake core-image-sato -c testimage
Then: runqemu core-image-sato
And: cd build/tmp/testimage/core-image-sato
./runexported.py testdata.json
The contents of build/tmp/testimage/core-image-sato can be moved on another machine as long as some paths are updated in the json.
The exported data contains paths to the build dir. We only care about DEPLOY_DIR/rpm ( if the rpm and smart tests are enabled), so running the tests on other machine means that the user has to move the contents and call runexported with --deploy-dir PATH:
./runexported.py --deploy-dir /path/on/another/machine testdata.jsonrunexported.py accepts other arguments as well, see --help.
[YOCTO #5613]
--- meta/lib/oeqa/runexported.py | 140 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100755 meta/lib/oeqa/runexported.py
meta/lib/oeqa/runexported.py new file +140 -0
@@ -0,0 +1,140 @@ +#!/usr/bin/env python + + +# Copyright (C) 2013 Intel Corporation +# +# Released under the MIT license (see COPYING.MIT) + +# This script should be used outside of the build system to run image tests. +# It needs a json file as input as exported by the build. +# E.g for an already built image: +#- export the tests: +# TEST_EXPORT_ONLY = "1" +# TEST_TARGET = "simpleremote" +# TEST_TARGET_IP = "192.168.7.2" +# TEST_SERVER_IP = "192.168.7.1" +# bitbake core-image-sato -c testimage +# Setup your target, e.g for qemu: runqemu core-image-sato +# cd build/tmp/testimage/core-image-sato +# ./runexported.py testdata.json + +import sys +import os +import time +from optparse import OptionParser + +try: + import simplejson as json +except ImportError: + import json + +sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "oeqa"))) + +from oeqa.oetest import runTests +from oeqa.utils.sshcontrol import SSHControl + +# this isn't pretty but we need a fake target object +# for running the tests externally as we don't care +# about deploy/start we only care about the connection methods (run, copy) +class FakeTarget(object): + def __init__(self, d): + self.connection = None + self.ip = None + self.server_ip = None + self.datetime = time.strftime('%Y%m%d%H%M%S',time.gmtime()) + self.testdir = d.getVar("TEST_LOG_DIR", True) + self.pn = d.getVar("PN", True) + + def exportStart(self): + self.sshlog = os.path.join(self.testdir, "ssh_target_log.%s" % self.datetime) + sshloglink = os.path.join(self.testdir, "ssh_target_log") + if os.path.islink(sshloglink): + os.unlink(sshloglink) + os.symlink(self.sshlog, sshloglink) + print("SSH log file: %s" % self.sshlog) + self.connection = SSHControl(self.ip, logfile=self.sshlog) + + def run(self, cmd, timeout=None): + return self.connection.run(cmd, timeout) + + def copy_to(self, localpath, remotepath): + return self.connection.copy_to(localpath, remotepath) + + def copy_from(self, remotepath, localpath): + return self.connection.copy_from(remotepath, localpath) + + +class MyDataDict(dict): + def getVar(self, key, unused = None): + return self.get(key, "") + +class TestContext(object): + def __init__(self): + self.d = None + self.target = None + +def main(): + + usage = "usage: %prog [options] <json file>" + parser = OptionParser(usage=usage) + parser.add_option("-t", "--target-ip", dest="ip", help="The IP address of the target machine. Use this to \ + overwrite the value determined from TEST_TARGET_IP at build time") + parser.add_option("-s", "--server-ip", dest="server_ip", help="The IP address of this machine. Use this to \ + overwrite the value determined from TEST_SERVER_IP at build time.") + parser.add_option("-d", "--deploy-dir", dest="deploy_dir", help="Full path to the package feeds, that this \ + the contents of what used to be DEPLOY_DIR on the build machine. If not specified it will use the value \ + specified in the json if that directory actually exists or it will error out.") + parser.add_option("-l", "--log-dir", dest="log_dir", help="This sets the path for TEST_LOG_DIR. If not specified \ + the current dir is used. This is used for usually creating a ssh log file and a scp test file.") + + (options, args) = parser.parse_args() + if len(args) != 1: + parser.error("Incorrect number of arguments. The one and only argument should be a json file exported by the build system") + + with open(args[0], "r") as f: + loaded = json.load(f) + + if options.ip: + loaded["target"]["ip"] = options.ip + if options.server_ip: + loaded["target"]["server_ip"] = options.server_ip + + d = MyDataDict() + for key in loaded["d"].keys(): + d[key] = loaded["d"][key] + + if options.log_dir: + d["TEST_LOG_DIR"] = options.log_dir + else: + d["TEST_LOG_DIR"] = os.path.abspath(os.path.dirname(__file__)) + if options.deploy_dir: + d["DEPLOY_DIR"] = options.deploy_dir + else: + if not os.path.isdir(d["DEPLOY_DIR"]): + raise Exception("The path to DEPLOY_DIR does not exists: %s" % d["DEPLOY_DIR"]) + + + target = FakeTarget(d) + for key in loaded["target"].keys(): + setattr(target, key, loaded["target"][key]) + + tc = TestContext() + setattr(tc, "d", d) + setattr(tc, "target", target) + for key in loaded.keys(): + if key != "d" and key != "target": + setattr(tc, key, loaded[key]) + + target.exportStart() + runTests(tc) + + return 0 + +if __name__ == "__main__": + try: + ret = main() + except Exception: + ret = 1 + import traceback + traceback.print_exc(5) + sys.exit(ret)
Signature · 1 line
1.8.5.3
PATCH 3/3 oeqa/utils: targetbuild: don't use bb.fetch anymore
In reply to Stefan StanacarWhen running tests outside of the build system we can't use bb.fetch anymore. It was nice but tests and their modules need to rely on the data storage only as that gets exported. This module is used by the oeqa/runtime/build* tests.
--- meta/lib/oeqa/utils/targetbuild.py | 27 ++++++++------------------- 1 file changed, 8 insertions(+), 19 deletions(-)
meta/lib/oeqa/utils/targetbuild.py +8 -19
@@ -4,11 +4,9 @@ # Provides a class for automating build tests for projects -from oeqa.oetest import oeRuntimeTest -import bb.fetch2 -import bb.data import os import re +import subprocess class TargetBuildProject():
@@ -16,26 +14,16 @@ class TargetBuildProject(): def __init__(self, target, uri, foldername=None): self.target = target self.uri = uri - self.targetdir = "/home/root/" - - self.localdata = bb.data.createCopy(oeRuntimeTest.tc.d) - bb.data.update_data(self.localdata) - - if not foldername: - self.archive = os.path.basename(uri) - self.fname = re.sub(r'.tar.bz2|tar.gz$', '', self.archive) - else: + self.targetdir = "~/" + self.archive = os.path.basename(uri) + self.localarchive = "/tmp/" + self.archive + self.fname = re.sub(r'.tar.bz2|tar.gz$', '', self.archive) + if foldername: self.fname = foldername def download_archive(self): - try: - self.localdata.delVar("BB_STRICT_CHECKSUM") - fetcher = bb.fetch2.Fetch([self.uri], self.localdata) - fetcher.download() - self.localarchive = fetcher.localpath(self.uri) - except bb.fetch2.BBFetchException: - raise Exception("Failed to download archive: %s" % self.uri) + subprocess.check_call("wget -O %s %s" % (self.localarchive, self.uri), shell=True) (status, output) = self.target.copy_to(self.localarchive, self.targetdir) if status != 0:
@@ -61,3 +49,4 @@ class TargetBuildProject(): def clean(self): self.target.run('rm -rf %s' % self.targetdir) + subprocess.call('rm -f %s' % self.localarchive, shell=True)
Signature · 1 line
1.8.5.3