Proposed implementation of a new runtime tests framework
v2 cover Proposed implementation of a new runtime tests framework
Hello,
This is the proposed implementation of a new runtime tests framework based on python unittest..
It's mean to ease qemu image testing and encourage developers to add more tests similar to the example tests provided (all tests are basically commands ran over ssh)
You can try it out like this:
- first build a qemu core-image-sato (a minimal wouldn't be interesting at all)
- add INHERIT += "testimage" in local.conf
- then bitbake core-image-sato -c testimage. That will run a standard suite of tests.
You can set TEST_SUITES = "ping ssh <test name>" in local.conf to force run only certain tests (order matters here, it's the order in which tests run). You can also append "auto" and it will also run whatever tests are suitable for the image (if that was a sato-sdk image and you set TEST_SUITE = "ping ssh rpm auto" you force run ping, ssh and rpm but you also get smart, connman and gcc tests).
Check the task log (log.do_testimage) in WORKDIR to see the results. Also a ssh log (what command is running, output and return codes) and qemu boot log are kept in WORKDIR/testimage/
There are some areas for improvement:
- a better way of getting the list of installed packages in a image (currently it uses the installed_packages.txt file in WORKDIR)
- qemu is started with the -snapshot option, we should create a copy of the original rootfs instead (so that we can keep copy of the tested image)
- when using TEST_SUITES = "auto" there is NO dependency at all between tests (rpm would run before ssh test)
- right know the class assumes qemu machines, plan is to abstract it to allow provision of real hardware too.
- other I don't remember right now :)
Changed in v2:
- renamed oeTelnetlib to oeQemuConsole
- added some tests for xorg and systemd
- increased timeouts for starting qemu a little bit
- move the unittest runner from the task to oetest module
- print and check DISPLAY value
- some comments and strings changes
I've run this on my AB instance with this applied http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=stefans/yab-master2&id=9cef75a5cc360629dd4ad32e121d9a9936cf2b5f
Comments and feedback are most welcome!
Regards,
Stefan
The following changes since commit 8a186a6b3853fc1a7dcf342d421c8926c38949c9:
bitbake: hob: save button from settings called a nonexisting method (2013-07-03 08:13:35 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib stefans/oeqa7
for you to fetch changes up to cee1fb0376c3440106552800af179cfb9df97b96:
lib/oeqa/runtime: add gcc test (2013-07-05 11:36:08 +0300)
----------------------------------------------------------------
Radu Moisan (2):
lib/oeqa/utils/qemurunner.py: class to handle qemu instance lib/oeqa/utils/decorators.py: decorators for test methods
Stefan Stanacar (6):
classes/testimage.bbclass: new class for image tests lib/oeqa/oetest.py: base module for all runtime unittests lib/oeqa/utils/sshcontrol.py: helper module for running remote commands lib/oeqa/utils/oeqemuconsole.py: handle qemu serial console connection lib/oeqa/runtime: image sanity tests
lib/oeqa/runtime: add gcc test
meta/classes/testimage.bbclass | 92 +++++++++++++++++ meta/lib/oeqa/__init__.py | 0 meta/lib/oeqa/oetest.py | 101 +++++++++++++++++++ meta/lib/oeqa/runtime/__init__.py | 0 meta/lib/oeqa/runtime/connman.py | 34 +++++++ meta/lib/oeqa/runtime/dmesg.py | 13 +++ meta/lib/oeqa/runtime/files/test.c | 26 +++++ meta/lib/oeqa/runtime/files/testmakefile | 5 + meta/lib/oeqa/runtime/gcc.py | 36 +++++++ meta/lib/oeqa/runtime/multilib.py | 14 +++ meta/lib/oeqa/runtime/ping.py | 11 +++ meta/lib/oeqa/runtime/rpm.py | 25 +++++ meta/lib/oeqa/runtime/smart.py | 27 +++++ meta/lib/oeqa/runtime/ssh.py | 16 +++ meta/lib/oeqa/runtime/systemd.py | 24 +++++ meta/lib/oeqa/runtime/xorg.py | 21 ++++ meta/lib/oeqa/utils/__init__.py | 0 meta/lib/oeqa/utils/decorators.py | 40 ++++++++ meta/lib/oeqa/utils/oeqemuconsole.py | 45 +++++++++ meta/lib/oeqa/utils/qemurunner.py | 164 +++++++++++++++++++++++++++++++ meta/lib/oeqa/utils/sshcontrol.py | 100 +++++++++++++++++++ scripts/runqemu | 2 +- 22 files changed, 795 insertions(+), 1 deletion(-) create mode 100644 meta/classes/testimage.bbclass create mode 100644 meta/lib/oeqa/__init__.py create mode 100644 meta/lib/oeqa/oetest.py create mode 100644 meta/lib/oeqa/runtime/__init__.py create mode 100644 meta/lib/oeqa/runtime/connman.py create mode 100644 meta/lib/oeqa/runtime/dmesg.py create mode 100644 meta/lib/oeqa/runtime/files/test.c create mode 100644 meta/lib/oeqa/runtime/files/testmakefile create mode 100644 meta/lib/oeqa/runtime/gcc.py create mode 100644 meta/lib/oeqa/runtime/multilib.py create mode 100644 meta/lib/oeqa/runtime/ping.py create mode 100644 meta/lib/oeqa/runtime/rpm.py create mode 100644 meta/lib/oeqa/runtime/smart.py create mode 100644 meta/lib/oeqa/runtime/ssh.py create mode 100644 meta/lib/oeqa/runtime/systemd.py create mode 100644 meta/lib/oeqa/runtime/xorg.py create mode 100644 meta/lib/oeqa/utils/__init__.py create mode 100644 meta/lib/oeqa/utils/decorators.py create mode 100644 meta/lib/oeqa/utils/oeqemuconsole.py create mode 100644 meta/lib/oeqa/utils/qemurunner.py create mode 100644 meta/lib/oeqa/utils/sshcontrol.py
Radu Moisan (2):
lib/oeqa/utils/qemurunner.py: class to handle qemu instance lib/oeqa/utils/decorators.py: decorators for test methods
Stefan Stanacar (6):
classes/testimage.bbclass: new class for image tests lib/oeqa/oetest.py: base module for all runtime unittests lib/oeqa/utils/sshcontrol.py: helper module for running remote
commands
lib/oeqa/utils/oeqemuconsole.py: handle qemu serial console connection lib/oeqa/runtime: image sanity tests
lib/oeqa/runtime: add gcc test
meta/classes/testimage.bbclass | 92 +++++++++++++++++ meta/lib/oeqa/__init__.py | 0 meta/lib/oeqa/oetest.py | 101 +++++++++++++++++++ meta/lib/oeqa/runtime/__init__.py | 0 meta/lib/oeqa/runtime/connman.py | 34 +++++++ meta/lib/oeqa/runtime/dmesg.py | 13 +++ meta/lib/oeqa/runtime/files/test.c | 26 +++++ meta/lib/oeqa/runtime/files/testmakefile | 5 + meta/lib/oeqa/runtime/gcc.py | 36 +++++++ meta/lib/oeqa/runtime/multilib.py | 14 +++ meta/lib/oeqa/runtime/ping.py | 11 +++ meta/lib/oeqa/runtime/rpm.py | 25 +++++ meta/lib/oeqa/runtime/smart.py | 27 +++++ meta/lib/oeqa/runtime/ssh.py | 16 +++ meta/lib/oeqa/runtime/systemd.py | 24 +++++ meta/lib/oeqa/runtime/xorg.py | 21 ++++ meta/lib/oeqa/utils/__init__.py | 0 meta/lib/oeqa/utils/decorators.py | 40 ++++++++ meta/lib/oeqa/utils/oeqemuconsole.py | 45 +++++++++ meta/lib/oeqa/utils/qemurunner.py | 164 +++++++++++++++++++++++++++++++ meta/lib/oeqa/utils/sshcontrol.py | 100 +++++++++++++++++++ scripts/runqemu | 2 +- 22 files changed, 795 insertions(+), 1 deletion(-) create mode 100644 meta/classes/testimage.bbclass create mode 100644 meta/lib/oeqa/__init__.py create mode 100644 meta/lib/oeqa/oetest.py create mode 100644 meta/lib/oeqa/runtime/__init__.py create mode 100644 meta/lib/oeqa/runtime/connman.py create mode 100644 meta/lib/oeqa/runtime/dmesg.py create mode 100644 meta/lib/oeqa/runtime/files/test.c create mode 100644 meta/lib/oeqa/runtime/files/testmakefile create mode 100644 meta/lib/oeqa/runtime/gcc.py create mode 100644 meta/lib/oeqa/runtime/multilib.py create mode 100644 meta/lib/oeqa/runtime/ping.py create mode 100644 meta/lib/oeqa/runtime/rpm.py create mode 100644 meta/lib/oeqa/runtime/smart.py create mode 100644 meta/lib/oeqa/runtime/ssh.py create mode 100644 meta/lib/oeqa/runtime/systemd.py create mode 100644 meta/lib/oeqa/runtime/xorg.py create mode 100644 meta/lib/oeqa/utils/__init__.py create mode 100644 meta/lib/oeqa/utils/decorators.py create mode 100644 meta/lib/oeqa/utils/oeqemuconsole.py create mode 100644 meta/lib/oeqa/utils/qemurunner.py create mode 100644 meta/lib/oeqa/utils/sshcontrol.py
Signature · 1 line
1.8.1.4
RFC PATCH v2 1/8 classes/testimage.bbclass: new class for image tests
Replacement class for imagetest-qemu.bbclass. It launches a qemu instance and runs test modules defined in TEST_SUITES.
--- meta/classes/testimage.bbclass | 92 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 meta/classes/testimage.bbclass
meta/classes/testimage.bbclass new file +92 -0
@@ -0,0 +1,92 @@ +TEST_LOG_DIR ?= "${WORKDIR}/testimage" + +DEFAULT_TEST_SUITES = "ping auto" +DEFAULT_TEST_SUITES_pn-core-image-minimal = "ping" +DEFAULT_TEST_SUITES_pn-core-image-sato = "ping ssh connman rpm smart xorg dmesg" +DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "ping ssh connman rpm smart gcc xorg dmesg" + +TEST_SUITES ?= "${DEFAULT_TEST_SUITES}" + +python do_testimage() { + testimage_main(d) +} +addtask testimage +do_testimage[nostamp] = "1" +do_testimage[depends] += "qemu-native:do_populate_sysroot" +do_testimage[depends] += "qemu-helper-native:do_populate_sysroot" + +def testimage_main(d): + import unittest + import os + import oeqa.runtime + import re + from oeqa.oetest import runTests + from oeqa.utils.sshcontrol import SSHControl + from oeqa.utils.qemurunner import QemuRunner + + testdir = d.getVar("TEST_LOG_DIR", True) + bb.utils.mkdirhier(testdir) + sshlog = os.path.join(testdir, "ssh_target_log.%s" % d.getVar('DATETIME', True)) + sshloglink = os.path.join(testdir, "ssh_target_log") + if os.path.islink(sshloglink): + os.unlink(sshloglink) + os.symlink(sshlog, sshloglink) + + # tests in TEST_SUITES become required tests + # they won't be skipped even if they aren't suitable for a default image (like xorg for minimal) + testsuites = d.getVar("TEST_SUITES", True) + # testslist is what we'll run and order matters + testslist = [ x for x in testsuites.split() if x != "auto" ] + # if we have auto search for other modules + if "auto" in testsuites: + for f in os.listdir(os.path.dirname(os.path.abspath(oeqa.runtime.__file__))): + if f.endswith('.py') and not f.startswith('_') and f[:-3] not in testslist: + testslist.append(f[:-3]) + + testslist = [ "oeqa.runtime." + x for x in testslist ] + + class TestContext: + def __init__(self): + self.d = d + self.testslist = testslist + self.testsrequired = testsuites.split() + self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files") + + # test context + tc = TestContext() + + # prepare qemu instance + # and boot each supported fs type + machine=d.getVar("MACHINE", True) + #will handle fs type eventually, stick with ext3 for now + rootfs=d.getVar("DEPLOY_DIR_IMAGE", True) + '/' + d.getVar("IMAGE_BASENAME",True) + '-' + machine + '.ext3' + + qemu = QemuRunner(machine, rootfs) + qemu.tmpdir = d.getVar("TMPDIR", True) + qemu.display = d.getVar("BB_ORIGENV", False).getVar("DISPLAY", True) + qemu.logfile = os.path.join(testdir, "qemu_boot_log.%s" % d.getVar('DATETIME', True)) + + bb.note("DISPLAY value: %s" % qemu.display) + bb.note("rootfs file: %s" % rootfs) + bb.note("Qemu logfile: %s" % qemu.logfile) + + #catch exceptions when loading or running tests (mostly our own errors) + try: + if qemu.launch(): + + # set more context - ssh instance and qemu + # we do these here because we needed qemu to boot and get the ip + tc.qemu = qemu + tc.target = SSHControl(host=qemu.ip,logfile=sshlog) + # run tests and get the results + result = runTests(tc) + + if result.wasSuccessful(): + bb.note("All required tests passed") + else: + raise bb.build.FuncFailed("Some tests failed. You should check the task log and the ssh log. (ssh log is %s" % sshlog) + + else: + raise bb.build.FuncFailed("Failed to start qemu. You should check the task log and the qemu boot log (qemu log is %s)" % qemu.logfile) + finally: + qemu.kill()
Signature · 1 line
1.8.1.4
RFC PATCH v2 2/8 lib/oeqa/oetest.py: base module for all runtime unittests
In reply to Stefan StanacarThis module contains the base class for all runtime tests and some helper methods.
--- meta/lib/oeqa/__init__.py | 0 meta/lib/oeqa/oetest.py | 101 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 meta/lib/oeqa/__init__.py create mode 100644 meta/lib/oeqa/oetest.py
meta/lib/oeqa/__init__.py new file +0 -0
meta/lib/oeqa/oetest.py new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
1.8.1.4
RFC PATCH v2 3/8 lib/oeqa/utils/sshcontrol.py: helper module for running remote commands
In reply to Stefan StanacarProvides a class for setting up ssh connections, running commands and copying files to/from a target.
--- meta/lib/oeqa/utils/__init__.py | 0 meta/lib/oeqa/utils/sshcontrol.py | 100 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+) create mode 100644 meta/lib/oeqa/utils/__init__.py create mode 100644 meta/lib/oeqa/utils/sshcontrol.py
meta/lib/oeqa/utils/__init__.py new file +0 -0
meta/lib/oeqa/utils/sshcontrol.py new file +100 -0
100 lines not on this page · raw message
Signature · 1 line
1.8.1.4
RFC PATCH v2 4/8 lib/oeqa/utils/oeqemuconsole.py: handle qemu serial console connection
In reply to Stefan StanacarPython's telnetlib Telnet class connects only to AF_INET sockets, but we want to use Unix domain socket for the qemu serial connection, so that's why we override it.
Also we add a new read_all_timeout method similar to Telnet's read_all, that read until a match or timeout and logs all output.
--- meta/lib/oeqa/utils/oeqemuconsole.py | 45 ++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 meta/lib/oeqa/utils/oeqemuconsole.py
meta/lib/oeqa/utils/oeqemuconsole.py new file +45 -0
@@ -0,0 +1,45 @@ +import socket +import time +import re +from telnetlib import Telnet + +class oeQemuConsole(Telnet): + + """ + Override Telnet class to use unix domain sockets, + Telnet uses AF_INET for socket, we don't want that. + Also, provide a read_all variant with timeout, that + returns whatever output there is. + """ + + def __init__(self, stream, logfile): + + Telnet.__init__(self, host=None) + self.stream = stream + self.logfile = logfile + self.eof = 0 + self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self.sock.connect(stream) + + def log(self, msg): + if self.logfile: + with open(self.logfile, "a") as f: + f.write("%s\n" % msg) + + + def read_all_timeout(self, match, timeout=200): + """Read until EOF or until timeout or until match. + """ + ret = False + self.process_rawq() + endtime = time.time() + timeout + while not self.eof and time.time() < endtime: + self.fill_rawq() + self.process_rawq() + if re.search(match, self.cookedq): + ret = True + break + buf = self.cookedq + self.cookedq = '' + self.log(buf) + return (ret, buf)
Signature · 1 line
1.8.1.4
RFC PATCH v2 5/8 lib/oeqa/utils/qemurunner.py: class to handle qemu instance
In reply to Stefan StanacarFrom: Radu Moisan <[email protected]>
Handles qemu instances (launch, kill, restart, serial connection, logging) Launch is blocking until login prompt and returns to the task. A qemu serial connection is used to save the boot log and get the ip from the image. Changed runqemu script not to error out when using custom serial option.
--- meta/lib/oeqa/utils/qemurunner.py | 164 ++++++++++++++++++++++++++++++++++++++ scripts/runqemu | 2 +- 2 files changed, 165 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/utils/qemurunner.py
meta/lib/oeqa/utils/qemurunner.py new file +164 -0
@@ -0,0 +1,164 @@ +import subprocess +import optparse +import sys +import os +import time +import signal +import re +import bb +from oeqa.utils.oeqemuconsole import oeQemuConsole + +class QemuRunner: + + def __init__(self, machine, rootfs, display = None, tmpdir = None, logfile = None): + # Popen object + self.runqemu = None + + self.machine = machine + self.rootfs = rootfs + + self.streampath = '/tmp/qemuconnection.%s' % os.getpid() + self.qemuparams = 'bootparams="console=ttyS0" qemuparams="-snapshot -serial unix:%s,server,nowait"' % self.streampath + self.qemupid = None + self.ip = None + + self.display = display + self.tmpdir = tmpdir + self.logfile = logfile + + def launch(self, qemuparams = None): + + if qemuparams: + self.qemuparams = self.qemuparams[:-1] + " " + qemuparams + " " + '\"' + + if self.display: + os.environ["DISPLAY"] = self.display + else: + bb.error("To start qemu I need a X desktop, please set DISPLAY correctly (e.g. DISPLAY=:1)") + return False + if not os.path.exists(self.rootfs): + bb.error("Invalid rootfs %s" % self.rootfs) + return False + if not os.path.exists(self.tmpdir): + bb.error("Invalid TMPDIR path %s" % self.tmpdir) + return False + else: + os.environ["OE_TMPDIR"] = self.tmpdir + + launch_cmd = 'runqemu %s %s %s' % (self.machine, self.rootfs, self.qemuparams) + self.runqemu = subprocess.Popen(launch_cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,preexec_fn=os.setpgrp) + + bb.note("runqemu started, pid is %s" % self.runqemu.pid) + # wait at most 30 seconds until qemu pid appears + bb.note("waiting at most 60 seconds for qemu pid") + endtime = time.time() + 60 + while not self.is_alive() and time.time() < endtime: + time.sleep(1) + + if self.is_alive(): + bb.note("qemu started - qemu procces pid is %s" % self.qemupid) + + console = oeQemuConsole(self.streampath, self.logfile) + bb.note("Waiting at most 200 seconds for login banner") + (match, text) = console.read_all_timeout("login:", 200) + + if match: + bb.note("Reached login banner") + console.write("root\n") + (index, match, text) = console.expect([r"(root@[\w-]+:~#)"],10) + if not match: + bb.note("Couldn't get prompt, all I got was:\n%s" % match.group(0)) + return False + console.write("ip addr show eth0 | sed -n '3p' | awk '{ print $2 }' | cut -f 1 -d \"/\"\n") + (index, match, text) = console.expect([r"((?:[0-9]{1,3}\.){3}[0-9]{1,3})"],10) + console.close() + if match: + self.ip = match.group(0) + bb.note("Ip found: %s" % self.ip) + else: + bb.note("Couldn't determine ip, all I got was:\n%s" % text) + return False + else: + console.close() + bb.note("Target didn't reached login boot in 120 seconds") + lines = "\n".join(text.splitlines()[-5:]) + bb.note("Last 5 lines of text:\n%s" % lines) + bb.note("Check full boot log: %s" % self.logfile) + return False + else: + bb.note("Qemu pid didn't appeared in 30 seconds") + self.runqemu.terminate() + self.runqemu.kill() + bb.note("Output from runqemu: %s " % self.runqemu.stdout.read()) + self.runqemu.stdout.close() + return False + + return self.is_alive() + + + def kill(self): + if self.runqemu: + os.kill(-self.runqemu.pid,signal.SIGTERM) + self.qemupid = None + self.ip = None + if os.path.exists(self.streampath): + os.remove(self.streampath) + + def restart(self, qemuparams = None): + if self.is_alive(): + self.kill() + bb.note("Qemu Restart required...") + return self.launch(qemuparams) + + def is_alive(self): + qemu_child = self.find_child(str(self.runqemu.pid)) + if qemu_child: + self.qemupid = qemu_child[0] + if os.path.exists("/proc/" + str(self.qemupid)) and os.path.exists(self.streampath): + return True + return False + + def find_child(self,parent_pid): + # + # Walk the process tree from the process specified looking for a qemu-system. Return its [pid'cmd] + # + ps = subprocess.Popen(['ps', 'axww', '-o', 'pid,ppid,command'], stdout=subprocess.PIPE).communicate()[0] + processes = ps.split('\n') + nfields = len(processes[0].split()) - 1 + pids = {} + commands = {} + for row in processes[1:]: + data = row.split(None, nfields) + if len(data) != 3: + continue + if data[1] not in pids: + pids[data[1]] = [] + + pids[data[1]].append(data[0]) + commands[data[0]] = data[2] + + if parent_pid not in pids: + sys.stderr.write("No children found matching %s\n" % parent_pid) + return [] + + parents = [] + newparents = pids[parent_pid] + while newparents: + next = [] + for p in newparents: + if p in pids: + for n in pids[p]: + if n not in parents and n not in next: + next.append(n) + if p not in parents: + parents.append(p) + newparents = next + #print "Children matching %s:" % str(parents) + for p in parents: + # Need to be careful here since runqemu-internal runs "ldd qemu-system-xxxx" + # Also, old versions of ldd (2.11) run "LD_XXXX qemu-system-xxxx" + basecmd = commands[p].split()[0] + basecmd = os.path.basename(basecmd) + if "qemu-system" in basecmd and "-serial unix" in commands[p]: + return [int(p),commands[p]] +
scripts/runqemu +1 -1
@@ -156,7 +156,7 @@ while true; do serial_option=`expr "$SCRIPT_QEMU_EXTRA_OPT" : '.*\(-serial\)'` kvm_option=`expr "$SCRIPT_QEMU_EXTRA_OPT" : '.*\(-enable-kvm\)'` [ ! -z "$serial_option" -o ! -z "$kvm_option" ] && \ - error "Please use simplified serial or kvm options instead" + echo "Please use simplified serial or kvm options instead" ;; "bootparams="*) SCRIPT_KERNEL_OPT="$SCRIPT_KERNEL_OPT ${arg##bootparams=}"
Signature · 1 line
1.8.1.4
RFC PATCH v2 6/8 lib/oeqa/utils/decorators.py: decorators for test methods
In reply to Stefan StanacarFrom: Radu Moisan <[email protected]>
Some skip decorators meant only for test methods, providing some kind of test methods dependency.
They are used together with a test method name not a condition. These are complementary to python's unittest skip decorators.
--- meta/lib/oeqa/utils/decorators.py | 40 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 meta/lib/oeqa/utils/decorators.py
meta/lib/oeqa/utils/decorators.py new file +40 -0
@@ -0,0 +1,40 @@ +from oeqa.oetest import * + +class skipIfFailure(object): + + def __init__(self,testcase): + self.testcase = testcase + + def __call__(self,f): + def wrapped_f(*args): + if self.testcase in (oeRuntimeTest.testFailures or oeRuntimeTest.testErrors): + raise unittest.SkipTest("Testcase dependency not met: %s" % self.testcase) + f(*args) + wrapped_f.__name__ = f.__name__ + return wrapped_f + +class skipIfSkipped(object): + + def __init__(self,testcase): + self.testcase = testcase + + def __call__(self,f): + def wrapped_f(*args): + if self.testcase in oeRuntimeTest.testSkipped: + raise unittest.SkipTest("Testcase dependency not met: %s" % self.testcase) + f(*args) + wrapped_f.__name__ = f.__name__ + return wrapped_f + +class skipUnlessPassed(object): + + def __init__(self,testcase): + self.testcase = testcase + + def __call__(self,f): + def wrapped_f(*args): + if self.testcase in (oeRuntimeTest.testSkipped, oeRuntimeTest.testFailures, oeRuntimeTest.testErrors): + raise unittest.SkipTest("Testcase dependency not met: %s" % self.testcase) + f(*args) + wrapped_f.__name__ = f.__name__ + return wrapped_f
Signature · 1 line
1.8.1.4
RFC PATCH v2 7/8 lib/oeqa/runtime: image sanity tests
In reply to Stefan StanacarThese are basic sanity tests. A test can be force run by setting TEST_SUITES = "ping ssh <module-name>" in local.conf. By default there are suites for minimal, sato and sato-sdk images.
--- meta/lib/oeqa/runtime/__init__.py | 0 meta/lib/oeqa/runtime/connman.py | 34 ++++++++++++++++++++++++++++++++++ meta/lib/oeqa/runtime/dmesg.py | 13 +++++++++++++ meta/lib/oeqa/runtime/multilib.py | 14 ++++++++++++++ meta/lib/oeqa/runtime/ping.py | 11 +++++++++++ meta/lib/oeqa/runtime/rpm.py | 25 +++++++++++++++++++++++++ meta/lib/oeqa/runtime/smart.py | 27 +++++++++++++++++++++++++++ meta/lib/oeqa/runtime/ssh.py | 16 ++++++++++++++++ meta/lib/oeqa/runtime/systemd.py | 24 ++++++++++++++++++++++++ meta/lib/oeqa/runtime/xorg.py | 21 +++++++++++++++++++++ 10 files changed, 185 insertions(+) create mode 100644 meta/lib/oeqa/runtime/__init__.py create mode 100644 meta/lib/oeqa/runtime/connman.py create mode 100644 meta/lib/oeqa/runtime/dmesg.py create mode 100644 meta/lib/oeqa/runtime/multilib.py create mode 100644 meta/lib/oeqa/runtime/ping.py create mode 100644 meta/lib/oeqa/runtime/rpm.py create mode 100644 meta/lib/oeqa/runtime/smart.py create mode 100644 meta/lib/oeqa/runtime/ssh.py create mode 100644 meta/lib/oeqa/runtime/systemd.py create mode 100644 meta/lib/oeqa/runtime/xorg.py
meta/lib/oeqa/runtime/__init__.py new file +0 -0
meta/lib/oeqa/runtime/connman.py new file +34 -0
@@ -0,0 +1,34 @@ +import unittest +from oeqa.oetest import oeRuntimeTest, skipModule +from oeqa.utils.decorators import * + +def setUpModule(): + if not oeRuntimeTest.hasPackage("connman"): + skipModule("No connman package in image") + + +class ConnmanTest(oeRuntimeTest): + + @skipUnlessPassed('test_ssh') + def test_connmand_help(self): + (status, output) = self.target.run('/usr/sbin/connmand --help') + self.assertEqual(status, 0, msg="status and output: %s and %s" % (status,output)) + + + @skipUnlessPassed('test_connmand_help') + def test_connmand_running(self): + status = self.target.run('ls -l `which ps` | grep busybox')[0] + if status == 0: + oeRuntimeTest.pscmd = 'ps' + else: + oeRuntimeTest.pscmd = 'ps -ef' + (status, output) = self.target.run(oeRuntimeTest.pscmd + ' | grep [c]onnmand') + self.assertEqual(status, 0, msg="no connmand process, ps output: %s" % self.target.run(oeRuntimeTest.pscmd)[1]) + + @skipUnlessPassed('test_connmand_running') + def test_connmand_unique(self): + self.target.run('/usr/sbin/connmand') + output = self.target.run(oeRuntimeTest.pscmd + ' | grep -c [c]onnmand')[1] + self.assertEqual(output, "1", msg="more than one connmand running in background, ps output: %s\n%s" % (output, self.target.run(oeRuntimeTest.pscmd)[1])) + +
meta/lib/oeqa/runtime/dmesg.py new file +13 -0
@@ -0,0 +1,13 @@ +import unittest +from oeqa.oetest import oeRuntimeTest +from oeqa.utils.decorators import * + +def setUpModule(): + skipModuleUnless(oeRuntimeTest.tc.target.run('which dmesg')[0] == 0, "No dmesg in image or no connection") + +class DmesgTest(oeRuntimeTest): + + @skipUnlessPassed('test_ssh') + def test_dmesg(self): + (status, output) = self.target.run('dmesg | grep -v mmci-pl18x | grep -i error') + self.assertEqual(status, 1, msg = "Error messages in dmesg log: %s" % output)
meta/lib/oeqa/runtime/multilib.py new file +14 -0
@@ -0,0 +1,14 @@ +import unittest +from oeqa.oetest import oeRuntimeTest, skipModule + +def setUpModule(): + multilibs = oeRuntimeTest.tc.d.getVar("MULTILIBS", True) or "" + if "multlib:lib32" not in multilibs: + skipModule("this isn't a multilib:lib32 image") + + +class MultilibFileTest(oeRuntimeTest): + + def test_file_connman(self): + (status, output) = self.target.run('file -L /usr/sbin/connmand | grep "ELF 32-bit LSB executable"') + self.assertEqual(status, 0, msg="status and output : %s and %s" % (status,output))
meta/lib/oeqa/runtime/ping.py new file +11 -0
@@ -0,0 +1,11 @@ +import subprocess +import unittest +import sys +from oeqa.oetest import oeRuntimeTest + +class PingTest(oeRuntimeTest): + + def test_ping(self): + status = subprocess.call("ping -w 30 -c 1 %s" % oeRuntimeTest.tc.qemu.ip, shell=True, stdout=subprocess.PIPE) + self.assertEqual(status, 0) +
meta/lib/oeqa/runtime/rpm.py new file +25 -0
@@ -0,0 +1,25 @@ +import unittest +from oeqa.oetest import oeRuntimeTest, skipModule +from oeqa.utils.decorators import * + +def setUpModule(): + if not oeRuntimeTest.hasFeature("package-management"): + skipModule("rpm module skipped: target doesn't have package-management in IMAGE_FEATURES") + if "package_rpm" != oeRuntimeTest.tc.d.getVar("PACKAGE_CLASSES", True).split()[0]: + skipModule("rpm module skipped: target doesn't have rpm as primary package manager") + + +class RpmHelpTest(oeRuntimeTest): + + @skipUnlessPassed('test_ssh') + def test_rpm_help(self): + (status, output) = self.target.run('rpm --help') + self.assertEqual(status, 0, msg="status and output: %s and %s" % (status,output)) + +class RpmQueryTest(oeRuntimeTest): + + @skipUnlessPassed('test_rpm_help') + def test_rpm_query(self): + (status, output) = self.target.run('rpm -q rpm') + self.assertEqual(status, 0, msg="status and output: %s and %s" % (status,output)) +
meta/lib/oeqa/runtime/smart.py new file +27 -0
@@ -0,0 +1,27 @@ +import unittest +from oeqa.oetest import oeRuntimeTest +from oeqa.utils.decorators import * + +def setUpModule(): + if not oeRuntimeTest.hasFeature("package-management"): + skipModule("Image doesn't have package management feature") + if not oeRuntimeTest.hasPackage("smart"): + skipModule("Image doesn't have smart installed") + +class SmartHelpTest(oeRuntimeTest): + + def test_smart_help(self): + status = self.target.run('smart --help')[0] + self.assertEqual(status, 0) + +class SmartQueryTest(oeRuntimeTest): + + @skipUnlessPassed('test_smart_help') + def test_smart_query(self): + (status, output) = self.target.run('smart query rpm') + self.assertEqual(status, 0, msg="smart query failed, output: %s" % output) + + @skipUnlessPassed('test_smart_query') + def test_smart_info(self): + (status, output) = self.target.run('smart info rpm') + self.assertEqual(status, 0, msg="smart info rpm failed, output: %s" % output)
meta/lib/oeqa/runtime/ssh.py new file +16 -0
@@ -0,0 +1,16 @@ +import subprocess +import unittest +import sys +from oeqa.oetest import oeRuntimeTest, skipModule +from oeqa.utils.decorators import * + +def setUpModule(): + if not (oeRuntimeTest.hasPackage("dropbear") or oeRuntimeTest.hasPackage("openssh")): + skipModule("No ssh package in image") + +class SshTest(oeRuntimeTest): + + @skipUnlessPassed('test_ping') + def test_ssh(self): + (status, output) = self.target.run('uname -a') + self.assertEqual(status, 0, msg="SSH Test failed: %s" % output)
meta/lib/oeqa/runtime/systemd.py new file +24 -0
@@ -0,0 +1,24 @@ +import unittest +from oeqa.oetest import oeRuntimeTest, skipModule +from oeqa.utils.decorators import * + +def setUpModule(): + if not oeRuntimeTest.hasFeature("systemd"): + skipModule("target doesn't have systemd in DISTRO_FEATURES") + if "systemd" != oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager", True): + skipModule("systemd is not the init manager for this image") + + +class SystemdBasicTest(oeRuntimeTest): + + @skipUnlessPassed('test_ssh') + def test_systemd_version(self): + (status, output) = self.target.run('systemctl --version') + self.assertEqual(status, 0, msg="status and output: %s and %s" % (status,output)) + +class SystemdTests(oeRuntimeTest): + + @skipUnlessPassed('test_systemd_version') + def test_systemd_failed(self): + (status, output) = self.target.run('systemctl --failed | grep "0 loaded units listed"') + self.assertEqual(status, 0, msg="Failed systemd services: %s" % self.target.run('systemctl --failed')[1])
meta/lib/oeqa/runtime/xorg.py new file +21 -0
@@ -0,0 +1,21 @@ +import unittest +from oeqa.oetest import oeRuntimeTest, skipModule +from oeqa.utils.decorators import * + +def setUpModule(): + if not oeRuntimeTest.hasFeature("x11-base"): + skipModule("target doesn't have x11 in IMAGE_FEATURES") + + +class XorgTest(oeRuntimeTest): + + @skipUnlessPassed('test_ssh') + def test_xorg_running(self): + (status, output) = self.target.run('ps | grep -v xinit | grep [X]org') + self.assertEqual(status, 0, msg="Xorg does not appear to be running %s" % self.target.run('ps')[1]) + + @skipUnlessPassed('test_ssh') + def test_xorg_error(self): + (status, output) = self.target.run('cat /var/log/Xorg.0.log | grep -v "(EE) error," | grep -v "PreInit" | grep -v "evdev:" | grep -v "glx" | grep "(EE)"') + self.assertEqual(status, 1, msg="Errors in Xorg log: %s" % output) +
Signature · 1 line
1.8.1.4
RFC PATCH v2 8/8 lib/oeqa/runtime: add gcc test
In reply to Stefan Stanacargcc compile test and support files.
--- meta/lib/oeqa/runtime/files/test.c | 26 +++++++++++++++++++++++ meta/lib/oeqa/runtime/files/testmakefile | 5 +++++ meta/lib/oeqa/runtime/gcc.py | 36 ++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) create mode 100644 meta/lib/oeqa/runtime/files/test.c create mode 100644 meta/lib/oeqa/runtime/files/testmakefile create mode 100644 meta/lib/oeqa/runtime/gcc.py
meta/lib/oeqa/runtime/files/test.c new file +26 -0
@@ -0,0 +1,26 @@ +#include <stdio.h> +#include <math.h> +#include <stdlib.h> + +double convert (long long l) +{ +return (double )l; +} + +int main (int argc,char * argv[]) { + +long long l =10 ; +double f; +double check =10.0 ; + + f =convert (l); +printf ("convert: %lld => %f\n" , l, f); +if ( f != check )exit (1 ); + + f =1234.67 ; + check =1234.0 ; +printf ("floorf(%f) = %f\n" , f,floorf (f)); +if (floorf (f) != check)exit (1 ); + +return 0 ; +}
meta/lib/oeqa/runtime/files/testmakefile new file +5 -0
@@ -0,0 +1,5 @@ +test: test.o + gcc -o test test.o -lm +test.o: test.c + gcc -c test.c +
meta/lib/oeqa/runtime/gcc.py new file +36 -0
@@ -0,0 +1,36 @@ +import unittest +import os +from oeqa.oetest import oeRuntimeTest, skipModule +from oeqa.utils.decorators import * + +def setUpModule(): + if not oeRuntimeTest.hasFeature("tools-sdk"): + skipModule("Image doesn't have tools-sdk in IMAGE_FEATURES") + + +class GccCompileTest(oeRuntimeTest): + + @classmethod + def setUpClass(self): + oeRuntimeTest.tc.target.copy_to(os.path.join(oeRuntimeTest.tc.filesdir, "test.c"), "/tmp/test.c") + oeRuntimeTest.tc.target.copy_to(os.path.join(oeRuntimeTest.tc.filesdir, "testmakefile"), "/tmp/testmakefile") + + def test_gcc_compile(self): + (status, output) = self.target.run('gcc /tmp/test.c -o /tmp/test -lm') + self.assertEqual(status, 0, msg="gcc compile failed, output: %s" % output) + (status, output) = self.target.run('/tmp/test') + self.assertEqual(status, 0, msg="running compiled file failed, output %s" % output) + + def test_gpp_compile(self): + (status, output) = self.target.run('g++ /tmp/test.c -o /tmp/test -lm') + self.assertEqual(status, 0, msg="g++ compile failed, output: %s" % output) + (status, output) = self.target.run('/tmp/test') + self.assertEqual(status, 0, msg="running compiled file failed, output %s" % output) + + def test_make(self): + (status, output) = self.target.run('cd /tmp; make -f testmakefile') + self.assertEqual(status, 0, msg="running make failed, output %s" % output) + + @classmethod + def tearDownClass(self): + oeRuntimeTest.tc.target.run("rm /tmp/test.c /tmp/test.o /tmp/test /tmp/testmakefile")
Signature · 1 line
1.8.1.4
Re: [RFC PATCH v2 3/8] lib/oeqa/utils/sshcontrol.py: helper module for running remote commands
In reply to Stefan StanacarQuoted 1 line · Stefan Stanacar
+ actualcmd = "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root %s '%s'" % (self.host, cmd)
This is going to fail if cmd contains a single quote. Personally, I *always* pass around subprocess arguments as an array. Anything else is going to lead to quoting problems.
Re: [RFC PATCH v2 3/8] lib/oeqa/utils/sshcontrol.py: helper module for running remote commands
Hi Colin,
Quoted 10 lines · Colin Walters
Quoted 1 line · Stefan Stanacar
+ actualcmd = "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root %s '%s'" % (self.host, cmd)
This is going to fail if cmd contains a single quote. Personally, I *always* pass around subprocess arguments as an array. Anything else is going to lead to quoting problems.
You're right, I just sent a patch that drops that and doesn't use shlex anymore.
Thanks,
Stefan