device-crash-test: device_add test mode
cover device-crash-test: device_add test mode
This series implements device_add testing on device-crash-test. The new test mode will be enabled by default in addition to the old test cases, but if only device_add testing is wanted, it can be specified using the '-t' command-line option, e.g.:
$ ./scripts/device-crash-test -t method=device_add
In comparison to the "-device" test cases, a full test run runs reasonably quickly (17 seconds on my machine for testing all hotpluggable devices on all 72 machine-types in qemu-system-x86_64). I believe it is a good candidate to be eventually included on "make check".
TODO:
* Fix the QEMU crash found using '-t plug_all=1' (see patch 5/5 for details)
* Test non-x86 architectures
* Compare results with Thomas' HMP unit test ("tests: Add a device_add/del HMP test" patch").
Based-on: [email protected] (Subject: [PATCH 0/4] qmp: query-device-type command)
Eduardo Habkost (6):
device-crash-test: Make whitelist code a bit more flexible
device-crash-test: Log detailed info on success too
device-crash-test: Allow checkOneCase() to report multiple results
device-crash-test: Exit immediately on fatal failures on quick mode
device-crash-test: Basic device_add support
device-crash-test: Multi-device device_add test
scripts/device-crash-test | 132 +++++++++++++++++++++++++++++++++------------- 1 file changed, 94 insertions(+), 38 deletions(-)
Signature · 1 line
2.13.5
RFC 1/6 device-crash-test: Make whitelist code a bit more flexible
Allow test results without the 'log' and 'exitcode' keys. The new device_add code won't report exitcode/log because multiple devices will be tested on a single QEMU run.
--- scripts/device-crash-test | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
scripts/device-crash-test +3 -2
@@ -284,9 +284,10 @@ def whitelistResultMatch(wl, r): """ assert whitelistTestCaseMatch(wl, r['testcase']) return ((wl.get('exitcode', 1) is None or - r['exitcode'] == wl.get('exitcode', 1)) and + r.get('exitcode') == wl.get('exitcode', 1)) and ('log' not in wl or - re.search(wl['log'], r['log'], re.MULTILINE))) + ('log' in r and + re.search(wl['log'], r['log'], re.MULTILINE)))) def checkResultWhitelist(r):
Signature · 1 line
2.13.5
RFC 2/6 device-crash-test: Log detailed info on success too
In reply to Eduardo HabkostInstead of making checkOneCase() return None on success, return detailed info even if no problems were found. This will make handling of device_add errors simpler, and improve debug output.
As logFailure() won't handle only failures, rename it to logResult() and make it a bit more flexible about missing fields.
--- scripts/device-crash-test | 61 ++++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 32 deletions(-)
scripts/device-crash-test +29 -32
@@ -234,6 +234,9 @@ ERROR_WHITELIST = [ {'exitcode':-11, 'device':'arm-gicv3', 'loglevel':logging.ERROR, 'expected':True}, {'exitcode':-11, 'machine':'isapc', 'device':'.*-iommu', 'loglevel':logging.ERROR, 'expected':True}, + # 0 exit code means test succeeded, so we just log it as logging.DEBUG: + {'exitcode':0}, + # everything else (including SIGABRT and SIGSEGV) will be a fatal error: {'exitcode':None, 'fatal':True, 'loglevel':logging.FATAL}, ]
@@ -381,8 +384,7 @@ def getBinaryInfo(args, binary): def checkOneCase(args, testcase): """Check one specific case - Returns a dictionary containing failure information on error, - or None on success + Returns a dictionary containing results of test case. """ binary = testcase['binary'] accel = testcase['accel']
@@ -397,26 +399,21 @@ def checkOneCase(args, testcase): dbg("will launch QEMU: %s", cmdline) vm = QEMUMachine(binary=binary, args=args) - exc_traceback = None + r = {'testcase':testcase, + 'cmdline':cmdline} + try: vm.launch() except KeyboardInterrupt: raise except: - exc_traceback = traceback.format_exc() + r['exc_traceback'] = traceback.format_exc() dbg("Exception while running test case") - finally: - vm.shutdown() - ec = vm.exitcode() - log = vm.get_log() - - if exc_traceback is not None or ec != 0: - return {'exc_traceback':exc_traceback, - 'exitcode':ec, - 'log':log, - 'testcase':testcase, - 'cmdline':cmdline} + vm.shutdown() + r['exitcode'] = vm.exitcode() + r['log'] = vm.get_log() + return r def binariesToTest(args, testcase): if args.qemu:
@@ -490,17 +487,18 @@ def casesToTest(args, testcase): return cases -def logFailure(f, level): +def logResult(f, level): t = f['testcase'] - logger.log(level, "failed: %s", formatTestCase(t)) + logger.log(level, "result: %s", formatTestCase(t)) logger.log(level, "cmdline: %s", f['cmdline']) - for l in f['log'].strip().split('\n'): + for l in f.get('log', '').strip().split('\n'): logger.log(level, "log: %s", l) - logger.log(level, "exit code: %r", f['exitcode']) - if f['exc_traceback']: + logger.log(level, "exit code: %r", f.get('exitcode')) + if 'exc_traceback' in f: logger.log(level, "exception:") for l in f['exc_traceback'].split('\n'): logger.log(level, " %s", l.rstrip('\n')) + dbg('raw result dictionary: %r', f) def main():
@@ -579,18 +577,17 @@ def main(): except KeyboardInterrupt: break - if f: - i, wl = checkResultWhitelist(f) - dbg("testcase: %r, whitelist match: %r", t, wl) - wl_stats.setdefault(i, []).append(f) - level = wl.get('loglevel', logging.DEBUG) - logFailure(f, level) - if wl.get('fatal') or (args.strict and level >= logging.WARN): - fatal_failures.append(f) - else: - dbg("success: %s", formatTestCase(t)) - if expected_match: - logger.warn("Didn't fail as expected: %s", formatTestCase(t)) + i, wl = checkResultWhitelist(f) + dbg("testcase: %r, whitelist match: %r", t, wl) + wl_stats.setdefault(i, []).append(f) + level = wl.get('loglevel', logging.DEBUG) + logResult(f, level) + + if wl.get('fatal') or (args.strict and level >= logging.WARN): + fatal_failures.append(f) + + if expected_match and expected_match[0] != i: + logger.warn("Didn't fail as expected: %s", formatTestCase(t)) logger.info("Total: %d test cases", total) if skipped:
Signature · 1 line
2.13.5
RFC 3/6 device-crash-test: Allow checkOneCase() to report multiple results
In reply to Eduardo HabkostThis will allow the test code to be improved to test multiple devices in a single run.
--- scripts/device-crash-test | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-)
scripts/device-crash-test +14 -15
@@ -384,7 +384,8 @@ def getBinaryInfo(args, binary): def checkOneCase(args, testcase): """Check one specific case - Returns a dictionary containing results of test case. + Generates a list of dictionaries containing results of testing. If + multiple devices are being tested, multiple entries can be returned. """ binary = testcase['binary'] accel = testcase['accel']
@@ -413,7 +414,7 @@ def checkOneCase(args, testcase): vm.shutdown() r['exitcode'] = vm.exitcode() r['log'] = vm.get_log() - return r + yield r def binariesToTest(args, testcase): if args.qemu:
@@ -572,22 +573,20 @@ def main(): if args.dry_run: continue - try: - f = checkOneCase(args, t) - except KeyboardInterrupt: - break - i, wl = checkResultWhitelist(f) - dbg("testcase: %r, whitelist match: %r", t, wl) - wl_stats.setdefault(i, []).append(f) - level = wl.get('loglevel', logging.DEBUG) - logResult(f, level) + for r in checkOneCase(args, t): + i, wl = checkResultWhitelist(r) + dbg("testcase: %r, whitelist match: %r", t, wl) + wl_stats.setdefault(i, []).append(r) + level = wl.get('loglevel', logging.DEBUG) + logResult(r, level) + + if wl.get('fatal') or (args.strict and level >= logging.WARN): + fatal_failures.append(r) - if wl.get('fatal') or (args.strict and level >= logging.WARN): - fatal_failures.append(f) + if expected_match and expected_match[0] != i: + logger.warn("Didn't fail as expected: %s", formatTestCase(t)) - if expected_match and expected_match[0] != i: - logger.warn("Didn't fail as expected: %s", formatTestCase(t)) logger.info("Total: %d test cases", total) if skipped:
Signature · 1 line
2.13.5
RFC 4/6 device-crash-test: Exit immediately on fatal failures on quick mode
In reply to Eduardo HabkostIf we're running on quick mode, we won't wait for the full test run, and will exit immediately if a fatal failure is found.
--- scripts/device-crash-test | 4 ++++ 1 file changed, 4 insertions(+)
scripts/device-crash-test +4 -0
@@ -583,10 +583,14 @@ def main(): if wl.get('fatal') or (args.strict and level >= logging.WARN): fatal_failures.append(r) + if args.quick: + break if expected_match and expected_match[0] != i: logger.warn("Didn't fail as expected: %s", formatTestCase(t)) + if fatal_failures and args.quick: + break logger.info("Total: %d test cases", total) if skipped:
Signature · 1 line
2.13.5
RFC 5/6 device-crash-test: Basic device_add support
In reply to Eduardo HabkostAdd a 'method' testcase argument that will test the device using device_add instead of -device. A new device_add_error whitelist key is now supported, to catch device_add errors.
--- scripts/device-crash-test | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-)
scripts/device-crash-test +33 -5
@@ -35,7 +35,7 @@ import argparse from itertools import chain sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts')) -from qemu import QEMUMachine +from qemu import QEMUMachine, MonitorResponseError logger = logging.getLogger('device-crash-test') dbg = logger.debug
@@ -203,6 +203,11 @@ ERROR_WHITELIST = [ {'log':r"core 0 already populated"}, {'log':r"could not find stage1 bootloader"}, + {'exitcode':None, 'device_add_error':r"^Bus '[\w.-]+' does not support hotplugging$"}, + {'exitcode':None, 'device_add_error':r"^No '[\w-]+' bus found for device '[\w-]+'$"}, + # There are too many ways device_add can fail, ignore all of them as long as QEMU doesn't crash: + {'exitcode':None, 'device_add_error':r".+"}, + # other exitcode=1 failures not listed above will just generate INFO messages: {'exitcode':1, 'loglevel':logging.INFO},
@@ -290,7 +295,10 @@ def whitelistResultMatch(wl, r): r.get('exitcode') == wl.get('exitcode', 1)) and ('log' not in wl or ('log' in r and - re.search(wl['log'], r['log'], re.MULTILINE)))) + re.search(wl['log'], r['log'], re.MULTILINE))) and + ('device_add_error' not in wl or + ('device_add_error' in r and + re.search(wl['device_add_error'], r['device_add_error'])))) def checkResultWhitelist(r):
@@ -339,6 +347,7 @@ class QemuBinaryInfo(object): self.alldevs = set(qomListTypeNames(vm, implements=devtype, abstract=False)) self.dev_info = dict((t, vm.command('query-device-type', typename=t)) for t in self.alldevs) self.user_devs = set(d for d in self.alldevs if self.dev_info[d]['user-creatable']) + self.hotpluggable_devs = set(d for d in self.alldevs if self.dev_info[d]['hotpluggable']) self.machines = list(m['name'] for m in vm.command('query-machines')) self.kvm_available = vm.command('query-kvm')['enabled'] finally:
@@ -391,11 +400,14 @@ def checkOneCase(args, testcase): accel = testcase['accel'] machine = testcase['machine'] device = testcase['device'] + method = testcase['method'] + assert method in ['-device', 'device_add'] dbg("will test: %r", testcase) - args = ['-S', '-machine', '%s,accel=%s' % (machine, accel), - '-device', qemuOptsEscape(device)] + args = ['-S', '-machine', '%s,accel=%s' % (machine, accel)] + if method == '-device': + args.extend(['-device', qemuOptsEscape(device)]) cmdline = ' '.join([binary] + args) dbg("will launch QEMU: %s", cmdline) vm = QEMUMachine(binary=binary, args=args)
@@ -405,6 +417,12 @@ def checkOneCase(args, testcase): try: vm.launch() + if method == 'device_add': + dbg('running device_add %s', device) + try: + vm.command('device_add', driver=device) + except MonitorResponseError, e: + r['device_add_error'] = e.reply['error']['desc'] except KeyboardInterrupt: raise except:
@@ -434,14 +452,22 @@ def machinesToTest(args, testcase): return getBinaryInfo(args, testcase['binary']).machines +def plugMethods(args, testcase): + return ['-device', 'device_add'] + + def devicesToTest(args, testcase): - return getBinaryInfo(args, testcase['binary']).user_devs + if testcase['method'] == 'device_add': + return getBinaryInfo(args, testcase['binary']).hotpluggable_devs + else: + return getBinaryInfo(args, testcase['binary']).user_devs TESTCASE_VARIABLES = [ ('binary', binariesToTest), ('accel', accelsToTest), ('machine', machinesToTest), + ('method', plugMethods), ('device', devicesToTest), ]
@@ -494,6 +520,8 @@ def logResult(f, level): logger.log(level, "cmdline: %s", f['cmdline']) for l in f.get('log', '').strip().split('\n'): logger.log(level, "log: %s", l) + if 'device_add_error' in f: + logger.log(level, "device_add error: %s", f['device_add_error']) logger.log(level, "exit code: %r", f.get('exitcode')) if 'exc_traceback' in f: logger.log(level, "exception:")
Signature · 1 line
2.13.5
RFC 6/6 device-crash-test: Multi-device device_add test
In reply to Eduardo HabkostWhen running device_add tests, test all devices in a single run instead of restarting QEMU every time.
There's a plug_all testcase argument that can be used to make the test code plug all devices at once. I'm adding this mode because there's a crash that was detected while testing the script without any device_del command, but the crash goes away if we device_del every device immediately:
$ ../scripts/device-crash-test ./x86_64-softmmu/qemu-system-x86_64 --quick -t method=device_add --strict -t plug_all=1
INFO: running test case: binary=./x86_64-softmmu/qemu-system-x86_64 plug_all=1 accel=kvm machine=pc-0.12 device=* method=device_add
WARNING: qemu received signal -6: ./x86_64-softmmu/qemu-system-x86_64 -chardev socket,id=mon,path=/var/tmp/qemu-23578-monitor.sock -mon chardev=mon,mode=control -display none -vga none -S -machine pc-0.12,accel=kvm
ERROR: result: binary=./x86_64-softmmu/qemu-system-x86_64 plug_all=1 accel=kvm machine=pc-0.12 device=* method=device_add
ERROR: cmdline: ./x86_64-softmmu/qemu-system-x86_64 -S -machine pc-0.12,accel=kvm
ERROR: log: audio: Could not init `oss' audio driver
ERROR: log: qemu-system-x86_64: .../qemu/migration/savevm.c:721: vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed.
ERROR: last device_add device: usb-net
ERROR: exit code: -6
ERROR: exception:
ERROR: Traceback (most recent call last):
ERROR: File "../scripts/device-crash-test", line 437, in checkOneCase
ERROR: vm.command('device_add', driver=device, id=devid)
ERROR: File ".../qemu/scripts/qemu.py", line 269, in command
ERROR: raise qmp.qmp.QMPError("Monitor is closed")
ERROR: QMPError: Monitor is closed
ERROR:
INFO: Total: 1 test cases
ERROR: Fatal failure: binary=./x86_64-softmmu/qemu-system-x86_64 plug_all=1 accel=kvm machine=pc-0.12 device=* method=device_add
ERROR: Fatal failures on some machine/device combinations--- scripts/device-crash-test | 41 ++++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-)
scripts/device-crash-test +34 -7
@@ -404,10 +404,19 @@ def checkOneCase(args, testcase): assert method in ['-device', 'device_add'] dbg("will test: %r", testcase) + if device == '*': + bi = getBinaryInfo(args, binary) + if method == '-device': + devices = list(bi.user_devs) + else: + devices = list(bi.hotpluggable_devs) + else: + devices = [device] args = ['-S', '-machine', '%s,accel=%s' % (machine, accel)] if method == '-device': - args.extend(['-device', qemuOptsEscape(device)]) + for device in devices: + args.extend(['-device', qemuOptsEscape(device)]) cmdline = ' '.join([binary] + args) dbg("will launch QEMU: %s", cmdline) vm = QEMUMachine(binary=binary, args=args)
@@ -418,11 +427,27 @@ def checkOneCase(args, testcase): try: vm.launch() if method == 'device_add': - dbg('running device_add %s', device) - try: - vm.command('device_add', driver=device) - except MonitorResponseError, e: - r['device_add_error'] = e.reply['error']['desc'] + plug_all = int(testcase.get('plug_all', 0)) + for device in devices: + r['device_add_device'] = device + dbg('running device_add %s', device) + devid = 'test-%s' % (device) + try: + vm.command('device_add', driver=device, id=devid) + if not plug_all: + vm.command('device_del', id=devid) + except MonitorResponseError, e: + r2 = r.copy() + r2['device_add_error'] = e.reply['error']['desc'] + yield r2 + + if plug_all: + for device in devices: + devid = 'test-%s' % (device) + vm.command('device_del', id=devid) + + except GeneratorExit: + raise except KeyboardInterrupt: raise except:
@@ -458,7 +483,7 @@ def plugMethods(args, testcase): def devicesToTest(args, testcase): if testcase['method'] == 'device_add': - return getBinaryInfo(args, testcase['binary']).hotpluggable_devs + return '*' # will test all devices in a single checkOneCase() call else: return getBinaryInfo(args, testcase['binary']).user_devs
@@ -520,6 +545,8 @@ def logResult(f, level): logger.log(level, "cmdline: %s", f['cmdline']) for l in f.get('log', '').strip().split('\n'): logger.log(level, "log: %s", l) + if 'device_add_device' in f: + logger.log(level, 'last device_add device: %s', f['device_add_device']) if 'device_add_error' in f: logger.log(level, "device_add error: %s", f['device_add_error']) logger.log(level, "exit code: %r", f.get('exitcode'))
Signature · 1 line
2.13.5
Re: [Qemu-devel] [RFC 6/6] device-crash-test: Multi-device device_add test
Quoted 9 lines · Eduardo Habkost
When running
device_addtests, test all devices in a single run instead of restarting QEMU every time.There's a
plug_alltestcase argument that can be used to make the test code plug all devices at once. I'm adding this mode because there's a crash that was detected while testing the script without anydevice_delcommand, but the crash goes away if wedevice_delevery device immediately:
Quoted 8 lines
$ ../scripts/device-crash-test ./x86_64-softmmu/qemu-system-x86_64 --quick -t method=device_add --strict -t plug_all=1 INFO: running test case: binary=./x86_64-softmmu/qemu-system-x86_64 plug_all=1 accel=kvm machine=pc-0.12 device=* method=device_add WARNING: qemu received signal -6: ./x86_64-softmmu/qemu-system-x86_64 -chardev socket,id=mon,path=/var/tmp/qemu-23578-monitor.sock -mon chardev=mon,mode=control -display none -vga none -S -machine pc-0.12,accel=kvm ERROR: result: binary=./x86_64-softmmu/qemu-system-x86_64 plug_all=1 accel=kvm machine=pc-0.12 device=* method=device_add ERROR: cmdline: ./x86_64-softmmu/qemu-system-x86_64 -S -machine pc-0.12,accel=kvm ERROR: log: audio: Could not init `oss' audio driver ERROR: log: qemu-system-x86_64: .../qemu/migration/savevm.c:721: vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed. ERROR: last device_add device: usb-net
FWIW, I've seen similar crashes while working on my HMP device_add tester, when I skipped the device_del and plugged-in the devices a couple of times instead. I think this happens because some devices are doing a vmstate_register() in their instance_init() or realize() function, instead of using dc->vmsd as they should. However, with the git master branch as of today (31bc1d8481af414cfa2857f905e), I am currently not able anymore to reproduce the problem with the HMP device_add tester, so one of the recent "set user_creatable = false" patches might have fixed the issue for me already...
Thomas
Re: [Qemu-devel] [RFC 6/6] device-crash-test: Multi-device device_add test
Quoted 19 lines · * Thomas Huth ([email protected]) wrote:
Quoted 9 lines · Eduardo Habkost
When running
device_addtests, test all devices in a single run instead of restarting QEMU every time.There's a
plug_alltestcase argument that can be used to make the test code plug all devices at once. I'm adding this mode because there's a crash that was detected while testing the script without anydevice_delcommand, but the crash goes away if wedevice_delevery device immediately:Quoted 8 lines
$ ../scripts/device-crash-test ./x86_64-softmmu/qemu-system-x86_64 --quick -t method=device_add --strict -t plug_all=1 INFO: running test case: binary=./x86_64-softmmu/qemu-system-x86_64 plug_all=1 accel=kvm machine=pc-0.12 device=* method=device_add WARNING: qemu received signal -6: ./x86_64-softmmu/qemu-system-x86_64 -chardev socket,id=mon,path=/var/tmp/qemu-23578-monitor.sock -mon chardev=mon,mode=control -display none -vga none -S -machine pc-0.12,accel=kvm ERROR: result: binary=./x86_64-softmmu/qemu-system-x86_64 plug_all=1 accel=kvm machine=pc-0.12 device=* method=device_add ERROR: cmdline: ./x86_64-softmmu/qemu-system-x86_64 -S -machine pc-0.12,accel=kvm ERROR: log: audio: Could not init `oss' audio driver ERROR: log: qemu-system-x86_64: .../qemu/migration/savevm.c:721: vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed. ERROR: last device_add device: usb-net
Quoted 9 lines
FWIW, I've seen similar crashes while working on my HMP
device_addtester, when I skipped thedevice_deland plugged-in the devices a couple of times instead. I think this happens because some devices are doing avmstate_register()in theirinstance_init()orrealize()function, instead of usingdc->vmsdas they should. However, with the git master branch as of today (31bc1d8481af414cfa2857f905e), I am currently not able anymore to reproduce the problem with the HMPdevice_addtester, so one of the recent "setuser_creatable= false" patches might have fixed the issue for me already...
I think I've seen this problem when two things end up trying to register with the same migration name; I'll be honest I can't remember the details, but I think the choice is either between having a full bus name, e.g.
pci-xx:yy.z:aa:bb:.c:...... then similar for the USB chain
or
mydevice + number
where number is the instance=id. So to hit this I think it's typically a case of messing up that long path and it no longer being unique.
Printing the path in vmstate_register_with_alias_id after the pstrcat might help.
I did add some sanity checking in there for a case I'd previously hit where the name just got truncated, but I don't think you should be able to hit that any more.
Dave
Quoted 1 line
Thomas
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK