tests/acceptance: Test case for detecting -object crashes
PATCH v2 1/1 tests/acceptance: Test case for detecting -object crashes
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.
---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports
--- tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 tests/acceptance/object_option.py
tests/acceptance/object_option.py new file +53 -0
@@ -0,0 +1,53 @@ +# Copyright (c) 2020 Red Hat, Inc. +# +# Author: +# Eduardo Habkost <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. +import shlex +import subprocess + +import avocado +import avocado_qemu + + +class ObjectOption(avocado_qemu.Test): + """Check if ``-object`` option behaves as expected""" + + def run(self, cmd, *args, **kwargs): + cmdstr = ' '.join(shlex.quote(c) for c in cmd) + self.log.info("Running command: %s", cmdstr) + return subprocess.run(cmd, universal_newlines=True, *args, **kwargs) + + def get_devices(self): + out = self.run([self.qemu_bin, '-object', 'help'], + check=True, stdout=subprocess.PIPE).stdout + lines = out.split('\n') + return [l.strip() for l in lines[1:] if l.strip()] + + @avocado.fail_on(subprocess.CalledProcessError) + def test_help(self): + """Check if ``-object ...,help`` behaves as expected""" + for device in self.get_devices(): + self.run([self.qemu_bin, '-object', '%s,help' % (device)], + check=True, + stdout=subprocess.DEVNULL) + + @avocado.fail_on(subprocess.CalledProcessError) + def test_crash(self): + """Check that QEMU doesn't crash when using ``-object ...``""" + for device in self.get_devices(): + r = self.run([self.qemu_bin, '-object', + '%s,id=obj0' % (device), + '-monitor', 'stdio'], + input='quit\n', + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE) + if r.returncode not in (0, 1): + self.log.warn("QEMU stderr: %s", r.stderr) + self.log.warn("QEMU exit code: %d", r.returncode) + if r.returncode < 0: + self.fail("QEMU crashed") + else: + self.fail("Unexpected exit code")
Signature · 1 line
2.26.2
Quoted 4 lines · Eduardo Habkost
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using
-object.Signed-off-by Eduardo Habkost <[email protected]>
Quoted 8 lines
---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Renamedevices()toget_devices()(Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Rewordtest_crash()docstring (Cleber)
* Reorder imports
Assuming:
Based-on: <[email protected]>
I get:
(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.19 s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.18 s)
Quoted 65 lines
--- tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+)create mode 100644
tests/acceptance/object_option.pytests/acceptance/object_option.py new file +53
-0@@ -0,0 +1,53 @@ +# Copyright (c) 2020 Red Hat, Inc. +# +# Author: +# Eduardo Habkost <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. +import shlex +import subprocess + +import avocado +import avocado_qemu + + +class ObjectOption(avocado_qemu.Test): + """Check if ``-object`` option behaves as expected""" + + def run(self, cmd, *args, **kwargs): + cmdstr = ' '.join(shlex.quote(c) for c in cmd) + self.log.info("Running command: %s", cmdstr) + return subprocess.run(cmd, universal_newlines=True, *args, **kwargs) + + def get_devices(self): + out = self.run([self.qemu_bin, '-object', 'help'], + check=True, stdout=subprocess.PIPE).stdout + lines = out.split('\n') + return [l.strip() for l in lines[1:] if l.strip()] + + @avocado.fail_on(subprocess.CalledProcessError) + def test_help(self): + """Check if ``-object ...,help`` behaves as expected""" + for device in self.get_devices(): + self.run([self.qemu_bin, '-object', '%s,help' % (device)], + check=True, + stdout=subprocess.DEVNULL) + + @avocado.fail_on(subprocess.CalledProcessError) + def test_crash(self): + """Check that QEMU doesn't crash when using ``-object ...``""" + for device in self.get_devices(): + r = self.run([self.qemu_bin, '-object', + '%s,id=obj0' % (device), + '-monitor', 'stdio'], + input='quit\n', + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE) + if r.returncode not in (0, 1): + self.log.warn("QEMU stderr: %s", r.stderr) + self.log.warn("QEMU exit code: %d", r.returncode) + if r.returncode < 0: + self.fail("QEMU crashed") + else: + self.fail("Unexpected exit code")
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using
-object.Signed-off-by Eduardo Habkost <[email protected]>Quoted 8 lines
---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Renamedevices()toget_devices()(Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Rewordtest_crash()docstring (Cleber)
* Reorder importsAssuming:
Based-on: <[email protected]>I get:
Quoted 11 lines
(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.18
s)
Did you influence (test parameter?) the QEMU binary to be used?
Thanks,
- Cleber.
Quoted 32 lines · Cleber Rosa
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using
-object.Signed-off-by Eduardo Habkost <[email protected]>Quoted 8 lines
---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Renamedevices()toget_devices()(Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Rewordtest_crash()docstring (Cleber)
* Reorder importsAssuming:
Based-on: <[email protected]>I get:
Quoted 11 lines
(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.18
s)
Quoted 1 line
Did you influence (test parameter?) the QEMU binary to be used?
I'm assuming this was triggered by "make check-acceptance". I will change the test case to use '-machine none', which should work on all architectures.
Signature · 1 line
Eduardo
Quoted 33 lines · Eduardo Habkost
Quoted 32 lines · Cleber Rosa
Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using
-object.Signed-off-by Eduardo Habkost <[email protected]>Quoted 8 lines
---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Renamedevices()toget_devices()(Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Rewordtest_crash()docstring (Cleber)
* Reorder importsAssuming:
Based-on: <[email protected]>I get:
(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.18
s)
Quoted 3 lines
Quoted 1 line
Did you influence (test parameter?) the QEMU binary to be used?
I'm assuming this was triggered by "make check-acceptance". I
The output shows test 1/2 and 2/2, so I don't think it was triggered by "make check-acceptance".
Quoted 3 lines
will change the test case to use '-machine none', which should work on all architectures.
+1, sounds good!
- Cleber.
Quoted 2 lines
Signature · 1 line
Eduardo
Quoted 34 lines · Cleber Rosa
Quoted 33 lines · Eduardo Habkost
Quoted 32 lines · Cleber Rosa
Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using
-object.Signed-off-by Eduardo Habkost <[email protected]>Quoted 8 lines
---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Renamedevices()toget_devices()(Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Rewordtest_crash()docstring (Cleber)
* Reorder importsAssuming:
Based-on: <[email protected]>I get:
(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL:CalledProcessError(1, ['./qemu-system-avr', '-object', 'help'])(0.18
s)
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Did you influence (test parameter?) the QEMU binary to be used?
I'm assuming this was triggered by "make check-acceptance". I
The output shows test 1/2 and 2/2, so I don't think it was triggered by "make check-acceptance".
I tested it in a build dir configured with:
../configure --target-list=avr-softmmu
I will test it again during the week.
Quoted 11 lines
Quoted 4 lines
will change the test case to use '-machine none', which should work on all architectures.
+1, sounds good!
- Cleber.Quoted 2 lines
Signature · 1 line
Eduardo
Hi Eduardo, Thomas,
Quoted 67 lines · Eduardo Habkost
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using
-object.Signed-off-by Eduardo Habkost <[email protected]>---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Renamedevices()toget_devices()(Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Rewordtest_crash()docstring (Cleber)
* Reorder imports--- tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+)create mode 100644
tests/acceptance/object_option.pytests/acceptance/object_option.py new file +43
-0@@ -0,0 +1,53 @@ +# Copyright (c) 2020 Red Hat, Inc. +# +# Author: +# Eduardo Habkost <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. +import shlex +import subprocess + +import avocado +import avocado_qemu + + +class ObjectOption(avocado_qemu.Test): + """Check if ``-object`` option behaves as expected""" + + def run(self, cmd, *args, **kwargs): + cmdstr = ' '.join(shlex.quote(c) for c in cmd) + self.log.info("Running command: %s", cmdstr) + return subprocess.run(cmd, universal_newlines=True, *args, **kwargs) + + def get_devices(self): + out = self.run([self.qemu_bin, '-object', 'help'], + check=True, stdout=subprocess.PIPE).stdout + lines = out.split('\n') + return [l.strip() for l in lines[1:] if l.strip()] + + @avocado.fail_on(subprocess.CalledProcessError) + def test_help(self): + """Check if ``-object ...,help`` behaves as expected""" + for device in self.get_devices(): + self.run([self.qemu_bin, '-object', '%s,help' % (device)], + check=True, + stdout=subprocess.DEVNULL) + + @avocado.fail_on(subprocess.CalledProcessError) + def test_crash(self): + """Check that QEMU doesn't crash when using ``-object ...``""" + for device in self.get_devices(): + r = self.run([self.qemu_bin, '-object', + '%s,id=obj0' % (device), + '-monitor', 'stdio'],
tests/acceptance/object_option.py line 53
Quoted 10 lines
tests/acceptance/object_option.py new file +10
-0+ input='quit\n', + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE) + if r.returncode not in (0, 1): + self.log.warn("QEMU stderr: %s", r.stderr) + self.log.warn("QEMU exit code: %d", r.returncode) + if r.returncode < 0: + self.fail("QEMU crashed") + else: + self.fail("Unexpected exit code")
Eduardo, what is the "acceptance" (functional) part of this test?
Thomas, could this be written using the QTest framework instead?
Thanks,
Phil.
Quoted 74 lines · Philippe Mathieu-Daudé
Hi Eduardo, Thomas,
Quoted 71 lines · Eduardo Habkost
Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using
-object.Signed-off-by Eduardo Habkost <[email protected]>---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Renamedevices()toget_devices()(Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Rewordtest_crash()docstring (Cleber)
* Reorder imports--- tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+)create mode 100644
tests/acceptance/object_option.pytests/acceptance/object_option.py new file +48
-0@@ -0,0 +1,53 @@ +# Copyright (c) 2020 Red Hat, Inc. +# +# Author: +# Eduardo Habkost <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. +import shlex +import subprocess + +import avocado +import avocado_qemu + + +class ObjectOption(avocado_qemu.Test): + """Check if ``-object`` option behaves as expected""" + + def run(self, cmd, *args, **kwargs): + cmdstr = ' '.join(shlex.quote(c) for c in cmd) + self.log.info("Running command: %s", cmdstr) + return subprocess.run(cmd, universal_newlines=True, *args, **kwargs) + + def get_devices(self): + out = self.run([self.qemu_bin, '-object', 'help'], + check=True, stdout=subprocess.PIPE).stdout + lines = out.split('\n') + return [l.strip() for l in lines[1:] if l.strip()] + + @avocado.fail_on(subprocess.CalledProcessError) + def test_help(self): + """Check if ``-object ...,help`` behaves as expected""" + for device in self.get_devices(): + self.run([self.qemu_bin, '-object', '%s,help' % (device)], + check=True, + stdout=subprocess.DEVNULL) + + @avocado.fail_on(subprocess.CalledProcessError) + def test_crash(self): + """Check that QEMU doesn't crash when using ``-object ...``""" + for device in self.get_devices(): + r = self.run([self.qemu_bin, '-object', + '%s,id=obj0' % (device), + '-monitor', 'stdio'], + input='quit\n', + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE) + if r.returncode not in (0, 1): + self.log.warn("QEMU stderr: %s", r.stderr)
Quoted 8 lines
tests/acceptance/object_option.py line 53
Quoted 6 lines
tests/acceptance/object_option.py new file +5
-0+ self.log.warn("QEMU exit code: %d", r.returncode) + if r.returncode < 0: + self.fail("QEMU crashed") + else: + self.fail("Unexpected exit code")Eduardo, what is the "acceptance" (functional) part of this test?
It is testing an external interface (the command line), and I'm pretty sure "not crashing when using -object with valid types" is a functional requirement.
Quoted 2 lines
Thomas, could this be written using the QTest framework instead?
I'm sure it can, but why would we?
I don't think rewriting test cases in C is a good use of our time.
Signature · 1 line
Eduardo