Skip to contentSkip to conversation

tests/acceptance: Test case for detecting -object crashes

PATCH In review Eduardo Habkost 3 participants 8 messages Oct 9, 2020 – Oct 26, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 tests/acceptance: Test case for detecting -object crashes

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

---
 tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)
 create mode 100644 tests/acceptance/object_option.py
tests/acceptance/object_option.py new file +53 -0
new file mode 100644
index 0000000000..511c03a36f
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Eduardo Habkost <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+import shlex
+import subprocess
+
+import avocado
+import avocado_qemu
+
+
+class ObjectOption(avocado_qemu.Test):
+    """Check if ``-object`` option behaves as expected"""
+
+    def run(self, cmd, *args, **kwargs):
+        cmdstr = ' '.join(shlex.quote(c) for c in cmd)
+        self.log.info("Running command: %s", cmdstr)
+        return subprocess.run(cmd, universal_newlines=True, *args, **kwargs)
+
+    def get_devices(self):
+        out = self.run([self.qemu_bin, '-object', 'help'],
+                       check=True, stdout=subprocess.PIPE).stdout
+        lines = out.split('\n')
+        return [l.strip() for l in lines[1:] if l.strip()]
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_help(self):
+        """Check if ``-object ...,help`` behaves as expected"""
+        for device in self.get_devices():
+            self.run([self.qemu_bin, '-object', '%s,help' % (device)],
+                     check=True,
+                     stdout=subprocess.DEVNULL)
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_crash(self):
+        """Check that QEMU doesn't crash when using ``-object ...``"""
+        for device in self.get_devices():
+            r = self.run([self.qemu_bin, '-object',
+                                '%s,id=obj0' % (device),
+                                '-monitor', 'stdio'],
+                         input='quit\n',
+                         stdout=subprocess.DEVNULL,
+                         stderr=subprocess.PIPE)
+            if r.returncode not in (0, 1):
+                self.log.warn("QEMU stderr: %s", r.stderr)
+                self.log.warn("QEMU exit code: %d", r.returncode)
+                if r.returncode < 0:
+                    self.fail("QEMU crashed")
+                else:
+                    self.fail("Unexpected exit code")
Signature · 1 line
2.26.2
#1 lore raw
Quoted 4 lines · Eduardo Habkost

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

Quoted 8 lines

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

Assuming:
Based-on: <[email protected]>

I get:

(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.19 s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.18 s)

Quoted 65 lines
---
  tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++
  1 file changed, 53 insertions(+)

create mode 100644 tests/acceptance/object_option.py

tests/acceptance/object_option.py new file +53 -0
new file mode 100644
index 0000000000..511c03a36f
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Eduardo Habkost <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+import shlex
+import subprocess
+
+import avocado
+import avocado_qemu
+
+
+class ObjectOption(avocado_qemu.Test):
+    """Check if ``-object`` option behaves as expected"""
+
+    def run(self, cmd, *args, **kwargs):
+        cmdstr = ' '.join(shlex.quote(c) for c in cmd)
+        self.log.info("Running command: %s", cmdstr)
+        return subprocess.run(cmd, universal_newlines=True, *args, **kwargs)
+
+    def get_devices(self):
+        out = self.run([self.qemu_bin, '-object', 'help'],
+                       check=True, stdout=subprocess.PIPE).stdout
+        lines = out.split('\n')
+        return [l.strip() for l in lines[1:] if l.strip()]
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_help(self):
+        """Check if ``-object ...,help`` behaves as expected"""
+        for device in self.get_devices():
+            self.run([self.qemu_bin, '-object', '%s,help' % (device)],
+                     check=True,
+                     stdout=subprocess.DEVNULL)
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_crash(self):
+        """Check that QEMU doesn't crash when using ``-object ...``"""
+        for device in self.get_devices():
+            r = self.run([self.qemu_bin, '-object',
+                                '%s,id=obj0' % (device),
+                                '-monitor', 'stdio'],
+                         input='quit\n',
+                         stdout=subprocess.DEVNULL,
+                         stderr=subprocess.PIPE)
+            if r.returncode not in (0, 1):
+                self.log.warn("QEMU stderr: %s", r.stderr)
+                self.log.warn("QEMU exit code: %d", r.returncode)
+                if r.returncode < 0:
+                    self.fail("QEMU crashed")
+                else:
+                    self.fail("Unexpected exit code")
#2 lore raw
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

Quoted 8 lines

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

Assuming:
Based-on: <[email protected]>

I get:

Quoted 11 lines

(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.18
s)

Did you influence (test parameter?) the QEMU binary to be used?

Thanks,
- Cleber.

#3 lore raw
Quoted 32 lines · Cleber Rosa
Quoted 19 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

Quoted 8 lines

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

Assuming:
Based-on: <[email protected]>

I get:

Quoted 11 lines

(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.18
s)

Quoted 1 line

Did you influence (test parameter?) the QEMU binary to be used?

I'm assuming this was triggered by "make check-acceptance". I will change the test case to use '-machine none', which should work on all architectures.

Signature · 1 line
Eduardo
#4 lore raw
Quoted 33 lines · Eduardo Habkost
Quoted 32 lines · Cleber Rosa
Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

Quoted 8 lines

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

Assuming:
Based-on: <[email protected]>

I get:

(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.18
s)

Quoted 3 lines
Quoted 1 line

Did you influence (test parameter?) the QEMU binary to be used?

I'm assuming this was triggered by "make check-acceptance". I

The output shows test 1/2 and 2/2, so I don't think it was triggered by "make check-acceptance".

Quoted 3 lines

will change the test case to use '-machine none', which should work on all architectures.

+1, sounds good!
- Cleber.

Quoted 2 lines
Signature · 1 line
Eduardo
#5 lore raw
Quoted 34 lines · Cleber Rosa
Quoted 33 lines · Eduardo Habkost
Quoted 32 lines · Cleber Rosa
Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Eduardo Habkost

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

Quoted 8 lines

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

Assuming:
Based-on: <[email protected]>

I get:

(1/2) tests/acceptance/object_option.py:ObjectOption.test_help:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.19
s)
(2/2) tests/acceptance/object_option.py:ObjectOption.test_crash:
qemu-system-avr: No machine specified, and there is no default Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) (0.18
s)

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Did you influence (test parameter?) the QEMU binary to be used?

I'm assuming this was triggered by "make check-acceptance". I

The output shows test 1/2 and 2/2, so I don't think it was triggered by "make check-acceptance".

I tested it in a build dir configured with:

  ../configure --target-list=avr-softmmu

I will test it again during the week.

Quoted 11 lines
Quoted 4 lines

will change the test case to use '-machine none', which should work on all architectures.

+1, sounds good!
- Cleber.

Quoted 2 lines
Signature · 1 line
Eduardo
#6 lore raw
In reply to Eduardo Habkost

Hi Eduardo, Thomas,

Quoted 67 lines · Eduardo Habkost

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

---
  tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++
  1 file changed, 53 insertions(+)

create mode 100644 tests/acceptance/object_option.py

tests/acceptance/object_option.py new file +43 -0
new file mode 100644
index 0000000000..511c03a36f
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Eduardo Habkost <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+import shlex
+import subprocess
+
+import avocado
+import avocado_qemu
+
+
+class ObjectOption(avocado_qemu.Test):
+    """Check if ``-object`` option behaves as expected"""
+
+    def run(self, cmd, *args, **kwargs):
+        cmdstr = ' '.join(shlex.quote(c) for c in cmd)
+        self.log.info("Running command: %s", cmdstr)
+        return subprocess.run(cmd, universal_newlines=True, *args, **kwargs)
+
+    def get_devices(self):
+        out = self.run([self.qemu_bin, '-object', 'help'],
+                       check=True, stdout=subprocess.PIPE).stdout
+        lines = out.split('\n')
+        return [l.strip() for l in lines[1:] if l.strip()]
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_help(self):
+        """Check if ``-object ...,help`` behaves as expected"""
+        for device in self.get_devices():
+            self.run([self.qemu_bin, '-object', '%s,help' % (device)],
+                     check=True,
+                     stdout=subprocess.DEVNULL)
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_crash(self):
+        """Check that QEMU doesn't crash when using ``-object ...``"""
+        for device in self.get_devices():
+            r = self.run([self.qemu_bin, '-object',
+                                '%s,id=obj0' % (device),
+                                '-monitor', 'stdio'],

tests/acceptance/object_option.py line 53

Quoted 10 lines
tests/acceptance/object_option.py new file +10 -0
+                         input='quit\n',
+                         stdout=subprocess.DEVNULL,
+                         stderr=subprocess.PIPE)
+            if r.returncode not in (0, 1):
+                self.log.warn("QEMU stderr: %s", r.stderr)
+                self.log.warn("QEMU exit code: %d", r.returncode)
+                if r.returncode < 0:
+                    self.fail("QEMU crashed")
+                else:
+                    self.fail("Unexpected exit code")

Eduardo, what is the "acceptance" (functional) part of this test?

Thomas, could this be written using the QTest framework instead?

Thanks,

Phil.

#7 lore raw
Quoted 74 lines · Philippe Mathieu-Daudé

Hi Eduardo, Thomas,

Quoted 71 lines · Eduardo Habkost

Add a simple test case that will run QEMU directly (without QMP) just to check for crashes when using -object.

---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports

---
  tests/acceptance/object_option.py | 53 +++++++++++++++++++++++++++++++
  1 file changed, 53 insertions(+)

create mode 100644 tests/acceptance/object_option.py

tests/acceptance/object_option.py new file +48 -0
new file mode 100644
index 0000000000..511c03a36f
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Eduardo Habkost <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+import shlex
+import subprocess
+
+import avocado
+import avocado_qemu
+
+
+class ObjectOption(avocado_qemu.Test):
+    """Check if ``-object`` option behaves as expected"""
+
+    def run(self, cmd, *args, **kwargs):
+        cmdstr = ' '.join(shlex.quote(c) for c in cmd)
+        self.log.info("Running command: %s", cmdstr)
+        return subprocess.run(cmd, universal_newlines=True, *args, **kwargs)
+
+    def get_devices(self):
+        out = self.run([self.qemu_bin, '-object', 'help'],
+                       check=True, stdout=subprocess.PIPE).stdout
+        lines = out.split('\n')
+        return [l.strip() for l in lines[1:] if l.strip()]
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_help(self):
+        """Check if ``-object ...,help`` behaves as expected"""
+        for device in self.get_devices():
+            self.run([self.qemu_bin, '-object', '%s,help' % (device)],
+                     check=True,
+                     stdout=subprocess.DEVNULL)
+
+    @avocado.fail_on(subprocess.CalledProcessError)
+    def test_crash(self):
+        """Check that QEMU doesn't crash when using ``-object ...``"""
+        for device in self.get_devices():
+            r = self.run([self.qemu_bin, '-object',
+                                '%s,id=obj0' % (device),
+                                '-monitor', 'stdio'],
+                         input='quit\n',
+                         stdout=subprocess.DEVNULL,
+                         stderr=subprocess.PIPE)
+            if r.returncode not in (0, 1):
+                self.log.warn("QEMU stderr: %s", r.stderr)
Quoted 8 lines

tests/acceptance/object_option.py line 53

Quoted 6 lines
tests/acceptance/object_option.py new file +5 -0
+                self.log.warn("QEMU exit code: %d", r.returncode)
+                if r.returncode < 0:
+                    self.fail("QEMU crashed")
+                else:
+                    self.fail("Unexpected exit code")

Eduardo, what is the "acceptance" (functional) part of this test?

It is testing an external interface (the command line), and I'm pretty sure "not crashing when using -object with valid types" is a functional requirement.

Quoted 2 lines

Thomas, could this be written using the QTest framework instead?

I'm sure it can, but why would we?

I don't think rewriting test cases in C is a good use of our time.

Signature · 1 line
Eduardo