Skip to contentSkip to conversation

machine.py fix for ppc64 tests + avocado changes

PATCH In review Daniel Henrique Barboza 3 participants 8 messages May 16, 2022 – May 23, 2022 Broadcom iproc Arm architecture

lore
Share & tools
lore
#0 lore raw

cover machine.py fix for ppc64 tests + avocado changes

Hello,

This series fixes a problem I'm having when running avocado tests in an IBM Power9 ppc64 host, without firmware modifications that breaks the default machine options of a pseries guest running KVM, and with --disable-tcg. The problem is described in detail in patch 02.

The proposed fix consists of checking whether we're running a pseries guest in the launch() method of machine.py. A simple fix, if we could rely on the QEMUSystemTest self.machine attribute to be set all the time a machine type is being set.

This is not the case for some tests, e.g. the empty_cpu_model.py test that, instead of using 'tags=machine:none' to set the 'none' machine type, it hardcodes '-machine none' via self.vm.add_args(). This doesn't set self.machine in the QEMUSystemTest class, and then we're left wondered whether the machine is being run with the binary defaults (i.e. no machine were set) or if the machine were set manually and we need to verify every -machine option to check it up. Which is not trivial by any
means: multiple machine options can be passed via avocado and you'll have to parse a string using all known machine types to see if it was set or not.

To fix the issue I needed to make the assumption that the machine type will be set in a way that self.machine is also set (i.e. using the avocado tag). Making this assumption for empty_cpu_model.py alone (patch
01) is enough for me to fix the issue I'm experiencing in patch 02.

I decided to make this assumption across the board in all tests that were setting the machine type by hand instead of using an avocado tag. So in the end, after this series, all avocado tests that sets a machine type are now setting via "avocado: tags=machine:<type>" annotation.

Daniel Henrique Barboza (5):
avocado/empty_cpu_model.py: use machine:none tag machine.py: add default pseries params in machine.py avocado/multiprocess.py: use tags=machine:pc|virt avocado/boot_linux.py: avocado tag fixes in BootLinuxAarch64 avocado/virtio-gpu.py: use tags=machine:pc

 python/qemu/machine/machine.py   | 13 +++++++++++++
 tests/avocado/boot_linux.py      |  7 +++----
 tests/avocado/empty_cpu_model.py |  5 ++++-
 tests/avocado/multiprocess.py    | 14 ++++++++------
 tests/avocado/virtio-gpu.py      |  6 ++++--
 5 files changed, 32 insertions(+), 13 deletions(-)
Signature · 1 line
2.32.0
#1 lore raw

PATCH 1/5 avocado/empty_cpu_model.py: use machine:none tag

Using tags=machine:none will do two things: it will avoid the need to passing '-machine none' via self.vm.add_args() and it will set the self.machine attribute of the parent QEMUSystemTest class (via its setUp() method).

We'll be relying on self.machine being set apropriately for an upcoming fix.

---
 tests/avocado/empty_cpu_model.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tests/avocado/empty_cpu_model.py +4 -1
index 22f504418d..723ecc73af 100644
@@ -10,8 +10,11 @@
 from avocado_qemu import QemuSystemTest
 
 class EmptyCPUModel(QemuSystemTest):
+    """
+    :avocado: tags=machine:none
+    """
     def test(self):
-        self.vm.add_args('-S', '-display', 'none', '-machine', 'none', '-cpu', '')
+        self.vm.add_args('-S', '-display', 'none', '-cpu', '')
         self.vm.set_qmp_monitor(enabled=False)
         self.vm.launch()
         self.vm.wait()
Signature · 1 line
2.32.0
#7 lore raw

PATCH 5/5 avocado/virtio-gpu.py: use tags=machine:pc

In reply to Daniel Henrique Barboza

Using tags=machine:pc will do two things: it will avoid the need to passing '-machine pc' via self.vm.add_args() and it will set the self.machine attribute of the parent QEMUSystemTest class (via its setUp() method).

---
 tests/avocado/virtio-gpu.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/avocado/virtio-gpu.py +4 -2
index 2a249a3a2c..a88f43d312 100644
@@ -59,6 +59,7 @@ def wait_for_console_pattern(self, success_message, vm=None):
     def test_virtio_vga_virgl(self):
         """
         :avocado: tags=device:virtio-vga-gl
+        :avocado: tags=machine:pc
         """
         # FIXME: should check presence of virtio, virgl etc
         self.require_accelerator('kvm')
@@ -68,7 +69,7 @@ def test_virtio_vga_virgl(self):
 
         self.vm.set_console()
         self.vm.add_args("-m", "2G")
-        self.vm.add_args("-machine", "pc,accel=kvm")
+        self.vm.add_args("-machine", "accel=kvm")
         self.vm.add_args("-device", "virtio-vga-gl")
         self.vm.add_args("-display", "egl-headless")
         self.vm.add_args(
@@ -94,6 +95,7 @@ def test_virtio_vga_virgl(self):
     def test_vhost_user_vga_virgl(self):
         """
         :avocado: tags=device:vhost-user-vga
+        :avocado: tags=machine:pc
         """
         # FIXME: should check presence of vhost-user-gpu, virgl, memfd etc
         self.require_accelerator('kvm')
@@ -131,7 +133,7 @@ def test_vhost_user_vga_virgl(self):
         self.vm.set_console()
         self.vm.add_args("-m", "2G")
         self.vm.add_args("-object", "memory-backend-memfd,id=mem,size=2G")
-        self.vm.add_args("-machine", "pc,memory-backend=mem,accel=kvm")
+        self.vm.add_args("-machine", "memory-backend=mem,accel=kvm")
         self.vm.add_args("-chardev", "socket,id=vug,fd=%d" % qemu_sock.fileno())
         self.vm.add_args("-device", "vhost-user-vga,chardev=vug")
         self.vm.add_args("-display", "egl-headless")
Signature · 1 line
2.32.0
#6 lore raw

PATCH 4/5 avocado/boot_linux.py: avocado tag fixes in BootLinuxAarch64

In reply to Daniel Henrique Barboza

BootLinuxAarch64 is already setting machine:virt in the avocado tags, meaning that we don't need to add '-machine virt' via add_args().

It's also adding an extra '-machine gic-version=2' parameter via an avocado tag, which is not ideal because:

- it prevents self.machine from QEMUSystemTest to be set since there are multiple 'machine' avocado tests being set;

- the tests are using different 'gic-version' setting, meaning that we're still needing to add '-machine gic-version=N' via add_args() regardless.

Removing the 'machine=gic-version=2' tag allows us to set 'self.machine' without adding extra work.

---
 tests/avocado/boot_linux.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
tests/avocado/boot_linux.py +3 -4
index ee584d2fdf..1fbedbab5c 100644
@@ -61,7 +61,6 @@ class BootLinuxAarch64(LinuxTest):
     """
     :avocado: tags=arch:aarch64
     :avocado: tags=machine:virt
-    :avocado: tags=machine:gic-version=2
     """
 
     def add_common_args(self):
@@ -80,7 +79,7 @@ def test_virt_tcg_gicv2(self):
         self.require_accelerator("tcg")
         self.vm.add_args("-accel", "tcg")
         self.vm.add_args("-cpu", "max,lpa2=off")
-        self.vm.add_args("-machine", "virt,gic-version=2")
+        self.vm.add_args("-machine", "gic-version=2")
         self.add_common_args()
         self.launch_and_wait(set_up_ssh_connection=False)
 
@@ -93,7 +92,7 @@ def test_virt_tcg_gicv3(self):
         self.require_accelerator("tcg")
         self.vm.add_args("-accel", "tcg")
         self.vm.add_args("-cpu", "max,lpa2=off")
-        self.vm.add_args("-machine", "virt,gic-version=3")
+        self.vm.add_args("-machine", "gic-version=3")
         self.add_common_args()
         self.launch_and_wait(set_up_ssh_connection=False)
 
@@ -104,7 +103,7 @@ def test_virt_kvm(self):
         """
         self.require_accelerator("kvm")
         self.vm.add_args("-accel", "kvm")
-        self.vm.add_args("-machine", "virt,gic-version=host")
+        self.vm.add_args("-machine", "gic-version=host")
         self.add_common_args()
         self.launch_and_wait(set_up_ssh_connection=False)
 
Signature · 1 line
2.32.0
#2 lore raw

PATCH 2/5 machine.py: add default pseries params in machine.py

In reply to Daniel Henrique Barboza

pSeries guests set a handful of machine capabilities on by default, all of them related to security mitigations, that aren't always available in the host.

This means that, as is today, running avocado in a Power9 server without the proper firmware support, and with --disable-tcg, this error will occur:

(1/1) tests/avocado/info_usernet.py:InfoUsernet.test_hostfwd: ERROR: ConnectError:
Failed to establish session: EOFError\n Exit code: 1\n (...)
(...)
Command: ./qemu-system-ppc64 -display none -vga none (...)
Output: qemu-system-ppc64: warning: netdev vnet has no peer
qemu-system-ppc64: Requested safe cache capability level not supported by KVM Try appending -machine cap-cfpc=broken

info_usernet.py happens to trigger this error first, but all tests would fail in this configuration because the host does not support the default 'cap-cfpc' capability.

A similar situation was already fixed a couple of years ago by Greg Kurz (commit 63d57c8f91d0) but it was focused on TCG warnings for these same capabilities and running C qtests. This commit ended up preventing the problem we're facing with avocado when running qtests with KVM support.

This patch does a similar approach by amending machine.py to disable these security capabilities in case we're running a pseries guest. The change is made in the _launch() callback to be sure that we're already commited into launching the guest. It's also worth noticing that we're relying on self._machine being set accordingly (i.e. via tag:machine), which is currently the case for all ppc64 related avocado tests.

---
 python/qemu/machine/machine.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)
python/qemu/machine/machine.py +13 -0
index 07ac5a710b..12e5e37bff 100644
@@ -51,6 +51,11 @@
 
 
 LOG = logging.getLogger(__name__)
+PSERIES_DEFAULT_CAPABILITIES = ("cap-cfpc=broken,"
+                                "cap-sbbc=broken,"
+                                "cap-ibs=broken,"
+                                "cap-ccf-assist=off,"
+                                "cap-fwnmi=off")
 
 
 class QEMUMachineError(Exception):
@@ -447,6 +452,14 @@ def _launch(self) -> None:
         """
         Launch the VM and establish a QMP connection
         """
+
+        # pseries needs extra machine options to disable Spectre/Meltdown
+        # KVM related capabilities that might not be available in the
+        # host.
+        if "qemu-system-ppc64" in self._binary:
+            if self._machine is None or "pseries" in self._machine:
+                self._args.extend(['-machine', PSERIES_DEFAULT_CAPABILITIES])
+
         self._pre_launch()
         LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args))
 
Signature · 1 line
2.32.0
#5 lore raw

PATCH 3/5 avocado/multiprocess.py: use tags=machine:pc|virt

In reply to Daniel Henrique Barboza

Assigning the machine type via the avocado tag will set self.machine from QEMUSystemTest and avoid the need to set the machine type by using self.vm.add_args().

do_test() was changed to receive a 'machine_opts' that will allow the aarch64 test to pass the additional '-machine gic-version=3' parameter it requires.

---
 tests/avocado/multiprocess.py | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
tests/avocado/multiprocess.py +8 -6
index 80a3b8f442..9a7cc6becb 100644
@@ -19,7 +19,7 @@ class Multiprocess(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
     def do_test(self, kernel_url, initrd_url, kernel_command_line,
-                machine_type):
+                machine_opts=None):
         """Main test method"""
         self.require_accelerator('kvm')
 
@@ -43,7 +43,8 @@ def do_test(self, kernel_url, initrd_url, kernel_command_line,
 
         # Create proxy process
         self.vm.set_console()
-        self.vm.add_args('-machine', machine_type)
+        if machine_opts:
+            self.vm.add_args('-machine', machine_opts)
         self.vm.add_args('-accel', 'kvm')
         self.vm.add_args('-cpu', 'host')
         self.vm.add_args('-object',
@@ -67,6 +68,7 @@ def do_test(self, kernel_url, initrd_url, kernel_command_line,
     def test_multiprocess_x86_64(self):
         """
         :avocado: tags=arch:x86_64
+        :avocado: tags=machine:pc
         """
         kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
                       '/linux/releases/31/Everything/x86_64/os/images'
@@ -76,12 +78,12 @@ def test_multiprocess_x86_64(self):
                       '/pxeboot/initrd.img')
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                                'console=ttyS0 rdinit=/bin/bash')
-        machine_type = 'pc'
-        self.do_test(kernel_url, initrd_url, kernel_command_line, machine_type)
+        self.do_test(kernel_url, initrd_url, kernel_command_line)
 
     def test_multiprocess_aarch64(self):
         """
         :avocado: tags=arch:aarch64
+        :avocado: tags=machine:virt
         """
         kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
                       '/linux/releases/31/Everything/aarch64/os/images'
@@ -91,5 +93,5 @@ def test_multiprocess_aarch64(self):
                       '/pxeboot/initrd.img')
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                                'rdinit=/bin/bash console=ttyAMA0')
-        machine_type = 'virt,gic-version=3'
-        self.do_test(kernel_url, initrd_url, kernel_command_line, machine_type)
+        machine_opts = 'gic-version=3'
+        self.do_test(kernel_url, initrd_url, kernel_command_line, machine_opts)
Signature · 1 line
2.32.0
#3 lore raw

Re: [PATCH 2/5] machine.py: add default pseries params in machine.py

In reply to Daniel Henrique Barboza
Quoted 73 lines · Daniel Henrique Barboza

pSeries guests set a handful of machine capabilities on by default, all of them related to security mitigations, that aren't always available in the host.

This means that, as is today, running avocado in a Power9 server without the proper firmware support, and with --disable-tcg, this error will occur:

(1/1) tests/avocado/info_usernet.py:InfoUsernet.test_hostfwd: ERROR:
ConnectError:
Failed to establish session: EOFError\n Exit code: 1\n (...) (...)
Command: ./qemu-system-ppc64 -display none -vga none (...)
Output: qemu-system-ppc64: warning: netdev vnet has no peer
qemu-system-ppc64: Requested safe cache capability level not supported by KVM
Try appending -machine cap-cfpc=broken

info_usernet.py happens to trigger this error first, but all tests would fail in this configuration because the host does not support the default 'cap-cfpc' capability.

A similar situation was already fixed a couple of years ago by Greg Kurz (commit 63d57c8f91d0) but it was focused on TCG warnings for these same capabilities and running C qtests. This commit ended up preventing the problem we're facing with avocado when running qtests with KVM support.

This patch does a similar approach by amending machine.py to disable these security capabilities in case we're running a pseries guest. The change is made in the _launch() callback to be sure that we're already commited into launching the guest. It's also worth noticing that we're relying on self._machine being set accordingly (i.e. via tag:machine), which is currently the case for all ppc64 related avocado tests.

---
 python/qemu/machine/machine.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)
python/qemu/machine/machine.py +13 -1
index 07ac5a710b..12e5e37bff 100644
@@ -51,6 +51,11 @@
 
 
 LOG = logging.getLogger(__name__)
+PSERIES_DEFAULT_CAPABILITIES = ("cap-cfpc=broken,"
+                                "cap-sbbc=broken,"
+                                "cap-ibs=broken,"
+                                "cap-ccf-assist=off,"
+                                "cap-fwnmi=off")
 
 
 class QEMUMachineError(Exception):
@@ -447,6 +452,14 @@ def _launch(self) -> None:
         """
         Launch the VM and establish a QMP connection
         """
+
+        # pseries needs extra machine options to disable Spectre/Meltdown
+        # KVM related capabilities that might not be available in the
+        # host.
+        if "qemu-system-ppc64" in self._binary:
+            if self._machine is None or "pseries" in self._machine:
+                self._args.extend(['-machine',
 PSERIES_DEFAULT_CAPABILITIES])
+
         self._pre_launch()
         LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args))
 
--
Quoted 2 lines

2.32.0

Hm, okay.

I have plans to try and factor the machine appliance out and into an upstream package in the near future, so I want to avoid more hardcoding of defaults.

Does avocado have a subclass of QEMUMachine where it might be more appropriate to stick this bandaid? Can we make one?

(I don't think iotests runs into this problem because we always use machine:none there, I think. VM tests might have a similar problem though, and then it'd be reasonable to want the bandaid here in machine.py ... well, boo. okay.)

My verdict is that it's a bandaid, but I'll accept it if the avocado folks agree to it and I'll sort it out later when I do my rewrite.

I don't think I have access to a power9 machine to test this with either, so I might want a tested-by from someone who does.

--js

Quoted 1 line
#4 lore raw

Re: [PATCH 2/5] machine.py: add default pseries params in machine.py

Quoted 87 lines · John Snow

On Mon, May 16, 2022, 12:53 PM Daniel Henrique Barboza <[email protected] <mailto:[email protected]>> wrote:

pSeries guests set a handful of machine capabilities on by default, all of them related to security mitigations, that aren't always available in the host.

This means that, as is today, running avocado in a Power9 server without the proper firmware support, and with --disable-tcg, this error will occur:

 (1/1) tests/avocado/info_usernet.py:InfoUsernet.test_hostfwd:
ERROR: ConnectError:
Failed to establish session: EOFError\n  Exit code: 1\n  (...) (...)
        Command: ./qemu-system-ppc64 -display none -vga none (...)
        Output: qemu-system-ppc64: warning: netdev vnet has no peer
qemu-system-ppc64: Requested safe cache capability level not supported by KVM
Try appending -machine cap-cfpc=broken

info_usernet.py happens to trigger this error first, but all tests would fail in this configuration because the host does not support the default 'cap-cfpc' capability.

A similar situation was already fixed a couple of years ago by Greg Kurz (commit 63d57c8f91d0) but it was focused on TCG warnings for these same capabilities and running C qtests. This commit ended up preventing the problem we're facing with avocado when running qtests with KVM support.

This patch does a similar approach by amending machine.py to disable these security capabilities in case we're running a pseries guest. The change is made in the _launch() callback to be sure that we're already commited into launching the guest. It's also worth noticing that we're relying on self._machine being set accordingly (i.e. via tag:machine), which is currently the case for all ppc64 related avocado tests.

<mailto:[email protected]>>
---
 python/qemu/machine/machine.py | 13 +++++++++++++ 1 file changed, 13 insertions(+)

diff --git a/python/qemu/machine/machine.py
b/python/qemu/machine/machine.py
index 07ac5a710b..12e5e37bff 100644
--- a/python/qemu/machine/machine.py
+++ b/python/qemu/machine/machine.py
@@ -51,6 +51,11 @@

 LOG = logging.getLogger(__name__)
+PSERIES_DEFAULT_CAPABILITIES = ("cap-cfpc=broken," +                                "cap-sbbc=broken," +                                "cap-ibs=broken," +                                "cap-ccf-assist=off," +                                "cap-fwnmi=off")

 class QEMUMachineError(Exception):
@@ -447,6 +452,14 @@ def _launch(self) -> None:
         """
         Launch the VM and establish a QMP connection
         """
+
+        # pseries needs extra machine options to disable Spectre/Meltdown
+        # KVM related capabilities that might not be available in the +        # host.
+        if "qemu-system-ppc64" in self._binary:
+            if self._machine is None or "pseries" in self._machine:
+                self._args.extend(['-machine', PSERIES_DEFAULT_CAPABILITIES])
+
         self._pre_launch()
         LOG.debug('VM launch command: %r', '
'.join(self._qemu_full_args))

--
2.32.0

Hm, okay.

I have plans to try and factor the machine appliance out and into an upstream package in the near future, so I want to avoid more hardcoding of defaults.

Does avocado have a subclass of QEMUMachine where it might be more appropriate to stick this bandaid? Can we make one?

Quoted 8 lines

(I don't think iotests runs into this problem because we always use machine:none there, I think. VM tests might have a similar problem though, and then it'd be reasonable to want the bandaid here in machine.py ... well, boo. okay.)

My verdict is that it's a bandaid, but I'll accept it if the avocado folks agree to it and I'll sort it out later when I do my rewrite.

I don't think I have access to a power9 machine to test this with either, so I might want a tested-by from someone who does.

--js

Unfortunately, none of our POWER9 machines had a firmware old enough to be affected by this issue. The closest I can test is a nested KVM-HV with L0 using cap-cfpc=broken, so the L1 receives the quoted message when running 'make check-avocado'.

With this setup I can confirm that the patch fixes this error, so

Thanks,
Matheus K. Ferst
Instituto de Pesquisas ELDORADO <http://www.eldorado.org.br/>
Analista de Software
Aviso Legal - Disclaimer <https://www.eldorado.org.br/disclaimer.html>