Skip to contentSkip to conversation

s390x: support virtio-mem-pci

PATCH In review David Hildenbrand 4 participants 8 messages Jan 28, 2025 – Feb 21, 2025

lore
Share & tools
lore
#0 lore raw

v2 cover s390x: support virtio-mem-pci

This is based-on [1], which adds MSI-X support to virtio-balloon-pci, but can be applied independently.

Turns out it is fairly easy to get virtio-mem-pci running on s390x. We only have to add MSI-X support to virtio-mem-pci, and wire-up the (un)plugging in the machine.

Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems to be working as expected.

The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM for it to work.

[1] https://lkml.kernel.org/r/[email protected]

v1 -> v2:
* There are no transitional/non_transitional devices for virtio-mem
* Spell out removal of "return;" in second patch

Cc Eduardo Habkost <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Cc Richard Henderson <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Ilya Leoshkevich <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Eric Farman <[email protected]>
Cc Thomas Huth <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Cornelia Huck <[email protected]>
Cc Boris Fiuczynski <[email protected]>
Cc Michal Privoznik <[email protected]>
Cc Mario Casquero <[email protected]>

David Hildenbrand (2):
virtio-mem-pci: Allow setting nvectors, so we can use MSI-X s390x/s390-virtio-ccw: Support plugging PCI-based virtio memory
devices

 hw/core/machine.c          |  1 +
 hw/s390x/s390-virtio-ccw.c | 20 ++++++++++++++------
 hw/virtio/virtio-mem-pci.c | 12 ++++++++++++
 3 files changed, 27 insertions(+), 6 deletions(-)
Signature · 1 line
2.48.1
#1 lore raw

PATCH v2 1/2 virtio-mem-pci: Allow setting nvectors, so we can use MSI-X

Let's do it similar as virtio-balloon-pci. With this change, we can use virtio-mem-pci on s390x, although plugging will still fail until properly wired up in the machine.

No need to worry about transitional/non_transitional devices, because they don't exist for virtio-mem.

---
 hw/core/machine.c          |  1 +
 hw/virtio/virtio-mem-pci.c | 12 ++++++++++++
 2 files changed, 13 insertions(+)
hw/core/machine.c +1 -0
index 8f396ef803..7b74cde10a 100644
@@ -41,6 +41,7 @@ GlobalProperty hw_compat_9_2[] = {
     { "virtio-balloon-pci", "vectors", "0" },
     { "virtio-balloon-pci-transitional", "vectors", "0" },
     { "virtio-balloon-pci-non-transitional", "vectors", "0" },
+    { "virtio-mem-pci", "vectors", "0" },
 };
 const size_t hw_compat_9_2_len = G_N_ELEMENTS(hw_compat_9_2);
 
hw/virtio/virtio-mem-pci.c +12 -0
index 1b4e9a3284..6cc5f0fd3b 100644
@@ -22,6 +22,10 @@ static void virtio_mem_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
     VirtIOMEMPCI *mem_pci = VIRTIO_MEM_PCI(vpci_dev);
     DeviceState *vdev = DEVICE(&mem_pci->vdev);
 
+    if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) {
+        vpci_dev->nvectors = 2;
+    }
+
     virtio_pci_force_virtio_1(vpci_dev);
     qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
 }
@@ -152,6 +156,13 @@ static void virtio_mem_pci_set_requested_size(Object *obj, Visitor *v,
     object_property_set(OBJECT(&pci_mem->vdev), name, v, errp);
 }
 
+static const Property virtio_mem_pci_class_properties[] = {
+    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
+                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
+    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
+                       DEV_NVECTORS_UNSPECIFIED),
+};
+
 static void virtio_mem_pci_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -164,6 +175,7 @@ static void virtio_mem_pci_class_init(ObjectClass *klass, void *data)
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
     pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
     pcidev_k->class_id = PCI_CLASS_OTHERS;
+    device_class_set_props(dc, virtio_mem_pci_class_properties);
 
     mdc->get_addr = virtio_mem_pci_get_addr;
     mdc->set_addr = virtio_mem_pci_set_addr;
Signature · 1 line
2.48.1
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] virtio-mem-pci: Allow setting nvectors, so we can use MSI-X

Quoted 4 lines · David Hildenbrand

Let's do it similar as virtio-balloon-pci. With this change, we can use virtio-mem-pci on s390x, although plugging will still fail until properly wired up in the machine.

Quoted 8 lines

No need to worry about transitional/non_transitional devices, because they don't exist for virtio-mem.

---
  hw/core/machine.c          |  1 +
  hw/virtio/virtio-mem-pci.c | 12 ++++++++++++
  2 files changed, 13 insertions(+)
#3 lore raw

PATCH v2 2/2 s390x/s390-virtio-ccw: Support plugging PCI-based virtio memory devices

In reply to David Hildenbrand

Let's just wire it up, unlocking virtio-mem-pci support on s390x.

While at it, drop the "return;" in s390_machine_device_unplug_request(), to make it look like the other handlers.

---
 hw/s390x/s390-virtio-ccw.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
hw/s390x/s390-virtio-ccw.c +14 -6
index 3af613d4e9..71f3443a53 100644
@@ -554,8 +554,7 @@ static void s390_machine_device_pre_plug(HotplugHandler *hotplug_dev,
     if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_CCW)) {
         virtio_ccw_md_pre_plug(VIRTIO_MD_CCW(dev), MACHINE(hotplug_dev), errp);
     } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) {
-        error_setg(errp,
-                   "PCI-attached virtio based memory devices not supported");
+        virtio_md_pci_pre_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp);
     }
 }
 
@@ -566,7 +565,8 @@ static void s390_machine_device_plug(HotplugHandler *hotplug_dev,
 
     if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
         s390_cpu_plug(hotplug_dev, dev, errp);
-    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_CCW)) {
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_CCW) ||
+               object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) {
         /*
          * At this point, the device is realized and set all memdevs mapped, so
          * qemu_maxrampagesize() will pick up the page sizes of these memdevs
@@ -580,7 +580,11 @@ static void s390_machine_device_plug(HotplugHandler *hotplug_dev,
                        " initial memory");
             return;
         }
-        virtio_ccw_md_plug(VIRTIO_MD_CCW(dev), MACHINE(hotplug_dev), errp);
+        if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_CCW)) {
+            virtio_ccw_md_plug(VIRTIO_MD_CCW(dev), MACHINE(hotplug_dev), errp);
+        } else {
+            virtio_md_pci_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp);
+        }
     }
 }
 
@@ -589,10 +593,12 @@ static void s390_machine_device_unplug_request(HotplugHandler *hotplug_dev,
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
         error_setg(errp, "CPU hot unplug not supported on this machine");
-        return;
     } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_CCW)) {
         virtio_ccw_md_unplug_request(VIRTIO_MD_CCW(dev), MACHINE(hotplug_dev),
                                      errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) {
+        virtio_md_pci_unplug_request(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev),
+                                     errp);
     }
 }
 
@@ -601,7 +607,9 @@ static void s390_machine_device_unplug(HotplugHandler *hotplug_dev,
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_CCW)) {
         virtio_ccw_md_unplug(VIRTIO_MD_CCW(dev), MACHINE(hotplug_dev), errp);
-     }
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) {
+        virtio_md_pci_unplug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp);
+    }
  }
 
 static CpuInstanceProperties s390_cpu_index_to_props(MachineState *ms,
Signature · 1 line
2.48.1
Tested-by on the v2 series
#4 lore raw
In reply to David Hildenbrand

This series has been successfully tested in s390x. Here the detailed test steps:

Boot up a VM already containing a memory object /home/qemu/build/qemu-system-s390x \
...
-m 4G,maxmem=20G \
-object memory-backend-ram,id=mem0,size=16G,reserve=off \
...

Check the memory devices
(qemu) info memory-devices
(qemu)

Check the PCI devices in the VM
[root@localhost ~]# lspci -v
[root@localhost ~]#

Hotplug a virtio-mem-pci device and check the memory devices (qemu) device_add virtio-mem-pci,id=vmem0,memdev=mem0,requested-size=1G (qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 1073741824
size: 1073741824
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0

Check the PCI device is correct in the guest
[root@localhost ~]# lspci -v
0001:00:00.0 Unclassified device [00ff]: Red Hat, Inc. virtio-mem (rev 01)
Subsystem: Red Hat, Inc. Device 1100
Physical Slot: 00000000
Flags: bus master, fast devsel, latency 0, IOMMU group 0 Memory at 4000000000000000 (32-bit, non-prefetchable) [virtual] [size=4K] Memory at 4001000000000000 (64-bit, prefetchable) [virtual] [size=16K]
Capabilities: [98] MSI-X: Enable+ Count=2 Masked-
Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
Capabilities: [70] Vendor Specific Information: VirtIO: Notify
Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
Capabilities: [50] Vendor Specific Information: VirtIO: ISR
Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci

Resize the virtio_mem device and verify the new size (qemu) qom-set vmem0 requested-size 4G (qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 4294967296
size: 4294967296
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0

Finally, try to perform a failed unplug, resize the device to 0, and unplug it seamlessly:
(qemu) device_del vmem0
Error: virtio-mem device cannot get unplugged while some of its memory is still plugged
(qemu) qom-set vmem0 requested-size 0
(qemu) device_del vmem0

Quoted 48 lines · David Hildenbrand

This is based-on [1], which adds MSI-X support to virtio-balloon-pci, but can be applied independently.

Turns out it is fairly easy to get virtio-mem-pci running on s390x. We only have to add MSI-X support to virtio-mem-pci, and wire-up the (un)plugging in the machine.

Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems to be working as expected.

The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM for it to work.

[1] https://lkml.kernel.org/r/[email protected]

v1 -> v2:
* There are no transitional/non_transitional devices for virtio-mem
* Spell out removal of "return;" in second patch

Cc Eduardo Habkost <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Cc Richard Henderson <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Ilya Leoshkevich <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Eric Farman <[email protected]>
Cc Thomas Huth <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Cornelia Huck <[email protected]>
Cc Boris Fiuczynski <[email protected]>
Cc Michal Privoznik <[email protected]>
Cc Mario Casquero <[email protected]>

David Hildenbrand (2):
virtio-mem-pci: Allow setting nvectors, so we can use MSI-X s390x/s390-virtio-ccw: Support plugging PCI-based virtio memory
devices

 hw/core/machine.c          |  1 +
 hw/s390x/s390-virtio-ccw.c | 20 ++++++++++++++------
 hw/virtio/virtio-mem-pci.c | 12 ++++++++++++
 3 files changed, 27 insertions(+), 6 deletions(-)

--
2.48.1

#5 lore raw
In reply to David Hildenbrand
Quoted 7 lines · David Hildenbrand

This is based-on [1], which adds MSI-X support to virtio-balloon-pci, but can be applied independently.

Turns out it is fairly easy to get virtio-mem-pci running on s390x. We only have to add MSI-X support to virtio-mem-pci, and wire-up the (un)plugging in the machine.

Quoted 7 lines

Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems to be working as expected.

The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM for it to work.

[1] https://lkml.kernel.org/r/[email protected]

Fails CI:

https://gitlab.com/mstredhat/qemu/-/jobs/9202574981

Quoted 31 lines

v1 -> v2:
* There are no transitional/non_transitional devices for virtio-mem
* Spell out removal of "return;" in second patch

Cc Eduardo Habkost <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Cc Richard Henderson <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Ilya Leoshkevich <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Eric Farman <[email protected]>
Cc Thomas Huth <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Cornelia Huck <[email protected]>
Cc Boris Fiuczynski <[email protected]>
Cc Michal Privoznik <[email protected]>
Cc Mario Casquero <[email protected]>

David Hildenbrand (2):
virtio-mem-pci: Allow setting nvectors, so we can use MSI-X s390x/s390-virtio-ccw: Support plugging PCI-based virtio memory
devices

 hw/core/machine.c          |  1 +
 hw/s390x/s390-virtio-ccw.c | 20 ++++++++++++++------
 hw/virtio/virtio-mem-pci.c | 12 ++++++++++++
 3 files changed, 27 insertions(+), 6 deletions(-)
Signature · 1 line
2.48.1
#6 lore raw
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · David Hildenbrand

This is based-on [1], which adds MSI-X support to virtio-balloon-pci, but can be applied independently.

Turns out it is fairly easy to get virtio-mem-pci running on s390x. We only have to add MSI-X support to virtio-mem-pci, and wire-up the (un)plugging in the machine.

Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems to be working as expected.

The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM for it to work.

Quoted 6 lines
Quoted 1 line

[1] https://lkml.kernel.org/r/[email protected]

Fails CI:

https://gitlab.com/mstredhat/qemu/-/jobs/9202574981

Hm, I thought the CI was happy when Thomas sent it upstream.

CRITICAL: log: qemu-system-i386: -device virtio-mem-pci: can't apply global virtio-mem-pci.vectors=0: Property 'virtio-mem-pci.vectors' not found

And the same for the virtio-balloon devices with a similar change.

Trying to run one offending cmdline on current upstream:

./qemu-system-i386 -display none -vga none -S -machine pc-q35-8.0,accel=tcg -device virtio-mem-pci
qemu-system-i386: -device virtio-mem-pci: the configuration is not prepared for memory devices (e.g., for memory hotplug), consider specifying the maxmem option

So I am not sure what is happening here? Is some commit in the pipeline breaking this?

Signature · 3 lines
Cheers,

David / dhildenb
#7 lore raw
Quoted 38 lines · David Hildenbrand
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · David Hildenbrand

This is based-on [1], which adds MSI-X support to virtio-balloon-pci, but can be applied independently.

Turns out it is fairly easy to get virtio-mem-pci running on s390x. We only have to add MSI-X support to virtio-mem-pci, and wire-up the (un)plugging in the machine.

Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems to be working as expected.

The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM for it to work.

Quoted 6 lines
Quoted 1 line

[1] https://lkml.kernel.org/r/[email protected]

Fails CI:

https://gitlab.com/mstredhat/qemu/-/jobs/9202574981

Hm, I thought the CI was happy when Thomas sent it upstream.

CRITICAL: log: qemu-system-i386: -device virtio-mem-pci: can't apply global virtio-mem-pci.vectors=0: Property 'virtio-mem-pci.vectors' not found

And the same for the virtio-balloon devices with a similar change.

Trying to run one offending cmdline on current upstream:

./qemu-system-i386 -display none -vga none -S -machine pc-q35-8.0,accel=tcg -device virtio-mem-pci
qemu-system-i386: -device virtio-mem-pci: the configuration is not prepared for memory devices (e.g., for memory hotplug), consider specifying the maxmem option

Quoted 2 lines

So I am not sure what is happening here? Is some commit in the pipeline breaking this?

Note that these two patches, and the virtio-balloon-pci MSI-X
one are already upstream, Thomas included them in his s390 MR. I see that you have them included on your branch:

https://gitlab.com/mstredhat/qemu/-/commits/v03b-20-02-2025?ref_type=heads

The following commit in that tree is messed up:

* virtio-mem-pci: Allow setting nvectors, so we can use MSI-X

Likely, you can just drop these two patches from your tree; they are alreayd upstream.

Signature · 3 lines
Cheers,

David / dhildenb