Skip to contentSkip to conversation

pc: Support coldplugging of virtio-pmem-pci devices on all buses

PATCH In review David Hildenbrand 5 participants 10 messages May 25, 2020 – Jun 9, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pc: Support coldplugging of virtio-pmem-pci devices on all buses

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/i386/pc.c +10 -8
index 2128f3d6fe..c740495eb6 100644
@@ -1663,13 +1663,13 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,
     HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
     Error *local_err = NULL;
 
-    if (!hotplug_dev2) {
+    if (!hotplug_dev2 && dev->hotplugged) {
         /*
          * Without a bus hotplug handler, we cannot control the plug/unplug
-         * order. This should never be the case on x86, however better add
-         * a safety net.
+         * order. We should never reach this point when hotplugging on x86,
+         * however, better add a safety net.
          */
-        error_setg(errp, "virtio-pmem-pci not supported on this bus.");
+        error_setg(errp, "virtio-pmem-pci hotplug not supported on this bus.");
         return;
     }
     /*
@@ -1678,7 +1678,7 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,
      */
     memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev), NULL,
                            &local_err);
-    if (!local_err) {
+    if (!local_err && hotplug_dev2) {
         hotplug_handler_pre_plug(hotplug_dev2, dev, &local_err);
     }
     error_propagate(errp, local_err);
@@ -1696,9 +1696,11 @@ static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev,
      * device bits.
      */
     memory_device_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
-    hotplug_handler_plug(hotplug_dev2, dev, &local_err);
-    if (local_err) {
-        memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+    if (hotplug_dev2) {
+        hotplug_handler_plug(hotplug_dev2, dev, &local_err);
+        if (local_err) {
+            memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+        }
     }
     error_propagate(errp, local_err);
 }
Signature · 1 line
2.25.4
Reviewed-by on v1 1/1
#1 lore raw
Quoted 65 lines

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/i386/pc.c +5 -6
index 2128f3d6fe..c740495eb6 100644
@@ -1663,13 +1663,13 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,
     HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
     Error *local_err = NULL;
 
-    if (!hotplug_dev2) {
+    if (!hotplug_dev2 && dev->hotplugged) {
         /*
          * Without a bus hotplug handler, we cannot control the plug/unplug
-         * order. This should never be the case on x86, however better add
-         * a safety net.
+         * order. We should never reach this point when hotplugging on x86,
+         * however, better add a safety net.
          */
-        error_setg(errp, "virtio-pmem-pci not supported on this bus.");
+        error_setg(errp, "virtio-pmem-pci hotplug not supported on this bus.");
         return;
     }
     /*
@@ -1678,7 +1678,7 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,
      */
     memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev), NULL,
                            &local_err);
-    if (!local_err) {
+    if (!local_err && hotplug_dev2) {
         hotplug_handler_pre_plug(hotplug_dev2, dev, &local_err);
     }
     error_propagate(errp, local_err);
@@ -1696,9 +1696,11 @@ static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev,
      * device bits.
      */
     memory_device_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
-    hotplug_handler_plug(hotplug_dev2, dev, &local_err);

hw/i386/pc.c static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev, · line 1706

Quoted 10 lines
hw/i386/pc.c +5 -2
-    if (local_err) {
-        memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+    if (hotplug_dev2) {
+        hotplug_handler_plug(hotplug_dev2, dev, &local_err);
+        if (local_err) {
+            memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+        }
     }
     error_propagate(errp, local_err);
 }

This looks good to me & will allow to cold-plug the virtio-pmem device on bus if they don't support hot-plug.

#2 lore raw
In reply to David Hildenbrand
Quoted 25 lines · David Hildenbrand

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Quoted 3 lines
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

Thanks for the patch David. I still seem to face a different error though.

2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem option

Following is my domain xml file.

Vivek

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>vm2-f30</name> <uuid>c15287f2-9e66-4867-99dc-d77ece6486c8</uuid>
<metadata>

    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://fedoraproject.org/fedora/30"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>67108864</memory>
  <currentMemory unit='KiB'>67108864</currentMemory>
  <vcpu placement='static'>64</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm2-f30.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:51:b6:34'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-fsdev'/>
    <qemu:arg value='local,id=extra1,path=/mnt/virtio-9p-dir,security_model=none'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='virtio-9p-pci,fsdev=extra1,mount_tag=hostShared'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-file,id=mem,size=64G,mem-path=/dev/shm,share=on'/>
    <qemu:arg value='-numa'/>
    <qemu:arg value='node,memdev=mem'/>
    <qemu:arg value='-chardev'/>
    <qemu:arg value='socket,id=char0,path=/tmp/vhostqemu'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vhost-user-fs-pci,id=myvirtiofs,chardev=char0,tag=myfs,cache-size=64G,queue-size=1024'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-file,id=pmem1,mem-path=/var/lib/libvirt/images/pmem-vm2-f30.img,share=on,size=4G'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='virtio-pmem-pci,memdev=pmem1,id=nv1'/>

</qemu:commandline>
</domain>

#3 lore raw
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Quoted 3 lines
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
Quoted 7 lines

Thanks for the patch David. I still seem to face a different error though.

2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem option

Following is my domain xml file.

Vivek

Hi Vivek,

you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.

<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>

(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)

@Pankaj, do we have a virtio-pmem doc somewhere describing how to set it up?

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw
Quoted 45 lines · David Hildenbrand
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Quoted 3 lines
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
Quoted 7 lines

Thanks for the patch David. I still seem to face a different error though.

2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem option

Following is my domain xml file.

Vivek

Hi Vivek,

you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.

Quoted 6 lines

<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>

(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)

Ok, tried that.

<maxMemory slots='1' unit='KiB'>134217728</maxMemory>

And now it complains about.

error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug

So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?

Thanks
Vivek

Quoted 7 lines

@Pankaj, do we have a virtio-pmem doc somewhere describing how to set it up?

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw
Quoted 55 lines · Vivek Goyal
Quoted 45 lines · David Hildenbrand
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Quoted 3 lines
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
Quoted 7 lines

Thanks for the patch David. I still seem to face a different error though.

2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem option

Following is my domain xml file.

Vivek

Hi Vivek,

you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.

Quoted 6 lines

<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>

(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)

Ok, tried that.

Quoted 8 lines

<maxMemory slots='1' unit='KiB'>134217728</maxMemory>

And now it complains about.

error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug

So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?

That's a libvirt error message. Maybe I am confused how libvirt maps these parameters to QEMU ...

NVDIMMs under libvirt seem to be easy:

https://www.redhat.com/archives/libvir-list/2016-August/msg00055.html

Maybe the issue is that virtio-pmem has not been properly integrated into libvirt yet:

https://www.redhat.com/archives/libvir-list/2019-August/msg00007.html

And you attempts to force virtio-pmem in via qemu args does not work properly.

Maybe maxMemory in libvirt does not directly map to the QEMU variant to define the maximum physical address space reserved also for any memory devices (DIMMs, NVDIMMs, virtio-pmem, ...). Any libvirt experts that can help?

@Pankaj, did you ever get it to run with libvirt?

Quoted 3 lines

Thanks
Vivek

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw
Quoted 68 lines · David Hildenbrand
Quoted 55 lines · Vivek Goyal
Quoted 45 lines · David Hildenbrand
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Quoted 3 lines
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
Quoted 7 lines

Thanks for the patch David. I still seem to face a different error though.

2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem option

Following is my domain xml file.

Vivek

Hi Vivek,

you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.

Quoted 6 lines

<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>

(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)

Ok, tried that.

Quoted 8 lines

<maxMemory slots='1' unit='KiB'>134217728</maxMemory>

And now it complains about.

error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug

So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?

That's a libvirt error message. Maybe I am confused how libvirt maps these parameters to QEMU ...

Quoted 8 lines

NVDIMMs under libvirt seem to be easy:

https://www.redhat.com/archives/libvir-list/2016-August/msg00055.html

Maybe the issue is that virtio-pmem has not been properly integrated into libvirt yet:

https://www.redhat.com/archives/libvir-list/2019-August/msg00007.html

While libvirt has generic pmem support, it doesn't have virtio-pmem:

https://libvirt.org/formatdomain.html#elementsMemory

eg

<memory model='nvdimm' access='shared'>

    <uuid>
    <source>
      <path>/dev/dax0.0</path>
      <alignsize unit='KiB'>2048</alignsize>
      <pmem/>
    </source>
    <target>
      <size unit='KiB'>524288</size>
      <node>1</node>
      <label>
        <size unit='KiB'>128</size>
      </label>
    </target>

</memory>

Quoted 4 lines

Maybe maxMemory in libvirt does not directly map to the QEMU variant to define the maximum physical address space reserved also for any memory devices (DIMMs, NVDIMMs, virtio-pmem, ...). Any libvirt experts that can help?

<maxMemory> reflects the upper limit on what you can hot-plug at runtime:

https://libvirt.org/formatdomain.html#elementsMemoryAllocation

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw
In reply to David Hildenbrand

Hi David, Vivek,
s

Quoted 39 lines
Quoted 15 lines
Quoted 6 lines

Hi Vivek,

you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.

Quoted 6 lines

<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>

(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)

Ok, tried that.

Quoted 8 lines

<maxMemory slots='1' unit='KiB'>134217728</maxMemory>

And now it complains about.

error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug

So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?

That's a libvirt error message. Maybe I am confused how libvirt maps these parameters to QEMU ...

NVDIMMs under libvirt seem to be easy:

https://www.redhat.com/archives/libvir-list/2016-August/msg00055.html

Maybe the issue is that virtio-pmem has not been properly integrated into libvirt yet:

https://www.redhat.com/archives/libvir-list/2019-August/msg00007.html

And you attempts to force virtio-pmem in via qemu args does not work properly.

Quoted 6 lines

Maybe maxMemory in libvirt does not directly map to the QEMU variant to define the maximum physical address space reserved also for any memory devices (DIMMs, NVDIMMs, virtio-pmem, ...). Any libvirt experts that can help?

@Pankaj, did you ever get it to run with libvirt?

I did not run virtio-pmem with libvirt. That requires work at libvirt side. Created [1] document to run from Qemu command line.

[1] https://github.com/qemu/qemu/blob/master/docs/virtio-pmem.rst

#8 lore raw
In reply to David Hildenbrand
Quoted 16 lines · David Hildenbrand

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Quoted 9 lines
Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>

I assume you are still debugging Vivek's issues, right? Let me know when you feel it's time to merge this ...

Quoted 52 lines
---
 hw/i386/pc.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/i386/pc.c +10 -8
index 2128f3d6fe..c740495eb6 100644
@@ -1663,13 +1663,13 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,
     HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
     Error *local_err = NULL;
 
-    if (!hotplug_dev2) {
+    if (!hotplug_dev2 && dev->hotplugged) {
         /*
          * Without a bus hotplug handler, we cannot control the plug/unplug
-         * order. This should never be the case on x86, however better add
-         * a safety net.
+         * order. We should never reach this point when hotplugging on x86,
+         * however, better add a safety net.
          */
-        error_setg(errp, "virtio-pmem-pci not supported on this bus.");
+        error_setg(errp, "virtio-pmem-pci hotplug not supported on this bus.");
         return;
     }
     /*
@@ -1678,7 +1678,7 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,
      */
     memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev), NULL,
                            &local_err);
-    if (!local_err) {
+    if (!local_err && hotplug_dev2) {
         hotplug_handler_pre_plug(hotplug_dev2, dev, &local_err);
     }
     error_propagate(errp, local_err);
@@ -1696,9 +1696,11 @@ static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev,
      * device bits.
      */
     memory_device_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
-    hotplug_handler_plug(hotplug_dev2, dev, &local_err);
-    if (local_err) {
-        memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+    if (hotplug_dev2) {
+        hotplug_handler_plug(hotplug_dev2, dev, &local_err);
+        if (local_err) {
+            memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+        }
     }
     error_propagate(errp, local_err);
 }
Signature · 1 line
2.25.4
#9 lore raw
Quoted 27 lines · Michael S. Tsirkin
Quoted 16 lines · David Hildenbrand

E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"

Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.

Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"

Quoted 9 lines
Cc Pankaj Gupta <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Quoted 2 lines

I assume you are still debugging Vivek's issues, right? Let me know when you feel it's time to merge this ...

The remain issue is lack of libvirt support. This is good to be merged from my point of view.

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb