pc: Support coldplugging of virtio-pmem-pci devices on all buses
PATCH 1/1 pc: Support coldplugging of virtio-pmem-pci devices on all buses
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"
Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before memory_device_plug() wired up the memory device bits.
Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"
Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"
--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
hw/i386/pc.c +10 -8
@@ -1663,13 +1663,13 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,HotplugHandler *hotplug_dev2 =qdev_get_bus_hotplug_handler (dev);Error *local_err =NULL ; -if (!hotplug_dev2) { +if (!hotplug_dev2 && dev->hotplugged) {/* * Without a bus hotplug handler, we cannot control the plug/unplug -* order. This should never be the case on x86, however better add -* a safety net. +* order. We should never reach this point when hotplugging on x86, +* however, better add a safety net. */ -error_setg (errp,"virtio-pmem-pci not supported on this bus." ); +error_setg (errp,"virtio-pmem-pci hotplug not supported on this bus." );return ; }/*
@@ -1678,7 +1678,7 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,*/ memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev),NULL , &local_err); -if (!local_err) { +if (!local_err && hotplug_dev2) {hotplug_handler_pre_plug (hotplug_dev2, dev, &local_err); }error_propagate (errp, local_err);
@@ -1696,9 +1696,11 @@ static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev,* device bits. */ memory_device_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); -hotplug_handler_plug (hotplug_dev2, dev, &local_err); -if (local_err) { -memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); +if (hotplug_dev2) { +hotplug_handler_plug (hotplug_dev2, dev, &local_err); +if (local_err) { +memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); + } }error_propagate (errp, local_err); }
Signature · 1 line
2.25.4
Quoted 65 lines
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)hw/i386/pc.c +5
-6@@ -1663,13 +1663,13 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,HotplugHandler *hotplug_dev2 =qdev_get_bus_hotplug_handler (dev);Error *local_err =NULL ; -if (!hotplug_dev2) { +if (!hotplug_dev2 && dev->hotplugged) {/* * Without a bus hotplug handler, we cannot control the plug/unplug -* order. This should never be the case on x86, however better add -* a safety net. +* order. We should never reach this point when hotplugging on x86, +* however, better add a safety net. */ -error_setg (errp,"virtio-pmem-pci not supported on this bus." ); +error_setg (errp,"virtio-pmem-pci hotplug not supported on this bus." );return ; }/* @@ -1678,7 +1678,7 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,*/ memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev),NULL , &local_err); -if (!local_err) { +if (!local_err && hotplug_dev2) {hotplug_handler_pre_plug (hotplug_dev2, dev, &local_err); }error_propagate (errp, local_err);@@ -1696,9 +1696,11 @@ static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev,* device bits. */ memory_device_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); -hotplug_handler_plug (hotplug_dev2, dev, &local_err);
hw/i386/pc.c static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev, · line 1706
Quoted 10 lines
hw/i386/pc.c +5
-2-if (local_err) { -memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); +if (hotplug_dev2) { +hotplug_handler_plug (hotplug_dev2, dev, &local_err); +if (local_err) { +memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); + } }error_propagate (errp, local_err); }
This looks good to me & will allow to cold-plug the virtio-pmem device on bus if they don't support hot-plug.
Quoted 25 lines · David Hildenbrand
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 3 lines
--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
Thanks for the patch David. I still seem to face a different error though.
2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem option
Following is my domain xml file.
Vivek
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>vm2-f30</name> <uuid>c15287f2-9e66-4867-99dc-d77ece6486c8</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://fedoraproject.org/fedora/30"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>67108864</memory> <currentMemory unit='KiB'>67108864</currentMemory> <vcpu placement='static'>64</vcpu> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='host-model' check='partial'> <model fallback='allow'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm2-f30.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='sda' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:51:b6:34'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'> <listen type='address'/> <image compression='off'/> </graphics> <sound model='ich9'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/urandom</backend> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </rng> </devices> <qemu:commandline> <qemu:arg value='-fsdev'/> <qemu:arg value='local,id=extra1,path=/mnt/virtio-9p-dir,security_model=none'/> <qemu:arg value='-device'/> <qemu:arg value='virtio-9p-pci,fsdev=extra1,mount_tag=hostShared'/> <qemu:arg value='-object'/> <qemu:arg value='memory-backend-file,id=mem,size=64G,mem-path=/dev/shm,share=on'/> <qemu:arg value='-numa'/> <qemu:arg value='node,memdev=mem'/> <qemu:arg value='-chardev'/> <qemu:arg value='socket,id=char0,path=/tmp/vhostqemu'/> <qemu:arg value='-device'/> <qemu:arg value='vhost-user-fs-pci,id=myvirtiofs,chardev=char0,tag=myfs,cache-size=64G,queue-size=1024'/> <qemu:arg value='-object'/> <qemu:arg value='memory-backend-file,id=pmem1,mem-path=/var/lib/libvirt/images/pmem-vm2-f30.img,share=on,size=4G'/> <qemu:arg value='-device'/> <qemu:arg value='virtio-pmem-pci,memdev=pmem1,id=nv1'/>
</qemu:commandline>
</domain>
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Quoted 3 lines
--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
Quoted 7 lines
Thanks for the patch David. I still seem to face a different error though.
2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem optionFollowing is my domain xml file.
Vivek
Hi Vivek,
you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.
<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>
(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)
@Pankaj, do we have a virtio-pmem doc somewhere describing how to set it up?
Signature · 3 lines
Thanks, David / dhildenb
Quoted 45 lines · David Hildenbrand
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Quoted 3 lines
--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)Quoted 7 lines
Thanks for the patch David. I still seem to face a different error though.
2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem optionFollowing is my domain xml file.
Vivek
Hi Vivek,
you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.
Quoted 6 lines
<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)
Ok, tried that.
<maxMemory slots='1' unit='KiB'>134217728</maxMemory>
And now it complains about.
error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug
So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?
Thanks
Vivek
Quoted 7 lines
@Pankaj, do we have a virtio-pmem doc somewhere describing how to set it up?
Signature · 3 lines
Thanks, David / dhildenb
Quoted 55 lines · Vivek Goyal
Quoted 45 lines · David Hildenbrand
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Quoted 3 lines
--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)Quoted 7 lines
Thanks for the patch David. I still seem to face a different error though.
2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem optionFollowing is my domain xml file.
Vivek
Hi Vivek,
you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.
Quoted 6 lines
<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)
Ok, tried that.
Quoted 8 lines
<maxMemory slots='1' unit='KiB'>134217728</maxMemory>
And now it complains about.
error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug
So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?
That's a libvirt error message. Maybe I am confused how libvirt maps these parameters to QEMU ...
NVDIMMs under libvirt seem to be easy:
https://www.redhat.com/archives/libvir-list/2016-August/msg00055.html
Maybe the issue is that virtio-pmem has not been properly integrated into libvirt yet:
https://www.redhat.com/archives/libvir-list/2019-August/msg00007.html
And you attempts to force virtio-pmem in via qemu args does not work properly.
Maybe maxMemory in libvirt does not directly map to the QEMU variant to define the maximum physical address space reserved also for any memory devices (DIMMs, NVDIMMs, virtio-pmem, ...). Any libvirt experts that can help?
@Pankaj, did you ever get it to run with libvirt?
Quoted 3 lines
Thanks
Vivek
Signature · 3 lines
Thanks, David / dhildenb
Quoted 68 lines · David Hildenbrand
Quoted 55 lines · Vivek Goyal
Quoted 45 lines · David Hildenbrand
Quoted 30 lines · Vivek Goyal
Quoted 25 lines · David Hildenbrand
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Quoted 3 lines
--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)Quoted 7 lines
Thanks for the patch David. I still seem to face a different error though.
2020-05-26T13:26:05.720617Z qemu-system-x86_64: -device virtio-pmem-pci,memdev=pmem1,id=nv1: memory devices (e.g. for memory hotplug) are not enabled, please specify the maxmem optionFollowing is my domain xml file.
Vivek
Hi Vivek,
you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.
Quoted 6 lines
<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)
Ok, tried that.
Quoted 8 lines
<maxMemory slots='1' unit='KiB'>134217728</maxMemory>
And now it complains about.
error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug
So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?
That's a libvirt error message. Maybe I am confused how libvirt maps these parameters to QEMU ...
Quoted 8 lines
NVDIMMs under libvirt seem to be easy:
https://www.redhat.com/archives/libvir-list/2016-August/msg00055.html
Maybe the issue is that virtio-pmem has not been properly integrated into libvirt yet:
https://www.redhat.com/archives/libvir-list/2019-August/msg00007.html
While libvirt has generic pmem support, it doesn't have virtio-pmem:
https://libvirt.org/formatdomain.html#elementsMemory
eg
<memory model='nvdimm' access='shared'>
<uuid>
<source>
<path>/dev/dax0.0</path>
<alignsize unit='KiB'>2048</alignsize>
<pmem/>
</source>
<target>
<size unit='KiB'>524288</size>
<node>1</node>
<label>
<size unit='KiB'>128</size>
</label>
</target></memory>
Quoted 4 lines
Maybe maxMemory in libvirt does not directly map to the QEMU variant to define the maximum physical address space reserved also for any memory devices (DIMMs, NVDIMMs, virtio-pmem, ...). Any libvirt experts that can help?
<maxMemory> reflects the upper limit on what you can hot-plug at runtime:
https://libvirt.org/formatdomain.html#elementsMemoryAllocation
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi David, Vivek,
s
Quoted 39 lines
Quoted 15 lines
Quoted 6 lines
Hi Vivek,
you have to declare the maxMemory option. Memory devices like virtio-pmem-pci reside in RAM like a pc-dimm or a nvdimm. If your virtio-pmem device will be 4GB, you have to add that to maxMemory.
Quoted 6 lines
<memory unit='GiB'>64</memory>
<maxMemory unit='GiB'>68</maxMemory>
<currentMemory unit='GiB'>64</currentMemory>(you might have to add "slots='0'" or "slots='1'" to maxMemory to make libvirt happy)
Ok, tried that.
Quoted 8 lines
<maxMemory slots='1' unit='KiB'>134217728</maxMemory>
And now it complains about.
error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug
So ultimately it seems to be wanting me to somehow enable memory hotplug to be able to use virtio-pmem?
That's a libvirt error message. Maybe I am confused how libvirt maps these parameters to QEMU ...
NVDIMMs under libvirt seem to be easy:
https://www.redhat.com/archives/libvir-list/2016-August/msg00055.html
Maybe the issue is that virtio-pmem has not been properly integrated into libvirt yet:
https://www.redhat.com/archives/libvir-list/2019-August/msg00007.html
And you attempts to force virtio-pmem in via qemu args does not work properly.
Quoted 6 lines
Maybe maxMemory in libvirt does not directly map to the QEMU variant to define the maximum physical address space reserved also for any memory devices (DIMMs, NVDIMMs, virtio-pmem, ...). Any libvirt experts that can help?
@Pankaj, did you ever get it to run with libvirt?
I did not run virtio-pmem with libvirt. That requires work at libvirt side. Created [1] document to run from Qemu command line.
[1] https://github.com/qemu/qemu/blob/master/docs/virtio-pmem.rst
Quoted 16 lines · David Hildenbrand
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"
Quoted 9 lines
Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
I assume you are still debugging Vivek's issues, right? Let me know when you feel it's time to merge this ...
Quoted 52 lines
--- hw/i386/pc.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)hw/i386/pc.c +10
-8@@ -1663,13 +1663,13 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,HotplugHandler *hotplug_dev2 =qdev_get_bus_hotplug_handler (dev);Error *local_err =NULL ; -if (!hotplug_dev2) { +if (!hotplug_dev2 && dev->hotplugged) {/* * Without a bus hotplug handler, we cannot control the plug/unplug -* order. This should never be the case on x86, however better add -* a safety net. +* order. We should never reach this point when hotplugging on x86, +* however, better add a safety net. */ -error_setg (errp,"virtio-pmem-pci not supported on this bus." ); +error_setg (errp,"virtio-pmem-pci hotplug not supported on this bus." );return ; }/* @@ -1678,7 +1678,7 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev,*/ memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev),NULL , &local_err); -if (!local_err) { +if (!local_err && hotplug_dev2) {hotplug_handler_pre_plug (hotplug_dev2, dev, &local_err); }error_propagate (errp, local_err);@@ -1696,9 +1696,11 @@ static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev,* device bits. */ memory_device_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); -hotplug_handler_plug (hotplug_dev2, dev, &local_err); -if (local_err) { -memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); +if (hotplug_dev2) { +hotplug_handler_plug (hotplug_dev2, dev, &local_err); +if (local_err) { +memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); + } }error_propagate (errp, local_err); }Signature · 1 line
2.25.4
Quoted 27 lines · Michael S. Tsirkin
Quoted 16 lines · David Hildenbrand
E.g., with "pc-q35-4.2", trying to coldplug a virtio-pmem-pci devices results in
"virtio-pmem-pci not supported on this bus"Reasons is, that the bus does not support hotplug and, therefore, does not have a hotplug handler. Let's allow coldplugging virtio-pmem devices on such buses. The hotplug order is only relevant for virtio-pmem-pci when the guest is already alive and the device is visible before
memory_device_plug()wired up the memory device bits.Hotplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Hotunplug attempts will still fail with:
"Error: Bus 'pcie.0' does not support hotplugging"Quoted 9 lines
Reported-by Vivek Goyal <[email protected]>Cc Pankaj Gupta <[email protected]>Cc Igor Mammedov <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 2 lines
I assume you are still debugging Vivek's issues, right? Let me know when you feel it's time to merge this ...
The remain issue is lack of libvirt support. This is good to be merged from my point of view.
Thanks!
Signature · 3 lines
Thanks, David / dhildenb