virtio-mem: Device unplug support
cover virtio-mem: Device unplug support
One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.
Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).
Briefly tested on both, x86_64 and aarch64.
David Hildenbrand (5):
pc: Properly handle unplug of virtio based memory devices arm/virt: Properly handle unplug of virtio based memory devices
virtio-mem: Prepare for unplug support of virtio-mem-pci devices
pc: Support unplug of virtio-mem-pci devices arm/virt: Support unplug of virtio-mem-pci devices
hw/arm/virt.c | 60 +++++++++++++++++++++++++++++-- hw/i386/pc.c | 66 ++++++++++++++++++++++++++++++---- hw/virtio/virtio-mem-pci.c | 42 ++++++++++++++++++++-- hw/virtio/virtio-mem-pci.h | 2 ++ hw/virtio/virtio-mem.c | 24 +++++++++++++ include/hw/virtio/virtio-mem.h | 2 ++ 6 files changed, 183 insertions(+), 13 deletions(-)
Signature · 1 line
2.40.1
PATCH 1/5 pc: Properly handle unplug of virtio based memory devices
While we fence unplug requests from the outside, the VM can still trigger unplug of virtio based memory devices, for example, in Linux doing:
# echo 0 > /sys/bus/pci/slots/3/power
While doing that is not really expected to work without harming the guest OS (e.g., removing a virtio-mem device while it still provides memory), let's make sure that we properly handle it on the QEMU side.
We'll support unplugging of virtio-mem devices in some configurations next.
--- hw/i386/pc.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
hw/i386/pc.c +19 -1
@@ -1559,7 +1559,25 @@ static void pc_virtio_md_pci_unplug_request(HotplugHandler *hotplug_dev,static void pc_virtio_md_pci_unplug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) { -/* We don't support hot unplug of virtio based memory devices */ +HotplugHandler *hotplug_dev2 =qdev_get_bus_hotplug_handler (dev); +Error *local_err =NULL ; + +/* Unplug the memory device while it is still realized. */ +memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); + +if (hotplug_dev2) { +hotplug_handler_unplug (hotplug_dev2, dev, &local_err); +if (local_err) { +/* Not expected to fail ... but still try to recover. */ +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); +error_propagate (errp, local_err); +return ; + } + }else { +/* Very unexpected, but let's just try to do the right thing. */ +warn_report ("Unexpected unplug of virtio based memory device" ); +qdev_unrealize (dev); + } } static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
Signature · 1 line
2.40.1
PATCH 2/5 arm/virt: Properly handle unplug of virtio based memory devices
In reply to David HildenbrandWhile we fence unplug requests from the outside, the VM can still trigger unplug of virtio based memory devices, for example, in Linux doing:
# echo 0 > /sys/bus/pci/slots/3/power
While doing that is not really expected to work without harming the guest OS (e.g., removing a virtio-mem device while it still provides memory), let's make sure that we properly handle it on the QEMU side.
We'll support unplugging of virtio-mem devices in some configurations next.
--- hw/arm/virt.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
hw/arm/virt.c +26 -0
@@ -2794,6 +2794,30 @@ static void virt_virtio_md_pci_plug(HotplugHandler *hotplug_dev,error_propagate (errp, local_err); } +static void virt_virtio_md_pci_unplug (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +HotplugHandler *hotplug_dev2 =qdev_get_bus_hotplug_handler (dev); +Error *local_err =NULL ; + +/* Unplug the memory device while it is still realized. */ +memory_device_unplug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); + +if (hotplug_dev2) { +hotplug_handler_unplug (hotplug_dev2, dev, &local_err); +if (local_err) { +/* Not expected to fail ... but still try to recover. */ +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (hotplug_dev)); +error_propagate (errp, local_err); +return ; + } + }else { +/* Very unexpected, but let's just try to do the right thing. */ +warn_report ("Unexpected unplug of virtio based memory device" ); +qdev_unrealize (dev); + } +} +static void virt_virtio_md_pci_unplug_request (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {
@@ -2932,6 +2956,8 @@ static void virt_machine_device_unplug_cb(HotplugHandler *hotplug_dev, {if (object_dynamic_cast (OBJECT (dev),TYPE_PC_DIMM )) {virt_dimm_unplug (hotplug_dev, dev, errp); + }else if (object_dynamic_cast (OBJECT (dev),TYPE_VIRTIO_MEM_PCI )) { +virt_virtio_md_pci_unplug (hotplug_dev, dev, errp); }else {error_setg (errp,"virt: device unplug for unsupported device" " type: %s" ,object_get_typename (OBJECT (dev)));
Signature · 1 line
2.40.1
PATCH 3/5 virtio-mem: Prepare for unplug support of virtio-mem-pci devices
In reply to David HildenbrandIn many cases, blindly unplugging a virtio-mem device is problematic. We can only safely remove a device once:
* The guest is not expecting to be able to read unplugged memory (unplugged-inaccessible == on)
* The virtio-mem device does not have memory plugged (size == 0)
* The virtio-mem device does not have outstanding requests to the VM to plug memory (requested-size == 0)
So let's add a helper to check for that from the unplug-request code from relevant machine hotplug handlers and disallow changing the requested-size once an unplug request is pending.
Disallowing requested-size changes handles corner cases such as
(1) pausing the VM (2) requesting device unplug and (3) adjusting the requested size. If the VM would plug memory (due to the requested size change) before processing the unplug request, we would be in trouble.
--- hw/virtio/virtio-mem-pci.c | 42 +++++++++++++++++++++++++++++++--- hw/virtio/virtio-mem-pci.h | 2 ++ hw/virtio/virtio-mem.c | 24 +++++++++++++++++++ include/hw/virtio/virtio-mem.h | 2 ++ 4 files changed, 67 insertions(+), 3 deletions(-)
hw/virtio/virtio-mem-pci.c +39 -3
@@ -93,6 +93,42 @@ static void virtio_mem_pci_size_change_notify(Notifier *notifier, void *data)g_free (qom_path); } +void virtio_mem_pci_unplug_request_check (VirtIOMEMPCI *pci_mem,Error **errp) +{ +virtio_mem_unplug_request_check (&pci_mem->vdev, errp); +} + +static void virtio_mem_pci_get_requested_size (Object *obj,Visitor *v, +const char *name,void *opaque, +Error **errp) +{ +VirtIOMEMPCI *pci_mem =VIRTIO_MEM_PCI (obj); + +object_property_get (OBJECT (&pci_mem->vdev), name, v, errp); +} + +static void virtio_mem_pci_set_requested_size (Object *obj,Visitor *v, +const char *name,void *opaque, +Error **errp) +{ +VirtIOMEMPCI *pci_mem =VIRTIO_MEM_PCI (obj); +DeviceState *dev =DEVICE (obj); + +/* +* If we passed virtio_mem_pci_unplug_request_check(), making sure that +* the requested size is 0, don't allow modifying the requested size +* anymore, otherwise the VM might end up hotplugging memory before +* handling the unplug request. +*/ +if (dev->pending_deleted_event) { +error_setg (errp,"'%s' cannot be changed if the device is in the" +" process of unplug" , name); +return ; + } + +object_property_set (OBJECT (&pci_mem->vdev), name, v, errp); +} +static void virtio_mem_pci_class_init (ObjectClass *klass,void *data) {DeviceClass *dc =DEVICE_CLASS (klass);
@@ -135,9 +171,9 @@ static void virtio_mem_pci_instance_init(Object *obj) OBJECT(&dev->vdev), VIRTIO_MEM_BLOCK_SIZE_PROP);object_property_add_alias (obj,VIRTIO_MEM_SIZE_PROP ,OBJECT (&dev->vdev),VIRTIO_MEM_SIZE_PROP ); -object_property_add_alias (obj,VIRTIO_MEM_REQUESTED_SIZE_PROP , -OBJECT (&dev->vdev), -VIRTIO_MEM_REQUESTED_SIZE_PROP ); +object_property_add (obj,VIRTIO_MEM_REQUESTED_SIZE_PROP ,"size" , + virtio_mem_pci_get_requested_size, + virtio_mem_pci_set_requested_size,NULL ,NULL ); }static const VirtioPCIDeviceTypeInfo virtio_mem_pci_info = {
hw/virtio/virtio-mem-pci.h +2 -0
@@ -32,4 +32,6 @@ struct VirtIOMEMPCI { Notifier size_change_notifier; }; +void virtio_mem_pci_unplug_request_check (VirtIOMEMPCI *pci_mem,Error **errp); +#endif /* QEMU_VIRTIO_MEM_PCI_H */
hw/virtio/virtio-mem.c +24 -0
@@ -1468,6 +1468,30 @@ static void virtio_mem_rdm_unregister_listener(RamDiscardManager *rdm,QLIST_REMOVE (rdl, next); } +void virtio_mem_unplug_request_check (VirtIOMEM *vmem,Error **errp) +{ +if (vmem->unplugged_inaccessible ==ON_OFF_AUTO_OFF ) { +/* +* We could allow it with a usable region size of 0, but let's just +* not care about that legacy setting. +*/ +error_setg (errp,"virtio-mem device cannot get unplugged while" +" '" VIRTIO_MEM_UNPLUGGED_INACCESSIBLE_PROP "' != 'on'" ); +return ; + } + +if (vmem->size) { +error_setg (errp,"virtio-mem device cannot get unplugged while" +" '" VIRTIO_MEM_SIZE_PROP "' != '0'" ); +return ; + } +if (vmem->requested_size) { +error_setg (errp,"virtio-mem device cannot get unplugged while" +" '" VIRTIO_MEM_REQUESTED_SIZE_PROP "' != '0'" ); +return ; + } +} +static void virtio_mem_class_init (ObjectClass *klass,void *data) {DeviceClass *dc =DEVICE_CLASS (klass);
include/hw/virtio/virtio-mem.h +2 -0
@@ -100,4 +100,6 @@ struct VirtIOMEMClass { void (*remove_size_change_notifier)(VirtIOMEM *vmem, Notifier *notifier); }; +void virtio_mem_unplug_request_check (VirtIOMEM *vmem,Error **errp); +#endif
Signature · 1 line
2.40.1
PATCH 4/5 pc: Support unplug of virtio-mem-pci devices
In reply to David HildenbrandLet's support unplug of virtio-mem-pci devices by granting (forwarding) unplug requests if it's safe to unplug a virtio-mem device.
--- hw/i386/pc.c | 48 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-)
hw/i386/pc.c +41 -7
@@ -1549,13 +1549,6 @@ static void pc_virtio_md_pci_plug(HotplugHandler *hotplug_dev,error_propagate (errp, local_err); } -static void pc_virtio_md_pci_unplug_request (HotplugHandler *hotplug_dev, -DeviceState *dev,Error **errp) -{ -/* We don't support hot unplug of virtio based memory devices */ -error_setg (errp,"virtio based memory devices cannot be unplugged." ); -} -static void pc_virtio_md_pci_unplug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {
@@ -1580,6 +1573,47 @@ static void pc_virtio_md_pci_unplug(HotplugHandler *hotplug_dev, } } +static void pc_virtio_md_pci_unplug_request (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +HotplugHandler *hotplug_dev2 =qdev_get_bus_hotplug_handler (dev); +HotplugHandlerClass *hdc; +Error *local_err =NULL ; + +if (object_dynamic_cast (OBJECT (dev),TYPE_VIRTIO_PMEM_PCI )) { +error_setg (errp,"virtio-pmem devices cannot be unplugged." ); +return ; + } + +if (!hotplug_dev2) { +error_setg (errp,"hotunplug of virtio based memory devices not" +"supported on this bus" ); +return ; + } + +/* Verify whether it is *currently* safe to unplug the virtio-mem device. */ +g_assert (object_dynamic_cast (OBJECT (dev),TYPE_VIRTIO_MEM_PCI )); +virtio_mem_pci_unplug_request_check (VIRTIO_MEM_PCI (dev), &local_err); +if (local_err) { +error_propagate (errp, local_err); +return ; + } + +/* +* Forward the async request or turn it into a sync request (handling it +* like qdev_unplug()). +*/ + hdc =HOTPLUG_HANDLER_GET_CLASS (hotplug_dev2); +if (hdc->unplug_request) { +hotplug_handler_unplug_request (hotplug_dev2, dev, &local_err); + }else { +pc_virtio_md_pci_unplug (hotplug_dev, dev, &local_err); +if (!local_err) { +object_unparent (OBJECT (dev)); + } + } +} +static void pc_machine_device_pre_plug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {
Signature · 1 line
2.40.1
PATCH 5/5 arm/virt: Support unplug of virtio-mem-pci devices
In reply to David HildenbrandLet's support unplug of virtio-mem-pci devices by granting (forwarding) unplug requests if it's safe to unplug a virtio-mem device.
--- hw/arm/virt.c | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-)
hw/arm/virt.c +31 -3
@@ -2821,10 +2821,38 @@ static void virt_virtio_md_pci_unplug(HotplugHandler *hotplug_dev,static void virt_virtio_md_pci_unplug_request (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) { -/* We don't support hot unplug of virtio based memory devices */ -error_setg (errp,"virtio based memory devices cannot be unplugged." ); -} +HotplugHandler *hotplug_dev2 =qdev_get_bus_hotplug_handler (dev); +HotplugHandlerClass *hdc; +Error *local_err =NULL ; + +if (!hotplug_dev2) { +error_setg (errp,"hotunplug of virtio based memory devices not" +"supported on this bus" ); +return ; + } + +/* Verify whether it is *currently* safe to unplug the virtio-mem device. */ +g_assert (object_dynamic_cast (OBJECT (dev),TYPE_VIRTIO_MEM_PCI )); +virtio_mem_pci_unplug_request_check (VIRTIO_MEM_PCI (dev), &local_err); +if (local_err) { +error_propagate (errp, local_err); +return ; + } +/* +* Forward the async request or turn it into a sync request (handling it +* like qdev_unplug()). +*/ + hdc =HOTPLUG_HANDLER_GET_CLASS (hotplug_dev2); +if (hdc->unplug_request) { +hotplug_handler_unplug_request (hotplug_dev2, dev, &local_err); + }else { +virt_virtio_md_pci_unplug (hotplug_dev, dev, &local_err); +if (!local_err) { +object_unparent (OBJECT (dev)); + } + } +}static void virt_machine_device_pre_plug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp)
Signature · 1 line
2.40.1
Quoted 9 lines · David Hildenbrand
One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.
Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).
Briefly tested on both,
x86_64and aarch64.
Quoted 9 lines
Cc Peter Maydell <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Cc Igor Mammedov <[email protected]>Cc Gavin Shan <[email protected]>
Lots of duplication pc/arm. Which is not new but do we have to keep growing this? Can't we put at least the new common code somewhere?
What do ARM maintainers think about it?
Quoted 17 lines
David Hildenbrand (5):
pc: Properly handle unplug of virtio based memory devices arm/virt: Properly handle unplug of virtio based memory devices
virtio-mem: Prepare for unplug support of virtio-mem-pci devices
pc: Support unplug of virtio-mem-pci devices arm/virt: Support unplug of virtio-mem-pci deviceshw/arm/virt.c | 60 +++++++++++++++++++++++++++++-- hw/i386/pc.c | 66 ++++++++++++++++++++++++++++++---- hw/virtio/virtio-mem-pci.c | 42 ++++++++++++++++++++-- hw/virtio/virtio-mem-pci.h | 2 ++ hw/virtio/virtio-mem.c | 24 +++++++++++++ include/hw/virtio/virtio-mem.h | 2 ++ 6 files changed, 183 insertions(+), 13 deletions(-)Signature · 1 line
2.40.1
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · David Hildenbrand
One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.
Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).
Briefly tested on both,
x86_64and aarch64.Quoted 9 lines
Cc Peter Maydell <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Cc Igor Mammedov <[email protected]>Cc Gavin Shan <[email protected]>
Quoted 2 lines
Lots of duplication pc/arm. Which is not new but do we have to keep growing this? Can't we put at least the new common code somewhere?
There are some minor differences in the code, but I guess we could factor the (un)plug handlers out.
hw/virtio/virtio-md.cinclude/hw/virtio/virtio-md.c
?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 21 lines · David Hildenbrand
Quoted 20 lines · Michael S. Tsirkin
Quoted 18 lines · David Hildenbrand
One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.
Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).
Briefly tested on both,
x86_64and aarch64.Cc Peter Maydell <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>Cc "Michael S. Tsirkin" <[email protected]>Cc Marcel Apfelbaum <[email protected]>Cc Igor Mammedov <[email protected]>Cc Gavin Shan <[email protected]>
Quoted 5 lines
Quoted 2 lines
Lots of duplication pc/arm. Which is not new but do we have to keep growing this? Can't we put at least the new common code somewhere?
There are some minor differences in the code, but I guess we could factor the (un)plug handlers out.
maybe with a callback.
Quoted 5 lines
hw/virtio/virtio-md.cinclude/hw/virtio/virtio-md.c?
Sure.
Quoted 4 lines
Signature · 3 lines
Cheers, David / dhildenb