Skip to contentSkip to conversation

virtio-mem: Device unplug support

PATCH Superseded David Hildenbrand 2 participants 9 messages Jun 13, 2023 – Jun 23, 2023

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#5 lore raw

PATCH 5/5 arm/virt: Support unplug of virtio-mem-pci devices

In reply to David Hildenbrand

Let's support unplug of virtio-mem-pci devices by granting (forwarding) unplug requests if it's safe to unplug a virtio-mem device.

---
 hw/arm/virt.c | 34 +++++++++++++++++++++++++++++++---
 1 file changed, 31 insertions(+), 3 deletions(-)
hw/arm/virt.c +31 -3
index ed5c3c8fc4..08d4eef4c1 100644
@@ -2821,10 +2821,38 @@ static void virt_virtio_md_pci_unplug(HotplugHandler *hotplug_dev,
 static void virt_virtio_md_pci_unplug_request(HotplugHandler *hotplug_dev,
                                               DeviceState *dev, Error **errp)
 {
-    /* We don't support hot unplug of virtio based memory devices */
-    error_setg(errp, "virtio based memory devices cannot be unplugged.");
-}
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    HotplugHandlerClass *hdc;
+    Error *local_err = NULL;
+
+    if (!hotplug_dev2) {
+        error_setg(errp, "hotunplug of virtio based memory devices not"
+                   "supported on this bus");
+        return;
+    }
+
+    /* Verify whether it is *currently* safe to unplug the virtio-mem device. */
+    g_assert(object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI));
+    virtio_mem_pci_unplug_request_check(VIRTIO_MEM_PCI(dev), &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
 
+    /*
+     * Forward the async request or turn it into a sync request (handling it
+     * like qdev_unplug()).
+     */
+    hdc = HOTPLUG_HANDLER_GET_CLASS(hotplug_dev2);
+    if (hdc->unplug_request) {
+        hotplug_handler_unplug_request(hotplug_dev2, dev, &local_err);
+    } else {
+        virt_virtio_md_pci_unplug(hotplug_dev, dev, &local_err);
+        if (!local_err) {
+            object_unparent(OBJECT(dev));
+        }
+    }
+}
 
 static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
                                             DeviceState *dev, Error **errp)
Signature · 1 line
2.40.1
#0 lore raw

cover virtio-mem: Device unplug support

One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.

Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).

Briefly tested on both, x86_64 and aarch64.

Cc Peter Maydell <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Gavin Shan <[email protected]>

David Hildenbrand (5):
pc: Properly handle unplug of virtio based memory devices arm/virt: Properly handle unplug of virtio based memory devices
virtio-mem: Prepare for unplug support of virtio-mem-pci devices
pc: Support unplug of virtio-mem-pci devices arm/virt: Support unplug of virtio-mem-pci devices

 hw/arm/virt.c                  | 60 +++++++++++++++++++++++++++++--
 hw/i386/pc.c                   | 66 ++++++++++++++++++++++++++++++----
 hw/virtio/virtio-mem-pci.c     | 42 ++++++++++++++++++++--
 hw/virtio/virtio-mem-pci.h     |  2 ++
 hw/virtio/virtio-mem.c         | 24 +++++++++++++
 include/hw/virtio/virtio-mem.h |  2 ++
 6 files changed, 183 insertions(+), 13 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH 1/5 pc: Properly handle unplug of virtio based memory devices

While we fence unplug requests from the outside, the VM can still trigger unplug of virtio based memory devices, for example, in Linux doing:

    # echo 0 > /sys/bus/pci/slots/3/power

While doing that is not really expected to work without harming the guest OS (e.g., removing a virtio-mem device while it still provides memory), let's make sure that we properly handle it on the QEMU side.

We'll support unplugging of virtio-mem devices in some configurations next.

---
 hw/i386/pc.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
hw/i386/pc.c +19 -1
index fc52772fdd..fdd7062929 100644
@@ -1559,7 +1559,25 @@ static void pc_virtio_md_pci_unplug_request(HotplugHandler *hotplug_dev,
 static void pc_virtio_md_pci_unplug(HotplugHandler *hotplug_dev,
                                     DeviceState *dev, Error **errp)
 {
-    /* We don't support hot unplug of virtio based memory devices */
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    Error *local_err = NULL;
+
+    /* Unplug the memory device while it is still realized. */
+    memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+
+    if (hotplug_dev2) {
+        hotplug_handler_unplug(hotplug_dev2, dev, &local_err);
+        if (local_err) {
+            /* Not expected to fail ... but still try to recover. */
+            memory_device_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+            error_propagate(errp, local_err);
+            return;
+        }
+    } else {
+        /* Very unexpected, but let's just try to do the right thing. */
+        warn_report("Unexpected unplug of virtio based memory device");
+        qdev_unrealize(dev);
+    }
 }
 
 static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
Signature · 1 line
2.40.1
#3 lore raw

PATCH 3/5 virtio-mem: Prepare for unplug support of virtio-mem-pci devices

In reply to David Hildenbrand

In many cases, blindly unplugging a virtio-mem device is problematic. We can only safely remove a device once:
* The guest is not expecting to be able to read unplugged memory (unplugged-inaccessible == on)
* The virtio-mem device does not have memory plugged (size == 0)
* The virtio-mem device does not have outstanding requests to the VM to plug memory (requested-size == 0)

So let's add a helper to check for that from the unplug-request code from relevant machine hotplug handlers and disallow changing the requested-size once an unplug request is pending.

Disallowing requested-size changes handles corner cases such as
(1) pausing the VM (2) requesting device unplug and (3) adjusting the requested size. If the VM would plug memory (due to the requested size change) before processing the unplug request, we would be in trouble.

---
 hw/virtio/virtio-mem-pci.c     | 42 +++++++++++++++++++++++++++++++---
 hw/virtio/virtio-mem-pci.h     |  2 ++
 hw/virtio/virtio-mem.c         | 24 +++++++++++++++++++
 include/hw/virtio/virtio-mem.h |  2 ++
 4 files changed, 67 insertions(+), 3 deletions(-)
hw/virtio/virtio-mem-pci.c +39 -3
index b85c12668d..dcbab9713c 100644
@@ -93,6 +93,42 @@ static void virtio_mem_pci_size_change_notify(Notifier *notifier, void *data)
     g_free(qom_path);
 }
 
+void virtio_mem_pci_unplug_request_check(VirtIOMEMPCI *pci_mem, Error **errp)
+{
+    virtio_mem_unplug_request_check(&pci_mem->vdev, errp);
+}
+
+static void virtio_mem_pci_get_requested_size(Object *obj, Visitor *v,
+                                              const char *name, void *opaque,
+                                              Error **errp)
+{
+    VirtIOMEMPCI *pci_mem = VIRTIO_MEM_PCI(obj);
+
+    object_property_get(OBJECT(&pci_mem->vdev), name, v, errp);
+}
+
+static void virtio_mem_pci_set_requested_size(Object *obj, Visitor *v,
+                                              const char *name, void *opaque,
+                                              Error **errp)
+{
+    VirtIOMEMPCI *pci_mem = VIRTIO_MEM_PCI(obj);
+    DeviceState *dev = DEVICE(obj);
+
+    /*
+     * If we passed virtio_mem_pci_unplug_request_check(), making sure that
+     * the requested size is 0, don't allow modifying the requested size
+     * anymore, otherwise the VM might end up hotplugging memory before
+     * handling the unplug request.
+     */
+    if (dev->pending_deleted_event) {
+        error_setg(errp, "'%s' cannot be changed if the device is in the"
+                   " process of unplug", name);
+        return;
+    }
+
+    object_property_set(OBJECT(&pci_mem->vdev), name, v, errp);
+}
+
 static void virtio_mem_pci_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -135,9 +171,9 @@ static void virtio_mem_pci_instance_init(Object *obj)
                               OBJECT(&dev->vdev), VIRTIO_MEM_BLOCK_SIZE_PROP);
     object_property_add_alias(obj, VIRTIO_MEM_SIZE_PROP, OBJECT(&dev->vdev),
                               VIRTIO_MEM_SIZE_PROP);
-    object_property_add_alias(obj, VIRTIO_MEM_REQUESTED_SIZE_PROP,
-                              OBJECT(&dev->vdev),
-                              VIRTIO_MEM_REQUESTED_SIZE_PROP);
+    object_property_add(obj, VIRTIO_MEM_REQUESTED_SIZE_PROP, "size",
+                        virtio_mem_pci_get_requested_size,
+                        virtio_mem_pci_set_requested_size, NULL, NULL);
 }
 
 static const VirtioPCIDeviceTypeInfo virtio_mem_pci_info = {
hw/virtio/virtio-mem-pci.h +2 -0
index e636e1a48d..f8e8bc523a 100644
@@ -32,4 +32,6 @@ struct VirtIOMEMPCI {
     Notifier size_change_notifier;
 };
 
+void virtio_mem_pci_unplug_request_check(VirtIOMEMPCI *pci_mem, Error **errp);
+
 #endif /* QEMU_VIRTIO_MEM_PCI_H */
hw/virtio/virtio-mem.c +24 -0
index 538b695c29..e0dc7f04ea 100644
@@ -1468,6 +1468,30 @@ static void virtio_mem_rdm_unregister_listener(RamDiscardManager *rdm,
     QLIST_REMOVE(rdl, next);
 }
 
+void virtio_mem_unplug_request_check(VirtIOMEM *vmem, Error **errp)
+{
+    if (vmem->unplugged_inaccessible == ON_OFF_AUTO_OFF) {
+        /*
+         * We could allow it with a usable region size of 0, but let's just
+         * not care about that legacy setting.
+         */
+        error_setg(errp, "virtio-mem device cannot get unplugged while"
+                   " '" VIRTIO_MEM_UNPLUGGED_INACCESSIBLE_PROP "' != 'on'");
+        return;
+    }
+
+    if (vmem->size) {
+        error_setg(errp, "virtio-mem device cannot get unplugged while"
+                   " '" VIRTIO_MEM_SIZE_PROP "' != '0'");
+        return;
+    }
+    if (vmem->requested_size) {
+        error_setg(errp, "virtio-mem device cannot get unplugged while"
+                   " '" VIRTIO_MEM_REQUESTED_SIZE_PROP "' != '0'");
+        return;
+    }
+}
+
 static void virtio_mem_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
include/hw/virtio/virtio-mem.h +2 -0
index f15e561785..dd5aec98f5 100644
@@ -100,4 +100,6 @@ struct VirtIOMEMClass {
     void (*remove_size_change_notifier)(VirtIOMEM *vmem, Notifier *notifier);
 };
 
+void virtio_mem_unplug_request_check(VirtIOMEM *vmem, Error **errp);
+
 #endif
Signature · 1 line
2.40.1
#2 lore raw

PATCH 2/5 arm/virt: Properly handle unplug of virtio based memory devices

In reply to David Hildenbrand

While we fence unplug requests from the outside, the VM can still trigger unplug of virtio based memory devices, for example, in Linux doing:

    # echo 0 > /sys/bus/pci/slots/3/power

While doing that is not really expected to work without harming the guest OS (e.g., removing a virtio-mem device while it still provides memory), let's make sure that we properly handle it on the QEMU side.

We'll support unplugging of virtio-mem devices in some configurations next.

---
 hw/arm/virt.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
hw/arm/virt.c +26 -0
index 9b9f7d9c68..ed5c3c8fc4 100644
@@ -2794,6 +2794,30 @@ static void virt_virtio_md_pci_plug(HotplugHandler *hotplug_dev,
     error_propagate(errp, local_err);
 }
 
+static void virt_virtio_md_pci_unplug(HotplugHandler *hotplug_dev,
+                                      DeviceState *dev, Error **errp)
+{
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    Error *local_err = NULL;
+
+    /* Unplug the memory device while it is still realized. */
+    memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+
+    if (hotplug_dev2) {
+        hotplug_handler_unplug(hotplug_dev2, dev, &local_err);
+        if (local_err) {
+            /* Not expected to fail ... but still try to recover. */
+            memory_device_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+            error_propagate(errp, local_err);
+            return;
+        }
+    } else {
+        /* Very unexpected, but let's just try to do the right thing. */
+        warn_report("Unexpected unplug of virtio based memory device");
+        qdev_unrealize(dev);
+    }
+}
+
 static void virt_virtio_md_pci_unplug_request(HotplugHandler *hotplug_dev,
                                               DeviceState *dev, Error **errp)
 {
@@ -2932,6 +2956,8 @@ static void virt_machine_device_unplug_cb(HotplugHandler *hotplug_dev,
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_dimm_unplug(hotplug_dev, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        virt_virtio_md_pci_unplug(hotplug_dev, dev, errp);
     } else {
         error_setg(errp, "virt: device unplug for unsupported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
Signature · 1 line
2.40.1
#4 lore raw

PATCH 4/5 pc: Support unplug of virtio-mem-pci devices

In reply to David Hildenbrand

Let's support unplug of virtio-mem-pci devices by granting (forwarding) unplug requests if it's safe to unplug a virtio-mem device.

---
 hw/i386/pc.c | 48 +++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)
hw/i386/pc.c +41 -7
index fdd7062929..996757794e 100644
@@ -1549,13 +1549,6 @@ static void pc_virtio_md_pci_plug(HotplugHandler *hotplug_dev,
     error_propagate(errp, local_err);
 }
 
-static void pc_virtio_md_pci_unplug_request(HotplugHandler *hotplug_dev,
-                                            DeviceState *dev, Error **errp)
-{
-    /* We don't support hot unplug of virtio based memory devices */
-    error_setg(errp, "virtio based memory devices cannot be unplugged.");
-}
-
 static void pc_virtio_md_pci_unplug(HotplugHandler *hotplug_dev,
                                     DeviceState *dev, Error **errp)
 {
@@ -1580,6 +1573,47 @@ static void pc_virtio_md_pci_unplug(HotplugHandler *hotplug_dev,
     }
 }
 
+static void pc_virtio_md_pci_unplug_request(HotplugHandler *hotplug_dev,
+                                            DeviceState *dev, Error **errp)
+{
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    HotplugHandlerClass *hdc;
+    Error *local_err = NULL;
+
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI)) {
+        error_setg(errp, "virtio-pmem devices cannot be unplugged.");
+        return;
+    }
+
+    if (!hotplug_dev2) {
+        error_setg(errp, "hotunplug of virtio based memory devices not"
+                   "supported on this bus");
+        return;
+    }
+
+    /* Verify whether it is *currently* safe to unplug the virtio-mem device. */
+    g_assert(object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI));
+    virtio_mem_pci_unplug_request_check(VIRTIO_MEM_PCI(dev), &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    /*
+     * Forward the async request or turn it into a sync request (handling it
+     * like qdev_unplug()).
+     */
+    hdc = HOTPLUG_HANDLER_GET_CLASS(hotplug_dev2);
+    if (hdc->unplug_request) {
+        hotplug_handler_unplug_request(hotplug_dev2, dev, &local_err);
+    } else {
+        pc_virtio_md_pci_unplug(hotplug_dev, dev, &local_err);
+        if (!local_err) {
+            object_unparent(OBJECT(dev));
+        }
+    }
+}
+
 static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
                                           DeviceState *dev, Error **errp)
 {
Signature · 1 line
2.40.1
#6 lore raw
In reply to David Hildenbrand
Quoted 9 lines · David Hildenbrand

One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.

Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).

Briefly tested on both, x86_64 and aarch64.

Quoted 9 lines
Cc Peter Maydell <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Gavin Shan <[email protected]>

Lots of duplication pc/arm. Which is not new but do we have to keep growing this? Can't we put at least the new common code somewhere?

What do ARM maintainers think about it?

Quoted 17 lines

David Hildenbrand (5):
pc: Properly handle unplug of virtio based memory devices arm/virt: Properly handle unplug of virtio based memory devices
virtio-mem: Prepare for unplug support of virtio-mem-pci devices
pc: Support unplug of virtio-mem-pci devices arm/virt: Support unplug of virtio-mem-pci devices

 hw/arm/virt.c                  | 60 +++++++++++++++++++++++++++++--
 hw/i386/pc.c                   | 66 ++++++++++++++++++++++++++++++----
 hw/virtio/virtio-mem-pci.c     | 42 ++++++++++++++++++++--
 hw/virtio/virtio-mem-pci.h     |  2 ++
 hw/virtio/virtio-mem.c         | 24 +++++++++++++
 include/hw/virtio/virtio-mem.h |  2 ++
 6 files changed, 183 insertions(+), 13 deletions(-)
Signature · 1 line
2.40.1
#7 lore raw
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · David Hildenbrand

One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.

Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).

Briefly tested on both, x86_64 and aarch64.

Quoted 9 lines
Cc Peter Maydell <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Gavin Shan <[email protected]>
Quoted 2 lines

Lots of duplication pc/arm. Which is not new but do we have to keep growing this? Can't we put at least the new common code somewhere?

There are some minor differences in the code, but I guess we could factor the (un)plug handlers out.

hw/virtio/virtio-md.c
include/hw/virtio/virtio-md.c

?

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw
Quoted 21 lines · David Hildenbrand
Quoted 20 lines · Michael S. Tsirkin
Quoted 18 lines · David Hildenbrand

One limitation of virtio-mem is that we cannot currently unplug virtio-mem devices that have all memory unplugged from the VM.

Let's properly handle forced unplug (as can be triggered by the VM) and add support for ordinary unplug (requests) of virtio-mem devices that are in a compatible state (no legacy mode, no plugged memory, no plug request).

Briefly tested on both, x86_64 and aarch64.

Cc Peter Maydell <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Marcel Apfelbaum <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Gavin Shan <[email protected]>
Quoted 5 lines
Quoted 2 lines

Lots of duplication pc/arm. Which is not new but do we have to keep growing this? Can't we put at least the new common code somewhere?

There are some minor differences in the code, but I guess we could factor the (un)plug handlers out.

maybe with a callback.

Quoted 5 lines

hw/virtio/virtio-md.c
include/hw/virtio/virtio-md.c

?

Sure.

Quoted 4 lines
Signature · 3 lines
Cheers,

David / dhildenb