Skip to contentSkip to conversation

NVMe namespace hotplug and drive reconnection support

PATCH Superseded mr-083 2 participants 5 messages Apr 9 – Apr 10 Block layer

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

mr-083

author
#0 lore raw

cover NVMe namespace hotplug and drive reconnection support

This series adds two features that together enable transparent NVMe disk hot-swap simulation in QEMU, matching the behavior of physical NVMe drives being pulled and reinserted in the same PCIe slot.

Problem:
Currently, hot-swapping an NVMe disk in QEMU requires removing the entire NVMe controller via device_del, which causes the Linux guest to assign a new controller number on re-add (e.g. nvme2 becomes nvme4). This breaks storage software that tracks drives by device name.

Solution:
Patch 1 adds hotplug support for nvme-ns devices on the NvmeBus, with proper Asynchronous Event Notification (AEN) so the guest kernel detects namespace changes. This allows namespace-level hot-swap without removing the NVMe controller.

Patch 2 adds a drive_insert HMP command that reconnects a host block device file to an existing guest device after drive_del. This is the counterpart to drive_del for non-removable devices where blockdev-change-medium cannot be used.

The recommended hot-swap sequence is:
1. drive_del <drive-id> # disconnect backing store
2. drive_insert <device> <file> # reconnect backing store
3. pcie_aer_inject_error <port> SDN # trigger controller reset

After this sequence, the guest sees the same controller and namespace names (e.g. /dev/nvme2n1 remains /dev/nvme2n1), and the NVMe driver recovers transparently via the standard AER recovery path.

Tested with:
- Linux 6.1 guest on QEMU aarch64 with HVF (macOS)
- NVMe subsystem model with multipath disabled
- DirectPV and MinIO AIStor storage stack

mr-083 (2):
hw/nvme: add namespace hotplug support
block/monitor: add drive_insert HMP command

 block/monitor/block-hmp-cmds.c | 59 +++++++++++++++++++++++
 hmp-commands.hx                | 18 +++++++
 hw/nvme/ctrl.c                 | 85 ++++++++++++++++++++++++++++++++++
 hw/nvme/ns.c                   |  1 +
 hw/nvme/subsys.c               |  2 +
 include/block/block-hmp-cmds.h |  1 +
 6 files changed, 166 insertions(+)

--
2.50.1 (Apple Git-155)

author
#1 lore raw

PATCH 1/2 hw/nvme: add namespace hotplug support

Add hotplug support for nvme-ns devices on the NvmeBus. This enables namespace-level hot-swap without removing the NVMe controller, which is how physical NVMe drives behave when hot-swapped in the same PCIe slot.

Mark nvme-ns devices as hotpluggable and register the NvmeBus as a hotplug handler with proper plug and unplug callbacks:

- plug: attach namespace to all started controllers and send an Asynchronous Event Notification (AEN) with NS_ATTR_CHANGED so the guest kernel rescans namespaces
- unplug: detach from all controllers, send AEN, remove from subsystem, then unrealize the device

The plug handler skips controllers that haven't started yet (qs_created == false) to avoid interfering with boot-time namespace attachment in nvme_start_ctrl().

Both the controller bus and subsystem bus are configured as hotplug handlers via qbus_set_bus_hotplug_handler() since nvme-ns devices may reparent to the subsystem bus during realize.

---
 hw/nvme/ctrl.c   | 85 ++++++++++++++++++++++++++++++++++++++++++++++++
 hw/nvme/ns.c     |  1 +
 hw/nvme/subsys.c |  2 ++
 3 files changed, 88 insertions(+)
hw/nvme/ctrl.c +85 -0
index be6c7028cb..5502e4ea2b 100644
@@ -206,6 +206,7 @@
 #include "system/hostmem.h"
 #include "hw/pci/msix.h"
 #include "hw/pci/pcie_sriov.h"
+#include "hw/core/qdev.h"
 #include "system/spdm-socket.h"
 #include "migration/vmstate.h"
 
@@ -9293,6 +9294,7 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
     }
 
     qbus_init(&n->bus, sizeof(NvmeBus), TYPE_NVME_BUS, dev, dev->id);
+    qbus_set_bus_hotplug_handler(BUS(&n->bus));
 
     if (nvme_init_subsys(n, errp)) {
         return;
@@ -9553,10 +9555,93 @@ static const TypeInfo nvme_info = {
     },
 };
 
+static void nvme_ns_hot_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                              Error **errp)
+{
+    NvmeNamespace *ns = NVME_NS(dev);
+    NvmeSubsystem *subsys = ns->subsys;
+    uint32_t nsid = ns->params.nsid;
+    int i;
+
+    /*
+     * Attach to all started controllers and notify via AEN.
+     * Skip controllers that haven't started yet (boot-time realize) —
+     * nvme_start_ctrl() will attach namespaces during controller init.
+     */
+    for (i = 0; i < NVME_MAX_CONTROLLERS; i++) {
+        NvmeCtrl *ctrl = nvme_subsys_ctrl(subsys, i);
+        if (!ctrl || !ctrl->qs_created) {
+            continue;
+        }
+
+        if (nvme_csi_supported(ctrl, ns->csi) && !ns->params.detached) {
+            nvme_attach_ns(ctrl, ns);
+            nvme_update_dsm_limits(ctrl, ns);
+
+            if (!test_and_set_bit(nsid, ctrl->changed_nsids)) {
+                nvme_enqueue_event(ctrl, NVME_AER_TYPE_NOTICE,
+                                   NVME_AER_INFO_NOTICE_NS_ATTR_CHANGED,
+                                   NVME_LOG_CHANGED_NSLIST);
+            }
+        }
+    }
+}
+
+static void nvme_ns_hot_unplug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                               Error **errp)
+{
+    NvmeNamespace *ns = NVME_NS(dev);
+    NvmeSubsystem *subsys = ns->subsys;
+    uint32_t nsid = ns->params.nsid;
+    int i;
+
+    /*
+     * Detach from all controllers and notify the guest via AEN.
+     * Must happen before unrealize to avoid use-after-free when the
+     * guest sends I/O to a freed namespace.
+     */
+    for (i = 0; i < NVME_MAX_CONTROLLERS; i++) {
+        NvmeCtrl *ctrl = nvme_subsys_ctrl(subsys, i);
+        if (!ctrl || !nvme_ns(ctrl, nsid)) {
+            continue;
+        }
+
+        nvme_detach_ns(ctrl, ns);
+        nvme_update_dsm_limits(ctrl, NULL);
+
+        if (!test_and_set_bit(nsid, ctrl->changed_nsids)) {
+            nvme_enqueue_event(ctrl, NVME_AER_TYPE_NOTICE,
+                               NVME_AER_INFO_NOTICE_NS_ATTR_CHANGED,
+                               NVME_LOG_CHANGED_NSLIST);
+        }
+    }
+
+    /* Remove from subsystem namespace list. */
+    subsys->namespaces[nsid] = NULL;
+
+    /*
+     * Unrealize: drain I/O, flush, cleanup structures, remove from QOM.
+     * nvme_ns_unrealize() handles drain/shutdown/cleanup internally.
+     */
+    qdev_unrealize(dev);
+}
+
+static void nvme_bus_class_init(ObjectClass *klass, const void *data)
+{
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(klass);
+    hc->plug = nvme_ns_hot_plug;
+    hc->unplug = nvme_ns_hot_unplug;
+}
+
 static const TypeInfo nvme_bus_info = {
     .name = TYPE_NVME_BUS,
     .parent = TYPE_BUS,
     .instance_size = sizeof(NvmeBus),
+    .class_init = nvme_bus_class_init,
+    .interfaces = (const InterfaceInfo[]) {
+        { TYPE_HOTPLUG_HANDLER },
+        { }
+    },
 };
 
 static void nvme_register_types(void)
hw/nvme/ns.c +1 -0
index b0106eaa5c..eb628c0734 100644
@@ -937,6 +937,7 @@ static void nvme_ns_class_init(ObjectClass *oc, const void *data)
     dc->bus_type = TYPE_NVME_BUS;
     dc->realize = nvme_ns_realize;
     dc->unrealize = nvme_ns_unrealize;
+    dc->hotpluggable = true;
     device_class_set_props(dc, nvme_ns_props);
     dc->desc = "Virtual NVMe namespace";
 }
hw/nvme/subsys.c +2 -0
index 777e1c620f..fa35055d3c 100644
@@ -9,6 +9,7 @@
 #include "qemu/osdep.h"
 #include "qemu/units.h"
 #include "qapi/error.h"
+#include "hw/core/qdev.h"
 
 #include "nvme.h"
 
@@ -205,6 +206,7 @@ static void nvme_subsys_realize(DeviceState *dev, Error **errp)
     NvmeSubsystem *subsys = NVME_SUBSYS(dev);
 
     qbus_init(&subsys->bus, sizeof(NvmeBus), TYPE_NVME_BUS, dev, dev->id);
+    qbus_set_bus_hotplug_handler(BUS(&subsys->bus));
 
     nvme_subsys_setup(subsys, errp);
 }
Signature · 1 line
2.50.1 (Apple Git-155)
author
#2 lore raw

PATCH 2/2 block/monitor: add drive_insert HMP command

In reply to mr-083

Add a drive_insert HMP command that reconnects a host block device file to an existing guest device whose backing store was previously removed with drive_del.

After drive_del, the BlockBackend remains attached to the guest device but has no BlockDriverState (shown as "[not inserted]" in info block). drive_insert opens the specified file, finds the device's BlockBackend by iterating all backends and matching the attached device ID, then calls blk_insert_bs() to reconnect the backing store.

This complements drive_del for non-removable devices (such as NVMe namespaces) where blockdev-change-medium cannot be used. Combined with PCIe AER Surprise Down error injection to trigger a controller reset, this enables complete NVMe disk hot-swap simulation where the guest sees the same device names throughout.

Example usage:
drive_del drv0 # remove backing store
drive_insert ns0 disk.qcow2 # reconnect backing pcie_aer_inject_error rp0 SDN # trigger controller reset

---
 block/monitor/block-hmp-cmds.c | 59 ++++++++++++++++++++++++++++++++++
 hmp-commands.hx                | 18 +++++++++++
 include/block/block-hmp-cmds.h |  1 +
 3 files changed, 78 insertions(+)
block/monitor/block-hmp-cmds.c +59 -0
index 1fd28d59eb..77e9662ead 100644
@@ -38,7 +38,9 @@
 #include "qemu/osdep.h"
 #include "hw/core/boards.h"
 #include "system/block-backend.h"
+#include "system/block-backend-global-state.h"
 #include "system/blockdev.h"
+#include "block/block-global-state.h"
 #include "qapi/qapi-commands-block.h"
 #include "qapi/qapi-commands-block-export.h"
 #include "qobject/qdict.h"
@@ -195,6 +197,63 @@ unlock:
     hmp_handle_error(mon, err);
 }
 
+void hmp_drive_insert(Monitor *mon, const QDict *qdict)
+{
+    const char *id = qdict_get_str(qdict, "id");
+    const char *filename = qdict_get_str(qdict, "filename");
+    BlockBackend *blk = NULL;
+    BlockBackend *iter;
+    BlockDriverState *bs;
+    Error *err = NULL;
+
+    GLOBAL_STATE_CODE();
+
+    /*
+     * After drive_del, the BlockBackend is removed from the monitor name
+     * registry but still attached to the device. Find it by iterating all
+     * BlockBackends and matching by the device ID shown in "info block".
+     */
+    for (iter = blk_all_next(NULL); iter; iter = blk_all_next(iter)) {
+        DeviceState *dev = blk_get_attached_dev(iter);
+        if (dev && dev->id && strcmp(dev->id, id) == 0) {
+            blk = iter;
+            break;
+        }
+    }
+
+    if (!blk) {
+        /* Fallback: try by block backend name */
+        blk = blk_by_name(id);
+    }
+
+    if (!blk) {
+        error_setg(&err, "Device '%s' not found", id);
+        goto out;
+    }
+
+    if (blk_bs(blk)) {
+        error_setg(&err, "Device '%s' already has a medium inserted", id);
+        goto out;
+    }
+
+    bs = bdrv_open(filename, NULL, NULL, BDRV_O_RDWR, &err);
+    if (!bs) {
+        goto out;
+    }
+
+    if (blk_insert_bs(blk, bs, &err) < 0) {
+        bdrv_unref(bs);
+        goto out;
+    }
+
+    bdrv_unref(bs);
+    monitor_printf(mon, "OK\n");
+    return;
+
+out:
+    hmp_handle_error(mon, err);
+}
+
 void hmp_commit(Monitor *mon, const QDict *qdict)
 {
     const char *device = qdict_get_str(qdict, "device");
hmp-commands.hx +18 -0
index 5cc4788f12..79af8e8988 100644
@@ -207,6 +207,24 @@ SRST
   actions (drive options rerror, werror).
 ERST
 
+    {
+        .name       = "drive_insert",
+        .args_type  = "id:B,filename:F",
+        .params     = "device filename",
+        .help       = "insert a host block device into an empty drive",
+        .cmd        = hmp_drive_insert,
+    },
+
+SRST
+``drive_insert`` *device* *filename*
+  Insert a host block device file into a drive that has been emptied by
+  ``drive_del``.  This reconnects the backing store without removing the
+  guest device, enabling transparent disk hot-swap for non-removable devices
+  such as NVMe namespaces.  Combined with PCIe AER Surprise Down error
+  injection (``pcie_aer_inject_error`` *device* ``SDN``), this enables
+  complete NVMe disk hot-swap simulation.
+ERST
+
     {
         .name       = "change",
         .args_type  = "device:B,force:-f,target:F,arg:s?,read-only-mode:s?",
include/block/block-hmp-cmds.h +1 -0
index 71113cd7ef..73c9607402 100644
@@ -21,6 +21,7 @@ void hmp_drive_add(Monitor *mon, const QDict *qdict);
 
 void hmp_commit(Monitor *mon, const QDict *qdict);
 void hmp_drive_del(Monitor *mon, const QDict *qdict);
+void hmp_drive_insert(Monitor *mon, const QDict *qdict);
 
 void hmp_drive_mirror(Monitor *mon, const QDict *qdict);
 void hmp_drive_backup(Monitor *mon, const QDict *qdict);
Signature · 1 line
2.50.1 (Apple Git-155)
#3 lore raw
In reply to mr-083
Quoted 9 lines · On Thu, Apr 09, 2026 at 09:01:09AM +0200, mr-083 wrote:

This series adds two features that together enable transparent NVMe disk hot-swap simulation in QEMU, matching the behavior of physical NVMe drives being pulled and reinserted in the same PCIe slot.

Problem:
Currently, hot-swapping an NVMe disk in QEMU requires removing the entire NVMe controller via device_del, which causes the Linux guest to assign a new controller number on re-add (e.g. nvme2 becomes nvme4). This breaks storage software that tracks drives by device name.

Hi mr-083,
Neat, I was looking for something like this recently!

Quoted 6 lines

Solution:
Patch 1 adds hotplug support for nvme-ns devices on the NvmeBus, with proper Asynchronous Event Notification (AEN) so the guest kernel detects namespace changes. This allows namespace-level hot-swap without removing the NVMe controller.

Quoted 8 lines

Patch 2 adds a drive_insert HMP command that reconnects a host block device file to an existing guest device after drive_del. This is the counterpart to drive_del for non-removable devices where blockdev-change-medium cannot be used.

The recommended hot-swap sequence is:
1. drive_del <drive-id> # disconnect backing store
2. drive_insert <device> <file> # reconnect backing store

Is it possible to achieve this with device_del + device_add instead of introducing a new monitor command?

device_del nvme-ns2
blockdev-del nvme-ns2-blk (or drive_del)
...
blockdev-add nvme-ns2-blk,... (or drive_add) device_add nvme-ns,id=nvme-ns2,nsid=2,drive=nvme-ns2-blk

Quoted 1 line

3. pcie_aer_inject_error <port> SDN # trigger controller reset

Is NVMe AEN insufficient to get the guest to recognize the Namespace change? I looked at the Linux NVMe driver code recently and got the impression it would process changes to the Namespace list upon receiving the NVMe AEN.

Quoted 24 lines

After this sequence, the guest sees the same controller and namespace names (e.g. /dev/nvme2n1 remains /dev/nvme2n1), and the NVMe driver recovers transparently via the standard AER recovery path.

Tested with:
- Linux 6.1 guest on QEMU aarch64 with HVF (macOS)
- NVMe subsystem model with multipath disabled
- DirectPV and MinIO AIStor storage stack

mr-083 (2):
hw/nvme: add namespace hotplug support
block/monitor: add drive_insert HMP command

 block/monitor/block-hmp-cmds.c | 59 +++++++++++++++++++++++
 hmp-commands.hx                | 18 +++++++
 hw/nvme/ctrl.c                 | 85 ++++++++++++++++++++++++++++++++++
 hw/nvme/ns.c                   |  1 +
 hw/nvme/subsys.c               |  2 +
 include/block/block-hmp-cmds.h |  1 +
 6 files changed, 166 insertions(+)

--
2.50.1 (Apple Git-155)

#4 lore raw

Thanks for the review!

Quoted 2 lines

Is it possible to achieve this with device_del + device_add instead of introducing a new monitor command?

Yes, device_del + device_add works. I tested it and the AEN properly notifies the guest kernel which rescans and adds/removes the block device.

However, when filesystems (XFS via DirectPV in our case) are mounted on the namespace, the old block device number is not reused on re-add. The kernel's IDA allocator only frees the ID when all references to the namespace head are released (nvme_free_ns_head), but the stale XFS mount holds a reference indefinitely.

Without mounted filesystems, the ID is reused correctly (/dev/nvme0n1 stays nvme0n1).

Quoted 2 lines

Is NVMe AEN insufficient to get the guest to recognize the Namespace change?

You're right AEN is sufficient. I confirmed that the Linux NVMe driver processes NVME_AER_NOTICE_NS_CHANGED and rescans automatically. The SDN was unnecessary.

I dropped Patch 2 (drive_insert) and sent v2 with just the namespace hotplug support. The commit message now documents the correct device_del + device_add flow.

Here is the link
https://mail.gnu.org/archive/html/qemu-devel/2026-04/msg01507.html

Thanks

Quoted 70 lines · Stefan Hajnoczi
Quoted 9 lines · On Thu, Apr 09, 2026 at 09:01:09AM +0200, mr-083 wrote:

This series adds two features that together enable transparent NVMe disk hot-swap simulation in QEMU, matching the behavior of physical NVMe drives being pulled and reinserted in the same PCIe slot.

Problem:
Currently, hot-swapping an NVMe disk in QEMU requires removing the entire NVMe controller via device_del, which causes the Linux guest to assign a new controller number on re-add (e.g. nvme2 becomes nvme4). This breaks storage software that tracks drives by device name.

Hi mr-083,
Neat, I was looking for something like this recently!

Quoted 6 lines

Solution:
Patch 1 adds hotplug support for nvme-ns devices on the NvmeBus, with proper Asynchronous Event Notification (AEN) so the guest kernel detects namespace changes. This allows namespace-level hot-swap without removing the NVMe controller.

Quoted 8 lines

Patch 2 adds a drive_insert HMP command that reconnects a host block device file to an existing guest device after drive_del. This is the counterpart to drive_del for non-removable devices where blockdev-change-medium cannot be used.

The recommended hot-swap sequence is:
1. drive_del <drive-id> # disconnect backing store
2. drive_insert <device> <file> # reconnect backing store

Is it possible to achieve this with device_del + device_add instead of introducing a new monitor command?

device_del nvme-ns2
blockdev-del nvme-ns2-blk (or drive_del)
...
blockdev-add nvme-ns2-blk,... (or drive_add) device_add nvme-ns,id=nvme-ns2,nsid=2,drive=nvme-ns2-blk

Quoted 1 line

3. pcie_aer_inject_error <port> SDN # trigger controller reset

Is NVMe AEN insufficient to get the guest to recognize the Namespace change? I looked at the Linux NVMe driver code recently and got the impression it would process changes to the Namespace list upon receiving the NVMe AEN.

Quoted 24 lines

After this sequence, the guest sees the same controller and namespace names (e.g. /dev/nvme2n1 remains /dev/nvme2n1), and the NVMe driver recovers transparently via the standard AER recovery path.

Tested with:
- Linux 6.1 guest on QEMU aarch64 with HVF (macOS)
- NVMe subsystem model with multipath disabled
- DirectPV and MinIO AIStor storage stack

mr-083 (2):
hw/nvme: add namespace hotplug support
block/monitor: add drive_insert HMP command

 block/monitor/block-hmp-cmds.c | 59 +++++++++++++++++++++++
 hmp-commands.hx                | 18 +++++++
 hw/nvme/ctrl.c                 | 85 ++++++++++++++++++++++++++++++++++
 hw/nvme/ns.c                   |  1 +
 hw/nvme/subsys.c               |  2 +
 include/block/block-hmp-cmds.h |  1 +
 6 files changed, 166 insertions(+)

--
2.50.1 (Apple Git-155)