nvme/pcie hot plug results in /dev name change
We have a customer complaining that when they hot unplug/plug their nvme-pci device the device name changes.
For example, when they hot remove /dev/nvme1n1 physically, then reinsert it, it gets identified as /dev/nvme4n1.
IIRC there is nothing about nvme_init_ctrl that guarantees the ctrl->instance will be the same because ida_alloc() will only return the next free id in the bitmap.
The fact that nvme_remove -> nvme_init_ctrl many times results in the same ctrl->instance is just a happen stance and there is nothing that guarantees /dev name consistency in Linux. Especially in the case of hot unplug/plug where the controller may very well be marked NVME_CTRL_DEAD due to the ungraceful disconnect.
True?
Signature · 4 lines
John A. Meneghini Senior Principal Platform Storage Engineer RHEL SST - Platform Storage Group [email protected]
Keith Busch
maintainerQuoted 4 lines · John Meneghini
We have a customer complaining that when they hot unplug/plug their nvme-pci device the device name changes.
For example, when they hot remove
/dev/nvme1n1physically, then reinsert it, it gets identified as/dev/nvme4n1.
Quoted 8 lines
IIRC there is nothing about
nvme_init_ctrlthat guarantees thectrl->instancewill be the same becauseida_alloc()will only return the next free id in the bitmap.The fact that
nvme_remove->nvme_init_ctrlmany times results in the samectrl->instanceis just a happen stance and there is nothing that guarantees /dev name consistency in Linux. Especially in the case of hot unplug/plug where the controller may very well be markedNVME_CTRL_DEADdue to the ungraceful disconnect.
That is correct. We don't know the identity of the device at the point we have to assign it an instance number, so the hot added one will just get the first available unique number. If you need a consistent name, we have the persistent naming rules that should create those links in /dev/disk/by-id/.
Quoted 5 lines · Keith Busch
That is correct. We don't know the identity of the device at the point we have to assign it an instance number, so the hot added one will just get the first available unique number. If you need a consistent name, we have the persistent naming rules that should create those links in
/dev/disk/by-id/.
Note that this a bit of a problem under a file system or stacking driver that handles failing drives (e.g. btrfs or md raid), that holds ontop the "old" device file, and then fails to find the new one. I had a customer complaint for that as well :)
The first hack was to force run the multipath code that can keep the node alive. That works, but is really ugly especially when dealing with corner cases such as overlapping nsids between different controllers.
In the long run I think we'll need to:
- send a notification to the holder if a device is hot removed from the block layer so that it can clean up
- make the upper layers look for the replugged devie
I've been working on some of this for a while but haven't made much progress due to other committments.
Quoted 12 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
That is correct. We don't know the identity of the device at the point we have to assign it an instance number, so the hot added one will just get the first available unique number. If you need a consistent name, we have the persistent naming rules that should create those links in
/dev/disk/by-id/.Note that this a bit of a problem under a file system or stacking driver that handles failing drives (e.g. btrfs or md raid), that holds ontop the "old" device file, and then fails to find the new one. I had a customer complaint for that as well :)
Quoted 8 lines
The first hack was to force run the multipath code that can keep the node alive. That works, but is really ugly especially when dealing with corner cases such as overlapping nsids between different controllers.
In the long run I think we'll need to:
- send a notification to the holder if a device is hot removed from the block layer so that it can clean up
When the disk is deleted, the notification has been sent to userspace via udev/kobj uevent, so user can umount the original FS or DM/MD userspace can handle the device removal.
Quoted 4 lines
- make the upper layers look for the replugged devie
I've been working on some of this for a while but haven't made much progress due to other committments.
block device persistent name is supposed to be supported by userspace, such as udev rule.
[1] https://wiki.archlinux.org/title/persistent_block_device_naming
Thanks,
Ming
Keith Busch
maintainerQuoted 26 lines · Ming Lei
Quoted 12 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
That is correct. We don't know the identity of the device at the point we have to assign it an instance number, so the hot added one will just get the first available unique number. If you need a consistent name, we have the persistent naming rules that should create those links in
/dev/disk/by-id/.Note that this a bit of a problem under a file system or stacking driver that handles failing drives (e.g. btrfs or md raid), that holds ontop the "old" device file, and then fails to find the new one. I had a customer complaint for that as well :)
Quoted 8 lines
The first hack was to force run the multipath code that can keep the node alive. That works, but is really ugly especially when dealing with corner cases such as overlapping nsids between different controllers.
In the long run I think we'll need to:
- send a notification to the holder if a device is hot removed from the block layer so that it can clean upWhen the disk is deleted, the notification has been sent to userspace via udev/kobj uevent, so user can umount the original FS or DM/MD userspace can handle the device removal.
Quoted 7 lines
Quoted 4 lines
- make the upper layers look for the replugged devie
I've been working on some of this for a while but haven't made much progress due to other committments.
block device persistent name is supposed to be supported by userspace, such as udev rule.
Come to think of it, I actually have heard many complaints about this behavior. Requiring user space deal with the teardown and restore of their open files and mount points on a transient link loss can be inconvenient. Example use cases are firmware activation requiring a Subsystem Reset, or a PCIe error containment event. Those cause the links to bounce, which can trigger hot plug events in some platforms.
The native nvme multipath looks like it could be leveraged to improving that user experience if we wanted to make that layer an option for non-multipath devices.
Quoted 32 lines · Keith Busch
Quoted 31 lines · Ming Lei
Quoted 12 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
That is correct. We don't know the identity of the device at the point we have to assign it an instance number, so the hot added one will just get the first available unique number. If you need a consistent name, we have the persistent naming rules that should create those links in
/dev/disk/by-id/.Note that this a bit of a problem under a file system or stacking driver that handles failing drives (e.g. btrfs or md raid), that holds ontop the "old" device file, and then fails to find the new one. I had a customer complaint for that as well :)
Quoted 8 lines
The first hack was to force run the multipath code that can keep the node alive. That works, but is really ugly especially when dealing with corner cases such as overlapping nsids between different controllers.
In the long run I think we'll need to:
- send a notification to the holder if a device is hot removed from the block layer so that it can clean upWhen the disk is deleted, the notification has been sent to userspace via udev/kobj uevent, so user can umount the original FS or DM/MD userspace can handle the device removal.
Quoted 4 lines
- make the upper layers look for the replugged devie
I've been working on some of this for a while but haven't made much progress due to other committments.
Quoted 6 lines
Quoted 2 lines
block device persistent name is supposed to be supported by userspace, such as udev rule.
Come to think of it, I actually have heard many complaints about this behavior. Requiring user space deal with the teardown and restore of their open files and mount points on a transient link loss can be inconvenient. Example use cases
If IO error is returned to FS, I guess umount may have to be done since it might be one meta IO. But if userspace has persistent device name, it is easy for userspace to handle the umount and re-setup.
Quoted 3 lines
are firmware activation requiring a Subsystem Reset, or a PCIe error containment event. Those cause the links to bounce, which can trigger hot plug events in some platforms.
The above isn't unique for nvme, and it is just easier for nvme-pci to handle timeout/err by removing device, IMO.
Quoted 3 lines
The native nvme multipath looks like it could be leveraged to improving that user experience if we wanted to make that layer an option for non-multipath devices.
Can you share the basic idea? Will nvme mulitpath hold the io error and not propagate to upper layer until new device is probed? What if the new device is probed late, and IO has been timed out and err is returned to upper layer?
Thanks,
Ming
Quoted 8 lines · Ming Lei
Quoted 3 lines
The native nvme multipath looks like it could be leveraged to improving that user experience if we wanted to make that layer an option for non-multipath devices.
Can you share the basic idea? Will nvme mulitpath hold the io error and not propagate to upper layer until new device is probed? What if the new device is probed late, and IO has been timed out and err is returned to upper layer?
Attached is what I did, but I'm really not sure it is the right final approach.
From 7016fc46f29463871b68348088384b3d0ceeba91 Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Sun, 30 Oct 2022 18:17:28 +0100
Subject: nvme: split ANA attribute handling from nvme_ns_id_attr_group
Use a separate attribute group for the ANA attributes that aren't shown on the multipath-device instead of just hiding them at runtime. While this is a little more code, it keeps the separation clear.
--- drivers/nvme/host/core.c | 22 +++++++------------- drivers/nvme/host/multipath.c | 38 ++++++++++++++++++++++++++++++++--- drivers/nvme/host/nvme.h | 14 ++----------- 3 files changed, 44 insertions(+), 30 deletions(-)
drivers/nvme/host/core.c +7 -15
@@ -3432,10 +3432,6 @@ static struct attribute *nvme_ns_id_attrs[] = { &dev_attr_nguid.attr, &dev_attr_eui.attr, &dev_attr_nsid.attr, -#ifdef CONFIG_NVME_MULTIPATH - &dev_attr_ana_grpid.attr, - &dev_attr_ana_state.attr, -#endif NULL, };
@@ -3458,24 +3454,20 @@ static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,if (!memchr_inv (ids->eui64,0 ,sizeof (ids->eui64)))return 0 ; } -#ifdef CONFIG_NVME_MULTIPATH -if (a == &dev_attr_ana_grpid.attr || a == &dev_attr_ana_state.attr) { -if (dev_to_disk (dev)->fops != &nvme_bdev_ops)/* per-path attr */ -return 0 ; -if (!nvme_ctrl_use_ana (nvme_get_ns_from_dev (dev)->ctrl)) -return 0 ; - } -#endif +return a->mode; } -static const struct attribute_group nvme_ns_id_attr_group = { +const struct attribute_group nvme_ns_id_attr_group = { .attrs = nvme_ns_id_attrs, .is_visible = nvme_ns_id_attrs_are_visible, }; -const struct attribute_group *nvme_ns_id_attr_groups[] = { +static const struct attribute_group *nvme_disk_attr_groups[] = { &nvme_ns_id_attr_group, +#ifdef CONFIG_NVME_MULTIPATH + &nvme_ns_ana_attr_group, +#endifNULL , };
@@ -4250,7 +4242,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)up_write (&ctrl->namespaces_rwsem);nvme_get_ctrl (ctrl); -if (device_add_disk (ctrl->device, ns->disk, nvme_ns_id_attr_groups)) +if (device_add_disk (ctrl->device, ns->disk, nvme_disk_attr_groups)) goto out_cleanup_ns_from_list;if (!nvme_ns_head_multipath (ns->head))
drivers/nvme/host/multipath.c +35 -3
@@ -50,6 +50,11 @@ void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys) subsys->iopolicy = iopolicy; } +static inline bool nvme_ctrl_use_ana (struct nvme_ctrl *ctrl) +{ +return ctrl->ana_log_buf !=NULL ; +} +void nvme_mpath_unfreeze (struct nvme_subsystem *subsys) {struct nvme_ns_head *h;
@@ -443,6 +448,11 @@ static const struct file_operations nvme_ns_head_chr_fops = { .uring_cmd_iopoll = nvme_ns_head_chr_uring_cmd_iopoll, }; +static const struct attribute_group *nvme_mpath_disk_attr_groups[] = { + &nvme_ns_id_attr_group, +NULL , +}; +static int nvme_add_ns_head_cdev (struct nvme_ns_head *head) {int ret;
@@ -539,7 +549,7 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)*/ if (!test_and_set_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) { rc =device_add_disk (&head->subsys->dev, head->disk, - nvme_ns_id_attr_groups); + nvme_mpath_disk_attr_groups);if (rc) {clear_bit (NVME_NSHEAD_DISK_LIVE , &ns->flags);return ;
@@ -780,7 +790,7 @@ static ssize_t ana_grpid_show(struct device *dev, struct device_attribute *attr, {return sysfs_emit (buf,"%d\n" ,nvme_get_ns_from_dev (dev)->ana_grpid); } -DEVICE_ATTR_RO (ana_grpid); +static DEVICE_ATTR_RO (ana_grpid);static ssize_t ana_state_show (struct device *dev,struct device_attribute *attr,char *buf)
@@ -789,7 +799,29 @@ static ssize_t ana_state_show(struct device *dev, struct device_attribute *attr,return sysfs_emit (buf,"%s\n" , nvme_ana_state_names[ns->ana_state]); } -DEVICE_ATTR_RO (ana_state); +static DEVICE_ATTR_RO (ana_state); + +static struct attribute *nvme_ns_ana_attrs[] = { + &dev_attr_ana_grpid.attr, + &dev_attr_ana_state.attr, +NULL , +}; + +static umode_t nvme_ns_ana_attr_is_visible (struct kobject *kobj, +struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + +if (!nvme_ctrl_use_ana (nvme_get_ns_from_dev (dev)->ctrl)) +return 0 ; + +return a->mode; +} + +const struct attribute_group nvme_ns_ana_attr_group = { + .attrs = nvme_ns_ana_attrs, + .is_visible = nvme_ns_ana_attr_is_visible, +};static int nvme_lookup_ana_group_desc (struct nvme_ctrl *ctrl,struct nvme_ana_group_desc *desc,void *data)
drivers/nvme/host/nvme.h +2 -12
@@ -855,17 +855,13 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,int nvme_getgeo (struct block_device *bdev,struct hd_geometry *geo);int nvme_dev_uring_cmd (struct io_uring_cmd *ioucmd,unsigned int issue_flags); -extern const struct attribute_group *nvme_ns_id_attr_groups[]; +extern const struct attribute_group nvme_ns_id_attr_group; +extern const struct attribute_group nvme_ns_ana_attr_group;extern const struct pr_ops nvme_pr_ops;extern const struct block_device_operations nvme_ns_head_ops;struct nvme_ns *nvme_find_path (struct nvme_ns_head *head);#ifdef CONFIG_NVME_MULTIPATH -static inline bool nvme_ctrl_use_ana (struct nvme_ctrl *ctrl) -{ -return ctrl->ana_log_buf !=NULL ; -} -void nvme_mpath_unfreeze (struct nvme_subsystem *subsys);void nvme_mpath_wait_freeze (struct nvme_subsystem *subsys);void nvme_mpath_start_freeze (struct nvme_subsystem *subsys);
@@ -894,16 +890,10 @@ static inline void nvme_trace_bio_complete(struct request *req) }extern bool multipath; -extern struct device_attribute dev_attr_ana_grpid; -extern struct device_attribute dev_attr_ana_state;extern struct device_attribute subsys_attr_iopolicy;#else #define multipath false -static inline bool nvme_ctrl_use_ana (struct nvme_ctrl *ctrl) -{ -return false; -}static inline void nvme_failover_req (struct request *req) { }
Signature · 1 line
2.39.0
From 76e56c0c5a1e6bb3d9d4d11255a5bc4db634e28b Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Sun, 30 Oct 2022 18:39:53 +0100
Subject: nvme: allow pinning shared namespaces
Add a sysfs attribute to pin a ns_head. While pinned the ns_head will not go away even if no namespace currently exists.
--- drivers/nvme/host/core.c | 11 ++++-- drivers/nvme/host/multipath.c | 63 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 1 + 3 files changed, 72 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +8 -3
@@ -3897,8 +3897,12 @@ static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,*/ if (h->ns_id != nsid || !nvme_is_unique_nsid (ctrl, h))continue ; -if (!list_empty (&h->list) &&nvme_tryget_ns_head (h)) -return h; +if (list_empty (&h->list) && + !test_bit (NVME_NSHEAD_PINNED , &h->flags)) +continue ; +if (!nvme_tryget_ns_head (h)) +continue ; +return h; }return NULL ;
@@ -4294,7 +4298,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)mutex_lock (&ns->ctrl->subsys->lock);list_del_rcu (&ns->siblings); -if (list_empty (&ns->head->list)) { +if (list_empty (&ns->head->list) && + !test_bit (NVME_NSHEAD_PINNED , &ns->head->flags)) {list_del_init (&ns->head->entry); last_path = true; }
drivers/nvme/host/multipath.c +63 -0
71 lines not on this page · raw message
drivers/nvme/host/nvme.h +1 -0
@@ -442,6 +442,7 @@ struct nvme_ns_head {struct mutex lock;unsigned long flags;#define NVME_NSHEAD_DISK_LIVE 0 +#define NVME_NSHEAD_PINNED 1struct nvme_ns __rcu *current_path[];#endif };
Signature · 1 line
2.39.0
From 9ceb1ab63aeab757dab5f78f62a2b467dc34e103 Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Mon, 31 Oct 2022 10:34:44 +0100
Subject: HACK: allow running multipath code for non-shared namespaces
--- drivers/nvme/host/core.c | 4 ++-- drivers/nvme/host/multipath.c | 3 +-- 2 files changed, 3 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +2 -2
@@ -1447,7 +1447,7 @@ static int nvme_ns_info_from_identify(struct nvme_ctrl *ctrl,if (ret)return ret; info->anagrpid = id->anagrpid; - info->is_shared = id->nmic &NVME_NS_NMIC_SHARED ; + info->is_shared = true; info->is_readonly = id->nsattr &NVME_NS_ATTR_RO ; info->is_ready = true;if (ctrl->quirks &NVME_QUIRK_BOGUS_NID ) {
@@ -1483,7 +1483,7 @@ static int nvme_ns_info_from_id_cs_indep(struct nvme_ctrl *ctrl, ret =nvme_submit_sync_cmd (ctrl->admin_q, &c, id,sizeof (*id));if (!ret) { info->anagrpid = id->anagrpid; - info->is_shared = id->nmic &NVME_NS_NMIC_SHARED ; + info->is_shared = true; info->is_readonly = id->nsattr &NVME_NS_ATTR_RO ; info->is_ready = id->nstat &NVME_NSTAT_NRDY ; }
drivers/nvme/host/multipath.c +1 -2
@@ -562,8 +562,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)* We also do this for private namespaces as the namespace sharing flag * could change after a rescan. */ -if (!(ctrl->subsys->cmic &NVME_CTRL_CMIC_MULTI_CTRL ) || - !nvme_is_unique_nsid (ctrl, head) || !multipath) +if (!nvme_is_unique_nsid (ctrl, head) || !multipath)return 0 ; head->disk =blk_alloc_disk (ctrl->numa_node);
Signature · 1 line
2.39.0
Keith Busch
maintainerQuoted 5 lines · Christoph Hellwig
Quoted 4 lines · Ming Lei
Quoted 3 lines
The native nvme multipath looks like it could be leveraged to improving that user experience if we wanted to make that layer an option for non-multipath devices.
Quoted 7 lines
Quoted 4 lines
Can you share the basic idea? Will nvme mulitpath hold the io error and not propagate to upper layer until new device is probed? What if the new device is probed late, and IO has been timed out and err is returned to upper layer?
Attached is what I did, but I'm really not sure it is the right final approach.
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the requeue_list while the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.
Sagi Grimberg
maintainerQuoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
The native nvme multipath looks like it could be leveraged to improving that user experience if we wanted to make that layer an option for non-multipath devices.
Quoted 7 lines
Quoted 4 lines
Can you share the basic idea? Will nvme mulitpath hold the io error and not propagate to upper layer until new device is probed? What if the new device is probed late, and IO has been timed out and err is returned to upper layer?
Attached is what I did, but I'm really not sure it is the right final approach.
Also not sure if this is going to work out, but it looks like a good start to me.
Quoted 7 lines
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.
Isn't this equivalent to dm-mpath queue_if_no_path or no_path_timeout ?
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
Keith Busch
maintainerQuoted 11 lines · Sagi Grimberg
Quoted 10 lines
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.
Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?
Similiar, but generic to non-multipath devices.
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
I don't think we're looking for any behavioral changes in the upper layers.
Quoted 12 lines · Keith Busch
Quoted 11 lines · Sagi Grimberg
Quoted 10 lines
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.
Quoted 8 lines
Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
I don't think we're looking for any behavioral changes in the upper layers.
That also means no matter if the nvme mpath layer is added or not, upper layer still has to handle this kind of failure, so what is the difference made from the added nvme-mpath?
Thanks,
Ming
Sagi Grimberg
maintainerQuoted 16 lines
Quoted 16 lines
Quoted 12 lines
Quoted 11 lines
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
I don't think we're looking for any behavioral changes in the upper layers.
That also means no matter if the nvme mpath layer is added or not, upper layer still has to handle this kind of failure, so what is the difference made from the added nvme-mpath?
What do you expect from the upper layer? The device went away...
The suggestion is that when the device comes back the upper layer will detect it and reconfigure it (restore it to a raid/dm or remount the FS)?
Quoted 18 lines · Sagi Grimberg
Quoted 18 lines
Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
I don't think we're looking for any behavioral changes in the upper layers.
That also means no matter if the nvme mpath layer is added or not, upper layer still has to handle this kind of failure, so what is the difference made from the added nvme-mpath?
What do you expect from the upper layer? The device went away...
From developer viewpoint, FS has to be re-configured(re-mount after new device comes back).
Quoted 4 lines
The suggestion is that when the device comes back the upper layer will detect it and reconfigure it (restore it to a raid/dm or remount the FS)?
I understand userspace is capable of recognizing if same device comes back, and userspace can get notified with device add/del. Why bother kernel to do that?
Also suppose the mpath device is invented for this purpose, we can't guarantee that the mapth dev won't be gone and back, so userspace has to deal with the situation, right?
Also the issue isn't nvme unique, and it shouldn't be implemented in nvme-mpath suppose agreement is reached to solve it in kernel.
Thanks,
Ming
Keith Busch
maintainerQuoted 17 lines · Ming Lei
Quoted 16 lines · Keith Busch
Quoted 11 lines · Sagi Grimberg
Quoted 10 lines
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
I don't think we're looking for any behavioral changes in the upper layers.
That also means no matter if the nvme mpath layer is added or not, upper layer still has to handle this kind of failure, so what is the difference made from the added nvme-mpath?
I don't understand. There's no difference here for the upper layers. It's just a different timing from when a disconnect starts failing IO. The upper layers can keep doing what they're already doing in either case.
Quoted 21 lines · Keith Busch
Quoted 20 lines · Ming Lei
Quoted 19 lines · Keith Busch
Quoted 11 lines · Sagi Grimberg
Quoted 10 lines
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't think we're looking for any behavioral changes in the upper layers.
That also means no matter if the nvme mpath layer is added or not, upper layer still has to handle this kind of failure, so what is the difference made from the added nvme-mpath?
I don't understand. There's no difference here for the upper layers. It's just a different timing from when a disconnect starts failing IO. The upper layers
If that isn't real disconnect, probably it is better to solve it in nvme error handling, and device removal should be done as the last straw.
Quoted 1 line
can keep doing what they're already doing in either case.
Because any device(include the to-be-invented mpath device) may be gone and back, userspace needs to take measures to cover them if persistent device name is required.
Thanks,
Ming
Sagi Grimberg
maintainerQuoted 12 lines
Quoted 12 lines
Quoted 11 lines
Also not sure if this is going to work out, but it looks like a good start to me.
Instead of user pinning the virtual device so that it never goes away though, I considered a user tunable "device missing delay" parameter to debounce link events. New IOs would be deferred to the
requeue_listwhile the timer is active, and then EIO'ed if the timer expires without a successful LIVE controller attachment. The use cases I'm considering are short bounces from transient link resets, so I'd expect timers to be from a few seconds to maybe a minute.
Quoted 8 lines
Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
I don't think we're looking for any behavioral changes in the upper layers.
I'm looking to what Christoph wrote, that in the long term upper layers will need to look for replugged devices. Let's take the simple example of a FS mounted disk that is going away, today the FS may get an error and go into RO or failure, needing a manual remount after the disk came back.
What exactly is the proposal for change here?
Quoted 3 lines · Keith Busch
Quoted 1 line
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Exactly!
Quoted 4 lines
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
I don't think we're looking for any behavioral changes in the upper layers.
There's really two different things, which would both a problem:
- make the device not go away if we expect it to come back by using the multipath code
- tell the upper layer that a devie went away when it actually did
Right now if user has a block device open, it will keep a reference to it forever. If we send a notification to the upper layer not only can it release that refrence, but it also knows the device went away. This really helps to e.g. rebalance data in a RAID or RAID-like setup.
Sagi Grimberg
maintainerQuoted 10 lines
Quoted 4 lines
Quoted 2 lines
Isn't this equivalent to dm-mpath
queue_if_no_pathorno_path_timeout?Similiar, but generic to non-multipath devices.
Exactly!
Quoted 3 lines
Quoted 2 lines
We can keep the mpath device around, but if not, what is the desired behavior from the upper layers?
Quoted 6 lines
Quoted 1 line
I don't think we're looking for any behavioral changes in the upper layers.
There's really two different things, which would both a problem:
- make the device not go away if we expect it to come back by using the multipath code
How do you know in the driver what to expect? how can anyone know what to expect? There is also a question of what to do with the I/O coming in, queue it or fail it and when...
Quoted 1 line
- tell the upper layer that a devie went away when it actually did
Don't we already send a udev event KOBJ_REMOVE when a block device goes away?
IIRC LVM already removes its stuff if the underlying block device goes away (although I do recall that the dm did leave the device handle open)...
Quoted 3 lines
Right now if user has a block device open, it will keep a reference to it forever. If we send a notification to the upper layer not only can it release that refrence, but it also knows the device went away.
--
KERNEL[520130.230115] remove /devices/virtual/bdi/252:0 (bdi)
KERNEL[520130.230261] remove /devices/virtual/block/nullb0 (block)
--
Quoted 1 line
This really helps to e.g. rebalance data in a RAID or RAID-like setup.
I thought that RAID already tracked these sort of events...
Keith Busch
maintainerQuoted 5 lines · Christoph Hellwig
Quoted 4 lines · Ming Lei
Quoted 3 lines
The native nvme multipath looks like it could be leveraged to improving that user experience if we wanted to make that layer an option for non-multipath devices.
Quoted 7 lines
Quoted 4 lines
Can you share the basic idea? Will nvme mulitpath hold the io error and not propagate to upper layer until new device is probed? What if the new device is probed late, and IO has been timed out and err is returned to upper layer?
Attached is what I did, but I'm really not sure it is the right final approach.
Just adding my proof-of-concept attempt to this below. It uses a hard-coded 10 second delay before proceeding with deletion. There may be some unlikely race conditions with this, but the concept works. I tested mounting a single path nvme namespace, and quick remove-add sequences while running file fio on the filesystem. Everything proceeded seemlessly with no need to remount.
---
drivers/nvme/host/core.c +4 -10
@@ -3974,7 +3974,7 @@ static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,*/ if (h->ns_id != nsid || !nvme_is_unique_nsid (ctrl, h))continue ; -if (!list_empty (&h->list) &&nvme_tryget_ns_head (h)) +if (nvme_tryget_ns_head (h))return h; }
@@ -4185,7 +4185,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info) } }else { ret = -EINVAL ; -if (!info->is_shared || !head->shared) { +if ((!info->is_shared || !head->shared) && + !list_empty (&head->list)) {dev_err (ctrl->device,"Duplicate unshared namespace %d\n" , info->nsid);
@@ -4350,8 +4351,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)static void nvme_ns_remove (struct nvme_ns *ns) { -bool last_path = false; -if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ;
@@ -4371,10 +4370,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)mutex_lock (&ns->ctrl->subsys->lock);list_del_rcu (&ns->siblings); -if (list_empty (&ns->head->list)) { -list_del_init (&ns->head->entry); - last_path = true; - }mutex_unlock (&ns->ctrl->subsys->lock);/* guarantee not available in head->list */
@@ -4388,8 +4383,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)list_del_init (&ns->list);up_write (&ns->ctrl->namespaces_rwsem); -if (last_path) -nvme_mpath_shutdown_disk (ns->head); +nvme_mpath_shutdown_disk (ns->head);nvme_put_ns (ns); }
drivers/nvme/host/multipath.c +24 -9
@@ -387,7 +387,8 @@ static void nvme_ns_head_submit_bio(struct bio *bio)trace_block_bio_remap (bio,disk_devt (ns->head->disk), bio->bi_iter.bi_sector);submit_bio_noacct (bio); - }else if (nvme_available_path (head)) { + }else if (nvme_available_path (head) || +delayed_work_pending (&head->remove_work)) {dev_warn_ratelimited (dev,"no usable path - requeuing I/O\n" );spin_lock_irq (&head->requeue_lock);
@@ -505,6 +506,23 @@ static void nvme_requeue_work(struct work_struct *work) } } +static void nvme_remove_head_work (struct work_struct *work) +{ +struct nvme_ns_head *head =container_of (to_delayed_work (work), + struct nvme_ns_head, remove_work); + +if (!list_empty (&head->list)) +return ; + +list_del_init (&head->entry); +kblockd_schedule_work (&head->requeue_work); +if (test_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) { +nvme_cdev_del (&head->cdev, &head->cdev_device); +del_gendisk (head->disk); + } +nvme_put_ns_head (head); +} +int nvme_mpath_alloc_disk (struct nvme_ctrl *ctrl,struct nvme_ns_head *head) {bool vwc = false;
@@ -513,14 +531,14 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)bio_list_init (&head->requeue_list);spin_lock_init (&head->requeue_lock);INIT_WORK (&head->requeue_work, nvme_requeue_work); +INIT_DELAYED_WORK (&head->remove_work, nvme_remove_head_work);/* * Add a multipath node if the subsystems supports multiple controllers. * We also do this for private namespaces as the namespace sharing flag * could change after a rescan. */ -if (!(ctrl->subsys->cmic &NVME_CTRL_CMIC_MULTI_CTRL ) || - !nvme_is_unique_nsid (ctrl, head) || !multipath) +if (!nvme_is_unique_nsid (ctrl, head) || !multipath)return 0 ; head->disk =blk_alloc_disk (ctrl->numa_node);
@@ -553,6 +571,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)if (ctrl->vwc &NVME_CTRL_VWC_PRESENT ) vwc = true;blk_queue_write_cache (head->disk->queue, vwc, vwc); +nvme_tryget_ns_head (head);return 0 ; }
@@ -871,13 +890,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)void nvme_mpath_shutdown_disk (struct nvme_ns_head *head) { -if (!head->disk) +if (!list_empty (&head->list) || !head->disk)return ; -kblockd_schedule_work (&head->requeue_work); -if (test_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) { -nvme_cdev_del (&head->cdev, &head->cdev_device); -del_gendisk (head->disk); - } +queue_delayed_work (nvme_wq, &head->remove_work,10 *HZ ); }void nvme_mpath_remove_disk (struct nvme_ns_head *head)
drivers/nvme/host/nvme.h +3 -0
@@ -446,6 +446,9 @@ struct nvme_ns_head {struct work_struct requeue_work;struct mutex lock;unsigned long flags; + +struct delayed_work remove_work; +#define NVME_NSHEAD_DISK_LIVE 0struct nvme_ns __rcu *current_path[];#endif
--