nvme: flush rescan worker before resetting
Subsystems (2)
cover nvme: flush rescan worker before resetting
Hi all,
There is a race condition between namespace rescanning and controller reset; during controller reset all namespaces are quiesed vie nams_stop_ctrl(), and after reset all namespaces are unquiesced again.
When namespace scanning was active by the time controller reset was triggered the rescan code will call nvme_ns_remove(), which then will cause a kernel crash in nvme_start_ctrl() as it'll trip over uninitialized namespaces.
To fix this two patches are required; the first one will skip the call to nvme_ns_remove() if the controller is not live, and the second one will flush the rescan worker before resetting to avoid this situation from occurring.
As usual, comments and reviews are welcome.
Hannes Reinecke (2):
nvme: Do not remove namespaces during reset
nvme: flush scan_work when resetting controller
drivers/nvme/host/core.c | 21 +++++++++++++++++++++ drivers/nvme/host/fc.c | 1 + drivers/nvme/host/rdma.c | 1 + drivers/nvme/host/tcp.c | 1 + 4 files changed, 24 insertions(+)
Signature · 1 line
2.16.4
PATCH 1/2 nvme: Do not remove namespaces during reset
When a controller is resetting or reconnecting there is no way how we could establish the validity of any given namespace. So do not call nvme_ns_remove() during resetting or reconnecting and rely on the call to nvme_scan_queue() after reset to fixup things.
--- drivers/nvme/host/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/nvme/host/core.c +11 -0
@@ -3358,6 +3358,17 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)static void nvme_ns_remove (struct nvme_ns *ns) { +/* +* We cannot make any assumptions about namespaces during +* reset; in particular we shouldn't attempt to remove them +* as I/O might still be queued to them. +* So ignore this call during reset and rely on the +* rescan after reset to clean up things again. +*/ +if (ns->ctrl->state ==NVME_CTRL_RESETTING || + ns->ctrl->state ==NVME_CTRL_CONNECTING ) +return ; +if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ;
Signature · 1 line
2.16.4
Sagi Grimberg
maintainerPATCH 1/2 nvme: Do not remove namespaces during reset
Quoted 21 lines
When a controller is resetting or reconnecting there is no way how we could establish the validity of any given namespace. So do not call
nvme_ns_remove()during resetting or reconnecting and rely on the call tonvme_scan_queue()after reset to fixup things.Signed-off-by Hannes Reinecke <hare at suse.com>--- drivers/nvme/host/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)drivers/nvme/host/core.c +1
-0@@ -3358,6 +3358,17 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)static void nvme_ns_remove (struct nvme_ns *ns) { +/*
drivers/nvme/host/core.c static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3371
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+* We cannot make any assumptions about namespaces during +* reset; in particular we shouldn't attempt to remove them +* as I/O might still be queued to them. +* So ignore this call during reset and rely on the +* rescan after reset to clean up things again. +*/ +if (ns->ctrl->state ==NVME_CTRL_RESETTING || + ns->ctrl->state ==NVME_CTRL_CONNECTING ) +return ; +
This is conflicting with the commit log, this condition needs to go to the call site.
PATCH 1/2 nvme: Do not remove namespaces during reset
In reply to Hannes ReineckeHi Hannes,
Could you explain a bit what the user visible issue is addressed by this patch?
Quoted 22 lines · Hannes Reinecke
When a controller is resetting or reconnecting there is no way how we could establish the validity of any given namespace. So do not call
nvme_ns_remove()during resetting or reconnecting and rely on the call tonvme_scan_queue()after reset to fixup things.Signed-off-by Hannes Reinecke <hare at suse.com>--- drivers/nvme/host/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)drivers/nvme/host/core.c +3
-0@@ -3358,6 +3358,17 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)static void nvme_ns_remove (struct nvme_ns *ns) { +/* +* We cannot make any assumptions about namespaces during +* reset; in particular we shouldn't attempt to remove them
drivers/nvme/host/core.c static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3373
Quoted 10 lines
drivers/nvme/host/core.c +8
-0+* as I/O might still be queued to them. +* So ignore this call during reset and rely on the +* rescan after reset to clean up things again. +*/ +if (ns->ctrl->state ==NVME_CTRL_RESETTING || + ns->ctrl->state ==NVME_CTRL_CONNECTING ) +return ; +if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ;
nvme_ns_remove() may be called from nvme_remove_invalid_namespaces() and nvme_remove_namespaces(), in which the 'ns' to be removed is retrieved & deleted from ctrl->namespaces.
That means if the 'ns' needs to be removed by the two mentioned functions from scan work context again after reset is done, the removal may never be done because the 'ns' can't be found in ctrl->namespaces.
If you want to avoid the use-after-free issue[1], I'd suggest to use the queue refcount.
https://lore.kernel.org/linux-block/20190424110221.17435-10-ming.lei at redhat.com/
Thanks,
Ming
PATCH 2/2 nvme: flush scan_work when resetting controller
In reply to Hannes ReineckeWhen resetting the controller there is no point whatsoever to have a scan run in parallel; we cannot access the controller and we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.
--- drivers/nvme/host/core.c | 10 ++++++++++ drivers/nvme/host/fc.c | 1 + drivers/nvme/host/rdma.c | 1 + drivers/nvme/host/tcp.c | 1 + 4 files changed, 13 insertions(+)
drivers/nvme/host/core.c +10 -0
@@ -1627,6 +1627,9 @@ static void nvme_update_disk_info(struct gendisk *disk,sector_t capacity =le64_to_cpu (id->nsze) << (ns->lba_shift -9 );unsigned short bs =1 << ns->lba_shift; +if (ns->ctrl->state !=NVME_CTRL_LIVE ) +return ; +if (ns->lba_shift >PAGE_SHIFT ) {/* unsupported block size, set capacity to 0 later */ bs = (1 <<9 );
@@ -1702,6 +1705,9 @@ static int nvme_revalidate_disk(struct gendisk *disk)return -ENODEV ; } +if (ctrl->state !=NVME_CTRL_LIVE ) +return 0 ; + id =nvme_identify_ns (ctrl, ns->head->ns_id);if (!id)return -ENODEV ;
@@ -3437,6 +3443,8 @@ static int nvme_scan_ns_list(struct nvme_ctrl *ctrl, unsigned nn)return -ENOMEM ;for (i =0 ; i < num_lists; i++) { +if (ctrl->state !=NVME_CTRL_LIVE ) + goto free; ret =nvme_identify_ns_list (ctrl, prev, ns_list);if (ret) goto free;
@@ -3515,6 +3523,8 @@ static void nvme_scan_work(struct work_struct *work)if (test_and_clear_bit (NVME_AER_NOTICE_NS_CHANGED , &ctrl->events)) {dev_info (ctrl->device,"rescanning namespaces.\n" );nvme_clear_changed_ns_log (ctrl); +if (ctrl->state !=NVME_CTRL_LIVE ) +return ; }if (nvme_identify_ctrl (ctrl, &id))
drivers/nvme/host/fc.c +1 -0
@@ -2909,6 +2909,7 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)__nvme_fc_terminate_io (ctrl); +flush_work (&ctrl->ctrl.scan_work);nvme_stop_ctrl (&ctrl->ctrl);if (ctrl->rport->remoteport.port_state ==FC_OBJSTATE_ONLINE )
drivers/nvme/host/rdma.c +1 -0
@@ -1839,6 +1839,7 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)nvme_stop_ctrl (&ctrl->ctrl);nvme_rdma_shutdown_ctrl (ctrl, false); +flush_work (&ctrl->ctrl.scan_work);if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING )) {/* state change failure should never happen */
drivers/nvme/host/tcp.c +1 -0
@@ -1872,6 +1872,7 @@ static void nvme_reset_ctrl_work(struct work_struct *work)nvme_stop_ctrl (ctrl);nvme_tcp_teardown_ctrl (ctrl, false); +flush_work (&ctrl->scan_work);if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_CONNECTING )) {/* state change failure is ok if we're in DELETING state */
Signature · 1 line
2.16.4
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 3 lines
When resetting the controller there is no point whatsoever to have a scan run in parallel;
There is also no point of trying to prevent it.
Quoted 6 lines
we cannot access the controller and
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.
What is this fixing? can we get a detailed description?
These ctrl->state conditions sprayed around that are not barriered by anything do not prevent any scan to run in parallel with resets.
Also, do note that resets will unquiesce the admin queue right away and not when the reset completes (rdma, tcp, fc). The only exception is pci that will unquiesce as soon as the reset fails.
So can you please clarify the problem?
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 6 lines · Sagi Grimberg
Quoted 3 lines
When resetting the controller there is no point whatsoever to have a scan run in parallel;
There is also no point of trying to prevent it.
Oh, I don't mind having a scan running in parallel once reset is underway; that will be properly short-circuited anyway.
What I object to is having scanning and resetting _starting_ at the same time, as then we incur all sorts of race conditions.
Like this one :-)
Quoted 7 lines
Quoted 6 lines
we cannot access the controller and
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.
Quoted 5 lines
What is this fixing? can we get a detailed description?
These
ctrl->stateconditions sprayed around that are not barriered by anything do not prevent any scan to run in parallel with resets.
The 'ctrl->state' conditions are required to avoid I/O to be stalled during reset. Problem here is that I/O will be held during reset, and resubmitted once reset is finished.
The transport drivers work around this problem to terminate any I/O prior to resetting the controller, but this will only cover I/O which had been pending _before_ reset was called.
Any I/O being started after that will still be held, and we would live-lock trying to flush the scan thread.
So essentially we guard any I/O with the state check, and short circuit it if we find ourselves in reset.
There is no need to protect the state check, as we will be aborting I/O when a state change to 'RESETTING' happens, and the update should be visible to all threads after that.
If you want I can separate this into two patches, one with flushing the scan function and one with the state check guards, and detail out the reasoning here.
Quoted 6 lines
Also, do note that resets will unquiesce the admin queue right away and not when the reset completes (rdma, tcp, fc). The only exception is pci that will unquiesce as soon as the reset fails.
So can you please clarify the problem?
Sure. During failover testing NetApp had been seeing several crashes like:
[67088.344034] WARNING: CPU: 4 PID: 25020 at ../lib/percpu-refcount.c:334 percpu_ref_kill_and_confirm+0x7a/0xa0 [67088.344035] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5
auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache af_packet iscsi_ibft iscsi_boot_sysfs xfs libcrc32c mgag200 ttm drm_kms_helper drm drm_panel_orientation_quirks iTCO_wdt syscopyarea iTCO_vendor_support sysfillrect igb sysimgblt fb_sys_fops be2net ptp pps_core i2c_algo_bit dca intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp ipmi_ssif coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper lpc_ich ipmi_si cryptd button mfd_core i2c_i801 pcspkr ipmi_devintf ipmi_msghandler ac shpchp btrfs xor raid6_pq sd_mod lpfc isci(X) ahci libsas nvmet_fc libahci nvmet ehci_pci scsi_transport_sas ehci_hcd crc32c_intel configfs libata usbcore nvme_fc nvme_fabrics nvme_core
[67088.344079] scsi_transport_fc sg scsi_mod autofs4
[67088.344082] Supported: No, Unreleased kernel
[67088.344085] CPU: 4 PID: 25020 Comm: kworker/u49:35 Tainted: G
4.12.14-2.g1a155c2-default #1 SLE15 (unreleased)
[67088.344085] Hardware name: FUJITSU PRIMERGY RX200 S7/D3032-A1, BIOS
V4.6.5.1 R1.8.0 for D3032-A1x 05/25/2012
[67088.344092] Workqueue: nvme-wq nvme_scan_work [nvme_core]
[67088.344094] task: ffff90a5dfa6c180 task.stack: ffff9e138d3d8000
[67088.344096] RIP: 0010:percpu_ref_kill_and_confirm+0x7a/0xa0
[67088.344097] RSP: 0018:ffff9e138d3dbc00 EFLAGS: 00010086
[67088.344099] RAX: 0000000000000055 RBX: ffff90a60d1f3ac0 RCX:
ffffffff9c066e08
[67088.344100] RDX: 0000000000000001 RSI: 0000000000000082 RDI:
0000000000000093
[67088.344100] RBP: 0000000000000246 R08: 0000000000035601 R09:
ffffffff9c83a8e0
[67088.344101] R10: ffff9e138633be98 R11: 0000000000000000 R12:
0000000000000000
[67088.344102] R13: ffff90a61e813100 R14: 0000000000000009 R15:
0000000000000200
[67088.344103] FS: 0000000000000000(0000) GS:ffff90a61fb00000(0000)
knlGS:0000000000000000
[67088.344104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[67088.344105] CR2: 00007fc607bb2860 CR3: 000000041700a004 CR4:
00000000000606e0
[67088.344106] Call Trace:
[67088.344112] blk_freeze_queue_start+0x2a/0x40
[67088.344114] blk_freeze_queue+0xe/0x40
[67088.344118] nvme_update_disk_info+0x36/0x260 [nvme_core]
[67088.344122] __nvme_revalidate_disk+0xca/0xf0 [nvme_core]
[67088.344125] nvme_revalidate_disk+0xa6/0x120 [nvme_core]
[67088.344127] ? blk_mq_get_tag+0xa3/0x220
[67088.344130] revalidate_disk+0x23/0xc0
[67088.344133] nvme_validate_ns+0x43/0x830 [nvme_core]
[67088.344137] ? wake_up_q+0x70/0x70
[67088.344139] ? blk_mq_free_request+0x12a/0x160
[67088.344142] ? __nvme_submit_sync_cmd+0x73/0xe0 [nvme_core]
[67088.344145] nvme_scan_work+0x2b3/0x350 [nvme_core]
[67088.344149] process_one_work+0x1da/0x400
[67088.344150] worker_thread+0x2b/0x3f0
[67088.344152] ? process_one_work+0x400/0x400The underlying reason is an imbalance in nvme namespace refcouting:nvme_ns_remove() will teardown the queue first, before removing it from the list of namespaces:
static void nvme_ns_remove(struct nvme_ns *ns)
{
if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags))
return;
nvme_fault_inject_fini(ns);
if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
del_gendisk(ns->disk);
blk_cleanup_queue(ns->queue);
if (blk_get_integrity(ns->disk))
blk_integrity_unregister(ns->disk);
}
mutex_lock(&ns->ctrl->subsys->lock);
list_del_rcu(&ns->siblings);
nvme_mpath_clear_current_path(ns);
mutex_unlock(&ns->ctrl->subsys->lock);
down_write(&ns->ctrl->namespaces_rwsem);
list_del_init(&ns->list);
up_write(&ns->ctrl->namespaces_rwsem);
synchronize_srcu(&ns->head->srcu);
nvme_mpath_check_last_path(ns);
nvme_put_ns(ns);
}when scanning is active during reset all namespaces are found to be unresponsive, and will be consequently removed:
static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)
{
struct nvme_ns *ns;
ns = nvme_find_get_ns(ctrl, nsid);
if (ns) {
if (ns->disk && revalidate_disk(ns->disk))
nvme_ns_remove(ns);
nvme_put_ns(ns);
} else
nvme_alloc_ns(ctrl, nsid);
}However, as the controller is resetting it will call nvme_stop_queues() at the start of the reset, and nvme_start_queues() once reset is complete.
Both iterate the namespace list like
void nvme_stop_queues(struct nvme_ctrl *ctrl)
{
struct nvme_ns *ns;
down_read(&ctrl->namespaces_rwsem);
list_for_each_entry(ns, &ctrl->namespaces, list)
blk_mq_quiesce_queue(ns->queue);
up_read(&ctrl->namespaces_rwsem);
}but this will only synchronize with the very last step of nvme_ns_remove(). At that time blk_cleanup_queue() already had been called, and consequently we crash in 'blk_mq_quiesce_queue()' (or 'blk_mq_unquiesce_queue()', depending on the timing).
This imbalance in nvme_ns_remove() is one issue which Ming Lei tried to fix up with his patchset 'blk-mq: fix races related with freezing queue'. However, he dropped the final patch
"nvme: hold request queue's refcount in ns's whole lifetime" on request from hch, as he felt that blk_cleanup_queue() was unsafe in general, and should need to include the final put for the request queue itself.
Sadly, this was precisely the patch for fixing this issue :-(
So instead of waiting for the big rewrite to happen I tried to approach things from the other end, namely stopping the race to happen in the first place.
And with these two patches the crashes are gone, so I thought to share it here if there were interest.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage hare at suse.com +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
When resetting the controller there is no point whatsoever to have a scan run in parallel;
There is also no point of trying to prevent it.
Oh, I don't mind having a scan running in parallel once reset is underway; that will be properly short-circuited anyway.
What I object to is having scanning and resetting _starting_ at the same time, as then we incur all sorts of race conditions.
But your patch does not inherently protect against it, these state conditions sprinkled can easily race the reset.
Note that I do agree with patch #1, but I don't understand how patch #2 is fixing the problem other than narrowing the window at best.
Quoted 10 lines
Like this one :-)
Quoted 7 lines
Quoted 6 lines
we cannot access the controller and
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.
Quoted 7 lines
Quoted 5 lines
What is this fixing? can we get a detailed description?
These
ctrl->stateconditions sprayed around that are not barriered by anything do not prevent any scan to run in parallel with resets.The '
ctrl->state' conditions are required to avoid I/O to be stalled during reset.
Not sure what you mean by stall, are you referring to the case that it takes a long time to reconnect?
The solution is to make sure we drain the inflight I/O, not half-way trying to prevent them from happening.
Quoted 5 lines
Problem here is that I/O will be held during reset, and resubmitted once reset is finished.
The transport drivers work around this problem to terminate any I/O prior to resetting the controller,
Its not a work around.
Quoted 4 lines
but this will only cover I/O which
had been pending _before_ reset was called.
Any I/O being started after that will still be held, and we would live-lock trying to flush the scan thread.
But resets do not flush the scan thread (before you restored it).
Quoted 2 lines
So essentially we guard any I/O with the state check, and short circuit it if we find ourselves in reset.
As I mentioned, patch #1 is fine. Please explain what patch #2 is exactly fixing.
Quoted 3 lines
There is no need to protect the state check, as we will be aborting I/O when a state change to 'RESETTING' happens, and the update should be visible to all threads after that.
I think this assumption is incorrect. Nothing guarantees when/who will have visibility to the state change.
Quoted 3 lines
If you want I can separate this into two patches, one with flushing the scan function and one with the state check guards, and detail out the reasoning here.
No need to split, if its needed, it needs to be in a single patch. But again, I want to understand after patch #1 is applied, what are the symptoms and why patch #2 is solving them.
Quoted 100 lines
Quoted 6 lines
Also, do note that resets will unquiesce the admin queue right away and not when the reset completes (rdma, tcp, fc). The only exception is pci that will unquiesce as soon as the reset fails.
So can you please clarify the problem?
Sure. During failover testing NetApp had been seeing several crashes like:
[67088.344034] WARNING: CPU: 4 PID: 25020 at ../lib/percpu-refcount.c:334 percpu_ref_kill_and_confirm+0x7a/0xa0 [67088.344035] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5
auth_rpcgssnfsv4dns_resolvernfs lockd grace sunrpc fscacheaf_packetiscsi_ibftiscsi_boot_sysfsxfs libcrc32c mgag200 ttmdrm_kms_helperdrmdrm_panel_orientation_quirksiTCO_wdtsyscopyareaiTCO_vendor_supportsysfillrect igb sysimgbltfb_sys_fopsbe2net ptppps_corei2c_algo_bitdcaintel_raplsb_edacx86_pkg_temp_thermalintel_powerclampipmi_ssifcoretempkvm_intelkvm irqbypasscrc32_pclmulghash_clmulni_intelpcbcaesni_intelaes_x86_64crypto_simdglue_helperlpc_ichipmi_sicryptd buttonmfd_corei2c_i801pcspkripmi_devintfipmi_msghandlerac shpchp btrfs xorraid6_pqsd_modlpfc isci(X) ahci libsasnvmet_fclibahci nvmetehci_pciscsi_transport_sasehci_hcdcrc32c_intelconfigfs libata usbcorenvme_fcnvme_fabricsnvme_core[67088.344079] scsi_transport_fc sg scsi_mod autofs4 [67088.344082] Supported: No, Unreleased kernel [67088.344085] CPU: 4 PID: 25020 Comm: kworker/u49:35 Tainted: G 4.12.14-2.g1a155c2-default #1 SLE15 (unreleased) [67088.344085] Hardware name: FUJITSU PRIMERGY RX200 S7/D3032-A1, BIOS V4.6.5.1 R1.8.0 for D3032-A1x 05/25/2012 [67088.344092] Workqueue: nvme-wq nvme_scan_work [nvme_core] [67088.344094] task: ffff90a5dfa6c180 task.stack: ffff9e138d3d8000 [67088.344096] RIP: 0010:percpu_ref_kill_and_confirm+0x7a/0xa0 [67088.344097] RSP: 0018:ffff9e138d3dbc00 EFLAGS: 00010086 [67088.344099] RAX: 0000000000000055 RBX: ffff90a60d1f3ac0 RCX: ffffffff9c066e08 [67088.344100] RDX: 0000000000000001 RSI: 0000000000000082 RDI: 0000000000000093 [67088.344100] RBP: 0000000000000246 R08: 0000000000035601 R09: ffffffff9c83a8e0 [67088.344101] R10: ffff9e138633be98 R11: 0000000000000000 R12: 0000000000000000 [67088.344102] R13: ffff90a61e813100 R14: 0000000000000009 R15: 0000000000000200 [67088.344103] FS: 0000000000000000(0000) GS:ffff90a61fb00000(0000) knlGS:0000000000000000 [67088.344104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [67088.344105] CR2: 00007fc607bb2860 CR3: 000000041700a004 CR4: 00000000000606e0 [67088.344106] Call Trace: [67088.344112] blk_freeze_queue_start+0x2a/0x40 [67088.344114] blk_freeze_queue+0xe/0x40 [67088.344118] nvme_update_disk_info+0x36/0x260 [nvme_core] [67088.344122] __nvme_revalidate_disk+0xca/0xf0 [nvme_core] [67088.344125] nvme_revalidate_disk+0xa6/0x120 [nvme_core] [67088.344127] ? blk_mq_get_tag+0xa3/0x220 [67088.344130] revalidate_disk+0x23/0xc0 [67088.344133] nvme_validate_ns+0x43/0x830 [nvme_core] [67088.344137] ? wake_up_q+0x70/0x70 [67088.344139] ? blk_mq_free_request+0x12a/0x160 [67088.344142] ? __nvme_submit_sync_cmd+0x73/0xe0 [nvme_core] [67088.344145] nvme_scan_work+0x2b3/0x350 [nvme_core] [67088.344149] process_one_work+0x1da/0x400 [67088.344150] worker_thread+0x2b/0x3f0 [67088.344152] ? process_one_work+0x400/0x400The underlying reason is an imbalance in nvme namespace refcouting:
nvme_ns_remove()will teardown the queue first, before removing it from the list of namespaces:static void
nvme_ns_remove(struct nvme_ns *ns){ if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) return; nvme_fault_inject_fini(ns); if (ns->disk && ns->disk->flags & GENHD_FL_UP) { del_gendisk(ns->disk); blk_cleanup_queue(ns->queue); if (blk_get_integrity(ns->disk)) blk_integrity_unregister(ns->disk); } mutex_lock(&ns->ctrl->subsys->lock); list_del_rcu(&ns->siblings); nvme_mpath_clear_current_path(ns); mutex_unlock(&ns->ctrl->subsys->lock); down_write(&ns->ctrl->namespaces_rwsem); list_del_init(&ns->list); up_write(&ns->ctrl->namespaces_rwsem); synchronize_srcu(&ns->head->srcu); nvme_mpath_check_last_path(ns); nvme_put_ns(ns); }when scanning is active during reset all namespaces are found to be unresponsive, and will be consequently removed:
static void
nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)
Quoted 12 lines
{
struct nvme_ns *ns;
ns = nvme_find_get_ns(ctrl, nsid);
if (ns) {
if (ns->disk && revalidate_disk(ns->disk))
nvme_ns_remove(ns);
nvme_put_ns(ns);
} else
nvme_alloc_ns(ctrl, nsid);
}Which is what patch #1 is for right?
Quoted 16 lines
However, as the controller is resetting it will call
nvme_stop_queues()at the start of the reset, andnvme_start_queues()once reset is complete.Both iterate the namespace list like
void
nvme_stop_queues(struct nvme_ctrl *ctrl){ struct nvme_ns *ns; down_read(&ctrl->namespaces_rwsem); list_for_each_entry(ns, &ctrl->namespaces, list) blk_mq_quiesce_queue(ns->queue); up_read(&ctrl->namespaces_rwsem); }
Quoted 4 lines
but this will only synchronize with the very last step of
nvme_ns_remove(). At that timeblk_cleanup_queue()already had been called, and consequently we crash in 'blk_mq_quiesce_queue()' (or 'blk_mq_unquiesce_queue()', depending on the timing).
I sent a patch that moves the rcu sync.
Please see:
[PATCH] nvme: fix possible io failures when removing multipathed ns
We can have an incremental patch that moves the ns removal from ctrl->namespaces also before.
Quoted 10 lines
This imbalance in
nvme_ns_remove()is one issue which Ming Lei tried to fix up with his patchset 'blk-mq: fix races related with freezing queue'. However, he dropped the final patch
"nvme: hold request queue's refcount in ns's whole lifetime" on request from hch, as he felt thatblk_cleanup_queue()was unsafe in general, and should need to include the final put for the request queue itself.
Sadly, this was precisely the patch for fixing this issue :-(
Quoted 6 lines
So instead of waiting for the big rewrite to happen I tried to approach things from the other end, namely stopping the race to happen in the first place.
And with these two patches the crashes are gone, so I thought to share it here if there were interest.
There is interest in fixing the crash, no doubt. But I just want to understand if this is the correct fix.
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 6 lines · Sagi Grimberg
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
When resetting the controller there is no point whatsoever to have a scan run in parallel;
There is also no point of trying to prevent it.
Quoted 8 lines
Quoted 4 lines
Oh, I don't mind having a scan running in parallel once reset is underway; that will be properly short-circuited anyway.
What I object to is having scanning and resetting _starting_ at the same time, as then we incur all sorts of race conditions.But your patch does not inherently protect against it, these state conditions sprinkled can easily race the reset.
The reasoning is:
- (a) I/O is started from the scan thread
- (b) reset triggers
- (b) reset abort I/O
- (a) I/O returns with an error
- (a) checks the state and skips remaining I/O
Quoted 3 lines
Note that I do agree with patch #1, but I don't understand how patch #2 is fixing the problem other than narrowing the window at best.
Only if the state is not propagated to the scan thread. My reasoning is that the thread will have a context switch (as it's doing I/O), and hence the state change will be propagated.
Quoted 12 lines
Quoted 12 lines
Like this one :-)
Quoted 9 lines
Quoted 6 lines
we cannot access the controller and
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.What is this fixing? can we get a detailed description?
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
These
ctrl->stateconditions sprayed around that are not barriered by anything do not prevent any scan to run in parallel with resets.The '
ctrl->state' conditions are required to avoid I/O to be stalled during reset.Not sure what you mean by stall, are you referring to the case that it takes a long time to reconnect?
The stall is here (to stick with the above example):
- (a) scan thread issues I/O
- (b) reset triggers
- (b) reset aborts all I/O
- (a) I/O returns with an error
- (a) scan thread issues next I/O
- (b) flushes scan thread
- (a) I/O is held by the core as the controller is in reset
- (b) flush hangs
Quoted 3 lines
The solution is to make sure we drain the inflight I/O, not half-way trying to prevent them from happening.
See above. Ths problem is that we're draining I/O only once, but the scan thread is issuing _several_ I/O, of which only _one_ will be aborted.
Quoted 8 lines
Quoted 5 lines
Problem here is that I/O will be held during reset, and resubmitted once reset is finished.
The transport drivers work around this problem to terminate any I/O prior to resetting the controller,
Its not a work around.
See above. Not a workaound, but not sufficient to let the scan thread making progress, either.
Quoted 7 lines
Quoted 4 lines
but this will only cover I/O which
had been pending _before_ reset was called.
Any I/O being started after that will still be held, and we would live-lock trying to flush the scan thread.But resets do not flush the scan thread (before you restored it).
Yes. But the scan thread will tear down namespaces when running in parallel with resets, and then we're running into a race condition.
Quoted 6 lines
Quoted 2 lines
So essentially we guard any I/O with the state check, and short circuit it if we find ourselves in reset.
As I mentioned, patch #1 is fine. Please explain what patch #2 is exactly fixing.
Yes, I'm trying to :-)
Quoted 7 lines
Quoted 3 lines
There is no need to protect the state check, as we will be aborting I/O when a state change to 'RESETTING' happens, and the update should be visible to all threads after that.
I think this assumption is incorrect. Nothing guarantees when/who will have visibility to the state change.
Which actually might be true, but we haven't seen the original issue anymore after these patches. So at the very least the race window becomes really small :-)
Quoted 17 lines
Quoted 3 lines
If you want I can separate this into two patches, one with flushing the scan function and one with the state check guards, and detail out the reasoning here.
No need to split, if its needed, it needs to be in a single patch. But again, I want to understand after patch #1 is applied, what are the symptoms and why patch #2 is solving them.
Quoted 9 lines
Quoted 6 lines
Also, do note that resets will unquiesce the admin queue right away and not when the reset completes (rdma, tcp, fc). The only exception is pci that will unquiesce as soon as the reset fails.
So can you please clarify the problem?
Sure. During failover testing NetApp had been seeing several crashes like:
[ .. ]
Quoted 17 lines
Quoted 7 lines
when scanning is active during reset all namespaces are found to be unresponsive, and will be consequently removed:
static void
nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid){
????structnvme_ns*ns;Quoted 9 lines
????ns =
nvme_find_get_ns(ctrl, nsid); ????if (ns) {
??????? if (ns->disk&&revalidate_disk(ns->disk)) ???????????nvme_ns_remove(ns);
???????nvme_put_ns(ns);
????} else
???????nvme_alloc_ns(ctrl, nsid);
}
Quoted 2 lines
Which is what patch #1 is for right?
That's what I thought initially, too, but it turned out to be not sufficient.
Quoted 22 lines
Quoted 16 lines
However, as the controller is resetting it will call
nvme_stop_queues()at the start of the reset, andnvme_start_queues()once reset is complete.Both iterate the namespace list like
void
nvme_stop_queues(struct nvme_ctrl *ctrl)
{
????structnvme_ns*ns;????down_read(&ctrl->namespaces_rwsem); ????list_for_each_entry(ns,
&ctrl->namespaces, list) ???????blk_mq_quiesce_queue(ns->queue); ????up_read(&ctrl->namespaces_rwsem);
}Quoted 5 lines
but this will only synchronize with the very last step of
nvme_ns_remove().
At that timeblk_cleanup_queue()already had been called, and consequently we crash in 'blk_mq_quiesce_queue()' (or 'blk_mq_unquiesce_queue()', depending on the timing).
Quoted 7 lines
I sent a patch that moves the rcu sync.
Please see:
[PATCH] nvme: fix possible io failures when removing multipathed nsWe can have an incremental patch that moves the ns removal from
ctrl->namespacesalso before.
Again, I'm not sure if that's sufficient.
Quoted 14 lines
Quoted 14 lines
This imbalance in
nvme_ns_remove()is one issue which Ming Lei tried to fix up with his patchset 'blk-mq: fix races related with freezing queue'. However, he dropped the final patch
"nvme: hold request queue's refcount in ns's whole lifetime" on request from hch, as he felt thatblk_cleanup_queue()was unsafe in general, and should need to include the final put for the request queue itself.
Sadly, this was precisely the patch for fixing this issue :-(So instead of waiting for the big rewrite to happen I tried to approach things from the other end, namely stopping the race to happen in the first place.
Quoted 5 lines
Quoted 2 lines
And with these two patches the crashes are gone, so I thought to share it here if there were interest.
There is interest in fixing the crash, no doubt. But I just want to understand if this is the correct fix.
Ultimately, it's the imbalance in nvme_ns_remove() which is causing this crash.nvme_ns_remove() will call blk_cleanup_queue() _before_ removing the namespace from the list.
which means that any call to nvme_stop_queues() or nvme_start_queues() happening between blk_cleanup_queue() and list_del() will be hitting this issue.
And without an explicit flush of the scan thread we cannot avoid this. Leaving us with the choice of either rebalance nvme_ns_remove() (which is what the patchset from Ming Lei tried to do) or we flush the scan thread, which is what I have been doing.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
When resetting the controller there is no point whatsoever to have a scan run in parallel;
There is also no point of trying to prevent it.
Quoted 8 lines
Quoted 4 lines
Oh, I don't mind having a scan running in parallel once reset is underway; that will be properly short-circuited anyway.
What I object to is having scanning and resetting _starting_ at the same time, as then we incur all sorts of race conditions.But your patch does not inherently protect against it, these state conditions sprinkled can easily race the reset.
The reasoning is:
- (a) I/O is started from the scan thread
- (b) reset triggers
- (b) reset abort I/O
- (a) I/O returns with an error
- (a) checks the state and skips remaining I/O
Quoted 6 lines
Quoted 3 lines
Note that I do agree with patch #1, but I don't understand how patch #2 is fixing the problem other than narrowing the window at best.
Only if the state is not propagated to the scan thread. My reasoning is that the thread will have a context switch (as it's doing I/O), and hence the state change will be propagated.
Still not guaranteed.
Quoted 10 lines
Quoted 3 lines
Not sure what you mean by stall, are you referring to the case that it takes a long time to reconnect?
The stall is here (to stick with the above example):
- (a) scan thread issues I/O
- (b) reset triggers
- (b) reset aborts all I/O
- (a) I/O returns with an error
- (a) scan thread issues next I/O
- (b) flushes scan thread
But resets do not flush the scan thread (before your patch). That is what I'm failing to understand. the scan work is only flushed in nvme_remove_namespaces().
Quoted 8 lines
- (a) I/O is held by the core as the controller is in reset
- (b) flush hangsQuoted 3 lines
The solution is to make sure we drain the inflight I/O, not half-way trying to prevent them from happening.
See above. Ths problem is that we're draining I/O only once, but the scan thread is issuing _several_ I/O, of which only _one_ will be aborted.
But with your patch #1 it will bail if it failed and the state was propagated.
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Problem here is that I/O will be held during reset, and resubmitted once reset is finished.
The transport drivers work around this problem to terminate any I/O prior to resetting the controller,
Its not a work around.
See above. Not a workaound, but not sufficient to let the scan thread making progress, either.
Quoted 5 lines
Quoted 4 lines
but this will only cover I/O which
had been pending _before_ reset was called.
Any I/O being started after that will still be held, and we would live-lock trying to flush the scan thread.
Quoted 4 lines
Quoted 2 lines
But resets do not flush the scan thread (before you restored it).
Yes. But the scan thread will tear down namespaces when running in parallel with resets, and then we're running into a race condition.
But where is the "hang" you are referring to which flushes the scan thread?
Quoted 2 lines
Quoted 2 lines
But again, I want to understand after patch #1 is applied, what are the symptoms and why patch #2 is solving them.
Can we take a step back here? I think we are mixing two different issues here. in your trace you refer to a use-after-free panic when this patch is trying to solve a "hang" condition.
Can we understand what is the phenomenon after patch #1 is applied?
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
static void
nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid){
????structnvme_ns*ns;????ns =
nvme_find_get_ns(ctrl, nsid); ????if (ns) {
??????? if (ns->disk&&revalidate_disk(ns->disk)) ???????????nvme_ns_remove(ns);
???????nvme_put_ns(ns);
????} else
???????nvme_alloc_ns(ctrl, nsid);
}
Quoted 4 lines
Quoted 2 lines
Which is what patch #1 is for right?
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
Quoted 4 lines
Quoted 3 lines
We can have an incremental patch that moves the ns removal from
ctrl->namespacesalso before.Again, I'm not sure if that's sufficient.
[...]
Quoted 4 lines
Ultimately, it's the imbalance in
nvme_ns_remove()which is causing this crash.nvme_ns_remove()will callblk_cleanup_queue()_before_ removing the namespace from the list.
This is what I suggest as an incremental change to the patch I referenced. blk_cleanup_queue() is called after the ns is removed from the list.
Quoted 3 lines
which means that any call to
nvme_stop_queues()ornvme_start_queues()happening betweenblk_cleanup_queue()andlist_del()will be hitting this issue.
This is the use-after-free again right?
Quoted 4 lines
And without an explicit flush of the scan thread we cannot avoid this. Leaving us with the choice of either rebalance
nvme_ns_remove()(which is what the patchset from Ming Lei tried to do) or we flush the scan thread, which is what I have been doing.
You may be right, I just need a little help getting there as so far I'm not convinced that there is no other way. I'm also not 100% clear on what is the phenomenon after patch #1 is applied, is it a crash or a hang?
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Ping?
Hannes did you get to look into my questions?
Quoted 145 lines · Sagi Grimberg
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
When resetting the controller there is no point whatsoever to have a scan run in parallel;
There is also no point of trying to prevent it.
Quoted 9 lines
Quoted 5 lines
Oh, I don't mind having a scan running in parallel once reset is underway; that will be properly short-circuited anyway. What I object to is having scanning and resetting _starting_ at the same
time, as then we incur all sorts of race conditions.But your patch does not inherently protect against it, these state conditions sprinkled can easily race the reset.
The reasoning is:
- (a) I/O is started from the scan thread
- (b) reset triggers
- (b) reset abort I/O
- (a) I/O returns with an error
- (a) checks the state and skips remaining I/OQuoted 6 lines
Quoted 3 lines
Note that I do agree with patch #1, but I don't understand how patch #2 is fixing the problem other than narrowing the window at best.
Only if the state is not propagated to the scan thread. My reasoning is that the thread will have a context switch (as it's doing I/O), and hence the state change will be propagated.
Still not guaranteed.
Quoted 10 lines
Quoted 3 lines
Not sure what you mean by stall, are you referring to the case that it takes a long time to reconnect?
The stall is here (to stick with the above example):
- (a) scan thread issues I/O
- (b) reset triggers
- (b) reset aborts all I/O
- (a) I/O returns with an error
- (a) scan thread issues next I/O
- (b) flushes scan threadBut resets do not flush the scan thread (before your patch). That is what I'm failing to understand. the scan work is only flushed in
nvme_remove_namespaces().Quoted 9 lines
- (a) I/O is held by the core as the controller is in reset
- (b) flush hangsQuoted 3 lines
The solution is to make sure we drain the inflight I/O, not half-way trying to prevent them from happening.
See above. Ths problem is that we're draining I/O only once, but the scan thread is issuing _several_ I/O, of which only _one_ will be aborted.
But with your patch #1 it will bail if it failed and the state was propagated.
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Problem here is that I/O will be held during reset, and resubmitted once reset is finished.
The transport drivers work around this problem to terminate any I/O prior to resetting the controller,
Its not a work around.
See above. Not a workaound, but not sufficient to let the scan thread making progress, either.
Quoted 5 lines
Quoted 4 lines
but this will only cover I/O which
had been pending _before_ reset was called.
Any I/O being started after that will still be held, and we would live-lock trying to flush the scan thread.Quoted 4 lines
Quoted 2 lines
But resets do not flush the scan thread (before you restored it).
Yes. But the scan thread will tear down namespaces when running in parallel with resets, and then we're running into a race condition.
But where is the "hang" you are referring to which flushes the scan thread?
Quoted 2 lines
Quoted 2 lines
But again, I want to understand after patch #1 is applied, what are the symptoms and why patch #2 is solving them.
Can we take a step back here? I think we are mixing two different issues here. in your trace you refer to a use-after-free panic when this patch is trying to solve a "hang" condition.
Can we understand what is the phenomenon after patch #1 is applied?
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
static void
nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid){
????structnvme_ns*ns;????ns =
nvme_find_get_ns(ctrl, nsid); ????if (ns) {
??????? if (ns->disk&&revalidate_disk(ns->disk)) ???????????nvme_ns_remove(ns);
???????nvme_put_ns(ns);
????} else
???????nvme_alloc_ns(ctrl, nsid);
}Quoted 4 lines
Quoted 2 lines
Which is what patch #1 is for right?
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
Quoted 4 lines
Quoted 3 lines
We can have an incremental patch that moves the ns removal from
ctrl->namespacesalso before.Again, I'm not sure if that's sufficient.
[...]
Quoted 4 lines
Ultimately, it's the imbalance in
nvme_ns_remove()which is causing this crash.nvme_ns_remove()will callblk_cleanup_queue()_before_ removing the namespace from the list.This is what I suggest as an incremental change to the patch I referenced.
blk_cleanup_queue()is called after the ns is removed from the list.Quoted 3 lines
which means that any call to
nvme_stop_queues()ornvme_start_queues()happening betweenblk_cleanup_queue()andlist_del()will be hitting this issue.This is the use-after-free again right?
Quoted 4 lines
And without an explicit flush of the scan thread we cannot avoid this. Leaving us with the choice of either rebalance
nvme_ns_remove()(which is what the patchset from Ming Lei tried to do) or we flush the scan thread, which is what I have been doing.You may be right, I just need a little help getting there as so far I'm not convinced that there is no other way. I'm also not 100% clear on what is the phenomenon after patch #1 is applied, is it a crash or a hang?
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 4 lines · Sagi Grimberg
Ping?
Hannes did you get to look into my questions?
Yeah, I did, but bugzilla was down on Friday so I couldn't lookup details. More to follow :-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
PATCH 2/2 nvme: flush scan_work when resetting controller
In reply to Sagi GrimbergQuoted 119 lines · Sagi Grimberg
Ping?
Hannes did you get to look into my questions?
Quoted 114 lines · Sagi Grimberg
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
When resetting the controller there is no point whatsoever to have a scan run in parallel;
There is also no point of trying to prevent it.
Quoted 9 lines
Quoted 5 lines
Oh, I don't mind having a scan running in parallel once reset is underway; that will be properly short-circuited anyway. What I object to is having scanning and resetting _starting_ at the same
time, as then we incur all sorts of race conditions.But your patch does not inherently protect against it, these state conditions sprinkled can easily race the reset.
The reasoning is:
- (a) I/O is started from the scan thread
- (b) reset triggers
- (b) reset abort I/O
- (a) I/O returns with an error
- (a) checks the state and skips remaining I/OQuoted 6 lines
Quoted 3 lines
Note that I do agree with patch #1, but I don't understand how patch #2 is fixing the problem other than narrowing the window at best.
Only if the state is not propagated to the scan thread. My reasoning is that the thread will have a context switch (as it's doing I/O), and hence the state change will be propagated.
Still not guaranteed.
Quoted 10 lines
Quoted 3 lines
Not sure what you mean by stall, are you referring to the case that it takes a long time to reconnect?
The stall is here (to stick with the above example):
- (a) scan thread issues I/O
- (b) reset triggers
- (b) reset aborts all I/O
- (a) I/O returns with an error
- (a) scan thread issues next I/O
- (b) flushes scan threadBut resets do not flush the scan thread (before your patch). That is what I'm failing to understand. the scan work is only flushed in
nvme_remove_namespaces().Quoted 9 lines
- (a) I/O is held by the core as the controller is in reset
- (b) flush hangsQuoted 3 lines
The solution is to make sure we drain the inflight I/O, not half-way trying to prevent them from happening.
See above. Ths problem is that we're draining I/O only once, but the scan thread is issuing _several_ I/O, of which only _one_ will be aborted.
But with your patch #1 it will bail if it failed and the state was propagated.
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Problem here is that I/O will be held during reset, and resubmitted once reset is finished.
The transport drivers work around this problem to terminate any I/O prior to resetting the controller,
Its not a work around.
See above. Not a workaound, but not sufficient to let the scan thread making progress, either.
Quoted 5 lines
Quoted 4 lines
but this will only cover I/O which
had been pending _before_ reset was called.
Any I/O being started after that will still be held, and we would live-lock trying to flush the scan thread.Quoted 4 lines
Quoted 2 lines
But resets do not flush the scan thread (before you restored it).
Yes. But the scan thread will tear down namespaces when running in parallel with resets, and then we're running into a race condition.
But where is the "hang" you are referring to which flushes the scan thread?
Quoted 2 lines
Quoted 2 lines
But again, I want to understand after patch #1 is applied, what are the symptoms and why patch #2 is solving them.
Can we take a step back here? I think we are mixing two different issues here. in your trace you refer to a use-after-free panic when this patch is trying to solve a "hang" condition.
Can we understand what is the phenomenon after patch #1 is applied?
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
static void
nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid){
????structnvme_ns*ns;????ns =
nvme_find_get_ns(ctrl, nsid); ????if (ns) {
??????? if (ns->disk&&revalidate_disk(ns->disk)) ???????????nvme_ns_remove(ns);
???????nvme_put_ns(ns);
????} else
???????nvme_alloc_ns(ctrl, nsid);
}Quoted 4 lines
Quoted 2 lines
Which is what patch #1 is for right?
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
It hangs, and we're seeing a warning:
kernel: [67088.344034] WARNING: CPU: 4 PID: 25020 at ../lib/percpu-refcount.c:334 percpu_ref_kill_and_confirm+0x7a/0xa0 [ .. ]
kernel: [67088.344106] Call Trace:
kernel: [67088.344112] blk_freeze_queue_start+0x2a/0x40
kernel: [67088.344114] blk_freeze_queue+0xe/0x40
kernel: [67088.344118] nvme_update_disk_info+0x36/0x260 [nvme_core]
kernel: [67088.344122] __nvme_revalidate_disk+0xca/0xf0 [nvme_core]
kernel: [67088.344125] nvme_revalidate_disk+0xa6/0x120 [nvme_core]
kernel: [67088.344127] ? blk_mq_get_tag+0xa3/0x220
kernel: [67088.344130] revalidate_disk+0x23/0xc0
kernel: [67088.344133] nvme_validate_ns+0x43/0x830 [nvme_core]
kernel: [67088.344137] ? wake_up_q+0x70/0x70
kernel: [67088.344139] ? blk_mq_free_request+0x12a/0x160
kernel: [67088.344142] ? __nvme_submit_sync_cmd+0x73/0xe0 [nvme_core]
kernel: [67088.344145] nvme_scan_work+0x2b3/0x350 [nvme_core]
kernel: [67088.344149] process_one_work+0x1da/0x400
Quoted 1 line
From which I've inferred that we're still running a scan in parallel to
reset, and that the scan thread is calling 'blk_freeze_queue()' on a queue which is already torn down.
Quoted 27 lines
Quoted 27 lines
Quoted 4 lines
Quoted 3 lines
We can have an incremental patch that moves the ns removal from
ctrl->namespacesalso before.Again, I'm not sure if that's sufficient.
[...]
Quoted 4 lines
Ultimately, it's the imbalance in
nvme_ns_remove()which is causing this crash.nvme_ns_remove()will callblk_cleanup_queue()_before_ removing the namespace from the list.This is what I suggest as an incremental change to the patch I referenced.
blk_cleanup_queue()is called after the ns is removed from the list.Quoted 3 lines
which means that any call to
nvme_stop_queues()ornvme_start_queues()happening betweenblk_cleanup_queue()andlist_del()will be hitting this issue.This is the use-after-free again right?
Quoted 4 lines
And without an explicit flush of the scan thread we cannot avoid this. Leaving us with the choice of either rebalance
nvme_ns_remove()(which is what the patchset from Ming Lei tried to do) or we flush the scan thread, which is what I have been doing.
Quoted 4 lines
Quoted 4 lines
You may be right, I just need a little help getting there as so far I'm not convinced that there is no other way. I'm also not 100% clear on what is the phenomenon after patch #1 is applied, is it a crash or a hang?
See above. The system hangs with a warning.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
It hangs, and we're seeing a warning:
kernel: [67088.344034] WARNING: CPU: 4 PID: 25020 at
../lib/percpu-refcount.c:334percpu_ref_kill_and_confirm+0x7a/0xa0[ .. ]
kernel: [67088.344106] Call Trace:
kernel: [67088.344112]blk_freeze_queue_start+0x2a/0x40
kernel: [67088.344114]blk_freeze_queue+0xe/0x40
kernel: [67088.344118]nvme_update_disk_info+0x36/0x260[nvme_core]
kernel: [67088.344122]__nvme_revalidate_disk+0xca/0xf0[nvme_core]
kernel: [67088.344125]nvme_revalidate_disk+0xa6/0x120[nvme_core]
kernel: [67088.344127] ?blk_mq_get_tag+0xa3/0x220
kernel: [67088.344130]revalidate_disk+0x23/0xc0
kernel: [67088.344133]nvme_validate_ns+0x43/0x830[nvme_core]
kernel: [67088.344137] ?wake_up_q+0x70/0x70
kernel: [67088.344139] ?blk_mq_free_request+0x12a/0x160
kernel: [67088.344142] ?__nvme_submit_sync_cmd+0x73/0xe0[nvme_core]
kernel: [67088.344145]nvme_scan_work+0x2b3/0x350[nvme_core]
kernel: [67088.344149]process_one_work+0x1da/0x400
Quoted 3 lines
From which I've inferred that we're still running a scan in parallel to
reset, and that the scan thread is calling 'blk_freeze_queue()' on a queue which is already torn down.
Where is the scan triggered from? there is no scan call from the reset path.
Is there a namespace removal or something else that triggers AEN to make this happen?
What exactly is the scenario?
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 31 lines · Sagi Grimberg
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
It hangs, and we're seeing a warning:
kernel: [67088.344034] WARNING: CPU: 4 PID: 25020 at
../lib/percpu-refcount.c:334percpu_ref_kill_and_confirm+0x7a/0xa0[ .. ]
kernel: [67088.344106] Call Trace:
kernel: [67088.344112]?blk_freeze_queue_start+0x2a/0x40
kernel: [67088.344114]?blk_freeze_queue+0xe/0x40
kernel: [67088.344118]?nvme_update_disk_info+0x36/0x260[nvme_core]
kernel: [67088.344122]?__nvme_revalidate_disk+0xca/0xf0[nvme_core]
kernel: [67088.344125]?nvme_revalidate_disk+0xa6/0x120[nvme_core]
kernel: [67088.344127]? ?blk_mq_get_tag+0xa3/0x220
kernel: [67088.344130]?revalidate_disk+0x23/0xc0
kernel: [67088.344133]?nvme_validate_ns+0x43/0x830[nvme_core]
kernel: [67088.344137]? ?wake_up_q+0x70/0x70
kernel: [67088.344139]? ?blk_mq_free_request+0x12a/0x160
kernel: [67088.344142]? ?__nvme_submit_sync_cmd+0x73/0xe0[nvme_core]
kernel: [67088.344145]?nvme_scan_work+0x2b3/0x350[nvme_core]
kernel: [67088.344149]?process_one_work+0x1da/0x400Quoted 3 lines
?From which I've inferred that we're still running a scan in parallel to reset, and that the scan thread is calling '
blk_freeze_queue()' on a queue which is already torn down.
Quoted 7 lines
Where is the scan triggered from? there is no scan call from the reset path.
Is there a namespace removal or something else that triggers AEN to make this happen?
What exactly is the scenario?
I believe this is from quick link bounce or logout/login testing on FC.
The loss of connectivity to the subsystem causes a reset to be done on the controller. Re-login causes the transport to send a discovery scan -or- the reconnect timer will fire, causing the controller to reconnect and issue the scan.
These resets and reconnects are right on the heels of each other - thus scan may not be finished before the reset occurs.? Reset will complete, as the ios it knows about are cleaned up.
-- james
PATCH 2/2 nvme: flush scan_work when resetting controller
In reply to Sagi GrimbergQuoted 26 lines · Sagi Grimberg
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
It hangs, and we're seeing a warning:
kernel: [67088.344034] WARNING: CPU: 4 PID: 25020 at
../lib/percpu-refcount.c:334percpu_ref_kill_and_confirm+0x7a/0xa0[ .. ]
kernel: [67088.344106] Call Trace:
kernel: [67088.344112]?blk_freeze_queue_start+0x2a/0x40
kernel: [67088.344114]?blk_freeze_queue+0xe/0x40
kernel: [67088.344118]?nvme_update_disk_info+0x36/0x260[nvme_core]
kernel: [67088.344122]?__nvme_revalidate_disk+0xca/0xf0[nvme_core]
kernel: [67088.344125]?nvme_revalidate_disk+0xa6/0x120[nvme_core]
kernel: [67088.344127]? ?blk_mq_get_tag+0xa3/0x220
kernel: [67088.344130]?revalidate_disk+0x23/0xc0
kernel: [67088.344133]?nvme_validate_ns+0x43/0x830[nvme_core]
kernel: [67088.344137]? ?wake_up_q+0x70/0x70
kernel: [67088.344139]? ?blk_mq_free_request+0x12a/0x160
kernel: [67088.344142]? ?__nvme_submit_sync_cmd+0x73/0xe0[nvme_core]
kernel: [67088.344145]?nvme_scan_work+0x2b3/0x350[nvme_core]
kernel: [67088.344149]?process_one_work+0x1da/0x400
Quoted 8 lines
Quoted 3 lines
?From which I've inferred that we're still running a scan in parallel to reset, and that the scan thread is calling '
blk_freeze_queue()' on a queue which is already torn down.Where is the scan triggered from? there is no scan call from the reset path.
It's triggered from AEN, being received around the same time when reset triggers.
There's actually a change that the AEN handling itself triggered the reset, but I haven't be able to decipher that from the crash dump.
Quoted 4 lines
Is there a namespace removal or something else that triggers AEN to make this happen?
What exactly is the scenario?
The scenario is multiple storage failover on NetApp OnTAP while I/O is running.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 39 lines
Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
It hangs, and we're seeing a warning:
kernel: [67088.344034] WARNING: CPU: 4 PID: 25020 at
../lib/percpu-refcount.c:334percpu_ref_kill_and_confirm+0x7a/0xa0[ .. ]
kernel: [67088.344106] Call Trace:
kernel: [67088.344112]?blk_freeze_queue_start+0x2a/0x40
kernel: [67088.344114]?blk_freeze_queue+0xe/0x40
kernel: [67088.344118]?nvme_update_disk_info+0x36/0x260[nvme_core]
kernel: [67088.344122]?__nvme_revalidate_disk+0xca/0xf0[nvme_core]
kernel: [67088.344125]?nvme_revalidate_disk+0xa6/0x120[nvme_core]
kernel: [67088.344127]? ?blk_mq_get_tag+0xa3/0x220
kernel: [67088.344130]?revalidate_disk+0x23/0xc0
kernel: [67088.344133]?nvme_validate_ns+0x43/0x830[nvme_core]
kernel: [67088.344137]? ?wake_up_q+0x70/0x70
kernel: [67088.344139]? ?blk_mq_free_request+0x12a/0x160
kernel: [67088.344142]? ?__nvme_submit_sync_cmd+0x73/0xe0[nvme_core]
kernel: [67088.344145]?nvme_scan_work+0x2b3/0x350[nvme_core]
kernel: [67088.344149]?process_one_work+0x1da/0x400Quoted 8 lines
Quoted 3 lines
?From which I've inferred that we're still running a scan in parallel to
reset, and that the scan thread is calling 'blk_freeze_queue()' on a queue which is already torn down.Where is the scan triggered from? there is no scan call from the reset path.
It's triggered from AEN, being received around the same time when reset triggers.
There's actually a change that the AEN handling itself triggered the reset, but I haven't be able to decipher that from the crash dump.
Quoted 7 lines
Quoted 4 lines
Is there a namespace removal or something else that triggers AEN to make this happen?
What exactly is the scenario?
The scenario is multiple storage failover on NetApp OnTAP while I/O is running.
Hannes,
I'm still not convinced that the transports need to flush the scan work on resets.
Does the below help as an alternative:
--
drivers/nvme/host/core.c +0 -0
@@ -1665,6 +1665,10 @@ static void __nvme_revalidate_disk(struct gendisk
*disk, struct nvme_id_ns *id)
{
struct nvme_ns *ns = disk->private_data;+ /* if ns is removing we cannot mangle with the request queue */
+ if (test_bit(NVME_NS_REMOVING, &ns->flags))
+ return;
+
/*
* If identify namespace failed, use default 512 byte block size so
* block layer can use before failing read/write for 0 capacity.
--
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 70 lines · Sagi Grimberg
Quoted 40 lines
Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
That's what I thought initially, too, but it turned out to be not sufficient.
Not sufficient because it hangs? or panics?
It hangs, and we're seeing a warning:
kernel: [67088.344034] WARNING: CPU: 4 PID: 25020 at
../lib/percpu-refcount.c:334percpu_ref_kill_and_confirm+0x7a/0xa0[ .. ]
kernel: [67088.344106] Call Trace:
kernel: [67088.344112]?blk_freeze_queue_start+0x2a/0x40
kernel: [67088.344114]?blk_freeze_queue+0xe/0x40
kernel: [67088.344118]?nvme_update_disk_info+0x36/0x260[nvme_core]
kernel: [67088.344122]?__nvme_revalidate_disk+0xca/0xf0[nvme_core]
kernel: [67088.344125]?nvme_revalidate_disk+0xa6/0x120[nvme_core]
kernel: [67088.344127]? ?blk_mq_get_tag+0xa3/0x220
kernel: [67088.344130]?revalidate_disk+0x23/0xc0
kernel: [67088.344133]?nvme_validate_ns+0x43/0x830[nvme_core]
kernel: [67088.344137]? ?wake_up_q+0x70/0x70
kernel: [67088.344139]? ?blk_mq_free_request+0x12a/0x160
kernel: [67088.344142]? ?__nvme_submit_sync_cmd+0x73/0xe0[nvme_core]
kernel: [67088.344145]?nvme_scan_work+0x2b3/0x350[nvme_core]
kernel: [67088.344149]?process_one_work+0x1da/0x400Quoted 9 lines
Quoted 4 lines
??From which I've inferred that we're still running a scan in parallel to
reset, and that the scan thread is calling 'blk_freeze_queue()' on a queue which is already torn down.Where is the scan triggered from? there is no scan call from the reset path.
It's triggered from AEN, being received around the same time when reset triggers.
There's actually a change that the AEN handling itself triggered the reset, but I haven't be able to decipher that from the crash dump.Quoted 7 lines
Quoted 4 lines
Is there a namespace removal or something else that triggers AEN to make this happen?
What exactly is the scenario?
The scenario is multiple storage failover on NetApp OnTAP while I/O is running.
Hannes,
I'm still not convinced that the transports need to flush the scan work on resets.
Does the below help as an alternative:
drivers/nvme/host/core.c +4
-0@@ -1665,6 +1665,10 @@ static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id) ?{ ??????? struct nvme_ns *ns = disk->private_data; +?????? /* if ns is removing we cannot mangle with the request queue */ +?????? if (test_bit(NVME_NS_REMOVING, &ns->flags)) +?????????????? return; + ??????? /*???????? * If identify namespace failed, use default 512 byte block
Quoted 3 lines
size so
???????? * block layer can use before failing read/write for 0 capacity.
Hmm. Not sure if that is sufficient, though.
We do issue several I/O in the course of nvme_revalidate_disk(), and as we're running fully asynchronous here the disk might be removed at any given time. Cf the above stack trace; we're getting an error for the _second_ I/O (nvme_identify_ns is issuing I/O, too, but went through without errors).
So if we were to go that route we'd need to protect each and every I/O with that test.
Which then would be similar to my patch, only the 'resetting' state check replaced by the 'removing' check.
Cheers,
Hanne
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 7 lines
Hmm. Not sure if that is sufficient, though.
We do issue several I/O in the course ofnvme_revalidate_disk(), and as we're running fully asynchronous here the disk might be removed at any given time. Cf the above stack trace; we're getting an error for the _second_ I/O (nvme_identify_nsis issuing I/O, too, but went through without errors).
Its not protecting against the I/O, but rather against queue freeze.
Quoted 2 lines
So if we were to go that route we'd need to protect each and every I/O with that test.
Why?
Sagi Grimberg
maintainerPATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 8 lines
Quoted 7 lines
Hmm. Not sure if that is sufficient, though.
We do issue several I/O in the course ofnvme_revalidate_disk(), and as we're running fully asynchronous here the disk might be removed at any given time. Cf the above stack trace; we're getting an error for the _second_ I/O (nvme_identify_nsis issuing I/O, too, but went through without errors).
Quoted 6 lines
Its not protecting against the I/O, but rather against queue freeze.
Quoted 2 lines
So if we were to go that route we'd need to protect each and every I/O with that test.
Why?
Hannes?
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 8 lines · Sagi Grimberg
Quoted 8 lines
Quoted 7 lines
Hmm. Not sure if that is sufficient, though.
We do issue several I/O in the course ofnvme_revalidate_disk(), and as we're running fully asynchronous here the disk might be removed at any given time. Cf the above stack trace; we're getting an error for the _second_ I/O (nvme_identify_nsis issuing I/O, too, but went through without errors).
Quoted 8 lines
Quoted 6 lines
Its not protecting against the I/O, but rather against queue freeze.
Quoted 2 lines
So if we were to go that route we'd need to protect each and every I/O with that test.
Why?
Hannes?
Ah, right.
So, I had been operating under the impression that we crash due to an invalid namespace access when doing I/O with the rescan thread. Your assumption is slightly different, namely that the scan thread is crashing trying to quiesce I/O, but normal I/O would handled just fine.
I'll see to give your patch a spin, will let you know the results.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
PATCH 2/2 nvme: flush scan_work when resetting controller
In reply to Hannes ReineckeQuoted 2 lines · Hannes Reinecke
When resetting the controller there is no point whatsoever to have a scan run in parallel; we cannot access the controller and
scan won't be run in parallel, because .scan_work is embedded in 'struct nvme_ctrl' which is per-HBA.
Quoted 5 lines
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.
This way may cause dead-lock.
1) nvme_revalidate_disk() might freeze queue in flush context, however any in-flight requests won't be completed until reset is done, so deadlock may be caused by flushing scans in reset context.
2) sync IO may be involved in revalidate_disk() which is called in scan context, so deadlock is caused for same reason with 1).
Thanks,
Ming
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 7 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
When resetting the controller there is no point whatsoever to have a scan run in parallel; we cannot access the controller and
scan won't be run in parallel, because .scan_work is embedded in 'struct
nvme_ctrl' which is per-HBA.
Wrong. We do.
Not sure why having it embedded in the controller structure might prevent this from happening; both reset and scan are embedded, but running on different queues:
void nvme_queue_scan(struct nvme_ctrl *ctrl)
{
/*
* Only new queue scan work when admin and IO queues are both alive
*/
if (ctrl->state == NVME_CTRL_LIVE)
queue_work(nvme_wq, &ctrl->scan_work);
}EXPORT_SYMBOL_GPL(nvme_queue_scan);
int nvme_reset_ctrl(struct nvme_ctrl *ctrl)
{
if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
return -EBUSY;
if (!queue_work(nvme_reset_wq, &ctrl->reset_work))
return -EBUSY;
return 0;
}EXPORT_SYMBOL_GPL(nvme_reset_ctrl);
So there's nothing stopping them to run in parallel.
Quoted 6 lines
Quoted 5 lines
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.
Quoted 6 lines
This way may cause dead-lock.
1)
nvme_revalidate_disk()might freeze queue in flush context, however any in-flight requests won't be completed until reset is done, so deadlock may be caused by flushing scans in reset context.
Which is why I'm checking the controller state; I've observed the deadlock plenty of times before introducing the controller state check.
Quoted 3 lines
2) sync IO may be involved in
revalidate_disk()which is called in scan context, so deadlock is caused for same reason with 1).
I/O during revalidate_disk() is protected by the state check, too, so we won't be issuing any I/O during resetting.
To be precise: any I/O in flight when reset is triggered will be terminated, and any subsequent I/O is short-circuited by the state check.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 5 lines · Hannes Reinecke
Quoted 4 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
When resetting the controller there is no point whatsoever to have a scan run in parallel; we cannot access the controller and
Quoted 7 lines
Quoted 3 lines
scan won't be run in parallel, because .scan_work is embedded in 'struct
nvme_ctrl' which is per-HBA.Wrong. We do.
Not sure why having it embedded in the controller structure might prevent this from happening; both reset and scan are embedded, but running on different queues:
I mean the scan_work function itself is run exclusively, but yes, it can be run when resetting is in-progress.
Quoted 29 lines
void
nvme_queue_scan(struct nvme_ctrl *ctrl){ /* * Only new queue scan work when admin and IO queues are both alive */ if (ctrl->state == NVME_CTRL_LIVE) queue_work(nvme_wq, &ctrl->scan_work); }
EXPORT_SYMBOL_GPL(nvme_queue_scan);int
nvme_reset_ctrl(struct nvme_ctrl *ctrl){ if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) return -EBUSY; if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) return -EBUSY; return 0; }
EXPORT_SYMBOL_GPL(nvme_reset_ctrl);So there's nothing stopping them to run in parallel.
Quoted 6 lines
Quoted 5 lines
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.
Quoted 8 lines
Quoted 6 lines
This way may cause dead-lock.
1)
nvme_revalidate_disk()might freeze queue in flush context, however any in-flight requests won't be completed until reset is done, so deadlock may be caused by flushing scans in reset context.Which is why I'm checking the controller state; I've observed the deadlock plenty of times before introducing the controller state check.
Your check can't help wrt. the deadlock, for example:
1) in scan work context:
- blk_mq_freeze_queue() is being started after passing the controller state check
2) timeout & reset is triggered in another context at the exact same time:
- all in-flight IOs won't be freed until disable controller & reset is done.
- now flush_work() in reset context can't move on, because blk_mq_freeze_queue() in scan context can't make progress.
Quoted 9 lines
Quoted 4 lines
2) sync IO may be involved in
revalidate_disk()which is called in scan context, so deadlock is caused for same reason with 1).I/O during
revalidate_disk()is protected by the state check, too, so we won't be issuing any I/O during resetting.To be precise: any I/O in flight when reset is triggered will be terminated, and any subsequent I/O is short-circuited by the state check.
No, any I/O in flight before resetting is just terminated from hardware, but still in blk-mq sw or scheduler queue, so either sync IO or queue freezing won't make progress.
Please see nvme_complete_rq(), all these IO will be retried usually.
Thanks,
Ming
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 62 lines · Ming Lei
Quoted 5 lines · Hannes Reinecke
Quoted 4 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
When resetting the controller there is no point whatsoever to have a scan run in parallel; we cannot access the controller and
Quoted 7 lines
Quoted 3 lines
scan won't be run in parallel, because .scan_work is embedded in 'struct
nvme_ctrl' which is per-HBA.Wrong. We do.
Not sure why having it embedded in the controller structure might prevent this from happening; both reset and scan are embedded, but running on different queues:I mean the
scan_workfunction itself is run exclusively, but yes, it can be run when resetting is in-progress.Quoted 29 lines
void
nvme_queue_scan(struct nvme_ctrl *ctrl){ /* * Only new queue scan work when admin and IO queues are both alive */ if (ctrl->state == NVME_CTRL_LIVE) queue_work(nvme_wq, &ctrl->scan_work); }
EXPORT_SYMBOL_GPL(nvme_queue_scan);int
nvme_reset_ctrl(struct nvme_ctrl *ctrl){ if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) return -EBUSY; if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) return -EBUSY; return 0; }
EXPORT_SYMBOL_GPL(nvme_reset_ctrl);So there's nothing stopping them to run in parallel.
Quoted 6 lines
Quoted 5 lines
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.Quoted 8 lines
Quoted 6 lines
This way may cause dead-lock.
1)
nvme_revalidate_disk()might freeze queue in flush context, however any in-flight requests won't be completed until reset is done, so deadlock may be caused by flushing scans in reset context.Which is why I'm checking the controller state; I've observed the deadlock plenty of times before introducing the controller state check.
Your check can't help wrt. the deadlock, for example:
1) in scan work context:
-
blk_mq_freeze_queue()is being started after passing the controller state check
Quoted 7 lines
2) timeout & reset is triggered in another context at the exact same time:
- all in-flight IOs won't be freed until disable controller & reset is done.
- now
flush_work()in reset context can't move on, becauseblk_mq_freeze_queue()in scan context can't make progress.
There might be a difference between RDMA and FC implementations; for FC we terminate all outstanding I/Os from the HW side, so each I/O will be returned with an aborted status.
Which for all tests I (and NetApp :-) did was enough to get 'blk_mq_freeze_queue()' unstuck and the flush_work to complete. We _did_ observed, however, that the state checks are absolutely critical to this, otherwise we indeed ended up with a stuck flush_work().
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
2) sync IO may be involved in
revalidate_disk()which is called in scan context, so deadlock is caused for same reason with 1).I/O during
revalidate_disk()is protected by the state check, too, so we won't be issuing any I/O during resetting.
Quoted 9 lines
Quoted 2 lines
To be precise: any I/O in flight when reset is triggered will be terminated, and any subsequent I/O is short-circuited by the state check.
No, any I/O in flight before resetting is just terminated from hardware, but still in blk-mq sw or scheduler queue, so either sync IO or queue freezing won't make progress.
Please see
nvme_complete_rq(), all these IO will be retried usually.
For the multipath case the request will be requeued via blk_steal_bios(), and the original request will be completed. So the queue indeed becomes free, and we can continue.
Things _might_ be different for the non-multipath case, though; I'll have to check.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
PATCH 2/2 nvme: flush scan_work when resetting controller
Quoted 15 lines
Quoted 8 lines
Your check can't help wrt. the deadlock, for example:
1) in scan work context:
-
blk_mq_freeze_queue()is being started after passing the controller state checkQuoted 7 lines
2) timeout & reset is triggered in another context at the exact same time:
- all in-flight IOs won't be freed until disable controller & reset is done.
- now
flush_work()in reset context can't move on, becauseblk_mq_freeze_queue()in scan context can't make progress.
Quoted 7 lines
There might be a difference between RDMA and FC implementations; for FC we terminate all outstanding I/Os from the HW side, so each I/O will be returned with an aborted status.
Which for all tests I (and NetApp :-) did was enough to get 'blk_mq_freeze_queue()' unstuck and theflush_workto complete. We _did_ observed, however, that the state checks are absolutely critical to this, otherwise we indeed ended up with a stuckflush_work().
RDMA and TCP does the same thing at what is basically the same point - as far as terminating all outstanding io. The difference is how they terminate.? RDMA and TCP use nvme_cancel_request() rather than a call that induces work on the link. nvme_cancel_request() is near immediate. FC will take longer for the i/o's to clear - that's the difference.
-- james
PATCH 2/2 nvme: flush scan_work when resetting controller
In reply to Hannes ReineckeQuoted 15 lines
Quoted 8 lines
Your check can't help wrt. the deadlock, for example:
1) in scan work context:
-
blk_mq_freeze_queue()is being started after passing the controller state checkQuoted 7 lines
2) timeout & reset is triggered in another context at the exact same time:
- all in-flight IOs won't be freed until disable controller & reset is done.
- now
flush_work()in reset context can't move on, becauseblk_mq_freeze_queue()in scan context can't make progress.
Quoted 7 lines
There might be a difference between RDMA and FC implementations; for FC we terminate all outstanding I/Os from the HW side, so each I/O will be returned with an aborted status.
Which for all tests I (and NetApp :-) did was enough to get 'blk_mq_freeze_queue()' unstuck and theflush_workto complete. We _did_ observed, however, that the state checks are absolutely critical to this, otherwise we indeed ended up with a stuckflush_work().
RDMA and TCP does the same thing at what is basically the same point - as far as terminating all outstanding io. The difference is how they terminate.? RDMA and TCP use nvme_cancel_request() rather than a call that induces work on the link. nvme_cancel_request() is near immediate. FC will take longer for the i/o's to clear - that's the difference.
-- james
PATCH 2/2 nvme: flush scan_work when resetting controller
In reply to Hannes ReineckeQuoted 70 lines · Hannes Reinecke
Quoted 62 lines · Ming Lei
Quoted 5 lines · Hannes Reinecke
Quoted 4 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
When resetting the controller there is no point whatsoever to have a scan run in parallel; we cannot access the controller and
Quoted 7 lines
Quoted 3 lines
scan won't be run in parallel, because .scan_work is embedded in 'struct
nvme_ctrl' which is per-HBA.Wrong. We do.
Not sure why having it embedded in the controller structure might prevent this from happening; both reset and scan are embedded, but running on different queues:I mean the
scan_workfunction itself is run exclusively, but yes, it can be run when resetting is in-progress.Quoted 29 lines
void
nvme_queue_scan(struct nvme_ctrl *ctrl){ /* * Only new queue scan work when admin and IO queues are both alive */ if (ctrl->state == NVME_CTRL_LIVE) queue_work(nvme_wq, &ctrl->scan_work); }
EXPORT_SYMBOL_GPL(nvme_queue_scan);int
nvme_reset_ctrl(struct nvme_ctrl *ctrl){ if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) return -EBUSY; if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) return -EBUSY; return 0; }
EXPORT_SYMBOL_GPL(nvme_reset_ctrl);So there's nothing stopping them to run in parallel.
Quoted 6 lines
Quoted 5 lines
we cannot tell which devices are present and which not. Additionally we'll run a scan after reset anyway.
So flush existing scans before reconnecting, ensuring to short-circuit the scan workqueue function if the controller state isn't live to avoid lockups.Quoted 8 lines
Quoted 6 lines
This way may cause dead-lock.
1)
nvme_revalidate_disk()might freeze queue in flush context, however any in-flight requests won't be completed until reset is done, so deadlock may be caused by flushing scans in reset context.Which is why I'm checking the controller state; I've observed the deadlock plenty of times before introducing the controller state check.
Your check can't help wrt. the deadlock, for example:
1) in scan work context:
-
blk_mq_freeze_queue()is being started after passing the controller state checkQuoted 7 lines
2) timeout & reset is triggered in another context at the exact same time:
- all in-flight IOs won't be freed until disable controller & reset is done.
- now
flush_work()in reset context can't move on, becauseblk_mq_freeze_queue()in scan context can't make progress.
Quoted 7 lines
There might be a difference between RDMA and FC implementations; for FC we terminate all outstanding I/Os from the HW side, so each I/O will be returned with an aborted status.
Which for all tests I (and NetApp :-) did was enough to get 'blk_mq_freeze_queue()' unstuck and theflush_workto complete. We _did_ observed, however, that the state checks are absolutely critical to this, otherwise we indeed ended up with a stuckflush_work().
I have explained that the check can't fix the issue completely, and it is like a workround.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
2) sync IO may be involved in
revalidate_disk()which is called in scan context, so deadlock is caused for same reason with 1).I/O during
revalidate_disk()is protected by the state check, too, so we won't be issuing any I/O during resetting.To be precise: any I/O in flight when reset is triggered will be terminated, and any subsequent I/O is short-circuited by the state check.
Quoted 8 lines
Quoted 6 lines
No, any I/O in flight before resetting is just terminated from hardware, but still in blk-mq sw or scheduler queue, so either sync IO or queue freezing won't make progress.
Please see
nvme_complete_rq(), all these IO will be retried usually.For the multipath case the request will be requeued via
blk_steal_bios(), and the original request will be completed.
The sync IO during scan is actually waiting for bio completion, not request.
So there is still the issue in case of multipath.
Thanks,
Ming