Skip to content

RFC nvme-tcp: fix hung issues for deleting

brookxu.cn linux-nvme lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest4 patches 15 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4nvme: unfreeze while exit from recovery or resettingdrivers/nvme/host/tcp.c+16 -80 replies
2/4nvme: donot retry request for NVME_CTRL_DELETING_NOIOdrivers/nvme/host/core.c+4 -10 replies
3/4nvme: optimize nvme_check_ready() for NVME_CTRL_DELETING_NOIOdrivers/nvme/host/nvme.h+2 -10 replies
4/4nvme-tcp: remove admin_q quiescing from nvme_tcp_teardown_io_queuesdrivers/nvme/host/tcp.c+0 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/4] nvme-tcp: fix hung issues for deleting

From: Chunguang Xu <[email protected]>

We found that nvme_remove_namespaces() may hang in flush_work(&ctrl->scan_work) while removing ctrl. The root cause may due to the state of ctrl changed to NVME_CTRL_DELETING while removing ctrl , which intterupt nvme_tcp_error_recovery_work()/ nvme_reset_ctrl_work()/nvme_tcp_reconnect_or_remove(). At this time, ctrl is freezed and queue is quiescing . Since scan_work may continue to issue IOs to load partition table, make it blocked, and lead to nvme_tcp_error_recovery_work() hang in flush_work(&ctrl->scan_work).

After analyzation, we found that there are mainly two case:
1. Since ctrl is freeze, scan_work hang in __bio_queue_enter() while it issue new IO to load partition table.
2. Since queus is quiescing, requeue timeouted IO may hang in hctx->dispatch queue, leading scan_work waiting for IO completion.

CallTrace:
Removing nvme_ctrl

[<0>] __flush_work+0x14c/0x280
[<0>] flush_work+0x14/0x20
[<0>] nvme_remove_namespaces+0x45/0x100
[<0>] nvme_do_delete_ctrl+0x79/0xa0
[<0>] nvme_sysfs_delete+0x6b/0x80
[<0>] dev_attr_store+0x18/0x30
[<0>] sysfs_kf_write+0x3f/0x50
[<0>] kernfs_fop_write_iter+0x141/0x1d0
[<0>] vfs_write+0x25b/0x3d0
[<0>] ksys_write+0x6b/0xf0
[<0>] __x64_sys_write+0x1e/0x30
[<0>] do_syscall_64+0x5d/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x72/0xdc

Scan_work:
Stack 0

[<0>] __bio_queue_enter+0x15a/0x210
[<0>] blk_mq_submit_bio+0x260/0x5e0
[<0>] __submit_bio+0xa6/0x1a0
[<0>] submit_bio_noacct_nocheck+0x2e5/0x390
[<0>] submit_bio_noacct+0x1cd/0x560
[<0>] submit_bio+0x3b/0x60
[<0>] submit_bh_wbc+0x137/0x160
[<0>] block_read_full_folio+0x24d/0x470
[<0>] blkdev_read_folio+0x1c/0x30
[<0>] filemap_read_folio+0x44/0x2a0
[<0>] do_read_cache_folio+0x135/0x390
[<0>] read_cache_folio+0x16/0x20
[<0>] read_part_sector+0x3e/0xd0
[<0>] sgi_partition+0x35/0x1d0
[<0>] bdev_disk_changed+0x1f6/0x650
[<0>] blkdev_get_whole+0x7e/0x90
[<0>] blkdev_get_by_dev+0x19c/0x2e0
[<0>] disk_scan_partitions+0x72/0x100
[<0>] device_add_disk+0x415/0x420
[<0>] nvme_scan_ns+0x636/0xcd0
[<0>] nvme_scan_work+0x26f/0x450
[<0>] process_one_work+0x21c/0x430
[<0>] worker_thread+0x4e/0x3c0
[<0>] kthread+0xfb/0x130
[<0>] ret_from_fork+0x29/0x50

Stack 1

[<0>] filemap_read_folio+0x195/0x2a0
[<0>] do_read_cache_folio+0x135/0x390
[<0>] read_cache_folio+0x16/0x20
[<0>] read_part_sector+0x3e/0xd0
[<0>] read_lba+0xcc/0x1b0
[<0>] efi_partition+0xec/0x7f0
[<0>] bdev_disk_changed+0x1f6/0x650
[<0>] blkdev_get_whole+0x7e/0x90
[<0>] blkdev_get_by_dev+0x19c/0x2e0
[<0>] disk_scan_partitions+0x72/0x100
[<0>] device_add_disk+0x433/0x440
[<0>] nvme_scan_ns+0x636/0xcd0
[<0>] nvme_scan_work+0x26f/0x450
[<0>] process_one_work+0x21c/0x430
[<0>] worker_thread+0x4e/0x3c0
[<0>] kthread+0xfb/0x130
[<0>] ret_from_fork+0x29/0x50

Here try to fix this issue by make sure ctrl is unfreezed and queue is quiescing while exit from error recovery or reset.

Chunguang Xu (4):
nvme: unfreeze while exit from recovery or resetting
nvme: donot retry request for NVME_CTRL_DELETING_NOIO
nvme: optimize nvme_check_ready() for NVME_CTRL_DELETING_NOIO
nvme-tcp: remove admin_q quiescing from nvme_tcp_teardown_io_queues

 drivers/nvme/host/core.c |  5 ++++-
 drivers/nvme/host/nvme.h |  3 ++-
 drivers/nvme/host/tcp.c  | 25 ++++++++++++++++---------
 3 files changed, 22 insertions(+), 11 deletions(-)
Signature · 1 line
2.25.1