NVMe: Force cancel commands on hot-removal
Subsystems (2)
PATCH 1/1 NVMe: Force cancel commands on hot-removal
On a surprise removal when pciehp is in use, the port services driver will usually notify the nvme driver to remove the device before the nvme polling thread detects it is gone. If this happens, the queues are not shutdown prior to deleting namespace gendisks, so there may be IO outstanding that will never complete. An unnecessarily long timeout has to happen in order to complete the IO's with failure status. This patch fixes that by clearing the queues first when we know the device is IO incapable.
--- drivers/block/nvme-core.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/block/nvme-core.c +12 -1
@@ -2402,8 +2402,19 @@ static void nvme_ns_remove(struct nvme_ns *ns) {bool kill =nvme_io_incapable (ns->dev) && !blk_queue_dying (ns->queue); -if (kill) +if (kill) { +int i; +struct blk_mq_hw_ctx *hctx; +blk_set_queue_dying (ns->queue); +queue_for_each_hw_ctx (ns->queue, hctx, i) { +if (!hctx->tags) +continue ; +blk_mq_all_tag_busy_iter (hctx->tags, + nvme_cancel_queue_ios, + hctx->driver_data); + } + }if (ns->disk->flags &GENHD_FL_UP ) {if (blk_get_integrity (ns->disk))blk_integrity_unregister (ns->disk);
Signature · 1 line
1.7.10.4
PATCH 1/1 NVMe: Force cancel commands on hot-removal
Keith,
With the patch issue is resolved.I do not see any IO timeouts and device removal process is also very quick.However i noticed below warning while running dd write IOs.I think this warning is OK since the IOs which are generated as part of flushing the dirty filesystem cache are failed.
Thanks for the patch.
------------------------
[49317.467067] pciehp 0000:0e:09.0:pcie24: pending interrupts 0x0108 from Slot Status [49317.467077] pciehp 0000:0e:09.0:pcie24: DPC Interrupt status is set:status = 0x89 [49317.467079] pciehp 0000:0e:09.0:pcie24: DPC is triggered:status = 0x89 [49317.467085] pciehp 0000:0e:09.0:pcie24: Card not present on Slot(9) [49317.467087] pciehp 0000:0e:09.0:pcie24: slot(9): Link Down event [49317.467098] pciehp 0000:0e:09.0:pcie24: Handle DPC Event on the slot(9) [49317.467100] pciehp 0000:0e:09.0:pcie24: handle_dpc_trigger_event: allocated memory :info =0xffff88102ddc1580 [49317.467105] pciehp 0000:0e:09.0:pcie24: pciehp_unconfigure_device: domain:bus:dev = 0000:15:00 [49317.467132] nvme_ns_remove :entry kill =1 [49317.467142] nvme 0000:15:00.0: Cancelling I/O 330 QID 4 [49317.467144] blk_update_request: 22 callbacks suppressed [49317.467145] blk_update_request: I/O error, dev nvme0n1, sector 5868608 [49317.467147] Buffer I/O error on dev nvme0n1, logical block 733576, lost async page write [49317.467151] nvme 0000:15:00.0: Cancelling I/O 331 QID 4 [49317.467153] blk_update_request: I/O error, dev nvme0n1, sector 5868616 [49317.467154] Buffer I/O error on dev nvme0n1, logical block 733577, lost async page write [49317.467156] nvme 0000:15:00.0: Cancelling I/O 332 QID 4 [49317.467157] blk_update_request: I/O error, dev nvme0n1, sector 5868624 [49317.467158] Buffer I/O error on dev nvme0n1, logical block 733578, lost async page write [49317.467161] nvme 0000:15:00.0: Cancelling I/O 334 QID 4 [49317.467162] blk_update_request: I/O error, dev nvme0n1, sector 5868640 [49317.467163] Buffer I/O error on dev nvme0n1, logical block 733580, lost async page write [49317.467166] nvme 0000:15:00.0: Cancelling I/O 336 QID 4 [49317.467167] blk_update_request: I/O error, dev nvme0n1, sector 5868656 [49317.467168] Buffer I/O error on dev nvme0n1, logical block 733582, lost async page write [49317.467170] nvme 0000:15:00.0: Cancelling I/O 339 QID 4 [49317.467171] blk_update_request: I/O error, dev nvme0n1, sector 5868680 [49317.467172] Buffer I/O error on dev nvme0n1, logical block 733585, lost async page write [49317.467175] nvme 0000:15:00.0: Cancelling I/O 341 QID 4 [49317.467176] blk_update_request: I/O error, dev nvme0n1, sector 5868696 [49317.467177] Buffer I/O error on dev nvme0n1, logical block 733587, lost async page write [49317.467179] nvme 0000:15:00.0: Cancelling I/O 344 QID 4 [49317.467180] blk_update_request: I/O error, dev nvme0n1, sector 5868720 [49317.467181] Buffer I/O error on dev nvme0n1, logical block 733590, lost async page write [49317.467184] nvme 0000:15:00.0: Cancelling I/O 348 QID 4 [49317.467185] blk_update_request: I/O error, dev nvme0n1, sector 5868752 [49317.467186] Buffer I/O error on dev nvme0n1, logical block 733594, lost async page write [49317.467188] nvme 0000:15:00.0: Cancelling I/O 351 QID 4 [49317.467189] blk_update_request: I/O error, dev nvme0n1, sector 5868776 [49317.467190] Buffer I/O error on dev nvme0n1, logical block 733597, lost async page write [49317.467193] nvme 0000:15:00.0: Cancelling I/O 353 QID 4 [49317.467196] nvme 0000:15:00.0: Cancelling I/O 355 QID 4 [49317.467198] nvme 0000:15:00.0: Cancelling I/O 358 QID 4 [49317.467200] nvme 0000:15:00.0: Cancelling I/O 360 QID 4 [49317.467203] nvme 0000:15:00.0: Cancelling I/O 363 QID 4 [49317.467206] nvme 0000:15:00.0: Cancelling I/O 366 QID 4 [49317.467209] nvme 0000:15:00.0: Cancelling I/O 368 QID 4 [49317.467211] nvme 0000:15:00.0: Cancelling I/O 371 QID 4 [49317.467213] nvme 0000:15:00.0: Cancelling I/O 373 QID 4 [49317.467216] nvme 0000:15:00.0: Cancelling I/O 374 QID 4 [49317.467219] nvme 0000:15:00.0: Cancelling I/O 376 QID 4 [49317.467221] nvme 0000:15:00.0: Cancelling I/O 377 QID 4 [49317.467224] nvme 0000:15:00.0: Cancelling I/O 379 QID 4 [49317.467227] nvme 0000:15:00.0: Cancelling I/O 380 QID 4 [49317.467230] nvme 0000:15:00.0: Cancelling I/O 383 QID 4 [49317.467233] nvme 0000:15:00.0: Cancelling I/O 384 QID 4 [49317.467236] nvme 0000:15:00.0: Cancelling I/O 386 QID 4 [49317.467239] nvme 0000:15:00.0: Cancelling I/O 387 QID 4 [49317.467242] nvme 0000:15:00.0: Cancelling I/O 389 QID 4 [49317.467244] nvme 0000:15:00.0: Cancelling I/O 390 QID 4 [49317.467247] nvme 0000:15:00.0: Cancelling I/O 392 QID 4 [49317.467250] nvme 0000:15:00.0: Cancelling I/O 393 QID 4 [49317.467253] nvme 0000:15:00.0: Cancelling I/O 395 QID 4 [49317.467255] nvme 0000:15:00.0: Cancelling I/O 396 QID 4 [49317.467257] nvme 0000:15:00.0: Cancelling I/O 398 QID 4 [49317.467260] nvme 0000:15:00.0: Cancelling I/O 400 QID 4 [49317.467262] nvme 0000:15:00.0: Cancelling I/O 402 QID 4 [49317.467264] nvme 0000:15:00.0: Cancelling I/O 404 QID 4 [49317.467266] nvme 0000:15:00.0: Cancelling I/O 406 QID 4 [49317.467269] nvme 0000:15:00.0: Cancelling I/O 408 QID 4 [49317.467271] nvme 0000:15:00.0: Cancelling I/O 410 QID 4 [49317.467273] nvme 0000:15:00.0: Cancelling I/O 411 QID 4 [49317.467275] nvme 0000:15:00.0: Cancelling I/O 412 QID 4 [49317.467278] nvme 0000:15:00.0: Cancelling I/O 413 QID 4 [49317.467280] nvme 0000:15:00.0: Cancelling I/O 414 QID 4 [49317.467282] nvme 0000:15:00.0: Cancelling I/O 415 QID 4 [49317.467284] nvme 0000:15:00.0: Cancelling I/O 416 QID 4 [49317.467287] nvme 0000:15:00.0: Cancelling I/O 417 QID 4 [49317.467290] nvme 0000:15:00.0: Cancelling I/O 419 QID 4 [49317.467293] nvme 0000:15:00.0: Cancelling I/O 420 QID 4 [49317.467295] nvme 0000:15:00.0: Cancelling I/O 423 QID 4 [49317.467298] nvme 0000:15:00.0: Cancelling I/O 426 QID 4 [49317.467301] nvme 0000:15:00.0: Cancelling I/O 429 QID 4 [49317.467304] nvme 0000:15:00.0: Cancelling I/O 432 QID 4 [49317.467305] nvme 0000:15:00.0: Cancelling I/O 434 QID 4 [49317.467308] nvme 0000:15:00.0: Cancelling I/O 435 QID 4 [49317.467311] nvme 0000:15:00.0: Cancelling I/O 440 QID 4 [49317.467314] nvme 0000:15:00.0: Cancelling I/O 441 QID 4 [49317.467316] nvme 0000:15:00.0: Cancelling I/O 442 QID 4 [49317.467319] nvme 0000:15:00.0: Cancelling I/O 443 QID 4 [49317.467320] nvme 0000:15:00.0: Cancelling I/O 444 QID 4 [49317.467322] nvme 0000:15:00.0: Cancelling I/O 445 QID 4 [49317.467324] nvme 0000:15:00.0: Cancelling I/O 446 QID 4 [49317.467327] nvme 0000:15:00.0: Cancelling I/O 447 QID 4 [49317.467329] nvme 0000:15:00.0: Cancelling I/O 448 QID 4 [49317.467331] nvme 0000:15:00.0: Cancelling I/O 449 QID 4 [49317.769832] device: 'nvme0n1': device_del [49317.769974] nvme_ns_remove :gen disk removed [49317.821140] ------------[ cut here ]------------ [49317.821150] WARNING: CPU: 2 PID: 50212 at fs/block_dev.c:57 __blkdev_put+0x1c1/0x200() [49317.821152] Modules linked in: nvme(OE) fuse(E) btrfs(E) xor(E)
raid6_pq(E) hfsplus(E) vfat(E) msdos(E) fat(E) jfs(E) reiserfs(E) ext4(E) crc16(E) jbd2(E) ext3(E) jbd(E) ext2(E) mbcache(E) xt_CHECKSUM(E) ipt_MASQUERADE(E) nf_nat_masquerade_ipv4(E) tun(E) af_packet(E) xt_tcpudp(E) ip6t_rpfilter(E) ip6t_REJECT(E) nf_reject_ipv6(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_conntrack(E) sr_mod(E) cdrom(E) ebtable_nat(E) ebtable_broute(E) bridge(E) stp(E) llc(E) ebtable_filter(E) ebtables(E) ip6table_nat(E) nf_conntrack_ipv6(E) nf_defrag_ipv6(E) nf_nat_ipv6(E) ip6table_mangle(E) ip6table_raw(E) ip6table_filter(E) ip6_tables(E) iptable_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) iptable_mangle(E) iptable_raw(E) iptable_filter(E) ip_tables(E) x_tables(E) dm_mirror(E)
[49317.821186] dm_region_hash(E) dm_log(E) coretemp(E)
x86_pkg_temp_thermal(E) kvm_intel(E) kvm(E) uas(E) ipmi_devintf(E) usb_storage(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) jitterentropy_rng(E) hmac(E) drbg(E) ansi_cprng(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) iTCO_wdt(E) glue_helper(E) iTCO_vendor_support(E) ablk_helper(E) mousedev(E) evdev(E) cryptd(E) mac_hid(E) tpm_tis(E) sb_edac(E) lpc_ich(E) pcspkr(E) ioatdma(E) edac_core(E) tpm(E) ipmi_si(E) i2c_i801(E) mfd_core(E) battery(E) ipmi_msghandler(E) thermal(E) wmi(E) acpi_pad(E) nfsd(E) button(E) processor(E) ac(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) sunrpc(E) hid_generic(E) usbhid(E) hid(E) sd_mod(E) ast(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) i2c_algo_bit(E) drm_kms_helper(E) ahci(E)
[49317.821219] libahci(E) ttm(E) libata(E) ixgbe(E) ehci_pci(E) drm(E) mdio(E) ehci_hcd(E) hwmon(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) e1000e(E) dca(E) usbcore(E) ptp(E) scsi_mod(E) usb_common(E) i2c_core(E) pps_core(E) ipv6(E) [last unloaded: nvme] [49317.821231] CPU: 2 PID: 50212 Comm: dd Tainted: G W OE 4.2.0 #5 [49317.821233] Hardware name: Seagate CS6000AC/Type2 - Board Product Summit Point, BIOS SummitPoint.v02.0009 08/13/2014 [49317.821234] ffffffff817397d3 ffff88103445fdb8 ffffffff81523669 0000000000000000 [49317.821236] 0000000000000000 ffff88103445fdf8 ffffffff810534aa 0000000000000000 [49317.821238] ffff880f226f00f0 ffff880f226f0000 ffff880f226f0170 ffff880f226f0018 [49317.821240] Call Trace: [49317.821247] [<ffffffff81523669>] dump_stack+0x45/0x57 [49317.821251] [<ffffffff810534aa>] warn_slowpath_common+0x8a/0xc0 [49317.821253] [<ffffffff8105359a>] warn_slowpath_null+0x1a/0x20 [49317.821255] [<ffffffff811d7961>] __blkdev_put+0x1c1/0x200 [49317.821256] [<ffffffff811d8220>] blkdev_put+0x50/0x120 [49317.821258] [<ffffffff811d83a5>] blkdev_close+0x25/0x30 [49317.821262] [<ffffffff811a1d3c>] __fput+0xdc/0x1e0 [49317.821264] [<ffffffff811a1e8e>] ____fput+0xe/0x10 [49317.821267] [<ffffffff8106cda5>] task_work_run+0x85/0xb0 [49317.821270] [<ffffffff81003798>] do_notify_resume+0x58/0x80 [49317.821273] [<ffffffff81529b02>] int_signal+0x12/0x17 [49317.821274] ---[ end trace 100510bdcaa3ba59 ]--- [49317.823245] device: '259:0': device_unregister [49317.823248] device: '259:0': device_del [49317.823299] device: '259:0': device_create_release [49317.823303] nvme_ns_remove :exit [49317.824749] nvme 0000:15:00.0: Cancelling I/O 1 QID 0 [49317.824756] nvme :Removed the namespaces [49317.824856] device: 'nvme0': device_unregister [49317.824857] device: 'nvme0': device_del [49317.825047] device: 'nvme0': device_create_release [49317.832985] nvme :nvme dev completly removed [49317.832993] device: '0000:15:00.0': device_del [49317.833043] pciehp 0000:0e:09.0:pcie24: pciehp_unconfigure_device: domain:bus:dev = 0000:15:00 [49317.833054] pcieport 0000:0e:09.0: Clear the DPC trigger status = 0x89
Quoted 42 lines · Keith Busch
On a surprise removal when pciehp is in use, the port services driver will usually notify the nvme driver to remove the device before the nvme polling thread detects it is gone. If this happens, the queues are not shutdown prior to deleting namespace gendisks, so there may be IO outstanding that will never complete. An unnecessarily long timeout has to happen in order to complete the IO's with failure status. This patch fixes that by clearing the queues first when we know the device is IO incapable.
Reported-by Mohana Goli <mohana.goli at seagate.com>Signed-off-by Keith Busch <keith.busch at intel.com>--- drivers/block/nvme-core.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)drivers/block/nvme-core.c +12
-2@@ -2402,8 +2402,19 @@ static void nvme_ns_remove(struct nvme_ns *ns) {bool kill =nvme_io_incapable (ns->dev) && !blk_queue_dying (ns->queue); -if (kill) +if (kill) { +int i; +struct blk_mq_hw_ctx *hctx; +blk_set_queue_dying (ns->queue); +queue_for_each_hw_ctx (ns->queue, hctx, i) { +if (!hctx->tags) +continue ; +blk_mq_all_tag_busy_iter (hctx->tags, + nvme_cancel_queue_ios, + hctx->driver_data); + } + }if (ns->disk->flags &GENHD_FL_UP ) {if (blk_get_integrity (ns->disk))blk_integrity_unregister (ns->disk); --1.7.10.4
PATCH 1/1 NVMe: Force cancel commands on hot-removal
Keith,
Don't we need to take a queue spin lock while processing the IOs on each request queue like the way we are doing in nvme_clear_queue ?
Quoted 1 line
Quoted 1 line
Quoted 1 line
blk_mq_all_tag_busy_iter(hctx->tags,
nvme_cancel_queue_ios,
hctx->driver_data);Thanks & Regards,
Mohan.
Quoted 238 lines · Mohana Goli
Keith,
With the patch issue is resolved.I do not see any IO timeouts and device removal process is also very quick.However i noticed below warning while running dd write IOs.I think this warning is OK since the IOs which are generated as part of flushing the dirty filesystem cache are failed.
Thanks for the patch.
Tested-by Mohana Rao Goli <mohana.goli at seagate.com>------------------------
[49317.467067] pciehp 0000:0e:09.0:pcie24: pending interrupts 0x0108 from Slot Status [49317.467077] pciehp 0000:0e:09.0:pcie24: DPC Interrupt status is set:status = 0x89 [49317.467079] pciehp 0000:0e:09.0:pcie24: DPC is triggered:status = 0x89 [49317.467085] pciehp 0000:0e:09.0:pcie24: Card not present on Slot(9) [49317.467087] pciehp 0000:0e:09.0:pcie24: slot(9): Link Down event [49317.467098] pciehp 0000:0e:09.0:pcie24: Handle DPC Event on the slot(9) [49317.467100] pciehp 0000:0e:09.0:pcie24: handle_dpc_trigger_event: allocated memory :info =0xffff88102ddc1580 [49317.467105] pciehp 0000:0e:09.0:pcie24: pciehp_unconfigure_device: domain:bus:dev = 0000:15:00 [49317.467132] nvme_ns_remove :entry kill =1 [49317.467142] nvme 0000:15:00.0: Cancelling I/O 330 QID 4 [49317.467144] blk_update_request: 22 callbacks suppressed [49317.467145] blk_update_request: I/O error, dev nvme0n1, sector 5868608 [49317.467147] Buffer I/O error on dev nvme0n1, logical block 733576, lost async page write [49317.467151] nvme 0000:15:00.0: Cancelling I/O 331 QID 4 [49317.467153] blk_update_request: I/O error, dev nvme0n1, sector 5868616 [49317.467154] Buffer I/O error on dev nvme0n1, logical block 733577, lost async page write [49317.467156] nvme 0000:15:00.0: Cancelling I/O 332 QID 4 [49317.467157] blk_update_request: I/O error, dev nvme0n1, sector 5868624 [49317.467158] Buffer I/O error on dev nvme0n1, logical block 733578, lost async page write [49317.467161] nvme 0000:15:00.0: Cancelling I/O 334 QID 4 [49317.467162] blk_update_request: I/O error, dev nvme0n1, sector 5868640 [49317.467163] Buffer I/O error on dev nvme0n1, logical block 733580, lost async page write [49317.467166] nvme 0000:15:00.0: Cancelling I/O 336 QID 4 [49317.467167] blk_update_request: I/O error, dev nvme0n1, sector 5868656 [49317.467168] Buffer I/O error on dev nvme0n1, logical block 733582, lost async page write [49317.467170] nvme 0000:15:00.0: Cancelling I/O 339 QID 4 [49317.467171] blk_update_request: I/O error, dev nvme0n1, sector 5868680 [49317.467172] Buffer I/O error on dev nvme0n1, logical block 733585, lost async page write [49317.467175] nvme 0000:15:00.0: Cancelling I/O 341 QID 4 [49317.467176] blk_update_request: I/O error, dev nvme0n1, sector 5868696 [49317.467177] Buffer I/O error on dev nvme0n1, logical block 733587, lost async page write [49317.467179] nvme 0000:15:00.0: Cancelling I/O 344 QID 4 [49317.467180] blk_update_request: I/O error, dev nvme0n1, sector 5868720 [49317.467181] Buffer I/O error on dev nvme0n1, logical block 733590, lost async page write [49317.467184] nvme 0000:15:00.0: Cancelling I/O 348 QID 4 [49317.467185] blk_update_request: I/O error, dev nvme0n1, sector 5868752 [49317.467186] Buffer I/O error on dev nvme0n1, logical block 733594, lost async page write [49317.467188] nvme 0000:15:00.0: Cancelling I/O 351 QID 4 [49317.467189] blk_update_request: I/O error, dev nvme0n1, sector 5868776 [49317.467190] Buffer I/O error on dev nvme0n1, logical block 733597, lost async page write [49317.467193] nvme 0000:15:00.0: Cancelling I/O 353 QID 4 [49317.467196] nvme 0000:15:00.0: Cancelling I/O 355 QID 4 [49317.467198] nvme 0000:15:00.0: Cancelling I/O 358 QID 4 [49317.467200] nvme 0000:15:00.0: Cancelling I/O 360 QID 4 [49317.467203] nvme 0000:15:00.0: Cancelling I/O 363 QID 4 [49317.467206] nvme 0000:15:00.0: Cancelling I/O 366 QID 4 [49317.467209] nvme 0000:15:00.0: Cancelling I/O 368 QID 4 [49317.467211] nvme 0000:15:00.0: Cancelling I/O 371 QID 4 [49317.467213] nvme 0000:15:00.0: Cancelling I/O 373 QID 4 [49317.467216] nvme 0000:15:00.0: Cancelling I/O 374 QID 4 [49317.467219] nvme 0000:15:00.0: Cancelling I/O 376 QID 4 [49317.467221] nvme 0000:15:00.0: Cancelling I/O 377 QID 4 [49317.467224] nvme 0000:15:00.0: Cancelling I/O 379 QID 4 [49317.467227] nvme 0000:15:00.0: Cancelling I/O 380 QID 4 [49317.467230] nvme 0000:15:00.0: Cancelling I/O 383 QID 4 [49317.467233] nvme 0000:15:00.0: Cancelling I/O 384 QID 4 [49317.467236] nvme 0000:15:00.0: Cancelling I/O 386 QID 4 [49317.467239] nvme 0000:15:00.0: Cancelling I/O 387 QID 4 [49317.467242] nvme 0000:15:00.0: Cancelling I/O 389 QID 4 [49317.467244] nvme 0000:15:00.0: Cancelling I/O 390 QID 4 [49317.467247] nvme 0000:15:00.0: Cancelling I/O 392 QID 4 [49317.467250] nvme 0000:15:00.0: Cancelling I/O 393 QID 4 [49317.467253] nvme 0000:15:00.0: Cancelling I/O 395 QID 4 [49317.467255] nvme 0000:15:00.0: Cancelling I/O 396 QID 4 [49317.467257] nvme 0000:15:00.0: Cancelling I/O 398 QID 4 [49317.467260] nvme 0000:15:00.0: Cancelling I/O 400 QID 4 [49317.467262] nvme 0000:15:00.0: Cancelling I/O 402 QID 4 [49317.467264] nvme 0000:15:00.0: Cancelling I/O 404 QID 4 [49317.467266] nvme 0000:15:00.0: Cancelling I/O 406 QID 4 [49317.467269] nvme 0000:15:00.0: Cancelling I/O 408 QID 4 [49317.467271] nvme 0000:15:00.0: Cancelling I/O 410 QID 4 [49317.467273] nvme 0000:15:00.0: Cancelling I/O 411 QID 4 [49317.467275] nvme 0000:15:00.0: Cancelling I/O 412 QID 4 [49317.467278] nvme 0000:15:00.0: Cancelling I/O 413 QID 4 [49317.467280] nvme 0000:15:00.0: Cancelling I/O 414 QID 4 [49317.467282] nvme 0000:15:00.0: Cancelling I/O 415 QID 4 [49317.467284] nvme 0000:15:00.0: Cancelling I/O 416 QID 4 [49317.467287] nvme 0000:15:00.0: Cancelling I/O 417 QID 4 [49317.467290] nvme 0000:15:00.0: Cancelling I/O 419 QID 4 [49317.467293] nvme 0000:15:00.0: Cancelling I/O 420 QID 4 [49317.467295] nvme 0000:15:00.0: Cancelling I/O 423 QID 4 [49317.467298] nvme 0000:15:00.0: Cancelling I/O 426 QID 4 [49317.467301] nvme 0000:15:00.0: Cancelling I/O 429 QID 4 [49317.467304] nvme 0000:15:00.0: Cancelling I/O 432 QID 4 [49317.467305] nvme 0000:15:00.0: Cancelling I/O 434 QID 4 [49317.467308] nvme 0000:15:00.0: Cancelling I/O 435 QID 4 [49317.467311] nvme 0000:15:00.0: Cancelling I/O 440 QID 4 [49317.467314] nvme 0000:15:00.0: Cancelling I/O 441 QID 4 [49317.467316] nvme 0000:15:00.0: Cancelling I/O 442 QID 4 [49317.467319] nvme 0000:15:00.0: Cancelling I/O 443 QID 4 [49317.467320] nvme 0000:15:00.0: Cancelling I/O 444 QID 4 [49317.467322] nvme 0000:15:00.0: Cancelling I/O 445 QID 4 [49317.467324] nvme 0000:15:00.0: Cancelling I/O 446 QID 4 [49317.467327] nvme 0000:15:00.0: Cancelling I/O 447 QID 4 [49317.467329] nvme 0000:15:00.0: Cancelling I/O 448 QID 4 [49317.467331] nvme 0000:15:00.0: Cancelling I/O 449 QID 4 [49317.769832] device: 'nvme0n1': device_del [49317.769974] nvme_ns_remove :gen disk removed [49317.821140] ------------[ cut here ]------------ [49317.821150] WARNING: CPU: 2 PID: 50212 at fs/block_dev.c:57 __blkdev_put+0x1c1/0x200() [49317.821152] Modules linked in: nvme(OE) fuse(E) btrfs(E) xor(E)
raid6_pq(E)hfsplus(E) vfat(E) msdos(E) fat(E) jfs(E) reiserfs(E) ext4(E) crc16(E) jbd2(E) ext3(E) jbd(E) ext2(E) mbcache(E)xt_CHECKSUM(E)ipt_MASQUERADE(E)nf_nat_masquerade_ipv4(E)tun(E)af_packet(E)xt_tcpudp(E)ip6t_rpfilter(E)ip6t_REJECT(E)nf_reject_ipv6(E)ipt_REJECT(E)nf_reject_ipv4(E)xt_conntrack(E)sr_mod(E)cdrom(E)ebtable_nat(E)ebtable_broute(E)bridge(E) stp(E) llc(E)ebtable_filter(E)ebtables(E)ip6table_nat(E)nf_conntrack_ipv6(E)nf_defrag_ipv6(E)nf_nat_ipv6(E)ip6table_mangle(E)ip6table_raw(E)ip6table_filter(E)ip6_tables(E)iptable_nat(E)nf_conntrack_ipv4(E)nf_defrag_ipv4(E)nf_nat_ipv4(E)nf_nat(E)nf_conntrack(E)iptable_mangle(E)iptable_raw(E)iptable_filter(E)ip_tables(E)x_tables(E)dm_mirror(E)[49317.821186] dm_region_hash(E) dm_log(E) coretemp(E)
x86_pkg_temp_thermal(E)kvm_intel(E)kvm(E) uas(E)ipmi_devintf(E)usb_storage(E)crct10dif_pclmul(E)crc32_pclmul(E)crc32c_intel(E)ghash_clmulni_intel(E)jitterentropy_rng(E)hmac(E) drbg(E)ansi_cprng(E)aesni_intel(E)aes_x86_64(E)lrw(E) gf128mul(E)iTCO_wdt(E)glue_helper(E)iTCO_vendor_support(E)ablk_helper(E)mousedev(E) evdev(E) cryptd(E)mac_hid(E)tpm_tis(E)sb_edac(E)lpc_ich(E)pcspkr(E) ioatdma(E)edac_core(E)tpm(E)ipmi_si(E)i2c_i801(E)mfd_core(E)battery(E)ipmi_msghandler(E)thermal(E) wmi(E)acpi_pad(E)nfsd(E) button(E) processor(E) ac(E)auth_rpcgss(E)nfs_acl(E)lockd(E) grace(E) sunrpc(E)hid_generic(E)usbhid(E) hid(E)sd_mod(E)ast(E) syscopyarea(E) sysfillrect(E) sysimgblt(E)i2c_algo_bit(E)drm_kms_helper(E)ahci(E)[49317.821219] libahci(E) ttm(E) libata(E) ixgbe(E) ehci_pci(E) drm(E) mdio(E) ehci_hcd(E) hwmon(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) e1000e(E) dca(E) usbcore(E) ptp(E) scsi_mod(E) usb_common(E) i2c_core(E) pps_core(E) ipv6(E) [last unloaded: nvme] [49317.821231] CPU: 2 PID: 50212 Comm: dd Tainted: G W OE 4.2.0 #5 [49317.821233] Hardware name: Seagate CS6000AC/Type2 - Board Product Summit Point, BIOS SummitPoint.v02.0009 08/13/2014 [49317.821234] ffffffff817397d3 ffff88103445fdb8 ffffffff81523669 0000000000000000 [49317.821236] 0000000000000000 ffff88103445fdf8 ffffffff810534aa 0000000000000000 [49317.821238] ffff880f226f00f0 ffff880f226f0000 ffff880f226f0170 ffff880f226f0018 [49317.821240] Call Trace: [49317.821247] [<ffffffff81523669>] dump_stack+0x45/0x57 [49317.821251] [<ffffffff810534aa>] warn_slowpath_common+0x8a/0xc0 [49317.821253] [<ffffffff8105359a>] warn_slowpath_null+0x1a/0x20 [49317.821255] [<ffffffff811d7961>] __blkdev_put+0x1c1/0x200 [49317.821256] [<ffffffff811d8220>] blkdev_put+0x50/0x120 [49317.821258] [<ffffffff811d83a5>] blkdev_close+0x25/0x30 [49317.821262] [<ffffffff811a1d3c>] __fput+0xdc/0x1e0 [49317.821264] [<ffffffff811a1e8e>] ____fput+0xe/0x10 [49317.821267] [<ffffffff8106cda5>] task_work_run+0x85/0xb0 [49317.821270] [<ffffffff81003798>] do_notify_resume+0x58/0x80 [49317.821273] [<ffffffff81529b02>] int_signal+0x12/0x17 [49317.821274] ---[ end trace 100510bdcaa3ba59 ]--- [49317.823245] device: '259:0': device_unregister [49317.823248] device: '259:0': device_del [49317.823299] device: '259:0': device_create_release [49317.823303] nvme_ns_remove :exit [49317.824749] nvme 0000:15:00.0: Cancelling I/O 1 QID 0 [49317.824756] nvme :Removed the namespaces [49317.824856] device: 'nvme0': device_unregister [49317.824857] device: 'nvme0': device_del [49317.825047] device: 'nvme0': device_create_release [49317.832985] nvme :nvme dev completly removed [49317.832993] device: '0000:15:00.0': device_del [49317.833043] pciehp 0000:0e:09.0:pcie24: pciehp_unconfigure_device: domain:bus:dev = 0000:15:00 [49317.833054] pcieport 0000:0e:09.0: Clear the DPC trigger status = 0x89Quoted 42 lines · Keith Busch
On a surprise removal when pciehp is in use, the port services driver will usually notify the nvme driver to remove the device before the nvme polling thread detects it is gone. If this happens, the queues are not shutdown prior to deleting namespace gendisks, so there may be IO outstanding that will never complete. An unnecessarily long timeout has to happen in order to complete the IO's with failure status. This patch fixes that by clearing the queues first when we know the device is IO incapable.
Reported-by Mohana Goli <mohana.goli at seagate.com>Signed-off-by Keith Busch <keith.busch at intel.com>--- drivers/block/nvme-core.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)drivers/block/nvme-core.c +12
-2@@ -2402,8 +2402,19 @@ static void nvme_ns_remove(struct nvme_ns *ns) {bool kill =nvme_io_incapable (ns->dev) && !blk_queue_dying (ns->queue); -if (kill) +if (kill) { +int i; +struct blk_mq_hw_ctx *hctx; +blk_set_queue_dying (ns->queue); +queue_for_each_hw_ctx (ns->queue, hctx, i) { +if (!hctx->tags) +continue ; +blk_mq_all_tag_busy_iter (hctx->tags, + nvme_cancel_queue_ios, + hctx->driver_data); + } + }if (ns->disk->flags &GENHD_FL_UP ) {if (blk_get_integrity (ns->disk))blk_integrity_unregister (ns->disk); --1.7.10.4
PATCH 1/1 NVMe: Force cancel commands on hot-removal
Quoted 9 lines · Mohana Goli
Keith,
Don't we need to take a queue spin lock while processing the IOs on each request queue like the way we are doing in
nvme_clear_queue?Quoted 1 line
Quoted 1 line
Quoted 1 line
blk_mq_all_tag_busy_iter(hctx->tags,
nvme_cancel_queue_ios, hctx->driver_data);
You're right, thanks for the catch.
PATCH 1/1 NVMe: Force cancel commands on hot-removal
Keith,
I ran another test after creating filesystem on the nvme devices.Unfortunately i have seen the IO Timeouts again. I used the following steps:
1.Created the 2 different filesystems on 2 different drives 2.Mounted the both the devices to different mount points 3.Initiated DD write IOs on both the mount points 4.Pulled one of the drive then i could see the IO timeouts.
But issue is not seen if i use the my fix ,doing nvme_dev_shutdown first and nvme_dev_remove in nvme_remove function.And calling blk_cleanup_queue(ns->queue) before calling del_gendisk in nvme_ns_remove function.
Probably this is uncovering another corner case.
Thanks & Regards,
Mohan.
-----------------------------------------------
[ 381.133461] pciehp 0000:0e:09.0:pcie24: pending interrupts 0x0108
from Slot Status
[ 381.133472] pciehp 0000:0e:09.0:pcie24: DPC Interrupt status is
set:status = 0x89
[ 381.133475] pciehp 0000:0e:09.0:pcie24: DPC is triggered:status = 0x89
[ 381.133483] pciehp 0000:0e:09.0:pcie24: Card not present on Slot(9)
[ 381.133486] pciehp 0000:0e:09.0:pcie24: slot(9): Link Down event
[ 381.133500] pciehp 0000:0e:09.0:pcie24: Handle DPC Event on the slot(9)
[ 381.133503] pciehp 0000:0e:09.0:pcie24: handle_dpc_trigger_event:
allocated memory :info =0xffff88102e8cde00
[ 381.133508] pciehp 0000:0e:09.0:pcie24: pciehp_unconfigure_device:
domain:bus:dev = 0000:15:00
[ 381.133547] nvme_ns_remove :entry kill =1
[ 381.133567] nvme 0000:15:00.0: Cancelling I/O 17 QID 7
[ 381.133571] blk_update_request: I/O error, dev nvme0n1, sector 25651808
[ 381.133749] Buffer I/O error on dev nvme0n1, logical block 3206476,
lost async page write
[ 381.133755] Buffer I/O error on dev nvme0n1, logical block 3206477,
lost async page write
[ 381.133757] Buffer I/O error on dev nvme0n1, logical block 3206478,
lost async page write
[ 381.133758] Buffer I/O error on dev nvme0n1, logical block 3206479,
lost async page write
[ 381.133760] Buffer I/O error on dev nvme0n1, logical block 3206480,
lost async page write
[ 381.133762] Buffer I/O error on dev nvme0n1, logical block 3206481,
lost async page write
[ 381.133763] Buffer I/O error on dev nvme0n1, logical block 3206482,
lost async page write
[ 381.133765] Buffer I/O error on dev nvme0n1, logical block 3206483,
lost async page write
[ 381.133766] Buffer I/O error on dev nvme0n1, logical block 3206484,
lost async page write
[ 381.133768] Buffer I/O error on dev nvme0n1, logical block 3206485,
lost async page write
[ 411.339718] nvme 0000:15:00.0: Timeout I/O 468 QID 1
[ 411.339740] nvme 0000:15:00.0: Aborting I/O 468 QID 1
[ 428.361007] nvme 0000:15:00.0: Timeout I/O 1 QID 67
[ 428.361023] nvme 0000:15:00.0: Aborting I/O 1 QID 67
[ 428.361029] nvme 0000:15:00.0: Timeout I/O 2 QID 67
[ 428.361033] nvme 0000:15:00.0: Aborting I/O 2 QID 67
[ 428.361038] nvme 0000:15:00.0: Timeout I/O 3 QID 67
[ 428.361042] nvme 0000:15:00.0: Aborting I/O 3 QID 67
[ 442.363914] nvme 0000:15:00.0: Timeout I/O 468 QID 1
[ 442.363929] nvme 0000:15:00.0: I/O 468 QID 1 timeout, reset controller
[ 442.366134] nvme 0000:15:00.0: Cancelling I/O 1 QID 67
[ 442.366140] nvme 0000:15:00.0: completing aborted command with status:4007
[ 442.366146] blk_update_request: I/O error, dev nvme0n1, sector 1
[ 442.366160] nvme 0000:15:00.0: Cancelling I/O 2 QID 67
[ 442.366163] nvme 0000:15:00.0: completing aborted command with status:4007
[ 442.366165] blk_update_request: I/O error, dev nvme0n1, sector 8
[ 442.366169] nvme 0000:15:00.0: Cancelling I/O 3 QID 67
[ 442.366172] nvme 0000:15:00.0: completing aborted command with status:4007
[ 442.366174] blk_update_request: I/O error, dev nvme0n1, sector 16
[ 442.366187] nvme 0000:15:00.0: Cancelling I/O 468 QID 1
[ 442.366189] nvme 0000:15:00.0: completing aborted command with status:4007
[ 442.366192] blk_update_request: I/O error, dev nvme0n1, sector 390911849
[ 442.366200] nvme 0000:15:00.0: Cancelling I/O 1 QID 0
[ 442.366204] nvme 0000:15:00.0: Abort status:7 result:288cf000
[ 442.366207] nvme 0000:15:00.0: Cancelling I/O 2 QID 0
[ 442.366209] nvme 0000:15:00.0: Abort status:7 result:288cf000
[ 442.366212] nvme 0000:15:00.0: Cancelling I/O 3 QID 0
[ 442.366215] nvme 0000:15:00.0: Abort status:7 result:288cf000
[ 442.366217] nvme 0000:15:00.0: Cancelling I/O 4 QID 0
[ 442.366220] nvme 0000:15:00.0: Abort status:7 result:288cf000
[ 442.366228] XFS (nvme0n1): metadata I/O error: block 0x174cd769
("xlog_iodone") error 5 numblks 64
[ 442.366233] XFS (nvme0n1): xfs_do_force_shutdown(0x2) called from
line 1177 of file fs/xfs/xfs_log.c. Return address =
0xffffffff8127cd00
[ 442.366253] XFS (nvme0n1): Log I/O Error Detected. Shutting down filesystem
[ 442.366255] XFS (nvme0n1): Please umount the filesystem and rectify
the problem(s)
[ 442.366292] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 442.366298] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 442.366665] device: 'nvme0n1': device_del
[ 442.366828] nvme_ns_remove :gen disk removed
[ 442.370177] device: '259:1': device_unregister
[ 442.370181] device: '259:1': device_del
[ 442.370231] device: '259:1': device_create_release
[ 442.370239] nvme_ns_remove :exit
[ 442.370264] nvme :Removed the namespaces
[ 442.370317] device: 'nvme0': device_unregister
[ 442.370319] device: 'nvme0': device_del
[ 442.370565] nvme :nvme dev completly removed
[ 442.370575] device: '0000:15:00.0': device_del
[ 442.370645] pciehp 0000:0e:09.0:pcie24: pciehp_unconfigure_device:
domain:bus:dev = 0000:15:00
[ 442.370663] pcieport 0000:0e:09.0: Clear the DPC trigger status = 0x89
[ 442.376962] pci 0000:15:00.0: Refused to change power state, currently in D3
[ 442.376971] pci 0000:15:00.0: can't enable device: BAR 0 [mem
0x95300000-0x95303fff 64bit] not claimed
[ 442.376975] pci 0000:15:00.0: Device failed to resume
[ 458.128229] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 488.231729] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 518.335223] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 548.438697] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 578.542194] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 608.645685] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 638.749164] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 668.852663] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 698.956143] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 729.059637] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 759.163119] XFS (nvme0n1): xfs_log_force: error -5 returned.
[ 789.266620] XFS (nvme0n1): xfs_log_force: error -5 returned.Quoted 14 lines · Keith Busch
Quoted 10 lines · Mohana Goli
Keith,
Don't we need to take a queue spin lock while processing the IOs on each request queue like the way we are doing in
nvme_clear_queue?Quoted 1 line
Quoted 1 line
Quoted 1 line
blk_mq_all_tag_busy_iter(hctx->tags,
nvme_cancel_queue_ios, hctx->driver_data);You're right, thanks for the catch.
PATCH 1/1 NVMe: Force cancel commands on hot-removal
Quoted 4 lines · Mohana Goli
But issue is not seen if i use the my fix ,doing
nvme_dev_shutdownfirst andnvme_dev_removeinnvme_removefunction.And callingblk_cleanup_queue(ns->queue)before callingdel_gendiskinnvme_ns_removefunction.
The problem with your way is an orderly removal won't be able to sync dirty data, which is something you _really_ want to happen if it at all possible.
PATCH 1/1 NVMe: Force cancel commands on hot-removal
Quoted 11 lines · Keith Busch
Quoted 5 lines · Mohana Goli
But issue is not seen if i use the my fix ,doing
nvme_dev_shutdownfirst andnvme_dev_removeinnvme_removefunction.And callingblk_cleanup_queue(ns->queue)before callingdel_gendiskinnvme_ns_removefunction.The problem with your way is an orderly removal won't be able to sync dirty data, which is something you _really_ want to happen if it at all possible.
I think in case of orderly removal user will ensure flushing the dirty data and unmounting the filesystem before physically remove the drive.However there is a problem unloading the driver where we need to sync the dirty data first.
For this i read csts register in nvme_remove,if that reads as "0xffffffff" (device is no longer present), then i go my way, otherwise your way.
PATCH 1/1 NVMe: Force cancel commands on hot-removal
In reply to Mohana GoliQuoted 3 lines · Mohana Goli
I ran another test after creating filesystem on the nvme devices.Unfortunately i have seen the IO Timeouts again. I used the following steps:
I guess we need stop the h/w queues first, then. I thought killing it would have sufficed, but inflight commands will make it to the driver.
A message that is not in the archive (sent off-list or not yet received): CAPYzm7S650StbfGhEiTsnXnvwFK_XvS5w1OeFAxZG88-oiaZig@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): CAPYzm7QwDTj+5eqcT4WGGNdLjEoAg1T7hDSH-dBQsCa++0Ostw@mail.gmail.com
PATCH 1/1 NVMe: Force cancel commands on hot-removal
Quoted 1 line · Mohana Goli
I called?nvme_freeze_queues before calling
nvme_ns_removein nvme_dev_remove,which seems to be solved ?the problem.
Great! I thought that would do it, but I've some tests that seem to get 1 IO lost somewhere in the block layer (thinking maybe it's in the ctx->rq_list, still trying to confirm). That failure doesn't happen very often though.
Quoted 17 lines · Mohana Goli
Yes,i think we should call "
nvme_freeze_queues" before deleting the gendisk.Right now that is no op since we are calling after deleting the name space.On Wed, Sep 9, 2015@10:00 PM, Keith Busch <[email protected]> wrote:
On Wed, 9 Sep 2015, Mohana Goli wrote:
I ran another test after creating filesystem on the nvme devices.Unfortunately i have seen the IO Timeouts again. I used the following steps:I guess we need stop the h/w queues first, then. I thought killing it would have sufficed, but inflight commands will make it to the driver.