Skip to contentSkip to conversation

nvme: fix doulbe irq free

PATCH In review Tong Zhang 3 participants 4 messages Sep 16, 2020 – Sep 17, 2020 NVMe

lore
Share & tools
lore

Tong Zhang

author
#0 lore raw

PATCH 1/1 nvme: fix doulbe irq free

the irq might already been released before reset work can run

[   81.137630] ------------[ cut here ]------------
[   81.137913] Trying to free already-free IRQ 11
[   81.138145] WARNING: CPU: 1 PID: 7 at kernel/irq/manage.c:1751 free_irq+0x389/0x590
[   81.138525] Modules linked in:
[   81.138681] CPU: 1 PID: 7 Comm: kworker/u4:0 Not tainted 5.9.0-rc4+ #83
[   81.139579] Workqueue: nvme-reset-wq nvme_reset_work
[   81.139828] RIP: 0010:free_irq+0x389/0x590
[   81.140035] Code: 8b 65 50 e8 69 6d 1d 00 48 83 c4 38 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 8b 74 24 24 48 c7 c7 20 1c 86 8a e8 d7 8e f6 ff <0f0e
[   81.140951] RSP: 0000:ffff88806c06fb18 EFLAGS: 00010082
[   81.141216] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[   81.141569] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffffed100d80df55
[   81.141924] RBP: ffff88806cac88c0 R08: 0000000000000001 R09: ffffed100d80df29
[   81.142278] R10: 0000000000000003 R11: ffffed100d80df28 R12: ffff88806cac8800
[   81.142632] R13: ffff88806cac8870 R14: ffff88806cac8840 R15: ffff88806cac8960
[   81.142987] FS:  0000000000000000(0000) GS:ffff88806d100000(0000) knlGS:0000000000000000
[   81.143388] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   81.143673] CR2: 0000000000000000 CR3: 0000000026c0e000 CR4: 00000000000006e0
[   81.144028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   81.144382] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   81.144734] Call Trace:
[   81.144862]  ? nvme_start_ctrl+0x180/0x180
[   81.145072]  pci_free_irq+0x13/0x20
[   81.145248]  nvme_reset_work+0xcf2/0x1ec0
[   81.145451]  ? __update_load_avg_cfs_rq+0x1a3/0x530
[   81.145695]  ? __update_load_avg_cfs_rq+0x4b/0x530
[   81.145935]  ? nvme_remove+0x1e0/0x1e0
[   81.146124]  ? ttwu_queue_wakelist+0x12e/0x150
[   81.146348]  ? try_to_wake_up+0x37c/0x900
[   81.146550]  ? migration_cpu_stop+0x1e0/0x1e0
[   81.146769]  ? __schedule+0x581/0xc40
[   81.146954]  ? read_word_at_a_time+0xe/0x20
[   81.147164]  ? strscpy+0xbf/0x190
[   81.147333]  process_one_work+0x4ad/0x7e0
[   81.147535]  worker_thread+0x73/0x690
[   81.147720]  ? process_one_work+0x7e0/0x7e0
[   81.147930]  kthread+0x199/0x1f0
[   81.148094]  ? kthread_create_on_node+0xd0/0xd0
[   81.148321]  ret_from_fork+0x22/0x30
[   81.148502] ---[ end trace 374ff42ced5b661f ]---
---
 drivers/nvme/host/pci.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
drivers/nvme/host/pci.c +11 -7
index 899d2f4d7ab6..c2b083dcfd17 100644
@@ -2086,6 +2086,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	unsigned int nr_io_queues;
 	unsigned long size;
 	int result;
+	int adminq_irq_enabled = test_bit(NVMEQ_ENABLED, &adminq->flags);
 
 	/*
 	 * Sample the module parameters once at reset time so that we have
@@ -2133,14 +2134,16 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	adminq->q_db = dev->dbs;
 
  retry:
-	/* Deregister the admin queue's interrupt */
-	pci_free_irq(pdev, 0, adminq);
+	if (adminq_irq_enabled) {
+		/* Deregister the admin queue's interrupt */
+		pci_free_irq(pdev, 0, adminq);
 
-	/*
-	 * If we enable msix early due to not intx, disable it again before
-	 * setting up the full range we need.
-	 */
-	pci_free_irq_vectors(pdev);
+		/*
+		 * If we enable msix early due to not intx, disable it again before
+		 * setting up the full range we need.
+		 */
+		pci_free_irq_vectors(pdev);
+	}
 
 	result = nvme_setup_irqs(dev, nr_io_queues);
 	if (result <= 0)
@@ -2160,6 +2163,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (result)
 		return result;
 	set_bit(NVMEQ_ENABLED, &adminq->flags);
+	adminq_irq_enabled = 1;
 
 	result = nvme_create_io_queues(dev);
 	if (result || dev->online_queues < 2)
Signature · 7 lines
2.25.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author
#2 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 1 line · Tong Zhang

the irq might already been released before reset work can run

If it is we have a problem with the state machine, as we shouldn't have two resets running at the same time. Can you share your reproducer?

Hi Christoph,
Thanks for replying.

It is possible that pci_free_irq() can be called twice for the same IRQ, the first call can be

[  124.413113] nvme nvme0: I/O 6 QID 0 timeout, disable controller
[  124.414317] Workqueue: kblockd blk_mq_timeout_work
[  124.414317] Call Trace:
[  124.414317]  dump_stack+0x7d/0xb0
[  124.414317]  nvme_suspend_queue.cold+0x11/0x58
[  124.414317]  nvme_dev_disable+0x116/0x7b0
[  124.414317]  nvme_timeout+0x309/0x320
[  124.414317]  ? nvme_shutdown+0x40/0x40
[  124.414317]  ? load_balance+0xe76/0x1450
[  124.414317]  ? deref_stack_reg+0x40/0x40
[  124.414317]  ? sched_clock_local+0x99/0xc0
[  124.414317]  blk_mq_check_expired+0x313/0x340
[  124.414317]  ? blk_mq_freeze_queue_wait+0x180/0x180
[  124.414317]  ? _find_next_bit.constprop.0+0x3e/0xf0
[  124.414317]  blk_mq_queue_tag_busy_iter+0x2e9/0x4a0
[  124.414317]  ? blk_mq_freeze_queue_wait+0x180/0x180
[  124.414317]  ? blk_mq_all_tag_iter+0x10/0x10
[  124.414317]  ? _raw_write_lock_irqsave+0xd0/0xd0
[  124.414317]  ? run_rebalance_domains+0x80/0x80
[  124.414317]  ? blk_mq_freeze_queue_wait+0x180/0x180
[  124.414317]  ? debug_object_deactivate.part.0+0x1c4/0x210
[  124.414317]  blk_mq_timeout_work+0xaa/0x1f0
[  124.414317]  ? __blk_mq_end_request+0x1f0/0x1f0
[  124.414317]  ? __schedule+0x581/0xc40
[  124.414317]  ? read_word_at_a_time+0xe/0x20
[  124.414317]  ? strscpy+0xbf/0x190
[  124.414317]  process_one_work+0x4ad/0x7e0
[  124.414317]  worker_thread+0x73/0x690
[  124.414317]  ? process_one_work+0x7e0/0x7e0
[  124.414317]  kthread+0x199/0x1f0
[  124.414317]  ? kthread_create_on_node+0xd0/0xd0
[  124.414317]  ret_from_fork+0x22/0x30

And the second call can be

[  149.763974] nvme nvme0: Failed to read smart log (error -4)
[  149.774307] Trying to free already-free IRQ 10
[  149.774385] Call Trace:
[  149.774385]  pci_free_irq+0x13/0x20
[  149.774385]  nvme_reset_work.cold+0x1be/0xa78
[  149.774385]  ? nvme_probe+0x8c0/0x8c0
[  149.774385]  ? check_preempt_curr+0x9d/0xf0
[  149.774385]  ? ttwu_do_wakeup.isra.0+0x176/0x190
[  149.774385]  ? try_to_wake_up+0x37c/0x900
[  149.774385]  ? migration_cpu_stop+0x1e0/0x1e0
[  149.774385]  ? __schedule+0x581/0xc40
[  149.774385]  ? read_word_at_a_time+0xe/0x20
[  149.774385]  ? strscpy+0xbf/0x190
[  149.774385]  process_one_work+0x4ad/0x7e0
[  149.774385]  worker_thread+0x73/0x690
[  149.774385]  ? process_one_work+0x7e0/0x7e0
[  149.774385]  kthread+0x199/0x1f0
[  149.774385]  ? kthread_create_on_node+0xd0/0xd0
[  149.774385]  ret_from_fork+0x22/0x30

and clearly something is not working as expected.

- Tong

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#3 lore raw
Quoted 69 lines · Tong Zhang
Quoted 7 lines · Christoph Hellwig
Quoted 1 line · Tong Zhang

the irq might already been released before reset work can run

If it is we have a problem with the state machine, as we shouldn't have two resets running at the same time. Can you share your reproducer?

Hi Christoph,
Thanks for replying.

It is possible that pci_free_irq() can be called twice for the same IRQ, the first call can be

[  124.413113] nvme nvme0: I/O 6 QID 0 timeout, disable controller
[  124.414317] Workqueue: kblockd blk_mq_timeout_work
[  124.414317] Call Trace:
[  124.414317]  dump_stack+0x7d/0xb0
[  124.414317]  nvme_suspend_queue.cold+0x11/0x58
[  124.414317]  nvme_dev_disable+0x116/0x7b0
[  124.414317]  nvme_timeout+0x309/0x320
[  124.414317]  ? nvme_shutdown+0x40/0x40
[  124.414317]  ? load_balance+0xe76/0x1450
[  124.414317]  ? deref_stack_reg+0x40/0x40
[  124.414317]  ? sched_clock_local+0x99/0xc0
[  124.414317]  blk_mq_check_expired+0x313/0x340
[  124.414317]  ? blk_mq_freeze_queue_wait+0x180/0x180
[  124.414317]  ? _find_next_bit.constprop.0+0x3e/0xf0
[  124.414317]  blk_mq_queue_tag_busy_iter+0x2e9/0x4a0
[  124.414317]  ? blk_mq_freeze_queue_wait+0x180/0x180
[  124.414317]  ? blk_mq_all_tag_iter+0x10/0x10
[  124.414317]  ? _raw_write_lock_irqsave+0xd0/0xd0
[  124.414317]  ? run_rebalance_domains+0x80/0x80
[  124.414317]  ? blk_mq_freeze_queue_wait+0x180/0x180
[  124.414317]  ? debug_object_deactivate.part.0+0x1c4/0x210
[  124.414317]  blk_mq_timeout_work+0xaa/0x1f0
[  124.414317]  ? __blk_mq_end_request+0x1f0/0x1f0
[  124.414317]  ? __schedule+0x581/0xc40
[  124.414317]  ? read_word_at_a_time+0xe/0x20
[  124.414317]  ? strscpy+0xbf/0x190
[  124.414317]  process_one_work+0x4ad/0x7e0
[  124.414317]  worker_thread+0x73/0x690
[  124.414317]  ? process_one_work+0x7e0/0x7e0
[  124.414317]  kthread+0x199/0x1f0
[  124.414317]  ? kthread_create_on_node+0xd0/0xd0
[  124.414317]  ret_from_fork+0x22/0x30

And the second call can be

[  149.763974] nvme nvme0: Failed to read smart log (error -4)
[  149.774307] Trying to free already-free IRQ 10
[  149.774385] Call Trace:
[  149.774385]  pci_free_irq+0x13/0x20
[  149.774385]  nvme_reset_work.cold+0x1be/0xa78
[  149.774385]  ? nvme_probe+0x8c0/0x8c0
[  149.774385]  ? check_preempt_curr+0x9d/0xf0
[  149.774385]  ? ttwu_do_wakeup.isra.0+0x176/0x190
[  149.774385]  ? try_to_wake_up+0x37c/0x900
[  149.774385]  ? migration_cpu_stop+0x1e0/0x1e0
[  149.774385]  ? __schedule+0x581/0xc40
[  149.774385]  ? read_word_at_a_time+0xe/0x20
[  149.774385]  ? strscpy+0xbf/0x190
[  149.774385]  process_one_work+0x4ad/0x7e0
[  149.774385]  worker_thread+0x73/0x690
[  149.774385]  ? process_one_work+0x7e0/0x7e0
[  149.774385]  kthread+0x199/0x1f0
[  149.774385]  ? kthread_create_on_node+0xd0/0xd0
[  149.774385]  ret_from_fork+0x22/0x30
Quoted 1 line

and clearly something is not working as expected.

And the problem is that nvme_hwmon_init() doesn't return an error, so it returns to continue initialization on a disabled controller. That's the bug that needs to be fixed.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme