Skip to contentSkip to conversation

blk-mq & nvme-pci: fix io failure in kdump kernel

PATCH In review Ming Lei 5 participants 12 messages Jul 8, 2023 – Jul 11, 2023

Subsystems (4)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

cover blk-mq & nvme-pci: fix io failure in kdump kernel

Hello,

This two patches fixes nvme-pci io failure on kdump kernel.

Thanks,
Ming

Ming Lei (2):
blk-mq: add blk_mq_max_nr_hw_queues()
nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

 block/blk-mq.c          | 9 +++++++++
 drivers/nvme/host/pci.c | 2 +-
 include/linux/blk-mq.h  | 1 +
 3 files changed, 11 insertions(+), 1 deletion(-)
Signature · 1 line
2.40.1
author
#1 lore raw

PATCH 1/2 blk-mq: add blk_mq_max_nr_hw_queues()

blk_mq_alloc_tag_set() may return less nr_hw_queues in case of kdump kernel. This way can cause trouble for driver, which needs to calculate nr_hw_queues first.

If blk_mq_alloc_tag_set() reduces nr_hw_queues for kdump kernel, it causes trouble for driver, cause real queue topo is actually changed, then IO may be dispatched to wrong queue.

Prepare for fixing this kind of issue by applying the added helper, so driver can take blk-mq max nr_hw_queues knowledge into account when calculating io queues.

---
 block/blk-mq.c         | 9 +++++++++
 include/linux/blk-mq.h | 1 +
 2 files changed, 10 insertions(+)
block/blk-mq.c +9 -0
index 5504719b970d..b764da69a416 100644
@@ -140,6 +140,15 @@ void blk_mq_freeze_queue_wait(struct request_queue *q)
 }
 EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_wait);
 
+/* Max nr_hw_queues for each hw queue type */
+unsigned int blk_mq_max_nr_hw_queues(void)
+{
+	if (is_kdump_kernel())
+		return 1;
+	return nr_cpu_ids;
+}
+EXPORT_SYMBOL_GPL(blk_mq_max_nr_hw_queues);
+
 int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
 				     unsigned long timeout)
 {
include/linux/blk-mq.h +1 -0
index 2b7fb8e87793..2407978fbc30 100644
@@ -713,6 +713,7 @@ int blk_mq_alloc_sq_tag_set(struct blk_mq_tag_set *set,
 		const struct blk_mq_ops *ops, unsigned int queue_depth,
 		unsigned int set_flags);
 void blk_mq_free_tag_set(struct blk_mq_tag_set *set);
+unsigned int blk_mq_max_nr_hw_queues(void);
 
 void blk_mq_free_request(struct request *rq);
 int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,
Signature · 1 line
2.40.1
author
#2 lore raw

PATCH 2/2 nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

In reply to Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

But blk-mq can only support single queue for kdump kernel, this way causes wrong queue mapping taken for handling IO, and IO timeout is triggered.

Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure.

---
 drivers/nvme/host/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/pci.c +1 -1
index 72725729cb6c..cb13ba203956 100644
@@ -2247,7 +2247,7 @@ static unsigned int nvme_max_io_queues(struct nvme_dev *dev)
 	 */
 	if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
 		return 1;
-	return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues;
+	return blk_mq_max_nr_hw_queues() + dev->nr_write_queues + dev->nr_poll_queues;
 }
 
 static int nvme_setup_io_queues(struct nvme_dev *dev)
Signature · 1 line
2.40.1
#3 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these setups, otherwise you'll have to paper over it in all kind of drivers.

author
#4 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1" simply keep one of CPU cores as online, and others as offline.

So Cc our arch(arm & ppc64) & kdump guys wrt. passing 'maxcpus=1' for kdump kernel.

Quoted 2 lines

setups, otherwise you'll have to paper over it in all kind of drivers.

The issue is only triggered for drivers which use managed irq & multiple hw queues.

Thanks,
Ming

maintainer
#5 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

Quoted 18 lines · Ming Lei
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1" simply keep one of CPU cores as online, and others as offline.

Quoted 8 lines

So Cc our arch(arm & ppc64) & kdump guys wrt. passing 'maxcpus=1' for kdump kernel.

Quoted 2 lines

setups, otherwise you'll have to paper over it in all kind of drivers.

The issue is only triggered for drivers which use managed irq & multiple hw queues.

Is the problem that the managed interrupt sets the effective irq affinity to an offline CPU? You mentioned observed timeouts; are you seeing the "completion polled" nvme message?

author
#6 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

Quoted 22 lines · Keith Busch
Quoted 21 lines · Ming Lei
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1" simply keep one of CPU cores as online, and others as offline.

So Cc our arch(arm & ppc64) & kdump guys wrt. passing 'maxcpus=1' for kdump kernel.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

setups, otherwise you'll have to paper over it in all kind of drivers.

The issue is only triggered for drivers which use managed irq & multiple hw queues.

Is the problem that the managed interrupt sets the effective irq affinity to an offline CPU? You mentioned observed timeouts; are you

Yes, the problem is that blk-mq only creates hctx0, so nvme-pci translate it into hctx0's nvme_queue, this way is actually wrong, cause blk-mq's view on queue topo isn't same with nvme's view.

Quoted 1 line

seeing the "completion polled" nvme message?

Yes, "completion polled" can be observed. Meantime the warning in __irq_startup_managed() can be triggered from nvme_timeout()->nvme_poll_irqdisable()->enable_irq().

Thanks,
Ming

#7 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

In reply to Ming Lei
Quoted 10 lines · Ming Lei
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

Quoted 7 lines
Quoted 2 lines

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1" simply keep one of CPU cores as online, and others as offline.

I don't know maxcpus on arm and ppc64 well. But maxcpus=1 or nr_cpus=1 are suggested parameter. Because usually nr_cpus=1 is enough to make kdump kernel work well to capture vmcore. However, user is allowed to specify nr_cpus=n (n>1) if they think multiple cpus are needed in kdump kernel. Your hard coding of cpu number in kdump kernel may be not so reasonable.

Please cc kexec mailing list when posting so that people can view the whole thread of discussion.

Thanks
Baoquan

Quoted 11 lines

So Cc our arch(arm & ppc64) & kdump guys wrt. passing 'maxcpus=1' for kdump kernel.

Quoted 2 lines

setups, otherwise you'll have to paper over it in all kind of drivers.

The issue is only triggered for drivers which use managed irq & multiple hw queues.

Quoted 9 lines

Thanks,
Ming

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#8 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

Hi Baoquan,

Quoted 19 lines · Baoquan He
Quoted 10 lines · Ming Lei
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

Quoted 7 lines
Quoted 2 lines

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1" simply keep one of CPU cores as online, and others as offline.

Quoted 6 lines

I don't know maxcpus on arm and ppc64 well. But maxcpus=1 or nr_cpus=1 are suggested parameter. Because usually nr_cpus=1 is enough to make kdump kernel work well to capture vmcore. However, user is allowed to specify nr_cpus=n (n>1) if they think multiple cpus are needed in kdump kernel. Your hard coding of cpu number in kdump kernel may be not so reasonable.

As I mentioned, for arm/ppc64, passing 'maxcpus=1' actually follows Documentation/admin-guide/kdump/kdump.rst.

'nr_cpus=N' just works fine, so not related with this topic.

After 'maxcpus=1' is passed, kernel only brings up one of cpu cores as online during booting, and others still can be put into online by userspace. Now this way causes IO timeout on some storage device which uses managed irq and supports multiple io queues.

Here the focus is if passing 'maxcpus=1' is valid for kdump kernel, that is we want to hear from our arch/kdump guys.

If yes, something needs to be fixed, such as, what this patchset is doing.

Quoted 3 lines

Please cc kexec mailing list when posting so that people can view the whole thread of discussion.

Already Cc kexe & arm/powerpc & irq list.

Thanks,
Ming

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

Quoted 34 lines · Ming Lei

Hi Baoquan,

Quoted 19 lines · Baoquan He
Quoted 10 lines · Ming Lei
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

Quoted 7 lines
Quoted 2 lines

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1" simply keep one of CPU cores as online, and others as offline.

Quoted 6 lines

I don't know maxcpus on arm and ppc64 well. But maxcpus=1 or nr_cpus=1 are suggested parameter. Because usually nr_cpus=1 is enough to make kdump kernel work well to capture vmcore. However, user is allowed to specify nr_cpus=n (n>1) if they think multiple cpus are needed in kdump kernel. Your hard coding of cpu number in kdump kernel may be not so reasonable.

As I mentioned, for arm/ppc64, passing 'maxcpus=1' actually follows Documentation/admin-guide/kdump/kdump.rst.

'nr_cpus=N' just works fine, so not related with this topic.

Quoted 7 lines

After 'maxcpus=1' is passed, kernel only brings up one of cpu cores as online during booting, and others still can be put into online by userspace. Now this way causes IO timeout on some storage device which uses managed irq and supports multiple io queues.

Here the focus is if passing 'maxcpus=1' is valid for kdump kernel, that is we want to hear from our arch/kdump guys.

Yes, 'maxcpus=1' is valid and suggested on ppc64 for kdump kernel if needed, because there's no 'nr_cpus=' support on ppc64 yet.

Quoted 14 lines

If yes, something needs to be fixed, such as, what this patchset is doing.

Quoted 3 lines

Please cc kexec mailing list when posting so that people can view the whole thread of discussion.

Already Cc kexe & arm/powerpc & irq list.

Thanks,
Ming

#10 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

In reply to Ming Lei
Quoted 5 lines · Ming Lei
Quoted 3 lines · Christoph Hellwig
Quoted 2 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

In fact, num_possible_cpus is not broken.

Quote from admin-guide/kernel-parameters.txt
maxcpus= [SMP] Maximum number of processors that an SMP kernel
will bring up during bootup. maxcpus=n : n >= 0 limits the kernel to bring up 'n' processors. Surely after bootup you can bring up the other plugged cpu
by executing
"echo 1 > /sys/devices/system/cpu/cpuX/online".
So maxcpus
only takes effect during system bootup. While n=0 is a special case, it is equivalent to "nosmp", which also disables the IO APIC.

Here, as it explained, maxcpus only affects the bootup, later, extra cpus can be online.

Quoted 3 lines

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1"

On aarch64 and x86, nr_cpus=1 is used, while on ppc64, due to the implementation, nr_cpus=1 can not be supported.

Thanks,

Pingfan

Quoted 12 lines

simply keep one of CPU cores as online, and others as offline.

So Cc our arch(arm & ppc64) & kdump guys wrt. passing 'maxcpus=1' for kdump kernel.

Quoted 2 lines

setups, otherwise you'll have to paper over it in all kind of drivers.

The issue is only triggered for drivers which use managed irq & multiple hw queues.

Quoted 9 lines

Thanks,
Ming

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw

Re: [PATCH 2/2] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues

In reply to Ming Lei

Hi Ming,

Having no [PATCH 1/2] blk-mq: add blk_mq_max_nr_hw_queues() in inbox. So I reply here.

At first glance, I think that the cpu hot plug callback hook should be the remedy for the newly introduced blk_mq_max_nr_hw_queues(), although it is more complicated.

Consider the scene where nr_cpus=4, which can speed up the dumping process, the blk_mq_max_nr_hw_queues() can not utilize the other three cpus.

Thanks,

Pingfan

Quoted 29 lines · Ming Lei
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Ming Lei

Take blk-mq's knowledge into account for calculating io queues.

Fix wrong queue mapping in case of kdump kernel.

On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see Documentation/admin-guide/kdump/kdump.rst, so num_possible_cpus() still returns all CPUs.

That's simply broken. Please fix the arch code to make sure it does not return a bogus num_possible_cpus value for these

That is documented in Documentation/admin-guide/kdump/kdump.rst.

On arm and ppc64, 'maxcpus=1' is passed for kdump kernel, and "maxcpu=1" simply keep one of CPU cores as online, and others as offline.

So Cc our arch(arm & ppc64) & kdump guys wrt. passing 'maxcpus=1' for kdump kernel.

Quoted 2 lines

setups, otherwise you'll have to paper over it in all kind of drivers.

The issue is only triggered for drivers which use managed irq & multiple hw queues.

Quoted 9 lines

Thanks,
Ming

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec