nvme-pci: adjust irq max_vector to avoid WARN()
cover nvme-pci: adjust irq max_vector to avoid WARN()
Hi All,
This patch series is to adjust the max_vector value from the nvme_setup_irqs() which provides the max_vector before recalculating the irq set_size in nvme_calc_irq_sets(). The third patch is mainly focused on avoding the WARN() which might occur in irq_build_affinity_masks() [1].
This series has been tested with combination of the following criteria:
1) write_queues=0..128
1) poll_queues=0..128
Please feel free to let me know if I have thought of something useless or feel this patch is useless.
Thanks in advance,
[1] WARN messages when modprobe nvme write_queues=32 poll_queues=0:
root at target:~/nvme# nproc
8
root at target:~/nvme# modprobe nvme write_queues=32 poll_queues=0
[ 17.925326] nvme nvme0: pci function 0000:00:04.0 [ 17.940601] WARNING: CPU: 3 PID: 1030 at kernel/irq/affinity.c:221 irq_create_affinity_masks+0x222/0x330 [ 17.940602] Modules linked in: nvme nvme_core [last unloaded: nvme] [ 17.940605] CPU: 3 PID: 1030 Comm: kworker/u17:4 Tainted: G W 5.1.0+ #156 [ 17.940605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [ 17.940608] Workqueue: nvme-reset-wq nvme_reset_work [nvme] [ 17.940609] RIP: 0010:irq_create_affinity_masks+0x222/0x330 [ 17.940611] Code: 4c 8d 4c 24 28 4c 8d 44 24 30 e8 c9 fa ff ff 89 44 24 18 e8 c0 38 fa ff 8b 44 24 18 44 8b 54 24 1c 5a 44 01 d0 41 39 c4 76 02 <0f> 0b 48 89 df 44 01 e5 e8 f1 ce 10 00 48 8b 34 24 44 89 f0 44 01 [ 17.940611] RSP: 0018:ffffc90002277c50 EFLAGS: 00010216 [ 17.940612] RAX: 0000000000000008 RBX: ffff88807ca48860 RCX: 0000000000000000 [ 17.940612] RDX: ffff88807bc03800 RSI: 0000000000000020 RDI: 0000000000000000 [ 17.940613] RBP: 0000000000000001 R08: ffffc90002277c78 R09: ffffc90002277c70 [ 17.940613] R10: 0000000000000008 R11: 0000000000000001 R12: 0000000000000020 [ 17.940614] R13: 0000000000025d08 R14: 0000000000000001 R15: ffff88807bc03800 [ 17.940614] FS: 0000000000000000(0000) GS:ffff88807db80000(0000) knlGS:0000000000000000 [ 17.940616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 17.940617] CR2: 00005635e583f790 CR3: 000000000240a000 CR4: 00000000000006e0 [ 17.940617] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 17.940618] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 17.940618] Call Trace: [ 17.940622] __pci_enable_msix_range+0x215/0x540 [ 17.940623] ? kernfs_put+0x117/0x160 [ 17.940625] pci_alloc_irq_vectors_affinity+0x74/0x110 [ 17.940626] nvme_reset_work+0xc30/0x1397 [nvme] [ 17.940628] ? __switch_to_asm+0x34/0x70 [ 17.940628] ? __switch_to_asm+0x40/0x70 [ 17.940629] ? __switch_to_asm+0x34/0x70 [ 17.940630] ? __switch_to_asm+0x40/0x70 [ 17.940630] ? __switch_to_asm+0x34/0x70 [ 17.940631] ? __switch_to_asm+0x40/0x70 [ 17.940632] ? nvme_irq_check+0x30/0x30 [nvme] [ 17.940633] process_one_work+0x20b/0x3e0 [ 17.940634] worker_thread+0x1f9/0x3d0 [ 17.940635] ? cancel_delayed_work+0xa0/0xa0 [ 17.940636] kthread+0x117/0x120 [ 17.940637] ? kthread_stop+0xf0/0xf0 [ 17.940638] ret_from_fork+0x3a/0x50 [ 17.940639] ---[ end trace aca8a131361cd42a ]--- [ 17.942124] nvme nvme0: 7/1/0 default/read/poll queues
Minwoo Im (3):
nvme-pci: remove unnecessary zero for static var
nvme-pci: remove queue_count_ops for write,poll_queues
nvme-pci: adjust irq max_vector using num_possible_cpus()
drivers/nvme/host/pci.c | 31 ++++++++----------------------- 1 file changed, 8 insertions(+), 23 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/3 nvme-pci: remove unnecessary zero for static var
poll_queues will be zero even without zero initialization here.
--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/pci.c +1 -1
@@ -80,7 +80,7 @@ MODULE_PARM_DESC(write_queues,"Number of queues to use for writes. If not set, reads and writes " "will share a queue set."); -static int poll_queues =0 ; +static int poll_queues;module_param_cb (poll_queues, &queue_count_ops, &poll_queues,0644 );MODULE_PARM_DESC (poll_queues,"Number of queues to use for polled IO." );
Signature · 1 line
2.21.0
PATCH 1/3 nvme-pci: remove unnecessary zero for static var
Looks good.
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Quoted 26 lines · Minwoo Im
poll_queueswill be zero even without zero initialization here.Cc Jens Axboe <axboe at fb.com>Cc Ming Lei <ming.lei at redhat.com>Cc Keith Busch <kbusch at kernel.org>Cc Christoph Hellwig <hch at lst.de>Cc Sagi Grimberg <sagi at grimberg.me>Signed-off-by Minwoo Im <minwoo.im.dev at gmail.com>--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/pci.c +1
-1@@ -80,7 +80,7 @@ MODULE_PARM_DESC(write_queues,"Number of queues to use for writes. If not set, reads and writes " "will share a queue set."); -static int poll_queues =0 ; +static int poll_queues;module_param_cb (poll_queues, &queue_count_ops, &poll_queues,0644 );MODULE_PARM_DESC (poll_queues,"Number of queues to use for polled IO." );
PATCH 2/3 nvme-pci: remove queue_count_ops for write,poll_queues
In reply to Minwoo Imqueue_count_set() seems like that it has been provided to limit the number of queue entries for write/poll queues. But, the queue_count_set() has been doing nothing but a parameter check even it has num_possible_cpus() which is nop.
This patch removes entire queue_count_ops from the write_queues and poll_queues.
--- drivers/nvme/host/pci.c | 23 ++--------------------- 1 file changed, 2 insertions(+), 21 deletions(-)
drivers/nvme/host/pci.c +2 -21
@@ -68,20 +68,14 @@ static int io_queue_depth = 1024;module_param_cb (io_queue_depth, &io_queue_depth_ops, &io_queue_depth,0644 );MODULE_PARM_DESC (io_queue_depth,"set io queue depth, should >= 2" ); -static int queue_count_set (const char *val,const struct kernel_param *kp); -static const struct kernel_param_ops queue_count_ops = { - .set = queue_count_set, - .get = param_get_int, -}; -static int write_queues; -module_param_cb (write_queues, &queue_count_ops, &write_queues,0644 ); +module_param (write_queues, int,0644 );MODULE_PARM_DESC (write_queues,"Number of queues to use for writes. If not set, reads and writes " "will share a queue set." );static int poll_queues; -module_param_cb (poll_queues, &queue_count_ops, &poll_queues,0644 ); +module_param (poll_queues, int,0644 );MODULE_PARM_DESC (poll_queues,"Number of queues to use for polled IO." );struct nvme_dev ;
@@ -146,19 +140,6 @@ static int io_queue_depth_set(const char *val, const struct kernel_param *kp)return param_set_int (val, kp); } -static int queue_count_set (const char *val,const struct kernel_param *kp) -{ -int n, ret; - - ret =kstrtoint (val,10 , &n); -if (ret) -return ret; -if (n >num_possible_cpus ()) - n =num_possible_cpus (); - -return param_set_int (val, kp); -} -static inline unsigned int sq_idx (unsigned int qid,u32 stride) {return qid *2 * stride;
Signature · 1 line
2.21.0
PATCH 2/3 nvme-pci: remove queue_count_ops for write,poll_queues
Quoted 4 lines · Minwoo Im
queue_count_set()seems like that it has been provided to limit the number of queue entries for write/poll queues. But, thequeue_count_set()has been doing nothing but a parameter check even it hasnum_possible_cpus()which is nop.
However, the check is valid, which shouldn't be nop, so could you fix the check instead of removing it?
Thanks,
Ming
PATCH 2/3 nvme-pci: remove queue_count_ops for write,poll_queues
Quoted 2 lines
However, the check is valid, which shouldn't be nop, so could you fix the check instead of removing it?
Hi, Ming.
I don't get what you really mean here. What do you mean "the check is
valid"? I don't see any valid checks in queue_count_set(), not just for check for the failure by kstrtoint(). I think current code is just checks the nr_cpus and do nothing after.
Instead fixing this check inside of this function, I have posted the next patch in this series to make sure the number of irqs requested not exceed the num_possible_num().
Could you please have a look the next patch in this series ?
If I missed something here, please let me know.
Thanks,
PATCH 2/3 nvme-pci: remove queue_count_ops for write,poll_queues
Quoted 5 lines · Minwoo Im
Quoted 2 lines
However, the check is valid, which shouldn't be nop, so could you fix the check instead of removing it?
Hi, Ming.
Quoted 8 lines
I don't get what you really mean here. What do you mean "the check is
valid"? I don't see any valid checks inqueue_count_set(), not just for check for the failure bykstrtoint(). I think current code is just checks thenr_cpusand do nothing after.Instead fixing this check inside of this function, I have posted the next patch in this series to make sure the number of irqs requested not exceed the
num_possible_num().
I suggest to cap 'write_queues' or 'poll_writes' to num_possible_num() from the beginning, instead of starting with invalid number.
thanks,
Ming
PATCH 2/3 nvme-pci: remove queue_count_ops for write,poll_queues
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
However, the check is valid, which shouldn't be nop, so could you fix the check instead of removing it?
Hi, Ming.
I don't get what you really mean here. What do you mean "the check is
valid"? I don't see any valid checks inqueue_count_set(), not just for check for the failure bykstrtoint(). I think current code is just checks thenr_cpusand do nothing after.
Quoted 6 lines
Quoted 3 lines
Instead fixing this check inside of this function, I have posted the next patch in this series to make sure the number of irqs requested not exceed the
num_possible_num().I suggest to cap '
write_queues' or 'poll_writes' tonum_possible_num()from the beginning, instead of starting with invalid number.
Ming,
Thanks for your review :)
I have already tried to limit the number of queues inside of queue_count_set() in [1]. But Christoph had suggested to limit the number in nvme_calc_irq_sets() instead. It might be my code was not really good at that time, but could you please have a look at the mail thread below and give some advices for me?
[1] http://lists.infradead.org/pipermail/linux-nvme/2019-May/023868.html
PATCH 2/3 nvme-pci: remove queue_count_ops for write, poll_queues
Quoted 20 lines · Minwoo Im
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
However, the check is valid, which shouldn't be nop, so could you fix the check instead of removing it?
Hi, Ming.
I don't get what you really mean here. What do you mean "the check is
valid"? I don't see any valid checks inqueue_count_set(), not just for check for the failure bykstrtoint(). I think current code is just checks thenr_cpusand do nothing after.Quoted 6 lines
Quoted 3 lines
Instead fixing this check inside of this function, I have posted the next patch in this series to make sure the number of irqs requested not exceed the
num_possible_num().I suggest to cap '
write_queues' or 'poll_writes' tonum_possible_num()from the beginning, instead of starting with invalid number.Ming,
Quoted 8 lines
Thanks for your review :)
I have already tried to limit the number of queues inside of
queue_count_set()in [1]. But Christoph had suggested to limit the number innvme_calc_irq_sets()instead. It might be my code was not really good at that time, but could you please have a look at the mail thread below and give some advices for me?[1] http://lists.infradead.org/pipermail/linux-nvme/2019-May/023868.html
Sorry for missing the previous discussion.
IMO, I prefer to the fix in above link. Cause it is the value of kernel_param_ops
to make 'write_queue/poll_queue' correct from the beginning. That is just like OOP's concept in which each method just does one thing. Then we don't need to bother nvme_calc_irq_sets() for verifying/fixing them.
Thanks,
Ming Lei
PATCH 2/3 nvme-pci: remove queue_count_ops for write, poll_queues
Quoted 32 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 3 lines
However, the check is valid, which shouldn't be nop, so could you fix the check instead of removing it?
Hi, Ming.
I don't get what you really mean here. What do you mean "the check is
valid"? I don't see any valid checks inqueue_count_set(), not just for check for the failure bykstrtoint(). I think current code is just checks thenr_cpusand do nothing after.Instead fixing this check inside of this function, I have posted the next patch in this series to make sure the number of irqs requested not exceed the
num_possible_num().Quoted 8 lines
Quoted 2 lines
I suggest to cap '
write_queues' or 'poll_writes' tonum_possible_num()from the beginning, instead of starting with invalid number.Ming,
Thanks for your review :)
I have already tried to limit the number of queues inside of
queue_count_set()Quoted 1 line
in [1]. But Christoph had suggested to limit the number in
nvme_calc_irq_sets()Quoted 1 line
instead. It might be my code was not really good at that time, but could
you
Quoted 3 lines
please have a look at the mail thread below and give some advices for me?
[1] http://lists.infradead.org/pipermail/linux-nvme/2019-May/023868.html
Quoted 7 lines
Sorry for missing the previous discussion.
IMO, I prefer to the fix in above link. Cause it is the value of
kernel_param_ops
to make 'write_queue/poll_queue' correct from the beginning. That is just like OOP's concept in which each method just does one thing. Then we don't need to bothernvme_calc_irq_sets()for verifying/fixing them.
Christoph,
What do you think about what Ming's suggested? I am actually okay with the concept that the function just does its own things.
It would be great if you can give some advices here :)
Thanks,
PATCH 2/3 nvme-pci: remove queue_count_ops for write,poll_queues
Quoted 40 lines · Minwoo Im
Quoted 32 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 3 lines
However, the check is valid, which shouldn't be nop, so could you fix the check instead of removing it?
Hi, Ming.
I don't get what you really mean here. What do you mean "the check is
valid"? I don't see any valid checks inqueue_count_set(), not just for check for the failure bykstrtoint(). I think current code is just checks thenr_cpusand do nothing after.Instead fixing this check inside of this function, I have posted the next patch in this series to make sure the number of irqs requested not exceed the
num_possible_num().Quoted 8 lines
Quoted 2 lines
I suggest to cap '
write_queues' or 'poll_writes' tonum_possible_num()from the beginning, instead of starting with invalid number.Ming,
Thanks for your review :)
I have already tried to limit the number of queues inside of
queue_count_set()Quoted 1 line
in [1]. But Christoph had suggested to limit the number in
nvme_calc_irq_sets()Quoted 1 line
instead. It might be my code was not really good at that time, but could
you
Quoted 3 lines
please have a look at the mail thread below and give some advices for me?
[1] http://lists.infradead.org/pipermail/linux-nvme/2019-May/023868.html
Quoted 7 lines
Sorry for missing the previous discussion.
IMO, I prefer to the fix in above link. Cause it is the value of
kernel_param_ops
to make 'write_queue/poll_queue' correct from the beginning. That is just like OOP's concept in which each method just does one thing. Then we don't need to bothernvme_calc_irq_sets()for verifying/fixing them.
Quoted 8 lines
Christoph,
What do you think about what Ming's suggested? I am actually okay with the concept that the function just does its own things.
It would be great if you can give some advices here :)
Thanks,
(Gentle ping)
Hi Christoph,
Could you please take a look at this patch ?
Thanks,
Christoph Hellwig
maintainerPATCH 2/3 nvme-pci: remove queue_count_ops for write,poll_queues
Quoted 3 lines · Minwoo Im
Hi Christoph,
Could you please take a look at this patch ?
I much prefer this simpler series. I'll queue it up for 5.3 after a little testing.
PATCH 3/3 nvme-pci: adjust irq max_vector using num_possible_cpus()
In reply to Minwoo ImIf the "irq_queues" are greater than num_possible_cpus(), nvme_calc_irq_sets() can have irq set_size for HCTX_TYPE_DEFAULT greater than it can be afforded.
2039 affd->set_size[HCTX_TYPE_DEFAULT] = nrirqs - nr_read_queues;
It might cause a WARN() from the irq_build_affinity_masks() like [1]:
220 if (nr_present < numvecs)
221 WARN_ON(nr_present + nr_others < numvecs);
This patch prevents it from the WARN() by adjusting the max_vector value from the nvme_setup_irqs().
[1] WARN messages when modprobe nvme write_queues=32 poll_queues=0:
root at target:~/nvme# nproc
8
root at target:~/nvme# modprobe nvme write_queues=32 poll_queues=0
[ 17.925326] nvme nvme0: pci function 0000:00:04.0 [ 17.940601] WARNING: CPU: 3 PID: 1030 at kernel/irq/affinity.c:221 irq_create_affinity_masks+0x222/0x330 [ 17.940602] Modules linked in: nvme nvme_core [last unloaded: nvme] [ 17.940605] CPU: 3 PID: 1030 Comm: kworker/u17:4 Tainted: G W 5.1.0+ #156 [ 17.940605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [ 17.940608] Workqueue: nvme-reset-wq nvme_reset_work [nvme] [ 17.940609] RIP: 0010:irq_create_affinity_masks+0x222/0x330 [ 17.940611] Code: 4c 8d 4c 24 28 4c 8d 44 24 30 e8 c9 fa ff ff 89 44 24 18 e8 c0 38 fa ff 8b 44 24 18 44 8b 54 24 1c 5a 44 01 d0 41 39 c4 76 02 <0f> 0b 48 89 df 44 01 e5 e8 f1 ce 10 00 48 8b 34 24 44 89 f0 44 01 [ 17.940611] RSP: 0018:ffffc90002277c50 EFLAGS: 00010216 [ 17.940612] RAX: 0000000000000008 RBX: ffff88807ca48860 RCX: 0000000000000000 [ 17.940612] RDX: ffff88807bc03800 RSI: 0000000000000020 RDI: 0000000000000000 [ 17.940613] RBP: 0000000000000001 R08: ffffc90002277c78 R09: ffffc90002277c70 [ 17.940613] R10: 0000000000000008 R11: 0000000000000001 R12: 0000000000000020 [ 17.940614] R13: 0000000000025d08 R14: 0000000000000001 R15: ffff88807bc03800 [ 17.940614] FS: 0000000000000000(0000) GS:ffff88807db80000(0000) knlGS:0000000000000000 [ 17.940616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 17.940617] CR2: 00005635e583f790 CR3: 000000000240a000 CR4: 00000000000006e0 [ 17.940617] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 17.940618] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 17.940618] Call Trace: [ 17.940622] __pci_enable_msix_range+0x215/0x540 [ 17.940623] ? kernfs_put+0x117/0x160 [ 17.940625] pci_alloc_irq_vectors_affinity+0x74/0x110 [ 17.940626] nvme_reset_work+0xc30/0x1397 [nvme] [ 17.940628] ? __switch_to_asm+0x34/0x70 [ 17.940628] ? __switch_to_asm+0x40/0x70 [ 17.940629] ? __switch_to_asm+0x34/0x70 [ 17.940630] ? __switch_to_asm+0x40/0x70 [ 17.940630] ? __switch_to_asm+0x34/0x70 [ 17.940631] ? __switch_to_asm+0x40/0x70 [ 17.940632] ? nvme_irq_check+0x30/0x30 [nvme] [ 17.940633] process_one_work+0x20b/0x3e0 [ 17.940634] worker_thread+0x1f9/0x3d0 [ 17.940635] ? cancel_delayed_work+0xa0/0xa0 [ 17.940636] kthread+0x117/0x120 [ 17.940637] ? kthread_stop+0xf0/0xf0 [ 17.940638] ret_from_fork+0x3a/0x50 [ 17.940639] ---[ end trace aca8a131361cd42a ]--- [ 17.942124] nvme nvme0: 7/1/0 default/read/poll queues
--- drivers/nvme/host/pci.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +5 -1
@@ -2051,6 +2051,7 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues) .priv = dev, };unsigned int irq_queues, this_p_queues; +unsigned int nr_cpus =num_possible_cpus ();/* * Poll queues don't need interrupts, but we need at least one IO
@@ -2061,7 +2062,10 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues) this_p_queues = nr_io_queues -1 ; irq_queues =1 ; }else { - irq_queues = nr_io_queues - this_p_queues +1 ; +if (nr_cpus < nr_io_queues - this_p_queues) + irq_queues = nr_cpus +1 ; +else + irq_queues = nr_io_queues - this_p_queues +1 ; } dev->io_queues[HCTX_TYPE_POLL ] = this_p_queues;
Signature · 1 line
2.21.0