nvme-pci: check CQ after batch submission for Microsoft device
v3 cover nvme-pci: check CQ after batch submission for Microsoft device
Hi,
The two patches fix one performance regression on Microsoft Corporation device. The root cause is that Microsoft device applies aggressive interrupt coalescing, so single job fio performance drops much after we removes checking cq in f9dde187fa92("nvme-pci: remove cq check after submission").
Turns out this issue is very specific on Microsoft device, so add a quirk for checking CQ on this device.
V3:
- replace spin_trylock_irqsave with spin_trylock_irq
- fix comment on cq_lock & cq_poll_lock
V2:
- only check CQ for Microsoft device.
Ming Lei (2):
nvme-pci: move sq/cq_poll lock initialization into nvme_init_queue
nvme-pci: check CQ after batch submission for Microsoft device
drivers/nvme/host/nvme.h | 6 +++ drivers/nvme/host/pci.c | 98 ++++++++++++++++++++++++++++++++++------ 2 files changed, 90 insertions(+), 14 deletions(-)
Signature · 7 lines
2.20.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v3 1/2 nvme-pci: move sq/cq_poll lock initialization into nvme_init_queue
Prepare for adding new cq lock for improving IO performance in case that device applies aggressive interrupt coalescing, and the new added cq_lock can share space with cq_poll_lock becasue both are mutually exclusive.
Move sq/cq_poll lock initialization into nvme_init_queue so that we may initialize the added cq_lock correctly and lockdep warning can be avoided.
This way is safe because no IO can be started until nvme_init_queue is returned.
--- drivers/nvme/host/pci.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +3 -2
@@ -1481,8 +1481,6 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth) goto free_cqdma; nvmeq->dev = dev; -spin_lock_init (&nvmeq->sq_lock); -spin_lock_init (&nvmeq->cq_poll_lock); nvmeq->cq_head =0 ; nvmeq->cq_phase =1 ; nvmeq->q_db = &dev->dbs[qid *2 * dev->db_stride];
@@ -1516,6 +1514,9 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid) {struct nvme_dev *dev = nvmeq->dev; +spin_lock_init (&nvmeq->sq_lock); +spin_lock_init (&nvmeq->cq_poll_lock); + nvmeq->sq_tail =0 ; nvmeq->last_sq_tail =0 ; nvmeq->cq_head =0 ;
Signature · 7 lines
2.20.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v3 2/2 nvme-pci: check CQ after batch submission for Microsoft device
In reply to Ming Leif9dde187fa92("nvme-pci: remove cq check after submission") removes cq check after submission, this change actually causes performance regression on Microsoft Corporation device in which aggressive interrupt coalescing is used, for example, the max single interrupt per second can reach 40~50K at most.
Checking CQ after submission can improve IO completion for Microsoft Corporation device:
1) the check is very cheap, and lockless check on cq is enough, see nvme_cqe_pending(). Especially the check can be done after batch submission is done. Compared with the big latency added by aggressive interrupt coalescing, cost of the check can be ignored.
2) when IO completion is observed via checking CQ in submission, the requst may be completed without interrupt involved, or the interrupt handler overload can be decreased.
3) when single sw queue is submiting IOs to this hw queue, if IOs completion is observed via this check, the IO can be completed locally, which is cheaper than remote completion.
Follows test result done on Azure L80sv2 guest with NVMe drive( Microsoft Corporation Device b111). This guest has 80 CPUs and 10 numa nodes, and each NVMe drive supports 8 hw queues.
1) test script:
fio --bs=4k --ioengine=libaio --iodepth=64 --filename=/dev/nvme0n1 \ --iodepth_batch_submit=16 --iodepth_batch_complete_min=16 \ --direct=1 --runtime=30 --numjobs=1 --rw=randread \ --name=test --group_reporting --gtod_reduce=1
2) test result:
| v5.3 | v5.3 with this patchset
-------------------------------------------
IOPS | 130K | 424K
Given IO is handled more efficiently in this way, the original report of CPU lockup[1] on Hyper-V can't be observed any more after this patch is applied, since this patch reduces IO completions much in interrupt handler.
[1] https://lore.kernel.org/lkml/[email protected]
--- drivers/nvme/host/nvme.h | 6 +++ drivers/nvme/host/pci.c | 95 ++++++++++++++++++++++++++++++++++------ 2 files changed, 88 insertions(+), 13 deletions(-)
drivers/nvme/host/nvme.h +6 -0
@@ -115,6 +115,12 @@ enum nvme_quirks {* Prevent tag overlap between queues */ NVME_QUIRK_SHARED_TAGS = (1 <<13 ), + +/* +* Check CQ in submission for improving performance since +* the device applies aggressive interrupt coalescing +*/ + NVME_QUIRK_CHECK_CQ = (1 << 14), };/*
drivers/nvme/host/pci.c +82 -13
190 lines not on this page · raw message
Signature · 7 lines
2.20.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerdrivers/nvme/host/pci.c static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, · line 923
Quoted 8 lines · Ming Lei
drivers/nvme/host/pci.c +4
-0@@ -912,6 +917,10 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,blk_mq_start_request (req);nvme_submit_cmd (nvmeq, &cmnd, bd->last); + +if (bd->last &&test_bit (NVMEQ_CHECK_CQ , &nvmeq->flags)) +nvme_check_cq (nvmeq); +
You've given this quirk a special completion handler, how about giving it a special queue_rq as well. Then you can check the completion queue during submission while holding the same lock as before.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 13 lines · Keith Busch
drivers/nvme/host/pci.c static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, · line 923
Quoted 8 lines · Ming Lei
drivers/nvme/host/pci.c +4
-0@@ -912,6 +917,10 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,blk_mq_start_request (req);nvme_submit_cmd (nvmeq, &cmnd, bd->last); + +if (bd->last &&test_bit (NVMEQ_CHECK_CQ , &nvmeq->flags)) +nvme_check_cq (nvmeq); +You've given this quirk a special completion handler, how about giving it a special
queue_rqas well. Then you can check the completion queue during submission while holding the same lock as before.
We can do that, either we have to pass one extra 'check_cq' parameter to common helper or duplicate a nvme_queue_rq_quick(). Not sure what benefit we can get by this way, given test_bit(nvmeq->flags) is always close to zero cost.
Also if same lock is held in both submission & completion for the quirk, the lock contention will be increased for the quirk since the lock has to cover command submission, which shouldn't be needed.
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 27 lines · Ming Lei
Hi,
The two patches fix one performance regression on Microsoft Corporation device. The root cause is that Microsoft device applies aggressive interrupt coalescing, so single job fio performance drops much after we removes checking cq in
f9dde187fa92("nvme-pci: remove cq check after submission").Turns out this issue is very specific on Microsoft device, so add a quirk for checking CQ on this device.
V3:
- replacespin_trylock_irqsavewithspin_trylock_irq
- fix comment oncq_lock&cq_poll_lockV2:
- only check CQ for Microsoft device.Ming Lei (2):
nvme-pci: move sq/cq_poll lock initialization intonvme_init_queue
nvme-pci: check CQ after batch submission for Microsoft devicedrivers/nvme/host/nvme.h | 6 +++ drivers/nvme/host/pci.c | 98 ++++++++++++++++++++++++++++++++++------ 2 files changed, 90 insertions(+), 14 deletions(-)
Quoted 8 lines
Cc Keith Busch <[email protected]>Cc Jens Axboe <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Long Li <[email protected]>Signature · 1 line
2.20.1
Hi Guys,
Ping...
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerQuoted 3 lines · Ming Lei
Hi Guys,
Ping...
I think everyone has told you that it is an invasive horrible hack that papers of the underlying problem(s). I'm not sure what more we can do here.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Ming Lei
Hi Guys,
Ping...
I think everyone has told you that it is an invasive horrible hack that papers of the underlying problem(s). I'm not sure what more we can do here.
The problem is that the NVMe driver applies aggressive interrupt coalescing clearly, that can be addressed exactly by this approach.
Also I don't understand why you think this patch is invasive and horrible.
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 4 lines · Ming Lei
Quoted 3 lines · Christoph Hellwig
Quoted 2 lines · Ming Lei
Hi Guys,
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Ping...
I think everyone has told you that it is an invasive horrible hack that papers of the underlying problem(s). I'm not sure what more we can do here.
The problem is that the NVMe driver applies aggressive interrupt coalescing clearly, that can be addressed exactly by this approach.
Can this default coalescing setting be turned off with a "set feature" command?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 11 lines · Keith Busch
Quoted 10 lines · Ming Lei
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Ming Lei
Hi Guys,
Ping...
I think everyone has told you that it is an invasive horrible hack that papers of the underlying problem(s). I'm not sure what more we can do here.
Quoted 6 lines
Quoted 2 lines
The problem is that the NVMe driver applies aggressive interrupt coalescing clearly, that can be addressed exactly by this approach.
Can this default coalescing setting be turned off with a "set feature" command?
At default, 'get feature -f 0x8' shows zero, and nothing changes after running 'set feature -f 0x8 -v 0'.
BTW, soft lockup from another Samsung NVMe can be fixed by this patch too. I am confirming if the Samsung NVMe applies aggressive interrupt coalescing too.
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerQuoted 10 lines · Ming Lei
Quoted 3 lines
Can this default coalescing setting be turned off with a "set feature" command?
At default, 'get feature -f
0x8' shows zero, and nothing changes after running 'set feature -f0x8-v 0'.BTW, soft lockup from another Samsung NVMe can be fixed by this patch too. I am confirming if the Samsung NVMe applies aggressive interrupt coalescing too.
I think we are missing up a few things here, and just polling the completion queue from submission context isn't the right answer for either.
The aggressive interrupt coalescing and resulting long run times of the irq handler really just means we need to stop processing them from hard irq context at all. NVMe already has support for threaded interrupts and we need to make that the default (and possibly even the only option). At that point we can do a cond_resched() in this handler to avoid soft lockups.
The next problem is drivers with less completion queues than cpu cores, as that will still overload the one cpu that the interrupt handler was assigned to. A dumb fix would be a cpu mask for the threaded interrupt handler that can be used for round robin scheduling, but that probably won't help with getting good performance. The other idea is to use "virtual" completion queues. NVMe allows free form command ids, so we could OR an index for the relative cpu number inside this queue into the command id and and then create one interrupt thread for each of them. Although I'd like to hear from Thomas on what he thinks of multiple threads per hardirq first.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 16 lines · Christoph Hellwig
Quoted 10 lines · Ming Lei
Quoted 3 lines
Can this default coalescing setting be turned off with a "set feature" command?
At default, 'get feature -f
0x8' shows zero, and nothing changes after running 'set feature -f0x8-v 0'.BTW, soft lockup from another Samsung NVMe can be fixed by this patch too. I am confirming if the Samsung NVMe applies aggressive interrupt coalescing too.
I think we are missing up a few things here, and just polling the completion queue from submission context isn't the right answer for either.
Quoted 6 lines
The aggressive interrupt coalescing and resulting long run times of the irq handler really just means we need to stop processing them from hard irq context at all. NVMe already has support for threaded interrupts and we need to make that the default (and possibly even the only option). At that point we can do a
cond_resched()in this handler to avoid soft lockups.
I am pretty sure that threaded interrupt will cause performance drop a lot, and Long has verified that.
Quoted 2 lines
The next problem is drivers with less completion queues than cpu cores,
IRQ matrix has balanced interrupt load among CPUs already. For example, on Azure, 8 cpu cores can be mapped to one hctx, however, there are only several CPUs which handles interrupts from at most two queues.
Quoted 2 lines
as that will still overload the one cpu that the interrupt handler was assigned to. A dumb fix would be a cpu mask for the threaded interrupt
Actually one CPU is fast enough to handle several drive's interrupt handling. Also there is per-queue depth limit, and the interrupt flood issue in network can't be serious on storage.
So far I only got three NVMe softlock tickets, two of them can be fixed by this patch, and another one turns out a IOMMU lock issue. Are there other NVMe softlock report?
At least Azure's NVMe applies aggressive interrupt coalescing, in which the softlock is usually triggered by interrupt delay handling, then there can be lots of requests to be handled in single interrupt handler some times. It doesn't mean CPU is saturated, and it is just sort of interrupt peak. And it is very specific with Azure's implementation.
Otherwise, this simple patch can't fix the issue.
Quoted 7 lines
handler that can be used for round robin scheduling, but that probably won't help with getting good performance. The other idea is to use "virtual" completion queues. NVMe allows free form command ids, so we could OR an index for the relative cpu number inside this queue into the command id and and then create one interrupt thread for each of them. Although I'd like to hear from Thomas on what he thinks of multiple threads per hardirq first.
As mentioned above, there isn't proof it is caused by interrupt loading saturating the CPU, so not sure we need to re-invent a wheel.
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 6 lines · Ming Lei
Quoted 2 lines
as that will still overload the one cpu that the interrupt handler was assigned to. A dumb fix would be a cpu mask for the threaded interrupt
Actually one CPU is fast enough to handle several drive's interrupt handling. Also there is per-queue depth limit, and the interrupt flood issue in network can't be serious on storage.
This is true today, but it won't be true in the future. Lets aim for a solution that's a little more future proof than just "enough today", if we're going to make changes in this area.
Signature · 6 lines
Jens Axboe _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 11 lines · Jens Axboe
Quoted 6 lines · Ming Lei
Quoted 2 lines
as that will still overload the one cpu that the interrupt handler was assigned to. A dumb fix would be a cpu mask for the threaded interrupt
Actually one CPU is fast enough to handle several drive's interrupt handling. Also there is per-queue depth limit, and the interrupt flood issue in network can't be serious on storage.
This is true today, but it won't be true in the future. Lets aim for a solution that's a little more future proof than just "enough today", if we're going to make changes in this area.
That should be a new feature for future hardware, and we don't know any performance details, and it can be hard to prepare for it now. Maybe such hardware or case never comes:
- storage device has queue depth, which limits the max in-flight requests to be handled in each queue's interrupt handler.
- Suppose such fast hardware comes, it isn't reasonable for them to support N:1 mapping(N is big).
- Also IRQ matrix has balanced interrupt handling loading already, that said most of times, one CPU is just responsible for handing one hw queue's interrupt. Even in Azure's case, 8 CPUs are mapped to one hw queue, but there is just several CPUs which is for responsible for at most 2 hw queues.
So could we focus on now and fix the regression first?
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 9 lines · Ming Lei
Quoted 8 lines · Jens Axboe
Quoted 6 lines · Ming Lei
Quoted 2 lines
as that will still overload the one cpu that the interrupt handler was assigned to. A dumb fix would be a cpu mask for the threaded interrupt
Actually one CPU is fast enough to handle several drive's interrupt handling. Also there is per-queue depth limit, and the interrupt flood issue in network can't be serious on storage.
Quoted 7 lines
Quoted 3 lines
This is true today, but it won't be true in the future. Lets aim for a solution that's a little more future proof than just "enough today", if we're going to make changes in this area.
That should be a new feature for future hardware, and we don't know any performance details, and it can be hard to prepare for it now. Maybe such hardware or case never comes:
Oh it'll surely come, and maybe sooner than you think. My point is that using "one CPU is fast enough to handle several drive interrupts" is very shortsighted, and probably not even true today.
Quoted 2 lines
- storage device has queue depth, which limits the max in-flight requests to be handled in each queue's interrupt handler.
Only if new requests aren't also coming in and completing while you are doing that work.
Quoted 3 lines
- Suppose such fast hardware comes, it isn't reasonable for them to support N:1 mapping(N is big).
Very true, in fact that's already pretty damn dumb today...
Quoted 6 lines
- Also IRQ matrix has balanced interrupt handling loading already, that said most of times, one CPU is just responsible for handing one hw queue's interrupt. Even in Azure's case, 8 CPUs are mapped to one hw queue, but there is just several CPUs which is for responsible for at most 2 hw queues.
So could we focus on now and fix the regression first?
As far as I could tell from the other message, sounds like they both have broken interrupt coalescing? Makes it harder to care, honestly...
But yes, I think we should do something about this. This really isn't a new issue, if a core gets overloaded just doing completions from interrupts, we should punt the work. NAPI has been doing that for ages, and the block layer also used to have support it, but nobody used it. Would be a great idea to make a blk-mq friendly version of that, with the kinds of IOPS and latencies in mind that we see today and in the coming years. I don't think hacking around this in the nvme driver is a very good way to go about it.
Signature · 6 lines
Jens Axboe _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 14 lines · Jens Axboe
Quoted 13 lines · Ming Lei
Quoted 11 lines · Jens Axboe
Quoted 6 lines · Ming Lei
Quoted 2 lines
as that will still overload the one cpu that the interrupt handler was assigned to. A dumb fix would be a cpu mask for the threaded interrupt
Actually one CPU is fast enough to handle several drive's interrupt handling. Also there is per-queue depth limit, and the interrupt flood issue in network can't be serious on storage.
This is true today, but it won't be true in the future. Lets aim for a solution that's a little more future proof than just "enough today", if we're going to make changes in this area.
Quoted 7 lines
Quoted 3 lines
That should be a new feature for future hardware, and we don't know any performance details, and it can be hard to prepare for it now. Maybe such hardware or case never comes:
Oh it'll surely come, and maybe sooner than you think. My point is that using "one CPU is fast enough to handle several drive interrupts" is very shortsighted, and probably not even true today.
That single CPU is responsible for handling more than one drives should only happen in case that the following condition is true:
nr_drives * nr_io_hw_queue > nr_cpus
Quoted 11 lines
Quoted 3 lines
- storage device has queue depth, which limits the max in-flight requests to be handled in each queue's interrupt handler.
Only if new requests aren't also coming in and completing while you are doing that work.
Quoted 3 lines
- Suppose such fast hardware comes, it isn't reasonable for them to support N:1 mapping(N is big).
Very true, in fact that's already pretty damn dumb today...
OK, I guess it is because lots of NVMe only supports limited hw queues(32).
Quoted 5 lines
Quoted 5 lines
- Also IRQ matrix has balanced interrupt handling loading already, that said most of times, one CPU is just responsible for handing one hw queue's interrupt. Even in Azure's case, 8 CPUs are mapped to one hw queue, but there is just several CPUs which is for responsible for at most 2 hw queues.
It also depends on how many drives are used on single machine. The issue is possible only when the number of drives is big enough. I guess it isn't unusual.
Quoted 5 lines
Quoted 2 lines
So could we focus on now and fix the regression first?
As far as I could tell from the other message, sounds like they both have broken interrupt coalescing? Makes it harder to care, honestly...
Yeah, I found two reports on two different drives, both can be fixed by this patch. Not see other reports which is caused by too much interrupt loading on single CPU. That is why I tried to avoid generic approach...
Quoted 9 lines
But yes, I think we should do something about this. This really isn't a new issue, if a core gets overloaded just doing completions from interrupts, we should punt the work. NAPI has been doing that for ages, and the block layer also used to have support it, but nobody used it. Would be a great idea to make a blk-mq friendly version of that, with the kinds of IOPS and latencies in mind that we see today and in the coming years. I don't think hacking around this in the nvme driver is a very good way to go about it.
OK, I will look at this approach, and Sagi has posted one such patch.
thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme