nvme pci interrupt handling improvements
Keith Busch
maintainerv3 cover nvme pci interrupt handling improvements
Here's the next revision to nvme pci interrupt handling. The series is attempting to address two issues that have been recently raised:
1. The nvme driver makes it possible to soft lockup a CPU due to high utilization in irq context. This is most prevalent when multiple CPUs are driving a single hardware queue on one or more controllers.
2. The threaded handler left interrupts unmasked, which breaks when used with level triggered interrupts, or unnecessarily runs in interrupt context when edge interrupts occur frequently.
Both issues are addressed by always configuring nvme interrupts to run the threaded handler with interrupts disabled. A hybrid approch to handling nvme completions in hard irq context and thread context is introduced so that there should not be a performance impact from removing the nvme.use_threaded_interrupts option.
The series appears to be a win or no impact on performance from what I can test. I would be greatful to hear if others can confirm this with other hardware.
I've dropped the fast MSIx masking. While I believe it's safe to skip flushing the memory write, I think this series mitigates the impact of the read back by ensuring the ratio of memory reads to IO is low enough to be negligable (AFAICT on hardware available to me).
I've changed the exit condition for the polling nvme irq thread to break out of the loop if we've wrapped the completion queue. Other irq threads may be affinitized to the same CPU, so we need to schedule out at some point, but I've been told multiple times that need_resched() or cond_resched() won't work as desired from the thread's fifo priority.
Keith Busch (4):
nvme/pci: Disable interrupts for threaded handler nvme/pci: Complete commands from primary handler nvme/pci: Remove use_threaded_interrupts nvme/pci: Poll threaded completions
drivers/nvme/host/pci.c | 56 ++++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 20 deletions(-)
Signature · 7 lines
2.21.0 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerPATCH v3 2/4 nvme/pci: Complete commands from primary handler
The context switch to wake the threaded interrupt handler has enough overhead to harm latency for low queue depth workloads. Complete one cycle through the completion queue in the primary handler to ensure threaded interrupts have no disadvantage compared to not using threaded interrupts. Wake the thread only if more completions remain after processing the queue once since that indicates a higher queue depth workload that may trigger frequent interrupts which would be better completed in the handler thread with device interrupts disabled.
Completing requests from two different contexts is safe since nvme driver disables entry to the primary handler when it returns IRQ_WAKE_THREAD.
--- drivers/nvme/host/pci.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/nvme/host/pci.c +4 -0
@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)if (!nvme_cqe_pending (nvmeq))return IRQ_NONE ; +nvme_irq (irq, data); +if (!nvme_cqe_pending (nvmeq)) +return IRQ_HANDLED ; +if (!to_pci_dev (nvmeq->dev->dev)->msix_enabled)writel (1 << nvmeq->cq_vector, nvmeq->dev->bar +NVME_REG_INTMS );
Signature · 7 lines
2.21.0 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerPATCH v3 1/4 nvme/pci: Disable interrupts for threaded handler
In reply to Keith BuschThe nvme driver had always left the device in a state capable of sending more interrupts while completions were being handled in the thread. Re-entering the primary handler disrupts the threaded handler since they run on the same CPU. The primary handler may also be detected as spurious when it returns IRQ_WAKE_THREAD if the threaded handler does not finish frequently enough due to completing many requests.
Disable the irq to ensure the primary handler will not be re-entered while the threaded handler is running. Use the nvme specific interrupt mask set/clear registers for MSI and legacy, and rely on the irq_flow_handler to mask the interrupt for MSIx.
--- drivers/nvme/host/pci.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-)
drivers/nvme/host/pci.c +22 -4
@@ -1036,12 +1036,30 @@ static irqreturn_t nvme_irq(int irq, void *data)return ret; } +static irqreturn_t nvme_irq_thread (int irq,void *data) +{ +struct nvme_queue *nvmeq = data; +irqreturn_t ret =nvme_irq (irq, data); + +if (!to_pci_dev (nvmeq->dev->dev)->msix_enabled) +writel (1 << nvmeq->cq_vector, nvmeq->dev->bar +NVME_REG_INTMC ); + +enable_irq (irq); +return ret; +} +static irqreturn_t nvme_irq_check (int irq,void *data) {struct nvme_queue *nvmeq = data; -if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; + +if (!nvme_cqe_pending (nvmeq)) +return IRQ_NONE ; + +if (!to_pci_dev (nvmeq->dev->dev)->msix_enabled) +writel (1 << nvmeq->cq_vector, nvmeq->dev->bar +NVME_REG_INTMS ); + +disable_irq_nosync (irq); +return IRQ_WAKE_THREAD ; }/*
@@ -1499,7 +1517,7 @@ static int queue_request_irq(struct nvme_queue *nvmeq)if (use_threaded_interrupts) {return pci_request_irq (pdev, nvmeq->cq_vector, nvme_irq_check, - nvme_irq, nvmeq,"nvme%dq%d" , nr, nvmeq->qid); + nvme_irq_thread, nvmeq,"nvme%dq%d" , nr, nvmeq->qid); }else {return pci_request_irq (pdev, nvmeq->cq_vector, nvme_irq,NULL , nvmeq,"nvme%dq%d" , nr, nvmeq->qid);
Signature · 7 lines
2.21.0 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerPATCH v3 3/4 nvme/pci: Remove use_threaded_interrupts
In reply to Keith BuschThe nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the use_threaded_interrupts nvme module parameter since leaving it disabled should not be providing a benefit.
--- drivers/nvme/host/pci.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
drivers/nvme/host/pci.c +2 -10
@@ -40,9 +40,6 @@#define NVME_MAX_KB_SZ 4096#define NVME_MAX_SEGS 127 -static int use_threaded_interrupts; -module_param (use_threaded_interrupts, int,0 ); -static bool use_cmb_sqes = true;module_param (use_cmb_sqes, bool,0444 );MODULE_PARM_DESC (use_cmb_sqes,"use controller's memory buffer for I/O SQes" );
@@ -1519,13 +1516,8 @@ static int queue_request_irq(struct nvme_queue *nvmeq)struct pci_dev *pdev =to_pci_dev (nvmeq->dev->dev);int nr = nvmeq->dev->ctrl.instance; -if (use_threaded_interrupts) { -return pci_request_irq (pdev, nvmeq->cq_vector, nvme_irq_check, - nvme_irq_thread, nvmeq,"nvme%dq%d" , nr, nvmeq->qid); - }else { -return pci_request_irq (pdev, nvmeq->cq_vector, nvme_irq, -NULL , nvmeq,"nvme%dq%d" , nr, nvmeq->qid); - } +return pci_request_irq (pdev, nvmeq->cq_vector, nvme_irq_check, + nvme_irq_thread, nvmeq,"nvme%dq%d" , nr, nvmeq->qid); }static void nvme_init_queue (struct nvme_queue *nvmeq,u16 qid)
Signature · 7 lines
2.21.0 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerPATCH v3 4/4 nvme/pci: Poll threaded completions
In reply to Keith BuschPoll for new completions in the nvme irq thread as long as it continues to observe new ones are available.
We need to ensure other irq threads affinitized to the same CPU will get a chance to run, so exit polling if we wrap the completion queue.
--- drivers/nvme/host/pci.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
drivers/nvme/host/pci.c +12 -10
@@ -1011,38 +1011,40 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start,return found; } -static irqreturn_t nvme_irq (int irq,void *data) +static int nvme_irq (int irq,void *data) {struct nvme_queue *nvmeq = data; -irqreturn_t ret =IRQ_NONE ;u16 start, end; +int found;/* * The rmb/wmb pair ensures we see all updates from a previous run of * the irq handler, even if that was on another CPU. */ rmb (); -nvme_process_cq (nvmeq, &start, &end,-1 ); + found =nvme_process_cq (nvmeq, &start, &end,-1 );wmb (); -if (start != end) { +if (start != end)nvme_complete_cqes (nvmeq, start, end); -return IRQ_HANDLED ; - } -return ret; +return found; }static irqreturn_t nvme_irq_thread (int irq,void *data) {struct nvme_queue *nvmeq = data; -irqreturn_t ret =nvme_irq (irq, data); +int found, total =0 ; + +do { + found =nvme_irq (irq, data); + total += found; + }while (found && total < nvmeq->q_depth *2 );if (!to_pci_dev (nvmeq->dev->dev)->msix_enabled)writel (1 << nvmeq->cq_vector, nvmeq->dev->bar +NVME_REG_INTMC ); -enable_irq (irq); -return ret; +return IRQ_HANDLED ; }static irqreturn_t nvme_irq_check (int irq,void *data)
Signature · 7 lines
2.21.0 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
In reply to Keith BuschHi Keith,
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056
Quoted 9 lines · Keith Busch
static
irqreturn_tnvme_irq_check(int irq, void *data){drivers/nvme/host/pci.c +3
-3struct nvme_queue *nvmeq = data; -if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; + +if (!nvme_cqe_pending (nvmeq)) +return IRQ_NONE ;
IIRC, returning IRQ_NONE tells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See __handle_irq_event_percpu, handle_irq_event_percpu and note_interrupt
Thanks,
Daniel
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
Quoted 17 lines · Daniel Wagner
Hi Keith,
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056
Quoted 9 lines · Keith Busch
static
irqreturn_tnvme_irq_check(int irq, void *data){drivers/nvme/host/pci.c +3
-3struct nvme_queue *nvmeq = data; -if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; + +if (!nvme_cqe_pending (nvmeq)) +return IRQ_NONE ;IIRC, returning
IRQ_NONEtells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See__handle_irq_event_percpu,handle_irq_event_percpuandnote_interrupt
This is okay / intended. If nvme_cqe_pending() does not recognise that this interrupt belongs to that device then it should return IRQ_NONE.
Quoted 2 lines
Thanks,
Daniel
Sebastian
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
Quoted 17 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Daniel Wagner
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056
Quoted 9 lines · Keith Busch
static
irqreturn_tnvme_irq_check(int irq, void *data){drivers/nvme/host/pci.c +3
-3struct nvme_queue *nvmeq = data; -if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; + +if (!nvme_cqe_pending (nvmeq)) +return IRQ_NONE ;IIRC, returning
IRQ_NONEtells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See__handle_irq_event_percpu,handle_irq_event_percpuandnote_interrupt
Quoted 2 lines
This is okay / intended. If
nvme_cqe_pending()does not recognise that this interrupt belongs to that device then it should returnIRQ_NONE.
Also note this patch doesn't change the condition when IRQ_NONE is returned. We've been doing that since nvme's beginning, as it really is a spurious interrupt from a likely malfunctioning device.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
Quoted 18 lines · Keith Busch
Quoted 17 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Daniel Wagner
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056
Quoted 9 lines · Keith Busch
static
irqreturn_tnvme_irq_check(int irq, void *data){drivers/nvme/host/pci.c +3
-3struct nvme_queue *nvmeq = data; -if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; + +if (!nvme_cqe_pending (nvmeq)) +return IRQ_NONE ;IIRC, returning
IRQ_NONEtells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See__handle_irq_event_percpu,handle_irq_event_percpuandnote_interrupt
Quoted 6 lines
Quoted 2 lines
This is okay / intended. If
nvme_cqe_pending()does not recognise that this interrupt belongs to that device then it should returnIRQ_NONE.Also note this patch doesn't change the condition when
IRQ_NONEis returned. We've been doing that since nvme's beginning, as it really is a spurious interrupt from a likely malfunctioning device.
Sure, I am just a bit confused how IRQ_NONE is used. The comment on nvme_cqe_pending says "We read the CQE phase first to check if the rest of the entry is valid", which I read as "check if we have anything to do" and not "this irq doesn't belong to this queue".
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
Quoted 22 lines · Daniel Wagner
Quoted 21 lines · Keith Busch
Quoted 19 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Daniel Wagner
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056
Quoted 9 lines · Keith Busch
static
irqreturn_tnvme_irq_check(int irq, void *data){drivers/nvme/host/pci.c +3
-3struct nvme_queue *nvmeq = data; -if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; + +if (!nvme_cqe_pending (nvmeq)) +return IRQ_NONE ;IIRC, returning
IRQ_NONEtells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See__handle_irq_event_percpu,handle_irq_event_percpuandnote_interruptThis is okay / intended. If
nvme_cqe_pending()does not recognise that this interrupt belongs to that device then it should returnIRQ_NONE.
Quoted 8 lines
Quoted 3 lines
Also note this patch doesn't change the condition when
IRQ_NONEis returned. We've been doing that since nvme's beginning, as it really is a spurious interrupt from a likely malfunctioning device.Sure, I am just a bit confused how
IRQ_NONEis used. The comment onnvme_cqe_pendingsays "We read the CQE phase first to check if the rest of the entry is valid", which I read as "check if we have anything to do" and not "this irq doesn't belong to this queue".
Those two interpretations sound like the same thing to me. :)
The interrupt notifies the driver it has something to do, and reading the CQE phase confirms whether or not that's true.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
Quoted 7 lines · Keith Busch
Quoted 4 lines · Daniel Wagner
Sure, I am just a bit confused how
IRQ_NONEis used. The comment onnvme_cqe_pendingsays "We read the CQE phase first to check if the rest of the entry is valid", which I read as "check if we have anything to do" and not "this irq doesn't belong to this queue".Those two interpretations sound like the same thing to me. :)
:)
I was confused why 'nothing to do' is then interpreted as IRQ was routed wrongly.
Quoted 2 lines
The interrupt notifies the driver it has something to do, and reading the CQE phase confirms whether or not that's true.
I think the thing I am missing the design idea behind it. I suppose the IRQ on only fires when previously work was posted.
Sorry for the noise.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 4/4] nvme/pci: Poll threaded completions
In reply to Keith BuschHi Keith,
Quoted 25 lines · Keith Busch
Poll for new completions in the nvme irq thread as long as it continues to observe new ones are available.
We need to ensure other irq threads affinitized to the same CPU will get a chance to run, so exit polling if we wrap the completion queue.
Signed-off-by Keith Busch <[email protected]>--- drivers/nvme/host/pci.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)drivers/nvme/host/pci.c +1
-2@@ -1011,38 +1011,40 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start,return found; } -static irqreturn_t nvme_irq (int irq,void *data) +static int nvme_irq (int irq,void *data) {struct nvme_queue *nvmeq = data; -irqreturn_t ret =IRQ_NONE ;u16 start, end;
drivers/nvme/host/pci.c static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, · line 1026
Quoted 10 lines
drivers/nvme/host/pci.c +2
-1+int found;/* * The rmb/wmb pair ensures we see all updates from a previous run of * the irq handler, even if that was on another CPU. */ rmb (); -nvme_process_cq (nvmeq, &start, &end,-1 ); + found =nvme_process_cq (nvmeq, &start, &end,-1 );wmb ();
Are the memory barries still at the right place after this change?
Thanks,
Daniel
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
In reply to Keith Buschdrivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1061
Quoted 13 lines
drivers/nvme/host/pci.c +4
-0@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)if (!nvme_cqe_pending (nvmeq))return IRQ_NONE ; +nvme_irq (irq, data); +if (!nvme_cqe_pending (nvmeq)) +return IRQ_HANDLED ; +
Question Keith,
If I have say 24 (or more) devices with a queue mapped to a cpu, and we
happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
Quoted 15 lines · Sagi Grimberg
drivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1060
Quoted 12 lines
drivers/nvme/host/pci.c +4
-0@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)if (!nvme_cqe_pending (nvmeq))return IRQ_NONE ; +nvme_irq (irq, data); +if (!nvme_cqe_pending (nvmeq)) +return IRQ_HANDLED ; +Question Keith,
Quoted 3 lines
If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?
I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
Quoted 11 lines
Quoted 6 lines
Question Keith,
If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?
I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return
IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.
Theoretically speaking, even if you limit to 1 cqe, the universe can align such that you will always reap in the primary handler right? The more devices we have in the system mapping a vector to the same cpu core will increase the likelihood of this happening.
I guess it is a corner-case, but I'm thinking we want to be a 100% covered. Usually what drivers do is defer to a kthread/tasklet/napi context without reaping in the primary handler.
So if we have this optimization, perhaps something else in the irq infrastructure should take care of cpu lockup prevention?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
Quoted 11 lines · Sagi Grimberg
Quoted 8 lines
Quoted 3 lines
If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?
I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return
IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.Theoretically speaking, even if you limit to 1 cqe, the universe can align such that you will always reap in the primary handler right?
Perhaps theoretically, though testing the limits of reason.
Quoted 2 lines
So if we have this optimization, perhaps something else in the irq infrastructure should take care of cpu lockup prevention?
Perhaps we can cycle the effective_affinity through the smp_affinity?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?
I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return
IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.
Quoted 4 lines
Quoted 2 lines
Theoretically speaking, even if you limit to 1 cqe, the universe can align such that you will always reap in the primary handler right?
Perhaps theoretically, though testing the limits of reason.
I know, maybe we should handle it when/if this really becomes a problem...
Quoted 4 lines
Quoted 2 lines
So if we have this optimization, perhaps something else in the irq infrastructure should take care of cpu lockup prevention?
Perhaps we can cycle the
effective_affinitythrough thesmp_affinity?
Not sure I follow your thoughts.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
Quoted 3 lines · Sagi Grimberg
Quoted 1 line
Perhaps we can cycle the
effective_affinitythrough thesmp_affinity?Not sure I follow your thoughts.
The only way the nvme driver's interrupt handler can saturate a cpu requires the smp_affinity have multiple online cpus set. The effective_affinity, however, is always a single cpu, which can be any of the online ones set in smp_affinity.
If we can detect a cpu is spending too many cycles in nvme irq, perhaps we can re-spread the effective_affinity to different cpus. I don't currently know of a good way to do this, though.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerRe: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
In reply to Keith Buschdrivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1061
Quoted 4 lines · Keith Busch
drivers/nvme/host/pci.c +4
-0+if (!to_pci_dev (nvmeq->dev->dev)->msix_enabled) +writel (1 << nvmeq->cq_vector, nvmeq->dev->bar +NVME_REG_INTMS ); + +disable_irq_nosync (irq);
Why do we need both the device-level disable and disable_irq_nosync for the non-MSI-X case here?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
In reply to Keith BuschQuoted 22 lines · Keith Busch
The context switch to wake the threaded interrupt handler has enough overhead to harm latency for low queue depth workloads. Complete one cycle through the completion queue in the primary handler to ensure threaded interrupts have no disadvantage compared to not using threaded interrupts. Wake the thread only if more completions remain after processing the queue once since that indicates a higher queue depth workload that may trigger frequent interrupts which would be better completed in the handler thread with device interrupts disabled.
Completing requests from two different contexts is safe since nvme driver disables entry to the primary handler when it returns
IRQ_WAKE_THREAD.Signed-off-by Keith Busch <[email protected]>--- drivers/nvme/host/pci.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/nvme/host/pci.c +0
-0
drivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1058
Quoted 4 lines
drivers/nvme/host/pci.c +1
-0@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)if (!nvme_cqe_pending (nvmeq))return IRQ_NONE ; +nvme_irq (irq, data);
I think we can just open code nvme_irq here. That also means we can use the return value of nvme_process_cq instead of the extra nvme_cqe_pending check above. This also needs a good comment explaining the scheme at least.
drivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1060
Quoted 2 lines
drivers/nvme/host/pci.c +2
-0+if (!nvme_cqe_pending (nvmeq)) +return IRQ_HANDLED ;
Including one explaining this check.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerRe: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts
In reply to Keith BuschQuoted 4 lines · Keith Busch
The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the
use_threaded_interruptsnvme module parameter since leaving it disabled should not be providing a benefit.
I think we need some careful benchmarking and numbers to justify the switch.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch
The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the
use_threaded_interruptsnvme module parameter since leaving it disabled should not be providing a benefit.I think we need some careful benchmarking and numbers to justify the switch.
Yes, I don't consider this 5.5 material, so we've some time.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler
In reply to Christoph HellwigQuoted 8 lines · Christoph Hellwig
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1061
Quoted 4 lines · Keith Busch
drivers/nvme/host/pci.c +4
-0+if (!to_pci_dev (nvmeq->dev->dev)->msix_enabled) +writel (1 << nvmeq->cq_vector, nvmeq->dev->bar +NVME_REG_INTMS ); + +disable_irq_nosync (irq);Why do we need both the device-level disable and
disable_irq_nosyncfor the non-MSI-X case here?
There's a race condition that needs to be handled here. We mask the interrupt on the device, but the device may have sent an MSI at the same time that the APIC hasn't seen yet, and that would retrigger the interrupt handler. We need it to see irqd_irq_disabled() is true, otherwise the nvme primary and thread may run at the same time, which we don't want in order to poll the cq from both.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
In reply to Keith BuschQuoted 7 lines · Keith Busch
Quoted 3 lines · Sagi Grimberg
Quoted 1 line
Perhaps we can cycle the
effective_affinitythrough thesmp_affinity?Not sure I follow your thoughts.
The only way the nvme driver's interrupt handler can saturate a cpu requires the
smp_affinityhave multiple online cpus set. The
As Sagi mentioned, there can be 24 drives, each drive can be 1:1 mapping, so each CPU may have to handle interrupts from 24 drives/queues, then the CPU may become slower compared with the 24 hardware/queues.
Quoted 7 lines
effective_affinity, however, is always a single cpu, which can be any of the online ones set insmp_affinity.If we can detect a cpu is spending too many cycles in nvme irq, perhaps we can re-spread the
effective_affinityto different cpus. I don't currently know of a good way to do this, though.
That shouldn't have been done just for nvme irq given other storage has similar issue.
I have tried toward that direction, and looks some cost has to be payed, such as reading one time of local clock in interrupt or io path has to be done:
https://lore.kernel.org/lkml/[email protected]/
It can be quite hard to figure out one efficient way to do that.
thanks,
Ming
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
Quoted 4 lines · Ming Lei
Quoted 3 lines · Keith Busch
Quoted 2 lines · Sagi Grimberg
Quoted 1 line
Perhaps we can cycle the
effective_affinitythrough thesmp_affinity?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Not sure I follow your thoughts.
The only way the nvme driver's interrupt handler can saturate a cpu requires the
smp_affinityhave multiple online cpus set. TheAs Sagi mentioned, there can be 24 drives, each drive can be 1:1 mapping, so each CPU may have to handle interrupts from 24 drives/queues, then the CPU may become slower compared with the 24 hardware/queues.
If it's 1:1 queue:cpu, then it doesn't matter how many devices you have. Processes can't submit commands on that cpu while that cpu is servicing device interrupts, but the device can't send interrupts if processes can't submit commands.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler
Quoted 10 lines · Keith Busch
Quoted 9 lines · Ming Lei
Quoted 7 lines · Keith Busch
Quoted 3 lines · Sagi Grimberg
Quoted 1 line
Perhaps we can cycle the
effective_affinitythrough thesmp_affinity?Not sure I follow your thoughts.
The only way the nvme driver's interrupt handler can saturate a cpu requires the
smp_affinityhave multiple online cpus set. The
Quoted 9 lines
Quoted 3 lines
As Sagi mentioned, there can be 24 drives, each drive can be 1:1 mapping, so each CPU may have to handle interrupts from 24 drives/queues, then the CPU may become slower compared with the 24 hardware/queues.
If it's 1:1 queue:cpu, then it doesn't matter how many devices you have. Processes can't submit commands on that cpu while that cpu is servicing device interrupts, but the device can't send interrupts if processes can't submit commands.
OK, but still not sure it is good to cycle the effective_affinity through the smp_affinity:
1) the top half(primary interrupt handler) still might saturate the CPU
1) task migration isn't cheap from one CPU to another one
2) in theory we only need to do that when one interrupt target CPU is saturated, but changing thread's affinity has to be done beforehand, so this way may hurt performance when the CPU is never saturated. I believe that should be usual case.
Thanks,
Ming
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts
In reply to Christoph HellwigQuoted 8 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch
The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the
use_threaded_interruptsnvme module parameter since leaving it disabled should not be providing a benefit.I think we need some careful benchmarking and numbers to justify the switch.
The patch can fix CPU lockup on Azure's NVMe, however, IOPS drops to ~600K from 3M+.
Thanks,
Ming
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts
Quoted 7 lines · Ming Lei
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch
The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the
use_threaded_interruptsnvme module parameter since leaving it disabled should not be providing a benefit.
Quoted 5 lines
Quoted 2 lines
I think we need some careful benchmarking and numbers to justify the switch.
The patch can fix CPU lockup on Azure's NVMe, however, IOPS drops to ~600K from 3M+.
Compared to current mainline, or is that 3M+ with the poll completions from submission?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts
Quoted 8 lines · Keith Busch
Quoted 7 lines · Ming Lei
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch
The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the
use_threaded_interruptsnvme module parameter since leaving it disabled should not be providing a benefit.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I think we need some careful benchmarking and numbers to justify the switch.
The patch can fix CPU lockup on Azure's NVMe, however, IOPS drops to ~600K from 3M+.
Compared to current mainline, or is that 3M+ with the poll completions from submission?
3M+ can be reached in mainline without poll completions from submission, which only improves IOPS in single fio job test.
This one is multi-job & multi-drive test:
fio --bs=4k --ioengine=libaio --iodepth=32 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=400 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
Thanks,
Ming
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme