Skip to contentSkip to conversation

nvme pci interrupt handling improvements

PATCH In review Keith Busch 6 participants 29 messages Dec 9, 2019 – Dec 19, 2019 NVMe

lore
Share & tools
lore

Keith Busch

author maintainer
#0 lore raw

v3 cover nvme pci interrupt handling improvements

Here's the next revision to nvme pci interrupt handling. The series is attempting to address two issues that have been recently raised:

1. The nvme driver makes it possible to soft lockup a CPU due to high utilization in irq context. This is most prevalent when multiple CPUs are driving a single hardware queue on one or more controllers.

2. The threaded handler left interrupts unmasked, which breaks when used with level triggered interrupts, or unnecessarily runs in interrupt context when edge interrupts occur frequently.

Both issues are addressed by always configuring nvme interrupts to run the threaded handler with interrupts disabled. A hybrid approch to handling nvme completions in hard irq context and thread context is introduced so that there should not be a performance impact from removing the nvme.use_threaded_interrupts option.

The series appears to be a win or no impact on performance from what I can test. I would be greatful to hear if others can confirm this with other hardware.

I've dropped the fast MSIx masking. While I believe it's safe to skip flushing the memory write, I think this series mitigates the impact of the read back by ensuring the ratio of memory reads to IO is low enough to be negligable (AFAICT on hardware available to me).

I've changed the exit condition for the polling nvme irq thread to break out of the loop if we've wrapped the completion queue. Other irq threads may be affinitized to the same CPU, so we need to schedule out at some point, but I've been told multiple times that need_resched() or cond_resched() won't work as desired from the thread's fifo priority.

Keith Busch (4):
nvme/pci: Disable interrupts for threaded handler nvme/pci: Complete commands from primary handler nvme/pci: Remove use_threaded_interrupts nvme/pci: Poll threaded completions

 drivers/nvme/host/pci.c | 56 ++++++++++++++++++++++++++---------------
 1 file changed, 36 insertions(+), 20 deletions(-)
Signature · 7 lines
2.21.0


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author maintainer
#10 lore raw

PATCH v3 2/4 nvme/pci: Complete commands from primary handler

The context switch to wake the threaded interrupt handler has enough overhead to harm latency for low queue depth workloads. Complete one cycle through the completion queue in the primary handler to ensure threaded interrupts have no disadvantage compared to not using threaded interrupts. Wake the thread only if more completions remain after processing the queue once since that indicates a higher queue depth workload that may trigger frequent interrupts which would be better completed in the handler thread with device interrupts disabled.

Completing requests from two different contexts is safe since nvme driver disables entry to the primary handler when it returns IRQ_WAKE_THREAD.

---
 drivers/nvme/host/pci.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/nvme/host/pci.c +4 -0
index a0138e3ca0b9..e415fadf7331 100644
@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
 	if (!nvme_cqe_pending(nvmeq))
 		return IRQ_NONE;
 
+	nvme_irq(irq, data);
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_HANDLED;
+
 	if (!to_pci_dev(nvmeq->dev->dev)->msix_enabled)
 		writel(1 << nvmeq->cq_vector, nvmeq->dev->bar + NVME_REG_INTMS);
 
Signature · 7 lines
2.21.0


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author maintainer
#1 lore raw

PATCH v3 1/4 nvme/pci: Disable interrupts for threaded handler

In reply to Keith Busch

The nvme driver had always left the device in a state capable of sending more interrupts while completions were being handled in the thread. Re-entering the primary handler disrupts the threaded handler since they run on the same CPU. The primary handler may also be detected as spurious when it returns IRQ_WAKE_THREAD if the threaded handler does not finish frequently enough due to completing many requests.

Disable the irq to ensure the primary handler will not be re-entered while the threaded handler is running. Use the nvme specific interrupt mask set/clear registers for MSI and legacy, and rely on the irq_flow_handler to mask the interrupt for MSIx.

---
 drivers/nvme/host/pci.c | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)
drivers/nvme/host/pci.c +22 -4
index 365a2ddbeaa7..a0138e3ca0b9 100644
@@ -1036,12 +1036,30 @@ static irqreturn_t nvme_irq(int irq, void *data)
 	return ret;
 }
 
+static irqreturn_t nvme_irq_thread(int irq, void *data)
+{
+	struct nvme_queue *nvmeq = data;
+	irqreturn_t ret = nvme_irq(irq, data);
+
+	if (!to_pci_dev(nvmeq->dev->dev)->msix_enabled)
+		writel(1 << nvmeq->cq_vector, nvmeq->dev->bar + NVME_REG_INTMC);
+
+	enable_irq(irq);
+	return ret;
+}
+
 static irqreturn_t nvme_irq_check(int irq, void *data)
 {
 	struct nvme_queue *nvmeq = data;
-	if (nvme_cqe_pending(nvmeq))
-		return IRQ_WAKE_THREAD;
-	return IRQ_NONE;
+
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_NONE;
+
+	if (!to_pci_dev(nvmeq->dev->dev)->msix_enabled)
+		writel(1 << nvmeq->cq_vector, nvmeq->dev->bar + NVME_REG_INTMS);
+
+	disable_irq_nosync(irq);
+	return IRQ_WAKE_THREAD;
 }
 
 /*
@@ -1499,7 +1517,7 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
 
 	if (use_threaded_interrupts) {
 		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
-				nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
+			nvme_irq_thread, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
 	} else {
 		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
 				NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
Signature · 7 lines
2.21.0


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author maintainer
#21 lore raw

PATCH v3 3/4 nvme/pci: Remove use_threaded_interrupts

In reply to Keith Busch

The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the use_threaded_interrupts nvme module parameter since leaving it disabled should not be providing a benefit.

---
 drivers/nvme/host/pci.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
drivers/nvme/host/pci.c +2 -10
index e415fadf7331..28e08c5ab412 100644
@@ -40,9 +40,6 @@
 #define NVME_MAX_KB_SZ	4096
 #define NVME_MAX_SEGS	127
 
-static int use_threaded_interrupts;
-module_param(use_threaded_interrupts, int, 0);
-
 static bool use_cmb_sqes = true;
 module_param(use_cmb_sqes, bool, 0444);
 MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -1519,13 +1516,8 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
 	struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
 	int nr = nvmeq->dev->ctrl.instance;
 
-	if (use_threaded_interrupts) {
-		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
-			nvme_irq_thread, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
-	} else {
-		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
-				NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
-	}
+	return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
+		nvme_irq_thread, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
 }
 
 static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
Signature · 7 lines
2.21.0


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author maintainer
#27 lore raw

PATCH v3 4/4 nvme/pci: Poll threaded completions

In reply to Keith Busch

Poll for new completions in the nvme irq thread as long as it continues to observe new ones are available.

We need to ensure other irq threads affinitized to the same CPU will get a chance to run, so exit polling if we wrap the completion queue.

---
 drivers/nvme/host/pci.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
drivers/nvme/host/pci.c +12 -10
index 28e08c5ab412..487c6b3858fe 100644
@@ -1011,38 +1011,40 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start,
 	return found;
 }
 
-static irqreturn_t nvme_irq(int irq, void *data)
+static int nvme_irq(int irq, void *data)
 {
 	struct nvme_queue *nvmeq = data;
-	irqreturn_t ret = IRQ_NONE;
 	u16 start, end;
+	int found;
 
 	/*
 	 * The rmb/wmb pair ensures we see all updates from a previous run of
 	 * the irq handler, even if that was on another CPU.
 	 */
 	rmb();
-	nvme_process_cq(nvmeq, &start, &end, -1);
+	found = nvme_process_cq(nvmeq, &start, &end, -1);
 	wmb();
 
-	if (start != end) {
+	if (start != end)
 		nvme_complete_cqes(nvmeq, start, end);
-		return IRQ_HANDLED;
-	}
 
-	return ret;
+	return found;
 }
 
 static irqreturn_t nvme_irq_thread(int irq, void *data)
 {
 	struct nvme_queue *nvmeq = data;
-	irqreturn_t ret = nvme_irq(irq, data);
+	int found, total = 0;
+
+	do {
+		found = nvme_irq(irq, data);
+		total += found;
+	} while (found && total < nvmeq->q_depth * 2);
 
 	if (!to_pci_dev(nvmeq->dev->dev)->msix_enabled)
 		writel(1 << nvmeq->cq_vector, nvmeq->dev->bar + NVME_REG_INTMC);
-
 	enable_irq(irq);
-	return ret;
+	return IRQ_HANDLED;
 }
 
 static irqreturn_t nvme_irq_check(int irq, void *data)
Signature · 7 lines
2.21.0


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#2 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

In reply to Keith Busch

Hi Keith,

drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056

Quoted 9 lines · Keith Busch

static irqreturn_t nvme_irq_check(int irq, void *data) {

drivers/nvme/host/pci.c +3 -3
 	struct nvme_queue *nvmeq = data;
-	if (nvme_cqe_pending(nvmeq))
-		return IRQ_WAKE_THREAD;
-	return IRQ_NONE;
+
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_NONE;

IIRC, returning IRQ_NONE tells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See __handle_irq_event_percpu, handle_irq_event_percpu and note_interrupt

Thanks,
Daniel

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#3 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

Quoted 17 lines · Daniel Wagner

Hi Keith,

drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056

Quoted 9 lines · Keith Busch

static irqreturn_t nvme_irq_check(int irq, void *data) {

drivers/nvme/host/pci.c +3 -3
 	struct nvme_queue *nvmeq = data;
-	if (nvme_cqe_pending(nvmeq))
-		return IRQ_WAKE_THREAD;
-	return IRQ_NONE;
+
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_NONE;

IIRC, returning IRQ_NONE tells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See __handle_irq_event_percpu, handle_irq_event_percpu and note_interrupt

This is okay / intended. If nvme_cqe_pending() does not recognise that this interrupt belongs to that device then it should return IRQ_NONE.

Quoted 2 lines

Thanks,
Daniel

Sebastian

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#4 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

Quoted 17 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Daniel Wagner

drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056

Quoted 9 lines · Keith Busch

static irqreturn_t nvme_irq_check(int irq, void *data) {

drivers/nvme/host/pci.c +3 -3
 	struct nvme_queue *nvmeq = data;
-	if (nvme_cqe_pending(nvmeq))
-		return IRQ_WAKE_THREAD;
-	return IRQ_NONE;
+
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_NONE;

IIRC, returning IRQ_NONE tells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See __handle_irq_event_percpu, handle_irq_event_percpu and note_interrupt

Quoted 2 lines

This is okay / intended. If nvme_cqe_pending() does not recognise that this interrupt belongs to that device then it should return IRQ_NONE.

Also note this patch doesn't change the condition when IRQ_NONE is returned. We've been doing that since nvme's beginning, as it really is a spurious interrupt from a likely malfunctioning device.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

Quoted 18 lines · Keith Busch
Quoted 17 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Daniel Wagner

drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056

Quoted 9 lines · Keith Busch

static irqreturn_t nvme_irq_check(int irq, void *data) {

drivers/nvme/host/pci.c +3 -3
 	struct nvme_queue *nvmeq = data;
-	if (nvme_cqe_pending(nvmeq))
-		return IRQ_WAKE_THREAD;
-	return IRQ_NONE;
+
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_NONE;

IIRC, returning IRQ_NONE tells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See __handle_irq_event_percpu, handle_irq_event_percpu and note_interrupt

Quoted 6 lines
Quoted 2 lines

This is okay / intended. If nvme_cqe_pending() does not recognise that this interrupt belongs to that device then it should return IRQ_NONE.

Also note this patch doesn't change the condition when IRQ_NONE is returned. We've been doing that since nvme's beginning, as it really is a spurious interrupt from a likely malfunctioning device.

Sure, I am just a bit confused how IRQ_NONE is used. The comment on nvme_cqe_pending says "We read the CQE phase first to check if the rest of the entry is valid", which I read as "check if we have anything to do" and not "this irq doesn't belong to this queue".

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#6 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

Quoted 22 lines · Daniel Wagner
Quoted 21 lines · Keith Busch
Quoted 19 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Daniel Wagner

drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1056

Quoted 9 lines · Keith Busch

static irqreturn_t nvme_irq_check(int irq, void *data) {

drivers/nvme/host/pci.c +3 -3
 	struct nvme_queue *nvmeq = data;
-	if (nvme_cqe_pending(nvmeq))
-		return IRQ_WAKE_THREAD;
-	return IRQ_NONE;
+
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_NONE;

IIRC, returning IRQ_NONE tells the irq core code that this is an unhandled IRQ. If the core sees too many of them it will print 'spurious irq' message. See __handle_irq_event_percpu, handle_irq_event_percpu and note_interrupt

This is okay / intended. If nvme_cqe_pending() does not recognise that this interrupt belongs to that device then it should return IRQ_NONE.

Quoted 8 lines
Quoted 3 lines

Also note this patch doesn't change the condition when IRQ_NONE is returned. We've been doing that since nvme's beginning, as it really is a spurious interrupt from a likely malfunctioning device.

Sure, I am just a bit confused how IRQ_NONE is used. The comment on nvme_cqe_pending says "We read the CQE phase first to check if the rest of the entry is valid", which I read as "check if we have anything to do" and not "this irq doesn't belong to this queue".

Those two interpretations sound like the same thing to me. :)

The interrupt notifies the driver it has something to do, and reading the CQE phase confirms whether or not that's true.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

Quoted 7 lines · Keith Busch
Quoted 4 lines · Daniel Wagner

Sure, I am just a bit confused how IRQ_NONE is used. The comment on nvme_cqe_pending says "We read the CQE phase first to check if the rest of the entry is valid", which I read as "check if we have anything to do" and not "this irq doesn't belong to this queue".

Those two interpretations sound like the same thing to me. :)

:)

I was confused why 'nothing to do' is then interpreted as IRQ was routed wrongly.

Quoted 2 lines

The interrupt notifies the driver it has something to do, and reading the CQE phase confirms whether or not that's true.

I think the thing I am missing the design idea behind it. I suppose the IRQ on only fires when previously work was posted.

Sorry for the noise.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#28 lore raw

Re: [PATCHv3 4/4] nvme/pci: Poll threaded completions

In reply to Keith Busch

Hi Keith,

Quoted 25 lines · Keith Busch

Poll for new completions in the nvme irq thread as long as it continues to observe new ones are available.

We need to ensure other irq threads affinitized to the same CPU will get a chance to run, so exit polling if we wrap the completion queue.

---
 drivers/nvme/host/pci.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
drivers/nvme/host/pci.c +1 -2
index 28e08c5ab412..487c6b3858fe 100644
@@ -1011,38 +1011,40 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start,
 	return found;
 }
 
-static irqreturn_t nvme_irq(int irq, void *data)
+static int nvme_irq(int irq, void *data)
 {
 	struct nvme_queue *nvmeq = data;
-	irqreturn_t ret = IRQ_NONE;
 	u16 start, end;

drivers/nvme/host/pci.c static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, · line 1026

Quoted 10 lines
drivers/nvme/host/pci.c +2 -1
+	int found;
 
 	/*
 	 * The rmb/wmb pair ensures we see all updates from a previous run of
 	 * the irq handler, even if that was on another CPU.
 	 */
 	rmb();
-	nvme_process_cq(nvmeq, &start, &end, -1);
+	found = nvme_process_cq(nvmeq, &start, &end, -1);
 	wmb();

Are the memory barries still at the right place after this change?

Thanks,
Daniel

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#11 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

In reply to Keith Busch

drivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1061

Quoted 13 lines
drivers/nvme/host/pci.c +4 -0
index a0138e3ca0b9..e415fadf7331 100644
@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
  	if (!nvme_cqe_pending(nvmeq))
  		return IRQ_NONE;
 
+	nvme_irq(irq, data);
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_HANDLED;
+

Question Keith,

If I have say 24 (or more) devices with a queue mapped to a cpu, and we
happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#12 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

Quoted 15 lines · Sagi Grimberg

drivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1060

Quoted 12 lines
drivers/nvme/host/pci.c +4 -0
index a0138e3ca0b9..e415fadf7331 100644
@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
  	if (!nvme_cqe_pending(nvmeq))
  		return IRQ_NONE;
+	nvme_irq(irq, data);
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_HANDLED;
+

Question Keith,

Quoted 3 lines

If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?

I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#13 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

Quoted 11 lines
Quoted 6 lines

Question Keith,

If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?

I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.

Theoretically speaking, even if you limit to 1 cqe, the universe can align such that you will always reap in the primary handler right? The more devices we have in the system mapping a vector to the same cpu core will increase the likelihood of this happening.

I guess it is a corner-case, but I'm thinking we want to be a 100% covered. Usually what drivers do is defer to a kthread/tasklet/napi context without reaping in the primary handler.

So if we have this optimization, perhaps something else in the irq infrastructure should take care of cpu lockup prevention?

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#14 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

Quoted 11 lines · Sagi Grimberg
Quoted 8 lines
Quoted 3 lines

If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?

I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.

Theoretically speaking, even if you limit to 1 cqe, the universe can align such that you will always reap in the primary handler right?

Perhaps theoretically, though testing the limits of reason.

Quoted 2 lines

So if we have this optimization, perhaps something else in the irq infrastructure should take care of cpu lockup prevention?

Perhaps we can cycle the effective_affinity through the smp_affinity?

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#15 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

If I have say 24 (or more) devices with a queue mapped to a cpu, and we happen to just reap in the primary handler for all devices, all the time, are we safe from cpu lockup?

I can't readily test that scenario, but I'm skeptical a workload can keep the primary handler running without ever seeing it return IRQ_WAKE_THREAD. If that is really a problem, we can mitigate it by limiting the number of CQEs the primary handler may process.

Quoted 4 lines
Quoted 2 lines

Theoretically speaking, even if you limit to 1 cqe, the universe can align such that you will always reap in the primary handler right?

Perhaps theoretically, though testing the limits of reason.

I know, maybe we should handle it when/if this really becomes a problem...

Quoted 4 lines
Quoted 2 lines

So if we have this optimization, perhaps something else in the irq infrastructure should take care of cpu lockup prevention?

Perhaps we can cycle the effective_affinity through the smp_affinity?

Not sure I follow your thoughts.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#16 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

Quoted 3 lines · Sagi Grimberg
Quoted 1 line

Perhaps we can cycle the effective_affinity through the smp_affinity?

Not sure I follow your thoughts.

The only way the nvme driver's interrupt handler can saturate a cpu requires the smp_affinity have multiple online cpus set. The effective_affinity, however, is always a single cpu, which can be any of the online ones set in smp_affinity.

If we can detect a cpu is spending too many cycles in nvme irq, perhaps we can re-spread the effective_affinity to different cpus. I don't currently know of a good way to do this, though.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#8 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

In reply to Keith Busch

drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1061

Quoted 4 lines · Keith Busch
drivers/nvme/host/pci.c +4 -0
+	if (!to_pci_dev(nvmeq->dev->dev)->msix_enabled)
+		writel(1 << nvmeq->cq_vector, nvmeq->dev->bar + NVME_REG_INTMS);
+
+	disable_irq_nosync(irq);

Why do we need both the device-level disable and disable_irq_nosync for the non-MSI-X case here?

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#20 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

In reply to Keith Busch
Quoted 22 lines · Keith Busch

The context switch to wake the threaded interrupt handler has enough overhead to harm latency for low queue depth workloads. Complete one cycle through the completion queue in the primary handler to ensure threaded interrupts have no disadvantage compared to not using threaded interrupts. Wake the thread only if more completions remain after processing the queue once since that indicates a higher queue depth workload that may trigger frequent interrupts which would be better completed in the handler thread with device interrupts disabled.

Completing requests from two different contexts is safe since nvme driver disables entry to the primary handler when it returns IRQ_WAKE_THREAD.

---
 drivers/nvme/host/pci.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/nvme/host/pci.c +0 -0
index a0138e3ca0b9..e415fadf7331 100644

drivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1058

Quoted 4 lines
drivers/nvme/host/pci.c +1 -0
@@ -1055,6 +1055,10 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
 	if (!nvme_cqe_pending(nvmeq))
 		return IRQ_NONE;
 
+	nvme_irq(irq, data);

I think we can just open code nvme_irq here. That also means we can use the return value of nvme_process_cq instead of the extra nvme_cqe_pending check above. This also needs a good comment explaining the scheme at least.

drivers/nvme/host/pci.c static irqreturn_t nvme_irq_check(int irq, void *data) · line 1060

Quoted 2 lines
drivers/nvme/host/pci.c +2 -0
+	if (!nvme_cqe_pending(nvmeq))
+		return IRQ_HANDLED;

Including one explaining this check.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#22 lore raw

Re: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts

In reply to Keith Busch
Quoted 4 lines · Keith Busch

The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the use_threaded_interrupts nvme module parameter since leaving it disabled should not be providing a benefit.

I think we need some careful benchmarking and numbers to justify the switch.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#23 lore raw

Re: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts

Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch

The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the use_threaded_interrupts nvme module parameter since leaving it disabled should not be providing a benefit.

I think we need some careful benchmarking and numbers to justify the switch.

Yes, I don't consider this 5.5 material, so we've some time.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#9 lore raw

Re: [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler

In reply to Christoph Hellwig
Quoted 8 lines · Christoph Hellwig

drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1061

Quoted 4 lines · Keith Busch
drivers/nvme/host/pci.c +4 -0
+	if (!to_pci_dev(nvmeq->dev->dev)->msix_enabled)
+		writel(1 << nvmeq->cq_vector, nvmeq->dev->bar + NVME_REG_INTMS);
+
+	disable_irq_nosync(irq);

Why do we need both the device-level disable and disable_irq_nosync for the non-MSI-X case here?

There's a race condition that needs to be handled here. We mask the interrupt on the device, but the device may have sent an MSI at the same time that the APIC hasn't seen yet, and that would retrigger the interrupt handler. We need it to see irqd_irq_disabled() is true, otherwise the nvme primary and thread may run at the same time, which we don't want in order to poll the cq from both.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#17 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

In reply to Keith Busch
Quoted 7 lines · Keith Busch
Quoted 3 lines · Sagi Grimberg
Quoted 1 line

Perhaps we can cycle the effective_affinity through the smp_affinity?

Not sure I follow your thoughts.

The only way the nvme driver's interrupt handler can saturate a cpu requires the smp_affinity have multiple online cpus set. The

As Sagi mentioned, there can be 24 drives, each drive can be 1:1 mapping, so each CPU may have to handle interrupts from 24 drives/queues, then the CPU may become slower compared with the 24 hardware/queues.

Quoted 7 lines

effective_affinity, however, is always a single cpu, which can be any of the online ones set in smp_affinity.

If we can detect a cpu is spending too many cycles in nvme irq, perhaps we can re-spread the effective_affinity to different cpus. I don't currently know of a good way to do this, though.

That shouldn't have been done just for nvme irq given other storage has similar issue.

I have tried toward that direction, and looks some cost has to be payed, such as reading one time of local clock in interrupt or io path has to be done:

https://lore.kernel.org/lkml/[email protected]/

It can be quite hard to figure out one efficient way to do that.

thanks,
Ming

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#18 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

Quoted 4 lines · Ming Lei
Quoted 3 lines · Keith Busch
Quoted 2 lines · Sagi Grimberg
Quoted 1 line

Perhaps we can cycle the effective_affinity through the smp_affinity?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Not sure I follow your thoughts.

The only way the nvme driver's interrupt handler can saturate a cpu requires the smp_affinity have multiple online cpus set. The

As Sagi mentioned, there can be 24 drives, each drive can be 1:1 mapping, so each CPU may have to handle interrupts from 24 drives/queues, then the CPU may become slower compared with the 24 hardware/queues.

If it's 1:1 queue:cpu, then it doesn't matter how many devices you have. Processes can't submit commands on that cpu while that cpu is servicing device interrupts, but the device can't send interrupts if processes can't submit commands.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#19 lore raw

Re: [PATCHv3 2/4] nvme/pci: Complete commands from primary handler

Quoted 10 lines · Keith Busch
Quoted 9 lines · Ming Lei
Quoted 7 lines · Keith Busch
Quoted 3 lines · Sagi Grimberg
Quoted 1 line

Perhaps we can cycle the effective_affinity through the smp_affinity?

Not sure I follow your thoughts.

The only way the nvme driver's interrupt handler can saturate a cpu requires the smp_affinity have multiple online cpus set. The

Quoted 9 lines
Quoted 3 lines

As Sagi mentioned, there can be 24 drives, each drive can be 1:1 mapping, so each CPU may have to handle interrupts from 24 drives/queues, then the CPU may become slower compared with the 24 hardware/queues.

If it's 1:1 queue:cpu, then it doesn't matter how many devices you have. Processes can't submit commands on that cpu while that cpu is servicing device interrupts, but the device can't send interrupts if processes can't submit commands.

OK, but still not sure it is good to cycle the effective_affinity through the smp_affinity:

1) the top half(primary interrupt handler) still might saturate the CPU

1) task migration isn't cheap from one CPU to another one

2) in theory we only need to do that when one interrupt target CPU is saturated, but changing thread's affinity has to be done beforehand, so this way may hurt performance when the CPU is never saturated. I believe that should be usual case.

Thanks,
Ming

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#24 lore raw

Re: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts

In reply to Christoph Hellwig
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch

The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the use_threaded_interrupts nvme module parameter since leaving it disabled should not be providing a benefit.

I think we need some careful benchmarking and numbers to justify the switch.

The patch can fix CPU lockup on Azure's NVMe, however, IOPS drops to ~600K from 3M+.

Thanks,
Ming

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#25 lore raw

Re: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts

Quoted 7 lines · Ming Lei
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch

The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the use_threaded_interrupts nvme module parameter since leaving it disabled should not be providing a benefit.

Quoted 5 lines
Quoted 2 lines

I think we need some careful benchmarking and numbers to justify the switch.

The patch can fix CPU lockup on Azure's NVMe, however, IOPS drops to ~600K from 3M+.

Compared to current mainline, or is that 3M+ with the poll completions from submission?

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#26 lore raw

Re: [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts

Quoted 8 lines · Keith Busch
Quoted 7 lines · Ming Lei
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch

The nvme threaded interrupt handling provides quick completions for latency sensitive workloads, and threaded handlers for more IOPS intensive ones. Remove the use_threaded_interrupts nvme module parameter since leaving it disabled should not be providing a benefit.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I think we need some careful benchmarking and numbers to justify the switch.

The patch can fix CPU lockup on Azure's NVMe, however, IOPS drops to ~600K from 3M+.

Compared to current mainline, or is that 3M+ with the poll completions from submission?

3M+ can be reached in mainline without poll completions from submission, which only improves IOPS in single fio job test.

This one is multi-job & multi-drive test:

fio --bs=4k --ioengine=libaio --iodepth=32 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=400 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

Thanks,
Ming

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme