Skip to content

nvme pci interrupt handling improvements

Keith Busch linux-nvme 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest4 patches 24 repliesIn review
  2. v22 patches 7 repliesSuperseded
  3. v14 patches 32 repliesSuperseded

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4nvme/pci: Disable interrupts for threaded handlerdrivers/nvme/host/pci.c+22 -48 replies
2/4nvme/pci: Complete commands from primary handlerdrivers/nvme/host/pci.c+4 -010 replies
3/4nvme/pci: Remove use_threaded_interruptsdrivers/nvme/host/pci.c+2 -105 replies
4/4nvme/pci: Poll threaded completionsdrivers/nvme/host/pci.c+12 -101 reply

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCHv3 0/4] nvme pci interrupt handling improvements

Here's the next revision to nvme pci interrupt handling. The series is attempting to address two issues that have been recently raised:

1. The nvme driver makes it possible to soft lockup a CPU due to high utilization in irq context. This is most prevalent when multiple CPUs are driving a single hardware queue on one or more controllers.

2. The threaded handler left interrupts unmasked, which breaks when used with level triggered interrupts, or unnecessarily runs in interrupt context when edge interrupts occur frequently.

Both issues are addressed by always configuring nvme interrupts to run the threaded handler with interrupts disabled. A hybrid approch to handling nvme completions in hard irq context and thread context is introduced so that there should not be a performance impact from removing the nvme.use_threaded_interrupts option.

The series appears to be a win or no impact on performance from what I can test. I would be greatful to hear if others can confirm this with other hardware.

I've dropped the fast MSIx masking. While I believe it's safe to skip flushing the memory write, I think this series mitigates the impact of the read back by ensuring the ratio of memory reads to IO is low enough to be negligable (AFAICT on hardware available to me).

I've changed the exit condition for the polling nvme irq thread to break out of the loop if we've wrapped the completion queue. Other irq threads may be affinitized to the same CPU, so we need to schedule out at some point, but I've been told multiple times that need_resched() or cond_resched() won't work as desired from the thread's fifo priority.

Keith Busch (4):
nvme/pci: Disable interrupts for threaded handler nvme/pci: Complete commands from primary handler nvme/pci: Remove use_threaded_interrupts nvme/pci: Poll threaded completions

 drivers/nvme/host/pci.c | 56 ++++++++++++++++++++++++++---------------
 1 file changed, 36 insertions(+), 20 deletions(-)
Signature · 7 lines
2.21.0


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme