Skip to content

pci: nvmet: support completion queue sharing by multiple submission queues

Wilfred Mallawa linux-nvme 1 version Applied

Status from this message: applied to nvme-6.16.

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest5 patches 14 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/5nvmet: add a helper function for cqid checkingdrivers/nvme/target/admin-cmd.c drivers/nvme/target/core.c drivers/nvme/target/nvmet.h+10 -121 reply
2/5nvmet: cq: prepare for completion queue sharingdrivers/nvme/target/admin-cmd.c drivers/nvme/target/core.c drivers/nvme/target/nvmet.h +1 more+66 -194 replies
3/5nvmet: fabrics: add CQ init and destroydrivers/nvme/target/fabrics-cmd.c drivers/nvme/target/fc.c drivers/nvme/target/loop.c +2 more+28 -21 reply
4/5nvmet: support completion queue sharingdrivers/nvme/target/admin-cmd.c drivers/nvme/target/core.c drivers/nvme/target/fc.c +5 more+39 -251 reply
5/5nvmet: Simplify nvmet_req_init() interfacedrivers/nvme/target/core.c drivers/nvme/target/fc.c drivers/nvme/target/loop.c +4 more+11 -191 reply

Reviews trailers per patch

All 5 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchDamien Le MoalChaitanya Kulkarni
SeriesR-b
1/5R-b
2/5R-b
3/5R-b
4/5R-b
5/5R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/5] pci: nvmet: support completion queue sharing by multiple submission queues

From: Wilfred Mallawa <[email protected]>

Hi all,

For the NVMe PCI transport, the NVMe specification allows different submission queues (SQs) to share completion queues (CQs), however, this is not supported in the current NVMe target implementation. Until now, the nvmet target implementation enforced a 1:1 relationship between SQs and CQs, which is not specification compliant for the NVMe PCI transport.

This patch series adds support for CQ sharing between multiple SQs in the NVMe target driver, in line with the NVMe PCI transport specification. This series implements reference counting for completion queues to ensure proper lifecycle management when shared across multiple submission queues. This ensures that we retain CQs until all referencing SQs are deleted first, thereby avoiding premature CQ deletions.

Sample callchain with CQ refcounting for the PCI endpoint target (pci-epf)
=================================================================

i. nvmet_execute_create_cq -> nvmet_pci_epf_create_cq -> nvmet_cq_create

     -> nvmet_cq_init			[cq refcount = 1]

ii. nvmet_execute_create_sq -> nvmet_pci_epf_create_sq -> nvmet_sq_create

     -> nvmet_sq_init -> nvmet_cq_get	[cq refcount = 2]

iii. nvmet_execute_delete_sq -> nvmet_pci_epf_delete_sq ->

     nvmet_sq_destroy -> nvmet_cq_put	[cq refcount = 1]

iv. nvmet_execute_delete_cq -> nvmet_pci_epf_delete_cq -> nvmet_cq_put
[cq refcount = 0]

For NVMe over fabrics, CQ sharing is not supported per specification, however, the fabrics drivers are updated to integrate the new API changes. No functional change is intended here.

Testing
=======

Core functionality changes were tested with a Rockchip-based Rock5B PCIe endpoint setup using the pci-epf driver. The host kernel was modified to support queue sharing. In the test setup, this resulted in IO SQs 1 & 2 using IO CQ 1 and IO SQ 3 & 4 using IO CQ 2.

Testing methodology includes:

For PCI:

1. Boot up host
2. Assert that the endpoint device is detected as an NVMe drive (IO CQs/SQs are created)
3. Run FIOs
4. Unbind NVMe driver (IO SQs then CQs are deleted)
5. Rebind NVMe driver (IO SQs then CQs are created)
6. Run FIOs

For NVMe over fabrics: Using NVMe loop driver:

Note that there is no queue sharing supported for fabrics.

1. Connect command (IO queues are created)
2. Run FIOs
3. Disconnect command (IO queues are deleted)

Thanks!

Wilfred Mallawa (5):
nvmet: add a helper function for cqid checking
nvmet: cq: prepare for completion queue sharing
nvmet: fabrics: add CQ init and destroy
nvmet: support completion queue sharing
nvmet: Simplify nvmet_req_init() interface

 drivers/nvme/target/admin-cmd.c   | 31 ++++------
 drivers/nvme/target/core.c        | 94 ++++++++++++++++++++++++-------
 drivers/nvme/target/fabrics-cmd.c |  8 +++
 drivers/nvme/target/fc.c          | 10 ++--
 drivers/nvme/target/loop.c        | 23 +++++---
 drivers/nvme/target/nvmet.h       | 24 +++++---
 drivers/nvme/target/pci-epf.c     | 11 ++--
 drivers/nvme/target/rdma.c        |  8 ++-
 drivers/nvme/target/tcp.c         |  8 ++-
 9 files changed, 147 insertions(+), 70 deletions(-)
Signature · 1 line
2.49.0