pci: nvmet: support completion queue sharing by multiple submission queues
Status from this message: applied to nvme-6.16.
Get series with b4
b4 am [email protected]Versions newest first
Patches in v1 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/5 | nvmet: add a helper function for cqid checkingdrivers/ drivers/ drivers/ | +10 | 1 reply | |
| 2/5 | nvmet: cq: prepare for completion queue sharingdrivers/ drivers/ drivers/ +1 more | +66 | 4 replies | |
| 3/5 | nvmet: fabrics: add CQ init and destroydrivers/ drivers/ drivers/ +2 more | +28 | 1 reply | |
| 4/5 | nvmet: support completion queue sharingdrivers/ drivers/ drivers/ +5 more | +39 | 1 reply | |
| 5/5 | nvmet: Simplify nvmet_req_init() interfacedrivers/ drivers/ drivers/ +4 more | +11 | 1 reply |
Reviews trailers per patch
All 5 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.
| Patch | Damien Le Moal | Chaitanya Kulkarni |
|---|---|---|
| Series | R-b | |
| 1/5 | R-b | |
| 2/5 | R-b | |
| 3/5 | R-b | |
| 4/5 | R-b | |
| 5/5 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH 0/5] pci: nvmet: support completion queue sharing by multiple submission queues
From: Wilfred Mallawa <[email protected]>
Hi all,
For the NVMe PCI transport, the NVMe specification allows different submission queues (SQs) to share completion queues (CQs), however, this is not supported in the current NVMe target implementation. Until now, the nvmet target implementation enforced a 1:1 relationship between SQs and CQs, which is not specification compliant for the NVMe PCI transport.
This patch series adds support for CQ sharing between multiple SQs in the NVMe target driver, in line with the NVMe PCI transport specification. This series implements reference counting for completion queues to ensure proper lifecycle management when shared across multiple submission queues. This ensures that we retain CQs until all referencing SQs are deleted first, thereby avoiding premature CQ deletions.
Sample callchain with CQ refcounting for the PCI endpoint target (pci-epf)
=================================================================
i. nvmet_execute_create_cq -> nvmet_pci_epf_create_cq -> nvmet_cq_create
-> nvmet_cq_init [cq refcount = 1]
ii. nvmet_execute_create_sq -> nvmet_pci_epf_create_sq -> nvmet_sq_create
-> nvmet_sq_init -> nvmet_cq_get [cq refcount = 2]
iii. nvmet_execute_delete_sq -> nvmet_pci_epf_delete_sq ->
nvmet_sq_destroy -> nvmet_cq_put [cq refcount = 1]
iv. nvmet_execute_delete_cq -> nvmet_pci_epf_delete_cq -> nvmet_cq_put
[cq refcount = 0]
For NVMe over fabrics, CQ sharing is not supported per specification, however, the fabrics drivers are updated to integrate the new API changes. No functional change is intended here.
Testing
=======
Core functionality changes were tested with a Rockchip-based Rock5B PCIe endpoint setup using the pci-epf driver. The host kernel was modified to support queue sharing. In the test setup, this resulted in IO SQs 1 & 2 using IO CQ 1 and IO SQ 3 & 4 using IO CQ 2.
Testing methodology includes:
For PCI:
1. Boot up host
2. Assert that the endpoint device is detected as an NVMe drive (IO CQs/SQs are created)
3. Run FIOs
4. Unbind NVMe driver (IO SQs then CQs are deleted)
5. Rebind NVMe driver (IO SQs then CQs are created)
6. Run FIOs
For NVMe over fabrics: Using NVMe loop driver:
Note that there is no queue sharing supported for fabrics.
1. Connect command (IO queues are created)
2. Run FIOs
3. Disconnect command (IO queues are deleted)
Thanks!
Wilfred Mallawa (5):
nvmet: add a helper function for cqid checking
nvmet: cq: prepare for completion queue sharing
nvmet: fabrics: add CQ init and destroy
nvmet: support completion queue sharing
nvmet: Simplify nvmet_req_init() interface
drivers/nvme/target/admin-cmd.c | 31 ++++------ drivers/nvme/target/core.c | 94 ++++++++++++++++++++++++------- drivers/nvme/target/fabrics-cmd.c | 8 +++ drivers/nvme/target/fc.c | 10 ++-- drivers/nvme/target/loop.c | 23 +++++--- drivers/nvme/target/nvmet.h | 24 +++++--- drivers/nvme/target/pci-epf.c | 11 ++-- drivers/nvme/target/rdma.c | 8 ++- drivers/nvme/target/tcp.c | 8 ++- 9 files changed, 147 insertions(+), 70 deletions(-)
Signature · 1 line
2.49.0