update RDMA controllers queue depth
Status from this message: applied to nvme-5.16.
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest3 patches 1 replyApplied
- v12 patches 16 repliesSuperseded
Changes in v2 from the cover letter
- added Reviewed-by signatures (Chaitanya)
- rename get_queue_size to get_max_queue_size (Sagi)
- add a fix to initiator side as well (Jason)
- move NVME_RDMA_MAX_QUEUE_SIZE to common header file for nvme-rdma
Patches in v2 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/3 | nvme-rdma: limit the maximal queue size for RDMA controllersdrivers/ include/ | +9 | 0 replies | |
| 2/3 | nvmet: add get_max_queue_size op for controllersdrivers/ drivers/ | +6 | 0 replies | |
| 3/3 | nvmet-rdma: implement get_max_queue_size controller opdrivers/ | +6 | 0 replies |
Reviews trailers per patch
2 of 3 patches have a Reviewed-by or Acked-by.
| Patch | Chaitanya Kulkarni |
|---|---|
| 1/3 | |
| 2/3 | R-b |
| 3/3 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH v2 0/3] update RDMA controllers queue depth
Hi all,
This series is solving the issue that was introduced by Mark Ruijter while testing SPDK initiators on Vmware-7.x while connecting to Linux RDMA target running on NVIDIA's ConnectX-6 Mellanox Technologies adapter. During connection establishment, the NVMf target controller expose a 1024 queue depth capability but wasn't able to satisfy this depth in reality. The reason for that is that the NVMf driver didn't take the underlying HW capabilities into consideration. For now, limit the RDMA queue depth to a value of 128 (that is the default and works for all the RDMA controllers probably). For that, introduce a new controller operation to return the possible queue size for a given HW. Other transport will continue with their old behaviour.
Also fix the other side of the coin. In case there is a target that is capable to expose a queue depth of 1024 (or any value > 128), limit the initiator side to open only queues upto 128 entries to avoid a situation of failing to allocate larger QP.
In the future, in order to increase this size, we'll need to create a special RDMA API to calculate a possible queue depth for ULPs. As we know, the RDMA IO operations sometimes are built from multiple WRs (such as memory registrations and invalidations) that the ULP driver should take this into consideration during device discovery and queue depth calculations.
Changes from V1:
- added Reviewed-by signatures (Chaitanya)
- rename get_queue_size to get_max_queue_size (Sagi)
- add a fix to initiator side as well (Jason)
- move NVME_RDMA_MAX_QUEUE_SIZE to common header file for nvme-rdma
Max Gurtovoy (3):
nvme-rdma: limit the maximal queue size for RDMA controllers
nvmet: add get_max_queue_size op for controllers
nvmet-rdma: implement get_max_queue_size controller op
drivers/nvme/host/rdma.c | 7 +++++++ drivers/nvme/target/core.c | 8 +++++--- drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/rdma.c | 6 ++++++ include/linux/nvme-rdma.h | 2 ++ 5 files changed, 21 insertions(+), 3 deletions(-)
Signature · 7 lines
2.18.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme