Skip to contentSkip to conversation

update RDMA controllers queue depth

PATCH Applied Max Gurtovoy 2 participants 5 messages Sep 22, 2021 – Oct 12, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover update RDMA controllers queue depth

Hi all,
This series is solving the issue that was introduced by Mark Ruijter while testing SPDK initiators on Vmware-7.x while connecting to Linux RDMA target running on NVIDIA's ConnectX-6 Mellanox Technologies adapter. During connection establishment, the NVMf target controller expose a 1024 queue depth capability but wasn't able to satisfy this depth in reality. The reason for that is that the NVMf driver didn't take the underlying HW capabilities into consideration. For now, limit the RDMA queue depth to a value of 128 (that is the default and works for all the RDMA controllers probably). For that, introduce a new controller operation to return the possible queue size for a given HW. Other transport will continue with their old behaviour.

Also fix the other side of the coin. In case there is a target that is capable to expose a queue depth of 1024 (or any value > 128), limit the initiator side to open only queues upto 128 entries to avoid a situation of failing to allocate larger QP.

In the future, in order to increase this size, we'll need to create a special RDMA API to calculate a possible queue depth for ULPs. As we know, the RDMA IO operations sometimes are built from multiple WRs (such as memory registrations and invalidations) that the ULP driver should take this into consideration during device discovery and queue depth calculations.

Changes from V1:
- added Reviewed-by signatures (Chaitanya)
- rename get_queue_size to get_max_queue_size (Sagi)
- add a fix to initiator side as well (Jason)
- move NVME_RDMA_MAX_QUEUE_SIZE to common header file for nvme-rdma

Max Gurtovoy (3):
nvme-rdma: limit the maximal queue size for RDMA controllers
nvmet: add get_max_queue_size op for controllers
nvmet-rdma: implement get_max_queue_size controller op

 drivers/nvme/host/rdma.c    | 7 +++++++
 drivers/nvme/target/core.c  | 8 +++++---
 drivers/nvme/target/nvmet.h | 1 +
 drivers/nvme/target/rdma.c  | 6 ++++++
 include/linux/nvme-rdma.h   | 2 ++
 5 files changed, 21 insertions(+), 3 deletions(-)
Signature · 7 lines
2.18.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw

PATCH v2 1/3 nvme-rdma: limit the maximal queue size for RDMA controllers

Corrent limit of 1024 isn't valid for some of the RDMA based ctrls. In case the target expose a cap of larger amount of entries (e.g. 1024), the initiator may fail to create a QP with this size. Thus limit to a value that works for all RDMA adapters.

Future general solution should use RDMA/core API to calculate this size according to device capabilities and number of WRs needed per NVMe IO request.

---
 drivers/nvme/host/rdma.c  | 7 +++++++
 include/linux/nvme-rdma.h | 2 ++
 2 files changed, 9 insertions(+)
drivers/nvme/host/rdma.c +7 -0
index a68704e39084..a2f62eefccd9 100644
@@ -1111,6 +1111,13 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new)
 			ctrl->ctrl.opts->queue_size, ctrl->ctrl.sqsize + 1);
 	}
 
+	if (ctrl->ctrl.sqsize + 1 > NVME_RDMA_MAX_QUEUE_SIZE) {
+		dev_warn(ctrl->ctrl.device,
+			"ctrl sqsize %u > max queue size %u, clamping down\n",
+			ctrl->ctrl.sqsize + 1, NVME_RDMA_MAX_QUEUE_SIZE);
+		ctrl->ctrl.sqsize = NVME_RDMA_MAX_QUEUE_SIZE - 1;
+	}
+
 	if (ctrl->ctrl.sqsize + 1 > ctrl->ctrl.maxcmd) {
 		dev_warn(ctrl->ctrl.device,
 			"sqsize %u > ctrl maxcmd %u, clamping down\n",
include/linux/nvme-rdma.h +2 -0
index 3ec8e50efa16..4dd7e6fe92fb 100644
@@ -6,6 +6,8 @@
 #ifndef _LINUX_NVME_RDMA_H
 #define _LINUX_NVME_RDMA_H
 
+#define NVME_RDMA_MAX_QUEUE_SIZE	128
+
 enum nvme_rdma_cm_fmt {
 	NVME_RDMA_CM_FMT_1_0 = 0x0,
 };
Signature · 7 lines
2.18.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#2 lore raw

PATCH v2 2/3 nvmet: add get_max_queue_size op for controllers

In reply to Max Gurtovoy

Some transports, such as RDMA, would like to set the queue size according to device/port/ctrl characteristics. Add a new nvmet transport op that is called during ctrl initialization. This will not effect transports that don't implement this option.

---
 drivers/nvme/target/core.c  | 8 +++++---
 drivers/nvme/target/nvmet.h | 1 +
 2 files changed, 6 insertions(+), 3 deletions(-)
drivers/nvme/target/core.c +5 -3
index b8425fa34300..93107af3310d 100644
@@ -1205,7 +1205,10 @@ static void nvmet_init_cap(struct nvmet_ctrl *ctrl)
 	/* CC.EN timeout in 500msec units: */
 	ctrl->cap |= (15ULL << 24);
 	/* maximum queue entries supported: */
-	ctrl->cap |= NVMET_QUEUE_SIZE - 1;
+	if (ctrl->ops->get_max_queue_size)
+		ctrl->cap |= ctrl->ops->get_max_queue_size(ctrl) - 1;
+	else
+		ctrl->cap |= NVMET_QUEUE_SIZE - 1;
 
 	if (nvmet_is_passthru_subsys(ctrl->subsys))
 		nvmet_passthrough_override_cap(ctrl);
@@ -1367,6 +1370,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 	mutex_init(&ctrl->lock);
 
 	ctrl->port = req->port;
+	ctrl->ops = req->ops;
 
 	INIT_WORK(&ctrl->async_event_work, nvmet_async_event_work);
 	INIT_LIST_HEAD(&ctrl->async_events);
@@ -1405,8 +1409,6 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 	}
 	ctrl->cntlid = ret;
 
-	ctrl->ops = req->ops;
-
 	/*
 	 * Discovery controllers may use some arbitrary high value
 	 * in order to cleanup stale discovery sessions
drivers/nvme/target/nvmet.h +1 -0
index 7143c7fa7464..f8e0ee131dc6 100644
@@ -309,6 +309,7 @@ struct nvmet_fabrics_ops {
 	u16 (*install_queue)(struct nvmet_sq *nvme_sq);
 	void (*discovery_chg)(struct nvmet_port *port);
 	u8 (*get_mdts)(const struct nvmet_ctrl *ctrl);
+	u16 (*get_max_queue_size)(const struct nvmet_ctrl *ctrl);
 };
 
 #define NVMET_MAX_INLINE_BIOVEC	8
Signature · 7 lines
2.18.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#3 lore raw

PATCH v2 3/3 nvmet-rdma: implement get_max_queue_size controller op

In reply to Max Gurtovoy

Limit the maximal queue size for RDMA controllers. Today, the target reports a limit of 1024 and this limit isn't valid for some of the RDMA based controllers. For now, limit RDMA transport to 128 entries (the max queue depth configured for Linux NVMe/RDMA host).

Future general solution should use RDMA/core API to calculate this size according to device capabilities and number of WRs needed per NVMe IO request.

---
 drivers/nvme/target/rdma.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/nvme/target/rdma.c +6 -0
index 891174ccd44b..8bf2fad429f9 100644
@@ -1975,6 +1975,11 @@ static u8 nvmet_rdma_get_mdts(const struct nvmet_ctrl *ctrl)
 	return NVMET_RDMA_MAX_MDTS;
 }
 
+static u16 nvmet_rdma_get_max_queue_size(const struct nvmet_ctrl *ctrl)
+{
+	return NVME_RDMA_MAX_QUEUE_SIZE;
+}
+
 static const struct nvmet_fabrics_ops nvmet_rdma_ops = {
 	.owner			= THIS_MODULE,
 	.type			= NVMF_TRTYPE_RDMA,
@@ -1986,6 +1991,7 @@ static const struct nvmet_fabrics_ops nvmet_rdma_ops = {
 	.delete_ctrl		= nvmet_rdma_delete_ctrl,
 	.disc_traddr		= nvmet_rdma_disc_port_addr,
 	.get_mdts		= nvmet_rdma_get_mdts,
+	.get_max_queue_size	= nvmet_rdma_get_max_queue_size,
 };
 
 static void nvmet_rdma_remove_one(struct ib_device *ib_device, void *client_data)
Signature · 7 lines
2.18.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme