Skip to contentSkip to conversation

nvme: enable PCI P2PDMA support for RDMA transport

RFC PATCH Superseded Shivaji Kant 3 participants 4 messages Apr 1

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 nvme: enable PCI P2PDMA support for RDMA transport

Enable BLK_FEAT_PCI_P2PDMA on the NVMe when the underlying RDMA controller supports it.

blk_stack_limits() currently filters out this feature bit because it is absent from BLK_FEAT_INHERIT_MASK. Manually re-assert the capability in nvme_update_ns_info() after the stacking operation.

Hardware reachability remains enforced by late-stage distance checks during DMA mapping.

---
 drivers/nvme/host/core.c |  9 +++++++++
 drivers/nvme/host/rdma.c | 15 +++++++++++++++
 2 files changed, 24 insertions(+)
drivers/nvme/host/core.c +9 -0
index a2da54f974fa..0d7b0f286895 100644
@@ -2205,6 +2205,15 @@ static int nvme_update_ns_info_generic(struct nvme_ns *ns,
 	nvme_set_ctrl_limits(ns->ctrl, &lim, false);
 
 	memflags = blk_mq_freeze_queue(ns->disk->queue);
+
+	/*
+	 * Explicitly check for P2PDMA support as BLK_FEAT_PCI_P2PDMA
+	 * is filtered out by queue_limits_stack_bdev().
+	 */
+	if (ns->ctrl->ops->supports_pci_p2pdma &&
+	   ns->ctrl->ops->supports_pci_p2pdma(ns->ctrl))
+		lim.features |= BLK_FEAT_PCI_P2PDMA;
+
 	ret = queue_limits_commit_update(ns->disk->queue, &lim);
 	set_disk_ro(ns->disk, nvme_ns_is_readonly(ns, info));
 	blk_mq_unfreeze_queue(ns->disk->queue, memflags);
drivers/nvme/host/rdma.c +15 -0
index 35c0822edb2d..3ce6f3e476b0 100644
@@ -2189,6 +2189,20 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
 	nvme_rdma_reconnect_or_remove(ctrl, ret);
 }
 
+static bool nvme_rdma_supports_pci_p2pdma(struct nvme_ctrl *ctrl)
+{
+	struct nvme_rdma_ctrl *r_ctrl = to_rdma_ctrl(ctrl);
+	bool supported = false;
+
+	if (r_ctrl && r_ctrl->device)
+		supported = ib_dma_pci_p2p_dma_supported(r_ctrl->device->dev);
+
+	dev_dbg(ctrl->device, "PCI P2PDMA support result: %s\n",
+			supported ? "PASSED" : "FAILED (HW/Driver restriction)");
+
+	return supported;
+}
+
 static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
 	.name			= "rdma",
 	.module			= THIS_MODULE,
@@ -2203,6 +2217,7 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
 	.get_address		= nvmf_get_address,
 	.stop_ctrl		= nvme_rdma_stop_ctrl,
 	.get_virt_boundary	= nvme_get_virt_boundary,
+	.supports_pci_p2pdma	= nvme_rdma_supports_pci_p2pdma,
 };
 
 /*
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#1 lore raw
Quoted 6 lines · Shivaji Kant

Enable BLK_FEAT_PCI_P2PDMA on the NVMe when the underlying RDMA controller supports it.

blk_stack_limits() currently filters out this feature bit because it is absent from BLK_FEAT_INHERIT_MASK. Manually re-assert the capability in nvme_update_ns_info() after the stacking operation.

This is really two different features/fixes and should be two patches. Note that Chaitanya also has an outstanding patch about p2p on multipath, so please work with him.

Quoted 2 lines

Hardware reachability remains enforced by late-stage distance checks during DMA mapping.

I don't know what this is supposed to mean. Callers need to check the reachability first before submitting P2P I/O.

drivers/nvme/host/rdma.c static void nvme_rdma_reset_ctrl_work(struct work_struct *work) · line 2197

Quoted 6 lines
drivers/nvme/host/rdma.c +6 -0
+static bool nvme_rdma_supports_pci_p2pdma(struct nvme_ctrl *ctrl)
+{
+	struct nvme_rdma_ctrl *r_ctrl = to_rdma_ctrl(ctrl);
+	bool supported = false;
+
+	if (r_ctrl && r_ctrl->device)

to_rdma_ctrl is a wrapper around container_of, so r_ctrl can't be NULL for a non-NULL ctrl. ->device also should not NULL because it is set up before namespaces are probed.

drivers/nvme/host/rdma.c static void nvme_rdma_reset_ctrl_work(struct work_struct *work) · line 2201

Quoted 4 lines
drivers/nvme/host/rdma.c +4 -0
+		supported = ib_dma_pci_p2p_dma_supported(r_ctrl->device->dev);
+
+	dev_dbg(ctrl->device, "PCI P2PDMA support result: %s\n",
+			supported ? "PASSED" : "FAILED (HW/Driver restriction)");

Overly long line, and screaming isn't really something we do in our messages. We also don't do that debug message in PCI, so please just drop it. IF you think this is important enough add a tracepoint in the core code in a separate patch.

#2 lore raw
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Shivaji Kant

Enable BLK_FEAT_PCI_P2PDMA on the NVMe when the underlying RDMA controller supports it.

Quoted 8 lines
Quoted 3 lines

blk_stack_limits() currently filters out this feature bit because it is absent from BLK_FEAT_INHERIT_MASK. Manually re-assert the capability in nvme_update_ns_info() after the stacking operation.

This is really two different features/fixes and should be two patches. Note that Chaitanya also has an outstanding patch about p2p on multipath, so please work with him.

Ack.
Shivaji, I believe this [1] is the patch Christoph's referring to.

Quoted 17 lines
Quoted 2 lines

Hardware reachability remains enforced by late-stage distance checks during DMA mapping.

I don't know what this is supposed to mean. Callers need to check the reachability first before submitting P2P I/O.

drivers/nvme/host/rdma.c static void nvme_rdma_reset_ctrl_work(struct work_struct *work) · line 2197

Quoted 6 lines
drivers/nvme/host/rdma.c +6 -0
+static bool nvme_rdma_supports_pci_p2pdma(struct nvme_ctrl *ctrl)
+{
+	struct nvme_rdma_ctrl *r_ctrl = to_rdma_ctrl(ctrl);
+	bool supported = false;
+
+	if (r_ctrl && r_ctrl->device)

to_rdma_ctrl is a wrapper around container_of, so r_ctrl can't be NULL for a non-NULL ctrl. ->device also should not NULL because it is set up before namespaces are probed.

Quoted 10 lines

drivers/nvme/host/rdma.c static void nvme_rdma_reset_ctrl_work(struct work_struct *work) · line 2201

Quoted 4 lines
drivers/nvme/host/rdma.c +4 -0
+		supported = ib_dma_pci_p2p_dma_supported(r_ctrl->device->dev);
+
+	dev_dbg(ctrl->device, "PCI P2PDMA support result: %s\n",
+			supported ? "PASSED" : "FAILED (HW/Driver restriction)");

Overly long line, and screaming isn't really something we do in our messages. We also don't do that debug message in PCI, so please just drop it. IF you think this is important enough add a tracepoint in the core code in a separate patch.

+1, we should drop the log and add a TP if necessary.

Thanks,
Praan

[1] https://lore.kernel.org/all/[email protected]/

#3 lore raw

Hi,
Thanks for the reviews.

Quoted 10 lines · Pranjal Shrivastava
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Shivaji Kant

Enable BLK_FEAT_PCI_P2PDMA on the NVMe when the underlying RDMA controller supports it.

blk_stack_limits() currently filters out this feature bit because it is absent from BLK_FEAT_INHERIT_MASK. Manually re-assert the capability in nvme_update_ns_info() after the stacking operation.

Quoted 7 lines
Quoted 4 lines

This is really two different features/fixes and should be two patches. Note that Chaitanya also has an outstanding patch about p2p on multipath, so please work with him.

Ack.
Shivaji, I believe this [1] is the patch Christoph's referring to.

Ack. Let me work with this.

Quoted 23 lines
Quoted 23 lines
Quoted 3 lines

Hardware reachability remains enforced by late-stage distance checks during DMA mapping.

I don't know what this is supposed to mean. Callers need to check the reachability first before submitting P2P I/O.

drivers/nvme/host/rdma.c static void nvme_rdma_reset_ctrl_work(struct work_struct *work) · line 2197

Quoted 6 lines
drivers/nvme/host/rdma.c +6 -0
+static bool nvme_rdma_supports_pci_p2pdma(struct nvme_ctrl *ctrl)
+{
+   struct nvme_rdma_ctrl *r_ctrl = to_rdma_ctrl(ctrl);
+   bool supported = false;
+
+   if (r_ctrl && r_ctrl->device)

to_rdma_ctrl is a wrapper around container_of, so r_ctrl can't be NULL for a non-NULL ctrl. ->device also should not NULL because it is set up before namespaces are probed.

Quoted 4 lines
drivers/nvme/host/rdma.c +4 -0
+           supported = ib_dma_pci_p2p_dma_supported(r_ctrl->device->dev);
+
+   dev_dbg(ctrl->device, "PCI P2PDMA support result: %s\n",
+                   supported ? "PASSED" : "FAILED (HW/Driver restriction)");
Quoted 7 lines
Quoted 5 lines

Overly long line, and screaming isn't really something we do in our messages. We also don't do that debug message in PCI, so please just drop it. IF you think this is important enough add a tracepoint in the core code in a separate patch.

+1, we should drop the log and add a TP if necessary.

Sure, sounds good. will incorporate these changes in v2. thanks.

Quoted 5 lines

Thanks,
Praan

[1] https://lore.kernel.org/all/[email protected]/

Regards
Shivaji