Skip to contentSkip to conversation

nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

PATCH Applied Niklas Cassel 4 participants 6 messages Aug 15, 2024 – Aug 26, 2024 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification allows you to populate the cntlid field in the RDMA_CM_REQUEST Private Data.

The cntlid is returned by the target on completion of the first RDMA_CM_REQUEST command (which creates the admin queue).

The cntlid field can then be populated by the host when the I/O queues are created (using additional RDMA_CM_REQUEST commands), such that the target can perform extra validation for additional RDMA_CM_REQUEST commands.

This additional error code and error message is also added, such that nvme_rdma_cm_msg() will display the proper error message if the target fails the RDMA_CM_REQUEST command because of this extra validation.

---
 drivers/nvme/host/rdma.c  | 2 ++
 include/linux/nvme-rdma.h | 6 +++++-
 2 files changed, 7 insertions(+), 1 deletion(-)
drivers/nvme/host/rdma.c +2 -0
index 2eb33842f971..e3520a91147b 100644
@@ -1876,6 +1876,8 @@ static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue)
 		 */
 		priv.hrqsize = cpu_to_le16(queue->queue_size);
 		priv.hsqsize = cpu_to_le16(queue->ctrl->ctrl.sqsize);
+		/* cntlid should only be set when creating an I/O queue */
+		priv.cntlid = cpu_to_le16(ctrl->ctrl.cntlid);
 	}
 
 	ret = rdma_connect_locked(queue->cm_id, &param);
include/linux/nvme-rdma.h +5 -1
index eb2f04d636c8..97c5f00b9aa3 100644
@@ -25,6 +25,7 @@ enum nvme_rdma_cm_status {
 	NVME_RDMA_CM_NO_RSC		= 0x06,
 	NVME_RDMA_CM_INVALID_IRD	= 0x07,
 	NVME_RDMA_CM_INVALID_ORD	= 0x08,
+	NVME_RDMA_CM_INVALID_CNTLID	= 0x09,
 };
 
 static inline const char *nvme_rdma_cm_msg(enum nvme_rdma_cm_status status)
@@ -46,6 +47,8 @@ static inline const char *nvme_rdma_cm_msg(enum nvme_rdma_cm_status status)
 		return "invalid IRD";
 	case NVME_RDMA_CM_INVALID_ORD:
 		return "Invalid ORD";
+	case NVME_RDMA_CM_INVALID_CNTLID:
+		return "invalid controller ID";
 	default:
 		return "unrecognized reason";
 	}
@@ -64,7 +67,8 @@ struct nvme_rdma_cm_req {
 	__le16		qid;
 	__le16		hrqsize;
 	__le16		hsqsize;
-	u8		rsvd[24];
+	__le16		cntlid;
+	u8		rsvd[22];
 };
 
 /**
Signature · 1 line
2.46.0
#2 lore raw
In reply to Niklas Cassel
Quoted 16 lines · Niklas Cassel

When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification allows you to populate the cntlid field in the RDMA_CM_REQUEST Private Data.

The cntlid is returned by the target on completion of the first RDMA_CM_REQUEST command (which creates the admin queue).

The cntlid field can then be populated by the host when the I/O queues are created (using additional RDMA_CM_REQUEST commands), such that the target can perform extra validation for additional RDMA_CM_REQUEST commands.

This additional error code and error message is also added, such that nvme_rdma_cm_msg() will display the proper error message if the target fails the RDMA_CM_REQUEST command because of this extra validation.

Quoted 5 lines
---
 drivers/nvme/host/rdma.c  | 2 ++
 include/linux/nvme-rdma.h | 6 +++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

Hello NVMe maintainers,

is there any chance of getting this picked up?

Have a nice day!

Kind regards,
Niklas

maintainer
#3 lore raw
In reply to Niklas Cassel
Quoted 17 lines · Niklas Cassel
Quoted 16 lines · Niklas Cassel

When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification allows you to populate the cntlid field in the RDMA_CM_REQUEST Private Data.

The cntlid is returned by the target on completion of the first RDMA_CM_REQUEST command (which creates the admin queue).

The cntlid field can then be populated by the host when the I/O queues are created (using additional RDMA_CM_REQUEST commands), such that the target can perform extra validation for additional RDMA_CM_REQUEST commands.

This additional error code and error message is also added, such that nvme_rdma_cm_msg() will display the proper error message if the target fails the RDMA_CM_REQUEST command because of this extra validation.

Quoted 8 lines
Quoted 5 lines
---
  drivers/nvme/host/rdma.c  | 2 ++
  include/linux/nvme-rdma.h | 6 +++++-
  2 files changed, 7 insertions(+), 1 deletion(-)

Hello NVMe maintainers,

is there any chance of getting this picked up?

Keith will probably pick it up for 6.12...

maintainer Applied
#5 lore raw
In reply to Niklas Cassel
Quoted 7 lines · Niklas Cassel

When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification allows you to populate the cntlid field in the RDMA_CM_REQUEST Private Data.

The cntlid is returned by the target on completion of the first RDMA_CM_REQUEST command (which creates the admin queue).

Quoted 8 lines

The cntlid field can then be populated by the host when the I/O queues are created (using additional RDMA_CM_REQUEST commands), such that the target can perform extra validation for additional RDMA_CM_REQUEST commands.

This additional error code and error message is also added, such that nvme_rdma_cm_msg() will display the proper error message if the target fails the RDMA_CM_REQUEST command because of this extra validation.

Thanks, applied to nvme-6.12.