Skip to contentSkip to conversation

nvme-rdma: Support 2 inline data SGEs for write commands.

PATCH New Parav Pandit 4 participants 8 messages Feb 7, 2017 – Feb 8, 2017 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

This patch allows to send data for write commands via maximum of 2 SGEs for block write commands using RDMA's inherent sges support.

Number of send SGEs have almost no relation to the amount of inline data size being advertised by the NVMe target. Though its likely that when NVMe target advertises more than 4K or page size worth of inline data size, multiple send sges can become more useful.

In future more advance implementation may be made available to refer to exposed inline data size to decide upon number of SGEs for IOQ. Until than just make use of RDMA adapters capability.
Most adapters known to support 3 send sges (cmd + data) or more. So this shouldn't break any current deployments.
It is also tested with ConnectX-4 RoCEv2 100Gb adapters, ext4 and xfs filesystem and raw block IOs.

Reviewed-by Max Gurtovoy <maxg at mellanox.com>
Signed-off-by Parav Pandit <parav at mellanox.com>
---
 drivers/nvme/host/rdma.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)
drivers/nvme/host/rdma.c +17 -10
index 557f29b..fc66c9c 100644
@@ -40,7 +40,7 @@
 
 #define NVME_RDMA_MAX_SEGMENTS		256
 
-#define NVME_RDMA_MAX_INLINE_SEGMENTS	1
+#define NVME_RDMA_MAX_INLINE_SEGMENTS	2
 
 static const char *const nvme_rdma_cm_status_strs[] = {
 	[NVME_RDMA_CM_INVALID_LEN]	= "invalid length",
@@ -913,16 +913,23 @@ static int nvme_rdma_set_sg_null(struct nvme_command *c)
 }
 
 static int nvme_rdma_map_sg_inline(struct nvme_rdma_queue *queue,
-		struct nvme_rdma_request *req, struct nvme_command *c)
+		struct nvme_rdma_request *req, struct nvme_command *c,
+		int count)
 {
 	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
-
-	req->sge[1].addr = sg_dma_address(req->sg_table.sgl);
-	req->sge[1].length = sg_dma_len(req->sg_table.sgl);
-	req->sge[1].lkey = queue->device->pd->local_dma_lkey;
+	int i;
+	u32 len = 0;
+
+	for (i = 0; i < count; i++) {
+		req->sge[i + 1].addr = sg_dma_address(&req->sg_table.sgl[i]);
+		req->sge[i + 1].length = sg_dma_len(&req->sg_table.sgl[i]);
+		req->sge[i + 1].lkey = queue->device->pd->local_dma_lkey;
+		len += sg_dma_len(&req->sg_table.sgl[i]);
+		req->num_sge++;
+	}
 
 	sg->addr = cpu_to_le64(queue->ctrl->ctrl.icdoff);
-	sg->length = cpu_to_le32(sg_dma_len(req->sg_table.sgl));
+	sg->length = cpu_to_le32(len);
 	sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;
 
 	req->inline_data = true;
@@ -1012,13 +1019,13 @@ static int nvme_rdma_map_data(struct nvme_rdma_queue *queue,
 		return -EIO;
 	}
 
-	if (count == 1) {
+	if (count <= NVME_RDMA_MAX_INLINE_SEGMENTS) {
 		if (rq_data_dir(rq) == WRITE && nvme_rdma_queue_idx(queue) &&
 		    blk_rq_payload_bytes(rq) <=
 				nvme_rdma_inline_data_size(queue))
-			return nvme_rdma_map_sg_inline(queue, req, c);
+			return nvme_rdma_map_sg_inline(queue, req, c, count);
 
-		if (dev->pd->flags & IB_PD_UNSAFE_GLOBAL_RKEY)
+		if (count == 1 && dev->pd->flags & IB_PD_UNSAFE_GLOBAL_RKEY)
 			return nvme_rdma_map_sg_single(queue, req, c);
 	}
 
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

This looks fine to me in general, but I'm a little curious how your arrived at that 2 number. Why not 4 or 8?

maintainer
#2 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

Quoted 3 lines

This looks fine to me in general, but I'm a little curious how your arrived at that 2 number. Why not 4 or 8?

Well, its preallocated with the request.

Maybe we can have a heuristic that will allocate dynamically according to the max in-capsule data capability reported by the target (one sge for each PAGE_SIZE)?

#3 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

In reply to Christoph Hellwig
Quoted 11 lines

-----Original Message-----
From: Christoph Hellwig [mailto:hch at lst.de]
Sent: Wednesday, February 8, 2017 2:28 AM
To: Parav Pandit <parav at mellanox.com>
Cc: hch at lst.de; sagi at grimberg.me; linux-nvme at lists.infradead.org; axboe at fb.com; keith.busch at intel.com
Subject: Re: [PATCH] nvme-rdma: Support 2 inline data SGEs for write commands.

Quoted 2 lines

This looks fine to me in general, but I'm a little curious how your arrived at that 2 number. Why not 4 or 8?

Some adapters doesn't support more than 5 sges. So increasing it to 4 or 8 would fail them.
Chuck reported 1 out of 3 adapters only support 3. Its likely due to min(send_sge, recv_sge).
With total of 3 SGEs (64B cmd sge + data sges), WQE fits in 64 bytes size at the provider driver level. So to strike balance between cache line usage and performance, and to be nice to those adapters which has low SGE (either due to min(send_sge, recv) or other otherwise, I picked 2 data sges.

Side note: I have split out send and recv sge in other unrelated patch which is in review internally from linux-rdma tree.

#4 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

Quoted 2 lines
Quoted 2 lines

This looks fine to me in general, but I'm a little curious how your arrived

at

Quoted 3 lines
Quoted 1 line

that 2 number. Why not 4 or 8?

Some adapters doesn't support more than 5 sges. So increasing it to 4 or 8

would

Quoted 2 lines

fail them.
Chuck reported 1 out of 3 adapters only support 3. Its likely due to

min(send_sge,

Quoted 2 lines

recv_sge).
With total of 3 SGEs (64B cmd sge + data sges), WQE fits in 64 bytes size at

the

Quoted 5 lines

provider driver level. So to strike balance between cache line usage and performance, and to be nice to those adapters which has low SGE (either due to min(send_sge, recv) or other otherwise, I picked 2 data sges.

Side note: I have split out send and recv sge in other unrelated patch which

is in

Quoted 1 line

review internally from linux-rdma tree.

I'm looking forward to this patch since cxgb4 has max_recv_sge of 4 and max_send_sge of 17!

Steve.

#5 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

Hi Steve,

Quoted 29 lines

-----Original Message-----
From: Steve Wise [mailto:swise at opengridcomputing.com]
Sent: Wednesday, February 8, 2017 9:41 AM
To: Parav Pandit <parav at mellanox.com>; 'Christoph Hellwig' <hch at lst.de>
Cc: axboe at fb.com; keith.busch at intel.com; sagi at grimberg.me; linux- nvme at lists.infradead.org
Subject: RE: [PATCH] nvme-rdma: Support 2 inline data SGEs for write commands.

Quoted 3 lines
Quoted 3 lines

This looks fine to me in general, but I'm a little curious how your arrived

at

Quoted 4 lines
Quoted 1 line

that 2 number. Why not 4 or 8?

Some adapters doesn't support more than 5 sges. So increasing it to 4 or 8

would

Quoted 2 lines

fail them.
Chuck reported 1 out of 3 adapters only support 3. Its likely due to

min(send_sge,

Quoted 3 lines

recv_sge).
With total of 3 SGEs (64B cmd sge + data sges), WQE fits in 64 bytes size at

the

Quoted 4 lines

provider driver level. So to strike balance between cache line usage and performance, and to be nice to those adapters which has low SGE (either due to min(send_sge, recv) or other otherwise, I picked 2 data sges.

Quoted 8 lines
Quoted 2 lines

Side note: I have split out send and recv sge in other unrelated patch which

is in

Quoted 1 line

review internally from linux-rdma tree.

I'm looking forward to this patch since cxgb4 has max_recv_sge of 4 and max_send_sge of 17!

Sure. I will send out shortly once done with review and testing with nvme target.

Quoted 1 line

Steve.

#6 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

Quoted 5 lines
Quoted 4 lines

I'm looking forward to this patch since cxgb4 has max_recv_sge of 4 and max_send_sge of 17!

Sure. I will send out shortly once done with review and testing with nvme

target.

Quoted 1 line

If you can send this and your 3 or 4 nvme/rdma patches as a series, I'll test them on cxgb4.

Steve.

maintainer
#7 lore raw

PATCH 1/1 nvme-rdma: Support 2 inline data SGEs for write commands.

In reply to Parav Pandit
Quoted 10 lines

static int nvme_rdma_map_sg_inline(struct nvme_rdma_queue *queue,

drivers/nvme/host/rdma.c +2 -5
-		struct nvme_rdma_request *req, struct nvme_command *c)
+		struct nvme_rdma_request *req, struct nvme_command *c,
+		int count)
 {
 	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
-
-	req->sge[1].addr = sg_dma_address(req->sg_table.sgl);
-	req->sge[1].length = sg_dma_len(req->sg_table.sgl);
-	req->sge[1].lkey = queue->device->pd->local_dma_lkey;

drivers/nvme/host/rdma.c static int nvme_rdma_set_sg_null(struct nvme_command *c) · line 929

Quoted 10 lines
drivers/nvme/host/rdma.c +10 -0
+	int i;
+	u32 len = 0;
+
+	for (i = 0; i < count; i++) {
+		req->sge[i + 1].addr = sg_dma_address(&req->sg_table.sgl[i]);
+		req->sge[i + 1].length = sg_dma_len(&req->sg_table.sgl[i]);
+		req->sge[i + 1].lkey = queue->device->pd->local_dma_lkey;
+		len += sg_dma_len(&req->sg_table.sgl[i]);
+		req->num_sge++;
+	}

Nit, you can also have sge iterator that starts in req->sge[1] instead.