Skip to contentSkip to conversation

rxe: Fix dma.length computation in wr_set_sge_list

PATCH In review Jared Holzman 3 participants 4 messages May 31 – Jun 8 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rxe: Fix dma.length computation in wr_set_sge_list

wr_set_sge_list() summed the SGE lengths with a loop that never advanced sg_list:

	while (num_sge--)
		tot_length += sg_list->length;

so tot_length ended up as num_sge * sg_list[0].length instead of the true sum, and wqe->dma.length / wqe->dma.resid were written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the preceding memcpy().

The kernel rxe driver requires dma.length == sum(sge[i].length) and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through the ibv_qp_ex builder API (ibv_wr_set_sge_list) on rxe completes with IB_WC_LOC_PROT_ERR once finish_packet()/copy_data() runs off the end of the SGE list.

The legacy ibv_post_send path (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.

Fix by computing the total with an indexed loop, matching the style already used in rxe_post_one_recv() and init_send_wqe() in this file.

Fixes 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")

PR: https://github.com/linux-rdma/rdma-core/pull/1744

---
 providers/rxe/rxe.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
providers/rxe/rxe.c +3 -2
index 423f834b1..6d7be1493 100644
@@ -1138,6 +1138,7 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge,
 	struct rxe_send_wqe *wqe = addr_from_index(qp->sq.queue,
 						   qp->cur_index - 1);
 	size_t tot_length = 0;
+	size_t i;
 
 	if (qp->err)
 		return;
@@ -1150,8 +1151,8 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge,
 	wqe->dma.num_sge = num_sge;
 	memcpy(wqe->dma.sge, sg_list, num_sge*sizeof(*sg_list));
 
-	while (num_sge--)
-		tot_length += sg_list->length;
+	for (i = 0; i < num_sge; i++)
+		tot_length += sg_list[i].length;
 
 	wqe->dma.length = tot_length;
 	wqe->dma.resid = tot_length;
Signature · 1 line
2.51.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 2 lines · Jared Holzman

wr_set_sge_list() summed the SGE lengths with a loop that never advanced sg_list:

Good catch! This is a clean and straightforward fix for a subtle but high-impact bug in the Soft-RoCE (rxe) user-space provider.

Zhu Yanjun

Quoted 51 lines
	while (num_sge--)
		tot_length += sg_list->length;

so tot_length ended up as num_sge * sg_list[0].length instead of the true sum, and wqe->dma.length / wqe->dma.resid were written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the preceding memcpy().

The kernel rxe driver requires dma.length == sum(sge[i].length) and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through the ibv_qp_ex builder API (ibv_wr_set_sge_list) on rxe completes with IB_WC_LOC_PROT_ERR once finish_packet()/copy_data() runs off the end of the SGE list.

The legacy ibv_post_send path (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.

Fix by computing the total with an indexed loop, matching the style already used in rxe_post_one_recv() and init_send_wqe() in this file.

Fixes 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")

PR: https://github.com/linux-rdma/rdma-core/pull/1744

---
  providers/rxe/rxe.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
providers/rxe/rxe.c +3 -2
index 423f834b1..6d7be1493 100644
@@ -1138,6 +1138,7 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge,
  	struct rxe_send_wqe *wqe = addr_from_index(qp->sq.queue,
  						   qp->cur_index - 1);
  	size_t tot_length = 0;
+	size_t i;
 
  	if (qp->err)
  		return;
@@ -1150,8 +1151,8 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge,
  	wqe->dma.num_sge = num_sge;
  	memcpy(wqe->dma.sge, sg_list, num_sge*sizeof(*sg_list));
 
-	while (num_sge--)
-		tot_length += sg_list->length;
+	for (i = 0; i < num_sge; i++)
+		tot_length += sg_list[i].length;
 
  	wqe->dma.length = tot_length;
  	wqe->dma.resid = tot_length;
maintainer
#2 lore raw
In reply to Jared Holzman
Quoted 20 lines · Jared Holzman

wr_set_sge_list() summed the SGE lengths with a loop that never advanced sg_list:

	while (num_sge--)
		tot_length += sg_list->length;

so tot_length ended up as num_sge * sg_list[0].length instead of the true sum, and wqe->dma.length / wqe->dma.resid were written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the preceding memcpy().

The kernel rxe driver requires dma.length == sum(sge[i].length) and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through the ibv_qp_ex builder API (ibv_wr_set_sge_list) on rxe completes with IB_WC_LOC_PROT_ERR once finish_packet()/copy_data() runs off the end of the SGE list.

The legacy ibv_post_send path (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.

Quoted 6 lines

Fix by computing the total with an indexed loop, matching the style already used in rxe_post_one_recv() and init_send_wqe() in this file.

Fixes 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")

PR: https://github.com/linux-rdma/rdma-core/pull/1744

I don't know what this is, upstream doesn't have this code qp_ex support or 1a894ca10105

The rdma-core thing looks OK though.

Jason

#3 lore raw
Quoted 27 lines · Jason Gunthorpe
Quoted 20 lines · Jared Holzman

wr_set_sge_list() summed the SGE lengths with a loop that never advanced sg_list:

	while (num_sge--)
		tot_length += sg_list->length;

so tot_length ended up as num_sge * sg_list[0].length instead of the true sum, and wqe->dma.length / wqe->dma.resid were written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the preceding memcpy().

The kernel rxe driver requires dma.length == sum(sge[i].length) and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through the ibv_qp_ex builder API (ibv_wr_set_sge_list) on rxe completes with IB_WC_LOC_PROT_ERR once finish_packet()/copy_data() runs off the end of the SGE list.

The legacy ibv_post_send path (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.

Quoted 6 lines

Fix by computing the total with an indexed loop, matching the style already used in rxe_post_one_recv() and init_send_wqe() in this file.

Fixes 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")

PR: https://github.com/linux-rdma/rdma-core/pull/1744

Quoted 6 lines

I don't know what this is, upstream doesn't have this code qp_ex support or 1a894ca10105

The rdma-core thing looks OK though.

Jason

It is only an rdma-core change.

The commit hash is from rdma-core repo: https://github.com/linux-rdma/rdma-core

Jared