rxe: Fix dma.length computation in wr_set_sge_list
PATCH 1/1 rxe: Fix dma.length computation in wr_set_sge_list
wr_set_sge_list() summed the SGE lengths with a loop that never advanced sg_list:
while (num_sge--) tot_length += sg_list->length;
so tot_length ended up as num_sge * sg_list[0].length instead of the true sum, and wqe->dma.length / wqe->dma.resid were written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the preceding memcpy().
The kernel rxe driver requires dma.length == sum(sge[i].length) and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through the ibv_qp_ex builder API (ibv_wr_set_sge_list) on rxe completes with IB_WC_LOC_PROT_ERR once finish_packet()/copy_data() runs off the end of the SGE list.
The legacy ibv_post_send path (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.
Fix by computing the total with an indexed loop, matching the style already used in rxe_post_one_recv() and init_send_wqe() in this file.
PR: https://github.com/linux-rdma/rdma-core/pull/1744
--- providers/rxe/rxe.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
providers/rxe/rxe.c +3 -2
@@ -1138,6 +1138,7 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge,struct rxe_send_wqe *wqe =addr_from_index (qp->sq.queue, qp->cur_index -1 );size_t tot_length =0 ; +size_t i;if (qp->err)return ;
@@ -1150,8 +1151,8 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge, wqe->dma.num_sge = num_sge;memcpy (wqe->dma.sge, sg_list, num_sge*sizeof (*sg_list)); -while (num_sge--) - tot_length += sg_list->length; +for (i =0 ; i < num_sge; i++) + tot_length += sg_list[i].length; wqe->dma.length = tot_length; wqe->dma.resid = tot_length;
Signature · 1 line
2.51.0
Quoted 2 lines · Jared Holzman
wr_set_sge_list()summed the SGE lengths with a loop that never advancedsg_list:
Good catch! This is a clean and straightforward fix for a subtle but high-impact bug in the Soft-RoCE (rxe) user-space provider.
Zhu Yanjun
Quoted 51 lines
while (num_sge--) tot_length += sg_list->length;so
tot_lengthended up asnum_sge* sg_list[0].length instead of the true sum, andwqe->dma.length/wqe->dma.residwere written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the precedingmemcpy().The kernel rxe driver requires dma.length ==
sum(sge[i].length)and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through theibv_qp_exbuilder API (ibv_wr_set_sge_list) on rxe completes withIB_WC_LOC_PROT_ERRonce finish_packet()/copy_data() runs off the end of the SGE list.The legacy
ibv_post_sendpath (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.Fix by computing the total with an indexed loop, matching the style already used in
rxe_post_one_recv()andinit_send_wqe()in this file.Fixes 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")Signed-off-by Jared Holzman <[email protected]>PR: https://github.com/linux-rdma/rdma-core/pull/1744
--- providers/rxe/rxe.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)providers/rxe/rxe.c +3
-2@@ -1138,6 +1138,7 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge,struct rxe_send_wqe *wqe =addr_from_index (qp->sq.queue, qp->cur_index -1 );size_t tot_length =0 ; +size_t i;if (qp->err)return ;@@ -1150,8 +1151,8 @@ static void wr_set_sge_list(struct ibv_qp_ex *ibqp, size_t num_sge, wqe->dma.num_sge = num_sge;memcpy (wqe->dma.sge, sg_list, num_sge*sizeof (*sg_list)); -while (num_sge--) - tot_length += sg_list->length; +for (i =0 ; i < num_sge; i++) + tot_length += sg_list[i].length; wqe->dma.length = tot_length; wqe->dma.resid = tot_length;
Jason Gunthorpe
maintainerQuoted 20 lines · Jared Holzman
wr_set_sge_list()summed the SGE lengths with a loop that never advancedsg_list:while (num_sge--) tot_length += sg_list->length;so
tot_lengthended up asnum_sge* sg_list[0].length instead of the true sum, andwqe->dma.length/wqe->dma.residwere written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the precedingmemcpy().The kernel rxe driver requires dma.length ==
sum(sge[i].length)and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through theibv_qp_exbuilder API (ibv_wr_set_sge_list) on rxe completes withIB_WC_LOC_PROT_ERRonce finish_packet()/copy_data() runs off the end of the SGE list.The legacy
ibv_post_sendpath (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.
Quoted 6 lines
Fix by computing the total with an indexed loop, matching the style already used in
rxe_post_one_recv()andinit_send_wqe()in this file.Fixes 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")Signed-off-by Jared Holzman <[email protected]>
I don't know what this is, upstream doesn't have this code qp_ex support or 1a894ca10105
The rdma-core thing looks OK though.
Jason
Quoted 27 lines · Jason Gunthorpe
Quoted 20 lines · Jared Holzman
wr_set_sge_list()summed the SGE lengths with a loop that never advancedsg_list:while (num_sge--) tot_length += sg_list->length;so
tot_lengthended up asnum_sge* sg_list[0].length instead of the true sum, andwqe->dma.length/wqe->dma.residwere written with that wrong value. The per-SGE entries themselves were unaffected because they are populated by the precedingmemcpy().The kernel rxe driver requires dma.length ==
sum(sge[i].length)and enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through theibv_qp_exbuilder API (ibv_wr_set_sge_list) on rxe completes withIB_WC_LOC_PROT_ERRonce finish_packet()/copy_data() runs off the end of the SGE list.The legacy
ibv_post_sendpath (init_send_wqe) is unaffected; it sums the lengths with an indexed for loop.Quoted 6 lines
Fix by computing the total with an indexed loop, matching the style already used in
rxe_post_one_recv()andinit_send_wqe()in this file.Fixes 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")Signed-off-by Jared Holzman <[email protected]>
Quoted 6 lines
I don't know what this is, upstream doesn't have this code
qp_exsupport or1a894ca10105The rdma-core thing looks OK though.
Jason
It is only an rdma-core change.
The commit hash is from rdma-core repo: https://github.com/linux-rdma/rdma-core
Jared