RDMA/rxe: Fix error path code in rxe_create_qp
Subsystems (2)
cover RDMA/rxe: Fix error path code in rxe_create_qp
If a call to rxe_create_qp() fails in rxe_qp_from_init() rxe_cleanup(qp) will be called. This code currently does not correctly handle cases where not all qp resources are allocated and can seg fault as reported below. The first two patches cleanup cases where this happens. The third patch corrects an error in rxe_srq.c where if caller requests a change in the srq size the correct new value is not returned to caller.
Bob Pearson (3):
RDMA/rxe: Move work queue code to subroutines
RDMA/rxe: Fix unsafe drain work queue code
RDMA/rxe: Fix rxe_modify_srq
drivers/infiniband/sw/rxe/rxe_comp.c | 4 + drivers/infiniband/sw/rxe/rxe_loc.h | 6 - drivers/infiniband/sw/rxe/rxe_qp.c | 163 ++++++++++++++++++--------- drivers/infiniband/sw/rxe/rxe_resp.c | 4 + drivers/infiniband/sw/rxe/rxe_srq.c | 55 +++++---- 5 files changed, 150 insertions(+), 82 deletions(-)
base-commit: 830f93f47068b1632cc127871fbf27e918efdf46
Signature · 1 line
2.39.2
PATCH 1/3 RDMA/rxe: Move work queue code to subroutines
This patch:
- Moves code to initialize a qp send work queue to a subroutine named rxe_init_sq.
- Moves code to initialize a qp recv work queue to a subroutine named rxe_init_rq.
- Moves initialization of qp request and response packet queues ahead of work queue initialization so that cleanup of a qp if it is not fully completed can successfully attempt to drain the packet queues without a seg fault.
- Makes minor whitespace cleanups.
--- drivers/infiniband/sw/rxe/rxe_qp.c | 163 +++++++++++++++++++---------- 1 file changed, 108 insertions(+), 55 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +108 -55
@@ -180,13 +180,63 @@ static void rxe_qp_init_misc(struct rxe_dev *rxe, struct rxe_qp *qp,atomic_set (&qp->skb_out,0 ); } +static int rxe_init_sq (struct rxe_qp *qp,struct ib_qp_init_attr *init, +struct ib_udata *udata, +struct rxe_create_qp_resp __user *uresp) +{ +struct rxe_dev *rxe =to_rdev (qp->ibqp.device); +int wqe_size; +int err; + + qp->sq.max_wr = init->cap.max_send_wr; + wqe_size =max_t (int, init->cap.max_send_sge *sizeof (struct ib_sge ), + init->cap.max_inline_data); + qp->sq.max_sge = wqe_size /sizeof (struct ib_sge ); + qp->sq.max_inline = wqe_size; + wqe_size +=sizeof (struct rxe_send_wqe ); + + qp->sq.queue =rxe_queue_init (rxe, &qp->sq.max_wr, wqe_size, +QUEUE_TYPE_FROM_CLIENT ); +if (!qp->sq.queue) { +rxe_err_qp (qp,"Unable to allocate send queue" ); + err = -ENOMEM ; + goto err_out; + } + +/* prepare info for caller to mmap send queue if user space qp */ + err =do_mmap_info (rxe, uresp ? &uresp->sq_mi :NULL , udata, + qp->sq.queue->buf, qp->sq.queue->buf_size, + &qp->sq.queue->ip); +if (err) { +rxe_err_qp (qp,"do_mmap_info failed, err = %d" , err); + goto err_free; + } + +/* return actual capabilities to caller which may be larger +* than requested +*/ + init->cap.max_send_wr = qp->sq.max_wr; + init->cap.max_send_sge = qp->sq.max_sge; + init->cap.max_inline_data = qp->sq.max_inline; + +return 0 ; + +err_free: +vfree (qp->sq.queue->buf); +kfree (qp->sq.queue); + qp->sq.queue =NULL ; +err_out: +return err; +} +static int rxe_qp_init_req (struct rxe_dev *rxe,struct rxe_qp *qp,struct ib_qp_init_attr *init,struct ib_udata *udata,struct rxe_create_qp_resp __user *uresp) {int err; -int wqe_size; -enum queue_type type; + +/* if we don't finish qp create make sure queue is valid */ +skb_queue_head_init (&qp->req_pkts); err =sock_create_kern (&init_net,AF_INET ,SOCK_DGRAM ,0 , &qp->sk);if (err <0 )
@@ -201,32 +251,10 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,* (0xc000 - 0xffff). */ qp->src_port =RXE_ROCE_V2_SPORT + (hash_32 (qp_num (qp),14 ) &0x3fff ); - qp->sq.max_wr = init->cap.max_send_wr; - -/* These caps are limited by rxe_qp_chk_cap() done by the caller */ - wqe_size =max_t (int, init->cap.max_send_sge *sizeof (struct ib_sge ), - init->cap.max_inline_data); - qp->sq.max_sge = init->cap.max_send_sge = - wqe_size /sizeof (struct ib_sge ); - qp->sq.max_inline = init->cap.max_inline_data = wqe_size; - wqe_size +=sizeof (struct rxe_send_wqe ); - - type =QUEUE_TYPE_FROM_CLIENT ; - qp->sq.queue =rxe_queue_init (rxe, &qp->sq.max_wr, - wqe_size, type); -if (!qp->sq.queue) -return -ENOMEM ; - err =do_mmap_info (rxe, uresp ? &uresp->sq_mi :NULL , udata, - qp->sq.queue->buf, qp->sq.queue->buf_size, - &qp->sq.queue->ip); - -if (err) { -vfree (qp->sq.queue->buf); -kfree (qp->sq.queue); - qp->sq.queue =NULL ; + err =rxe_init_sq (qp, init, udata, uresp); +if (err)return err; - } qp->req.wqe_index =queue_get_producer (qp->sq.queue,QUEUE_TYPE_FROM_CLIENT );
@@ -234,8 +262,6 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, qp->req.opcode =-1 ; qp->comp.opcode =-1 ; -skb_queue_head_init (&qp->req_pkts); -rxe_init_task (&qp->req.task, qp, rxe_requester);rxe_init_task (&qp->comp.task, qp, rxe_completer);
@@ -247,40 +273,67 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,return 0 ; } +static int rxe_init_rq (struct rxe_qp *qp,struct ib_qp_init_attr *init, +struct ib_udata *udata, +struct rxe_create_qp_resp __user *uresp) +{ +struct rxe_dev *rxe =to_rdev (qp->ibqp.device); +int wqe_size; +int err; + + qp->rq.max_wr = init->cap.max_recv_wr; + qp->rq.max_sge = init->cap.max_recv_sge; + wqe_size =sizeof (struct rxe_recv_wqe ) + + qp->rq.max_sge*sizeof (struct ib_sge ); + + qp->rq.queue =rxe_queue_init (rxe, &qp->rq.max_wr, wqe_size, +QUEUE_TYPE_FROM_CLIENT ); +if (!qp->rq.queue) { +rxe_err_qp (qp,"Unable to allocate recv queue" ); + err = -ENOMEM ; + goto err_out; + } + +/* prepare info for caller to mmap recv queue if user space qp */ + err =do_mmap_info (rxe, uresp ? &uresp->rq_mi :NULL , udata, + qp->rq.queue->buf, qp->rq.queue->buf_size, + &qp->rq.queue->ip); +if (err) { +rxe_err_qp (qp,"do_mmap_info failed, err = %d" , err); + goto err_free; + } + +/* return actual capabilities to caller which may be larger +* than requested +*/ + init->cap.max_recv_wr = qp->rq.max_wr; + +return 0 ; + +err_free: +vfree (qp->rq.queue->buf); +kfree (qp->rq.queue); + qp->rq.queue =NULL ; +err_out: +return err; +} +static int rxe_qp_init_resp (struct rxe_dev *rxe,struct rxe_qp *qp,struct ib_qp_init_attr *init,struct ib_udata *udata,struct rxe_create_qp_resp __user *uresp) {int err; -int wqe_size; -enum queue_type type; + +/* if we don't finish qp create make sure queue is valid */ +skb_queue_head_init (&qp->resp_pkts);if (!qp->srq) { - qp->rq.max_wr = init->cap.max_recv_wr; - qp->rq.max_sge = init->cap.max_recv_sge; - - wqe_size =rcv_wqe_size (qp->rq.max_sge); - - type =QUEUE_TYPE_FROM_CLIENT ; - qp->rq.queue =rxe_queue_init (rxe, &qp->rq.max_wr, - wqe_size, type); -if (!qp->rq.queue) -return -ENOMEM ; - - err =do_mmap_info (rxe, uresp ? &uresp->rq_mi :NULL , udata, - qp->rq.queue->buf, qp->rq.queue->buf_size, - &qp->rq.queue->ip); -if (err) { -vfree (qp->rq.queue->buf); -kfree (qp->rq.queue); - qp->rq.queue =NULL ; + err =rxe_init_rq (qp, init, udata, uresp); +if (err)return err; - } } -skb_queue_head_init (&qp->resp_pkts); -rxe_init_task (&qp->resp.task, qp, rxe_responder); qp->resp.opcode =OPCODE_NONE ;
@@ -307,10 +360,10 @@ int rxe_qp_from_init(struct rxe_dev *rxe, struct rxe_qp *qp, struct rxe_pd *pd,if (srq)rxe_get (srq); - qp->pd = pd; - qp->rcq = rcq; - qp->scq = scq; - qp->srq = srq; + qp->pd = pd; + qp->rcq = rcq; + qp->scq = scq; + qp->srq = srq;atomic_inc (&rcq->num_wq);atomic_inc (&scq->num_wq);
Signature · 1 line
2.39.2
PATCH 2/3 RDMA/rxe: Fix unsafe drain work queue code
In reply to Bob PearsonIf create_qp does not fully succeed it is possible for qp cleanup code to attempt to drain the send or recv work queues before the queues have been created causing a seg fault. This patch checks to see if the queues exist before attempting to drain them.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++++ drivers/infiniband/sw/rxe/rxe_resp.c | 4 ++++ 2 files changed, 8 insertions(+)
drivers/infiniband/sw/rxe/rxe_comp.c +4 -0
@@ -594,6 +594,10 @@ static void flush_send_queue(struct rxe_qp *qp, bool notify)struct rxe_queue *q = qp->sq.queue;int err; +/* send queue never got created. nothing to do. */ +if (!qp->sq.queue) +return ; +while ((wqe =queue_head (q, q->type))) {if (notify) { err =flush_send_wqe (qp, wqe);
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
@@ -1467,6 +1467,10 @@ static void flush_recv_queue(struct rxe_qp *qp, bool notify)return ; } +/* recv queue not created. nothing to do. */ +if (!qp->rq.queue) +return ; +while ((wqe =queue_head (q, q->type))) {if (notify) { err =flush_recv_wqe (qp, wqe);
Signature · 1 line
2.39.2
PATCH 3/3 RDMA/rxe: Fix rxe_m-dify_srq
In reply to Bob PearsonThis patch corrects an error in rxe_modify_srq where if the caller changes the srq size the actual new value is not returned to the caller since it may be larger than what is requested. Additionally it open codes the subroutine rcv_wqe_size() which adds very little value. And makes some whitespace changes.
--- drivers/infiniband/sw/rxe/rxe_loc.h | 6 ---- drivers/infiniband/sw/rxe/rxe_srq.c | 55 ++++++++++++++++++----------- 2 files changed, 34 insertions(+), 27 deletions(-)
drivers/infiniband/sw/rxe/rxe_loc.h +0 -6
@@ -136,12 +136,6 @@ static inline int qp_mtu(struct rxe_qp *qp)return IB_MTU_4096 ; } -static inline int rcv_wqe_size (int max_sge) -{ -return sizeof (struct rxe_recv_wqe ) + - max_sge *sizeof (struct ib_sge ); -} -void free_rd_atomic_resource (struct resp_res *res);static inline void rxe_advance_resp_resource (struct rxe_qp *qp)
drivers/infiniband/sw/rxe/rxe_srq.c +34 -21
103 lines not on this page · raw message
Signature · 1 line
2.39.2
kernel test robot
build botRe: [PATCH for-next 3/3] RDMA/rxe: Fix rxe_m-dify_srq
kernel test robot noticed the following build warnings:
Hi Bob,
kernel test robot noticed the following build warnings:
[auto build test WARNING on 830f93f47068b1632cc127871fbf27e918efdf46]
url: https://github.com/intel-lab-lkp/linux/commits/Bob-Pearson/RDMA-rxe-Move-work-queue-code-to-subroutines/20230620-042342 base: 830f93f47068b1632cc127871fbf27e918efdf46 patch link: https://lore.kernel.org/r/20230619202110.45680-4-rpearsonhpe%40gmail.com patch subject: [PATCH for-next 3/3] RDMA/rxe: Fix rxe_m-dify_srq config: x86_64-randconfig-a005-20230620 (https://download.01.org/0day-ci/archive/20230620/[email protected]/config) compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a) reproduce: (https://download.01.org/0day-ci/archive/20230620/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> drivers/infiniband/sw/rxe/rxe_srq.c:73:18: warning: variable 'q' is uninitialized when used here [-Wuninitialized]
srq->rq.queue = q;
^
drivers/infiniband/sw/rxe/rxe_srq.c:50:21: note: initialize the variable 'q' to silence this warning
struct rxe_queue *q;
^
= NULL1 warning generated.
vim +/q +73 drivers/infiniband/sw/rxe/rxe_srq.c
8700e3e7c4857d Moni Shoua 2016-06-16 43
8700e3e7c4857d Moni Shoua 2016-06-16 44 int rxe_srq_from_init(struct rxe_dev *rxe, struct rxe_srq *srq,
ff23dfa134576e Shamir Rabinovitch 2019-03-31 45 struct ib_srq_init_attr *init, struct ib_udata *udata,
0c43ab371bcb07 Jason Gunthorpe 2018-03-13 46 struct rxe_create_srq_resp __user *uresp)
8700e3e7c4857d Moni Shoua 2016-06-16 47 {
8700e3e7c4857d Moni Shoua 2016-06-16 48 int err;
06e1f13ffd8b02 Bob Pearson 2023-06-19 49 int wqe_size;
8700e3e7c4857d Moni Shoua 2016-06-16 50 struct rxe_queue *q;
8700e3e7c4857d Moni Shoua 2016-06-16 51
8700e3e7c4857d Moni Shoua 2016-06-16 52 srq->ibsrq.event_handler = init->event_handler;
8700e3e7c4857d Moni Shoua 2016-06-16 53 srq->ibsrq.srq_context = init->srq_context;
8700e3e7c4857d Moni Shoua 2016-06-16 54 srq->limit = init->attr.srq_limit;
02827b67085162 Bob Pearson 2021-11-03 55 srq->srq_num = srq->elem.index;
8700e3e7c4857d Moni Shoua 2016-06-16 56 srq->rq.max_wr = init->attr.max_wr;
8700e3e7c4857d Moni Shoua 2016-06-16 57 srq->rq.max_sge = init->attr.max_sge;
8700e3e7c4857d Moni Shoua 2016-06-16 58
06e1f13ffd8b02 Bob Pearson 2023-06-19 59 wqe_size = sizeof(struct rxe_recv_wqe) +
06e1f13ffd8b02 Bob Pearson 2023-06-19 60 srq->rq.max_sge*sizeof(struct ib_sge);
8700e3e7c4857d Moni Shoua 2016-06-16 61
8700e3e7c4857d Moni Shoua 2016-06-16 62 spin_lock_init(&srq->rq.producer_lock);
8700e3e7c4857d Moni Shoua 2016-06-16 63 spin_lock_init(&srq->rq.consumer_lock);
8700e3e7c4857d Moni Shoua 2016-06-16 64
06e1f13ffd8b02 Bob Pearson 2023-06-19 65 srq->rq.queue = rxe_queue_init(rxe, &srq->rq.max_wr, wqe_size,
06e1f13ffd8b02 Bob Pearson 2023-06-19 66 QUEUE_TYPE_FROM_CLIENT);
06e1f13ffd8b02 Bob Pearson 2023-06-19 67 if (!srq->rq.queue) {
0e6090024b3ebf Bob Pearson 2022-11-03 68 rxe_dbg_srq(srq, "Unable to allocate queue\n");
06e1f13ffd8b02 Bob Pearson 2023-06-19 69 err = -ENOMEM;
06e1f13ffd8b02 Bob Pearson 2023-06-19 70 goto err_out;
8700e3e7c4857d Moni Shoua 2016-06-16 71 }
8700e3e7c4857d Moni Shoua 2016-06-16 72
8700e3e7c4857d Moni Shoua 2016-06-16 @73 srq->rq.queue = q;
8700e3e7c4857d Moni Shoua 2016-06-16 74
ff23dfa134576e Shamir Rabinovitch 2019-03-31 75 err = do_mmap_info(rxe, uresp ? &uresp->mi : NULL, udata, q->buf,
8700e3e7c4857d Moni Shoua 2016-06-16 76 q->buf_size, &q->ip);
aae0484e15f062 Zhu Yanjun 2018-09-30 77 if (err) {
06e1f13ffd8b02 Bob Pearson 2023-06-19 78 rxe_dbg_srq(srq, "Unable to init mmap info for caller\n");
06e1f13ffd8b02 Bob Pearson 2023-06-19 79 goto err_free;
aae0484e15f062 Zhu Yanjun 2018-09-30 80 }
8700e3e7c4857d Moni Shoua 2016-06-16 81
06e1f13ffd8b02 Bob Pearson 2023-06-19 82 init->attr.max_wr = srq->rq.max_wr;
06e1f13ffd8b02 Bob Pearson 2023-06-19 83
0c43ab371bcb07 Jason Gunthorpe 2018-03-13 84 if (uresp) {
0c43ab371bcb07 Jason Gunthorpe 2018-03-13 85 if (copy_to_user(&uresp->srq_num, &srq->srq_num,
aae0484e15f062 Zhu Yanjun 2018-09-30 86 sizeof(uresp->srq_num))) {
aae0484e15f062 Zhu Yanjun 2018-09-30 87 rxe_queue_cleanup(q);
8700e3e7c4857d Moni Shoua 2016-06-16 88 return -EFAULT;
8700e3e7c4857d Moni Shoua 2016-06-16 89 }
aae0484e15f062 Zhu Yanjun 2018-09-30 90 }
0c43ab371bcb07 Jason Gunthorpe 2018-03-13 91
8700e3e7c4857d Moni Shoua 2016-06-16 92 return 0;
06e1f13ffd8b02 Bob Pearson 2023-06-19 93
06e1f13ffd8b02 Bob Pearson 2023-06-19 94 err_free:
06e1f13ffd8b02 Bob Pearson 2023-06-19 95 vfree(q->buf);
06e1f13ffd8b02 Bob Pearson 2023-06-19 96 kfree(q);
06e1f13ffd8b02 Bob Pearson 2023-06-19 97 err_out:
06e1f13ffd8b02 Bob Pearson 2023-06-19 98 return err;
8700e3e7c4857d Moni Shoua 2016-06-16 99 }8700e3e7c4857d Moni Shoua 2016-06-16 100
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Zhu Yanjun
maintainerQuoted 9 lines · Bob Pearson
If a call to
rxe_create_qp()fails inrxe_qp_from_init()rxe_cleanup(qp)will be called. This code currently does not correctly handle cases where not all qp resources are allocated and can seg fault as reported below. The first two patches cleanup cases where this happens. The third patch corrects an error inrxe_srq.cwhere if caller requests a change in the srq size the correct new value is not returned to caller.
Quoted 5 lines
Reported-by [email protected]Fixes 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c")Fixes fbdeb828a21f ("RDMA/rxe: Cleanup error state handling in rxe_comp.c")Signed-off-by Bob Pearson <[email protected]>
Can not apply these commits to Linux 6.4-rc7.
Zhu Yanjun
Quoted 18 lines
Bob Pearson (3):
RDMA/rxe: Move work queue code to subroutines
RDMA/rxe: Fix unsafe drain work queue code
RDMA/rxe: Fixrxe_modify_srqdrivers/infiniband/sw/rxe/rxe_comp.c | 4 + drivers/infiniband/sw/rxe/rxe_loc.h | 6 - drivers/infiniband/sw/rxe/rxe_qp.c | 163 ++++++++++++++++++--------- drivers/infiniband/sw/rxe/rxe_resp.c | 4 + drivers/infiniband/sw/rxe/rxe_srq.c | 55 +++++---- 5 files changed, 150 insertions(+), 82 deletions(-)base-commit:
830f93f47068b1632cc127871fbf27e918efdf46--
2.39.2
Quoted 16 lines · Zhu Yanjun
Quoted 9 lines · Bob Pearson
If a call to
rxe_create_qp()fails inrxe_qp_from_init()rxe_cleanup(qp)will be called. This code currently does not correctly handle cases where not all qp resources are allocated and can seg fault as reported below. The first two patches cleanup cases where this happens. The third patch corrects an error inrxe_srq.cwhere if caller requests a change in the srq size the correct new value is not returned to caller.Quoted 5 lines
Reported-by [email protected]Fixes 49dc9c1f0c7e ("RDMA/rxe: Cleanup reset state handling in rxe_resp.c")Fixes fbdeb828a21f ("RDMA/rxe: Cleanup error state handling in rxe_comp.c")Signed-off-by Bob Pearson <[email protected]>
Quoted 22 lines
Can not apply these commits to Linux 6.4-rc7.
Zhu Yanjun
Quoted 18 lines
Bob Pearson (3):
RDMA/rxe: Move work queue code to subroutines
RDMA/rxe: Fix unsafe drain work queue code
RDMA/rxe: Fixrxe_modify_srqdrivers/infiniband/sw/rxe/rxe_comp.c | 4 + drivers/infiniband/sw/rxe/rxe_loc.h | 6 - drivers/infiniband/sw/rxe/rxe_qp.c | 163 ++++++++++++++++++--------- drivers/infiniband/sw/rxe/rxe_resp.c | 4 + drivers/infiniband/sw/rxe/rxe_srq.c | 55 +++++---- 5 files changed, 150 insertions(+), 82 deletions(-)base-commit:
830f93f47068b1632cc127871fbf27e918efdf46--
2.39.2
They applied to current for-next.