Misc changes for siw
Subsystems (2)
v2 cover Misc changes for siw
V2 changes:
1. add Fixes lines for the first two patches per Leon
Hi,
The first one fix below calltrace which could happen if siw_connect goto error (I manually set rv to -1 after siw_send_mpareqrep to trigger it) after cep is allocated.
[ 97.341035] ------------[ cut here ]------------ [ 97.341037] WARNING: CPU: 0 PID: 143 at drivers/infiniband/sw/siw/siw_cm.c:444 siw_cep_put+0x1c5/0x1e0 [siw] ... [ 97.341126] CPU: 0 PID: 143 Comm: kworker/u4:4 Tainted: G OE 6.5.0-rc3+ #16 [ 97.341128] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.0-0-gd239552c-rebuilt.opensuse.org 04/01/2014 [ 97.341130] Workqueue: rdma_cm cma_work_handler [rdma_cm] [ 97.341137] RIP: 0010:siw_cep_put+0x1c5/0x1e0 [siw] ... [ 97.341159] Call Trace: [ 97.341160] <TASK> [ 97.341162] ? show_regs+0x72/0x90 [ 97.341166] ? siw_cep_put+0x1c5/0x1e0 [siw] [ 97.341170] ? __warn+0x8d/0x1a0 [ 97.341175] ? siw_cep_put+0x1c5/0x1e0 [siw] [ 97.341180] ? report_bug+0x1f9/0x250 [ 97.341185] ? handle_bug+0x46/0x90 [ 97.341188] ? exc_invalid_op+0x19/0x80 [ 97.341190] ? asm_exc_invalid_op+0x1b/0x20 [ 97.341196] ? siw_cep_put+0x1c5/0x1e0 [siw] [ 97.341204] siw_connect+0x474/0x780 [siw] [ 97.341211] iw_cm_connect+0x1ca/0x250 [iw_cm] [ 97.341216] rdma_connect_locked+0x1bf/0x940 [rdma_cm] [ 97.341227] nvme_rdma_cm_handler+0x5d7/0x9c0 [nvme_rdma] [ 97.341235] cma_cm_event_handler+0x4f/0x170 [rdma_cm] [ 97.341241] cma_work_handler+0x6a/0xe0 [rdma_cm] [ 97.341247] process_one_work+0x2bd/0x590
...
The second one make the debug message consistent with the condition, and the last one cleanup code a bit. Pls help to review them.
Thanks,
Guoqing
Guoqing Jiang (3):
RDMA/siw: Balance the reference of cep->kref in the error path RDMA/siw: Correct wrong debug message
RDMA/siw: Call llist_reverse_order in siw_run_sq
drivers/infiniband/sw/siw/siw_cm.c | 1 - drivers/infiniband/sw/siw/siw_qp_tx.c | 12 +----------- drivers/infiniband/sw/siw/siw_verbs.c | 2 +- 3 files changed, 2 insertions(+), 13 deletions(-)
Signature · 1 line
2.35.3
PATCH v2 1/3 RDMA/siw: Balance the reference of cep->kref in the error path
The siw_connect can go to err in below after cep is allocated successfully:
1. If siw_cm_alloc_work returns failure. In this case socket is not assoicated with cep so siw_cep_put can't be called by siw_socket_disassoc. We need to call siw_cep_put twice since cep->kref is increased once after it was initialized.
2. If siw_cm_queue_work can't find a work, which means siw_cep_get is not called in siw_cm_queue_work, so cep->kref is increased twice by siw_cep_get and when associate socket with cep after it was initialized. So we need to call siw_cep_put three times (one in siw_socket_disassoc).
3. siw_send_mpareqrep returns error, this scenario is similar as 2.
So we need to remove one siw_cep_put in the error path.
--- drivers/infiniband/sw/siw/siw_cm.c | 1 - 1 file changed, 1 deletion(-)
drivers/infiniband/sw/siw/siw_cm.c +0 -1
@@ -1501,7 +1501,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) cep->cm_id =NULL ; id->rem_ref (id); -siw_cep_put (cep); qp->cep =NULL ;siw_cep_put (cep);
Signature · 1 line
2.35.3
RE: [PATCH V2 1/3] RDMA/siw: Balance the reference of cep->kref in the error path
Quoted 45 lines
-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Monday, 21 August 2023 10:48
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]
Subject: [EXTERNAL] [PATCH V2 1/3] RDMA/siw: Balance the reference of cep-Quoted 1 line
kref in the error path
The
siw_connectcan go to err in below after cep is allocated successfully:1. If
siw_cm_alloc_workreturns failure. In this case socket is not assoicated with cep sosiw_cep_putcan't be called bysiw_socket_disassoc. We need to callsiw_cep_puttwice sincecep->krefis increased once after it was initialized.2. If
siw_cm_queue_workcan't find a work, which meanssiw_cep_getis not called insiw_cm_queue_work, socep->krefis increased twice bysiw_cep_getand when associate socket with cep after it was initialized. So we need to callsiw_cep_putthree times (one insiw_socket_disassoc).3.
siw_send_mpareqrepreturns error, this scenario is similar as 2.So we need to remove one
siw_cep_putin the error path.Fixes 6c52fdc244b5 ("rdma/siw: connection management")Signed-off-by Guoqing Jiang <[email protected]>--- drivers/infiniband/sw/siw/siw_cm.c | 1 - 1 file changed, 1 deletion(-)drivers/infiniband/sw/siw/siw_cm.c +0
-1@@ -1501,7 +1501,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) cep->cm_id = NULL; id->rem_ref (id); -siw_cep_put (cep); qp->cep =NULL ;
Quoted 3 lines
siw_cep_put(cep);
--
2.35.3
That's correct, thank you!
PATCH v2 2/3 RDMA/siw: Correct wrong debug message
In reply to Guoqing JiangWe need to print num_sle first then pbl->max_buf per the condition. Also replace mem->pbl with pbl while at it.
--- drivers/infiniband/sw/siw/siw_verbs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
@@ -1494,7 +1494,7 @@ int siw_map_mr_sg(struct ib_mr *base_mr, struct scatterlist *sl, int num_sle,if (pbl->max_buf < num_sle) {siw_dbg_mem (mem,"too many SGE's: %d > %d\n" , - mem->pbl->max_buf, num_sle); + num_sle, pbl->max_buf);return -ENOMEM ; }for_each_sg (sl, slp, num_sle, i) {
Signature · 1 line
2.35.3
RE: [PATCH V2 2/3] RDMA/siw: Correct wrong debug message
Quoted 32 lines
-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Monday, 21 August 2023 10:48
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]
Subject: [EXTERNAL] [PATCH V2 2/3] RDMA/siw: Correct wrong debug messageWe need to print
num_slefirst thenpbl->max_bufper the condition. Also replacemem->pblwith pbl while at it.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Guoqing Jiang <[email protected]>--- drivers/infiniband/sw/siw/siw_verbs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/infiniband/sw/siw/siw_verbs.c +1
-1@@ -1494,7 +1494,7 @@ int siw_map_mr_sg(struct ib_mr *base_mr, struct scatterlist *sl, int num_sle, if (pbl->max_buf < num_sle) { siw_dbg_mem(mem, "too many SGE's: %d > %d\n", - mem->pbl->max_buf, num_sle); + num_sle, pbl->max_buf);return -ENOMEM ; }
Quoted 3 lines
for_each_sg(sl, slp, num_sle, i){
--
2.35.3
makes sense, thank you!
PATCH v2 3/3 RDMA/siw: Call llist_reverse_order in siw_run_sq
In reply to Guoqing JiangWe can call the function to get fifo list.
--- drivers/infiniband/sw/siw/siw_qp_tx.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)
drivers/infiniband/sw/siw/siw_qp_tx.c +1 -11
@@ -1229,17 +1229,7 @@ int siw_run_sq(void *data)break ; active =llist_del_all (&tx_task->active); -/* -* llist_del_all returns a list with newest entry first. -* Re-order list for fairness among QP's. -*/ -while (active) { -struct llist_node *tmp = active; - - active =llist_next (active); - tmp->next = fifo_list; - fifo_list = tmp; - } + fifo_list =llist_reverse_order (active);while (fifo_list) { qp = container_of(fifo_list, struct siw_qp, tx_list); fifo_list =llist_next (fifo_list);
Signature · 1 line
2.35.3
RE: [PATCH V2 3/3] RDMA/siw: Call llist_reverse_order in siw_run_sq
Quoted 42 lines
-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Monday, 21 August 2023 10:48
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]
Subject: [EXTERNAL] [PATCH V2 3/3] RDMA/siw: Callllist_reverse_orderinsiw_run_sqWe can call the function to get fifo list.
Signed-off-by Guoqing Jiang <[email protected]>--- drivers/infiniband/sw/siw/siw_qp_tx.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)drivers/infiniband/sw/siw/siw_qp_tx.c +1
-12@@ -1229,17 +1229,7 @@ int siw_run_sq(void *data)break ; active =llist_del_all (&tx_task->active); -/* -* llist_del_all returns a list with newest entry first. -* Re-order list for fairness among QP's. -*/ -while (active) { -struct llist_node *tmp = active; - - active =llist_next (active); - tmp->next = fifo_list; - fifo_list = tmp; - } + fifo_list =llist_reverse_order (active);while (fifo_list) { qp = container_of(fifo_list, struct siw_qp, tx_list); fifo_list =llist_next (fifo_list); --
Quoted 1 line
2.35.3
Oh yes, that function already exists. Many thanks! I'd keep the comment, since it might be not obvious why we reverse the list.
Re: [PATCH V2 3/3] RDMA/siw: Call llist_reverse_order in siw_run_sq
Quoted 37 lines · Bernard Metzler
Quoted 37 lines
-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Monday, 21 August 2023 10:48
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]
Subject: [EXTERNAL] [PATCH V2 3/3] RDMA/siw: Callllist_reverse_orderinsiw_run_sqWe can call the function to get fifo list.
Signed-off-by Guoqing Jiang <[email protected]>--- drivers/infiniband/sw/siw/siw_qp_tx.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)drivers/infiniband/sw/siw/siw_qp_tx.c +1
-11@@ -1229,17 +1229,7 @@ int siw_run_sq(void *data)break ; active =llist_del_all (&tx_task->active); -/* -* llist_del_all returns a list with newest entry first. -* Re-order list for fairness among QP's. -*/ -while (active) { -struct llist_node *tmp = active; - - active =llist_next (active); - tmp->next = fifo_list; - fifo_list = tmp; - } + fifo_list =llist_reverse_order (active);
Quoted 8 lines
Quoted 5 lines
drivers/infiniband/sw/siw/siw_qp_tx.c +0
-1while (fifo_list) { qp = container_of(fifo_list, struct siw_qp, tx_list); fifo_list =llist_next (fifo_list); --2.35.3
Oh yes, that function already exists. Many thanks! I'd keep the comment, since it might be not obvious why we reverse the list.
Ok, will add them back.
Quoted 1 line
Acked-by Bernard Metzler <[email protected]>
Appreciate for your review!
Thanks,
Guoqing