Skip to contentSkip to conversation

Misc update for rtrs

PATCH Applied Jack Wang 3 participants 24 messages Dec 17, 2020 – Jan 15, 2021

Subsystems (2)
lore
Share & tools
lore

Jack Wang

author
#0 lore raw

v2 cover Misc update for rtrs

Hi Jason, hi Doug,

Please consider to include following changes to the next merge window.

It contains a few bugfix and cleanup:
- Fix memory leak incase of failure in kobj_init_and_add in both clt/srv
- reduce memory footprint by set proper limit when create QP
- fix missing wr_cqe in a few cases on srv, it could lead to crash in error case.
- remove the SIGNAL flag for heartbeat, otherwise it could mess around the send_wr_awail accouting.
- flush on going session closing to release the memory presure on server.
- other misc fix and cleanup.

The patches are created based on rdma/for-next.

V2->V1
* more descprition for the patches above as requested by Jason, also include Fixes tag for bugfix.
* suppress the lockdep warning for PATCH 2 Occasionally flush ongoing session closing with comment.
* new bugfix PATCH 19 RDMA/rtrs: Fix KASAN: stack-out-of-bounds bug

Thanks!

Guoqing Jiang (8):
RDMA/rtrs-srv: Jump to dereg_mr label if allocate iu fails RDMA/rtrs: Call kobject_put in the failure path RDMA/rtrs-clt: Consolidate rtrs_clt_destroy_sysfs_root_{folder,files} RDMA/rtrs-clt: Kill wait_for_inflight_permits RDMA/rtrs-clt: Remove unnecessary 'goto out' RDMA/rtrs-clt: Kill rtrs_clt_change_state
RDMA/rtrs-clt: Rename __rtrs_clt_change_state to rtrs_clt_change_state RDMA/rtrs-clt: Refactor the failure cases in alloc_clt

Jack Wang (11):
RDMA/rtrs: Extend ibtrs_cq_qp_create RMDA/rtrs-srv: Occasionally flush ongoing session closing RDMA/rtrs-srv: Release lock before call into close_sess RDMA/rtrs-srv: Use sysfs_remove_file_self for disconnect RDMA/rtrs-clt: Set mininum limit when create QP RDMA/rtrs-srv: Fix missing wr_cqe
RDMA/rtrs: Do not signal for heatbeat RDMA/rtrs-clt: Use bitmask to check sess->flags RDMA/rtrs-srv: Do not signal REG_MR RDMA/rtrs-srv: Init wr_cnt as 1
RDMA/rtrs: Fix KASAN: stack-out-of-bounds bug

 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c |  11 +-
 drivers/infiniband/ulp/rtrs/rtrs-clt.c       | 120 +++++++++----------
 drivers/infiniband/ulp/rtrs/rtrs-clt.h       |   3 +-
 drivers/infiniband/ulp/rtrs/rtrs-pri.h       |   5 +-
 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c |   5 +-
 drivers/infiniband/ulp/rtrs/rtrs-srv.c       |  34 ++++--
 drivers/infiniband/ulp/rtrs/rtrs.c           |  32 ++---
 7 files changed, 110 insertions(+), 100 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v2 1/19 RDMA/rtrs: Extend ibtrs_cq_qp_create

rtrs does not have same limit for both max_send_wr and max_recv_wr, To allow client and server set different values, export in a separate parameter for rtrs_cq_qp_create.

Also fix the type accordingly, u32 should be used instead of u16.

Fixes c0894b3ea69d ("RDMA/rtrs: core: lib functions shared between client and server modules")
---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c |  4 ++--
 drivers/infiniband/ulp/rtrs/rtrs-pri.h |  5 +++--
 drivers/infiniband/ulp/rtrs/rtrs-srv.c |  5 +++--
 drivers/infiniband/ulp/rtrs/rtrs.c     | 14 ++++++++------
 4 files changed, 16 insertions(+), 12 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +2 -2
index 67f86c405a26..719254fc83a1 100644
@@ -1511,7 +1511,7 @@ static void destroy_con(struct rtrs_clt_con *con)
 static int create_con_cq_qp(struct rtrs_clt_con *con)
 {
 	struct rtrs_clt_sess *sess = to_clt_sess(con->c.sess);
-	u16 wr_queue_size;
+	u32 wr_queue_size;
 	int err, cq_vector;
 	struct rtrs_msg_rkey_rsp *rsp;
 
@@ -1573,7 +1573,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
 	cq_vector = con->cpu % sess->s.dev->ib_dev->num_comp_vectors;
 	err = rtrs_cq_qp_create(&sess->s, &con->c, sess->max_send_sge,
 				 cq_vector, wr_queue_size, wr_queue_size,
-				 IB_POLL_SOFTIRQ);
+				 wr_queue_size, IB_POLL_SOFTIRQ);
 	/*
 	 * In case of error we do not bother to clean previous allocations,
 	 * since destroy_con_cq_qp() must be called.
drivers/infiniband/ulp/rtrs/rtrs-pri.h +3 -2
index 3f2918671dbe..d5621e6fad1b 100644
@@ -303,8 +303,9 @@ int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe,
 				   struct ib_send_wr *head);
 
 int rtrs_cq_qp_create(struct rtrs_sess *rtrs_sess, struct rtrs_con *con,
-		      u32 max_send_sge, int cq_vector, u16 cq_size,
-		      u16 wr_queue_size, enum ib_poll_context poll_ctx);
+		      u32 max_send_sge, int cq_vector, int cq_size,
+		      u32 max_send_wr, u32 max_recv_wr,
+		      enum ib_poll_context poll_ctx);
 void rtrs_cq_qp_destroy(struct rtrs_con *con);
 
 void rtrs_init_hb(struct rtrs_sess *sess, struct ib_cqe *cqe,
drivers/infiniband/ulp/rtrs/rtrs-srv.c +3 -2
index c42fd470c4eb..ed4628f032bb 100644
@@ -1586,7 +1586,7 @@ static int create_con(struct rtrs_srv_sess *sess,
 	struct rtrs_sess *s = &sess->s;
 	struct rtrs_srv_con *con;
 
-	u16 cq_size, wr_queue_size;
+	u32 cq_size, wr_queue_size;
 	int err, cq_vector;
 
 	con = kzalloc(sizeof(*con), GFP_KERNEL);
@@ -1630,7 +1630,8 @@ static int create_con(struct rtrs_srv_sess *sess,
 
 	/* TODO: SOFTIRQ can be faster, but be careful with softirq context */
 	err = rtrs_cq_qp_create(&sess->s, &con->c, 1, cq_vector, cq_size,
-				 wr_queue_size, IB_POLL_WORKQUEUE);
+				 wr_queue_size, wr_queue_size,
+				 IB_POLL_WORKQUEUE);
 	if (err) {
 		rtrs_err(s, "rtrs_cq_qp_create(), err: %d\n", err);
 		goto free_con;
drivers/infiniband/ulp/rtrs/rtrs.c +8 -6
index 2e3a849e0a77..df52427f1710 100644
@@ -231,14 +231,14 @@ static int create_cq(struct rtrs_con *con, int cq_vector, u16 cq_size,
 }
 
 static int create_qp(struct rtrs_con *con, struct ib_pd *pd,
-		     u16 wr_queue_size, u32 max_sge)
+		     u32 max_send_wr, u32 max_recv_wr, u32 max_sge)
 {
 	struct ib_qp_init_attr init_attr = {NULL};
 	struct rdma_cm_id *cm_id = con->cm_id;
 	int ret;
 
-	init_attr.cap.max_send_wr = wr_queue_size;
-	init_attr.cap.max_recv_wr = wr_queue_size;
+	init_attr.cap.max_send_wr = max_send_wr;
+	init_attr.cap.max_recv_wr = max_recv_wr;
 	init_attr.cap.max_recv_sge = 1;
 	init_attr.event_handler = qp_event_handler;
 	init_attr.qp_context = con;
@@ -260,8 +260,9 @@ static int create_qp(struct rtrs_con *con, struct ib_pd *pd,
 }
 
 int rtrs_cq_qp_create(struct rtrs_sess *sess, struct rtrs_con *con,
-		       u32 max_send_sge, int cq_vector, u16 cq_size,
-		       u16 wr_queue_size, enum ib_poll_context poll_ctx)
+		       u32 max_send_sge, int cq_vector, int cq_size,
+		       u32 max_send_wr, u32 max_recv_wr,
+		       enum ib_poll_context poll_ctx)
 {
 	int err;
 
@@ -269,7 +270,8 @@ int rtrs_cq_qp_create(struct rtrs_sess *sess, struct rtrs_con *con,
 	if (err)
 		return err;
 
-	err = create_qp(con, sess->dev->ib_pd, wr_queue_size, max_send_sge);
+	err = create_qp(con, sess->dev->ib_pd, max_send_wr, max_recv_wr,
+			max_send_sge);
 	if (err) {
 		ib_free_cq(con->cq);
 		con->cq = NULL;
Signature · 1 line
2.25.1
author
#2 lore raw

PATCH v2 2/19 RMDA/rtrs-srv: Occasionally flush ongoing session closing

In reply to Jack Wang

If there are many establishments/teardowns, we need to make sure we do not consume too much system memory. Thus let on going session closing to finish before accepting new connection.

In cma_ib_req_handler, the conn_id is newly created holding handler_mutex when call this function, and flush_workqueue wait for close_work to finish, in close_work rdma_destroy_id will be called, which will hold the handler_mutex, but they are mutex for different rdma_cm_id.

To avoid circular locking lockdep warnings, disable lockdep around flush_workqueue in rtrs_rdma_connect, it's false positive.

Inspired by commit 777dc82395de ("nvmet-rdma: occasionally flush ongoing controller teardown")

---
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +14 -0
index ed4628f032bb..eb7bb93884e7 100644
@@ -1791,6 +1791,20 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id,
 		err = -ENOMEM;
 		goto reject_w_err;
 	}
+	if (!cid) {
+		/* Let inflight session teardown complete *
+		 *
+		 * Avoid circular locking lockdep warnings.
+		 * cma_ib_req_handler, the conn_id is newly created holding
+		 * handler_mutex when call this function, and flush_workqueue
+		 * wait for close_work to finish, in close_work rdma_destroy_id
+		 * will be called, which will hold the handler_mutex, but they
+		 * are mutex for different rdma_cm_id.
+		 */
+		lockdep_off();
+		flush_workqueue(rtrs_wq);
+		lockdep_on();
+	}
 	mutex_lock(&srv->paths_mutex);
 	sess = __find_sess(srv, &msg->sess_uuid);
 	if (sess) {
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCHv2 for-next 02/19] RMDA/rtrs-srv: Occasionally flush ongoing session closing

Quoted 3 lines · Jack Wang

If there are many establishments/teardowns, we need to make sure we do not consume too much system memory. Thus let on going session closing to finish before accepting new connection.

Then just limit it, why this scheme?

Quoted 5 lines

In cma_ib_req_handler, the conn_id is newly created holding handler_mutex when call this function, and flush_workqueue wait for close_work to finish, in close_work rdma_destroy_id will be called, which will hold the handler_mutex, but they are mutex for different rdma_cm_id.

No, there are multiple handler locks held here, and the new one is already marked nested, so isn't even the thing triggering lockdep.

The locking for CM is already bonkers, I don't want to see drivers turning off lockdep. How are you sure that work queue doesn't become (and won't ever in the future) become entangled with the listening handler_mutex?

Jason

author
#4 lore raw

Re: [PATCHv2 for-next 02/19] RMDA/rtrs-srv: Occasionally flush ongoing session closing

Hi Jason,

Quoted 7 lines · Jason Gunthorpe
Quoted 3 lines · Jack Wang

If there are many establishments/teardowns, we need to make sure we do not consume too much system memory. Thus let on going session closing to finish before accepting new connection.

Then just limit it, why this scheme?

Will think about it, thanks for the suggestion.

Quoted 7 lines
Quoted 6 lines

In cma_ib_req_handler, the conn_id is newly created holding handler_mutex when call this function, and flush_workqueue wait for close_work to finish, in close_work rdma_destroy_id will be called, which will hold the handler_mutex, but they are mutex for different rdma_cm_id.

Quoted 7 lines

No, there are multiple handler locks held here, and the new one is already marked nested, so isn't even the thing triggering lockdep.

The locking for CM is already bonkers, I don't want to see drivers turning off lockdep. How are you sure that work queue doesn't become (and won't ever in the future) become entangled with the listening handler_mutex?

IIUC, only new created conn_id is passed and save in ULP. but I understand your concerns,
I will drop this patch, think about other solution. Do you need a resend/rebase for the reset of the patchset?

Quoted 2 lines

Jason

Thanks!

author
#5 lore raw

PATCH v2 3/19 RDMA/rtrs-srv: Release lock before call into close_sess

In reply to Jack Wang

In this error case, we don't need hold mutex to call close_sess.

Fixes 9cb837480424 ("RDMA/rtrs: server: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +1 -1
index eb7bb93884e7..6771fb4a1110 100644
@@ -1877,8 +1877,8 @@ static int rtrs_rdma_connect(struct rdma_cm_id *cm_id,
 	return rtrs_rdma_do_reject(cm_id, -ECONNRESET);
 
 close_and_return_err:
-	close_sess(sess);
 	mutex_unlock(&srv->paths_mutex);
+	close_sess(sess);
 
 	return err;
 }
Signature · 1 line
2.25.1
author
#6 lore raw

PATCH v2 4/19 RDMA/rtrs-srv: Use sysfs_remove_file_self for disconnect

In reply to Jack Wang

Remove self first to avoid deadlock, we don't want to use close_work to remove sess sysfs.

Fixes 91b11610af8d ("RDMA/rtrs: server: sysfs interface functions")
---
 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c +2 -0
index d2edff3b8f0d..cca3a0acbabc 100644
@@ -51,6 +51,8 @@ static ssize_t rtrs_srv_disconnect_store(struct kobject *kobj,
 	sockaddr_to_str((struct sockaddr *)&sess->s.dst_addr, str, sizeof(str));
 
 	rtrs_info(s, "disconnect for path %s requested\n", str);
+	/* first remove sysfs itself to avoid deadlock */
+	sysfs_remove_file_self(&sess->kobj, &attr->attr);
 	close_sess(sess);
 
 	return count;
Signature · 1 line
2.25.1
author
#7 lore raw

PATCH v2 5/19 RDMA/rtrs-clt: Set mininum limit when create QP

In reply to Jack Wang

Currently rtrs when create_qp use a coarse numbers (bigger in general), which leads to hardware create more resources which only waste memory with no benefits.

- SERVICE con,
For max_send_wr/max_recv_wr, it's 2 times SERVICE_CON_QUEUE_DEPTH + 2

- IO con
For max_send_wr/max_recv_wr, it's sess->queue_depth * 3 + 1

Fixes 6a98d71daea1 ("RDMA/rtrs: client: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +12 -7
index 719254fc83a1..b3fb5fb93815 100644
@@ -1511,7 +1511,7 @@ static void destroy_con(struct rtrs_clt_con *con)
 static int create_con_cq_qp(struct rtrs_clt_con *con)
 {
 	struct rtrs_clt_sess *sess = to_clt_sess(con->c.sess);
-	u32 wr_queue_size;
+	u32 max_send_wr, max_recv_wr, cq_size;
 	int err, cq_vector;
 	struct rtrs_msg_rkey_rsp *rsp;
 
@@ -1523,7 +1523,8 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
 		 * + 2 for drain and heartbeat
 		 * in case qp gets into error state
 		 */
-		wr_queue_size = SERVICE_CON_QUEUE_DEPTH * 3 + 2;
+		max_send_wr = SERVICE_CON_QUEUE_DEPTH * 2 + 2;
+		max_recv_wr = SERVICE_CON_QUEUE_DEPTH * 2 + 2;
 		/* We must be the first here */
 		if (WARN_ON(sess->s.dev))
 			return -EINVAL;
@@ -1555,25 +1556,29 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
 
 		/* Shared between connections */
 		sess->s.dev_ref++;
-		wr_queue_size =
+		max_send_wr =
 			min_t(int, sess->s.dev->ib_dev->attrs.max_qp_wr,
 			      /* QD * (REQ + RSP + FR REGS or INVS) + drain */
 			      sess->queue_depth * 3 + 1);
+		max_recv_wr =
+			min_t(int, sess->s.dev->ib_dev->attrs.max_qp_wr,
+			      sess->queue_depth * 3 + 1);
 	}
 	/* alloc iu to recv new rkey reply when server reports flags set */
 	if (sess->flags == RTRS_MSG_NEW_RKEY_F || con->c.cid == 0) {
-		con->rsp_ius = rtrs_iu_alloc(wr_queue_size, sizeof(*rsp),
+		con->rsp_ius = rtrs_iu_alloc(max_recv_wr, sizeof(*rsp),
 					      GFP_KERNEL, sess->s.dev->ib_dev,
 					      DMA_FROM_DEVICE,
 					      rtrs_clt_rdma_done);
 		if (!con->rsp_ius)
 			return -ENOMEM;
-		con->queue_size = wr_queue_size;
+		con->queue_size = max_recv_wr;
 	}
+	cq_size = max_send_wr + max_recv_wr;
 	cq_vector = con->cpu % sess->s.dev->ib_dev->num_comp_vectors;
 	err = rtrs_cq_qp_create(&sess->s, &con->c, sess->max_send_sge,
-				 cq_vector, wr_queue_size, wr_queue_size,
-				 wr_queue_size, IB_POLL_SOFTIRQ);
+				 cq_vector, cq_size, max_send_wr,
+				 max_recv_wr, IB_POLL_SOFTIRQ);
 	/*
 	 * In case of error we do not bother to clean previous allocations,
 	 * since destroy_con_cq_qp() must be called.
Signature · 1 line
2.25.1
author
#8 lore raw

PATCH v2 6/19 RDMA/rtrs-srv: Jump to dereg_mr label if allocate iu fails

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

The rtrs_iu_free is called in rtrs_iu_alloc if memory is limited, so we don't need to free the same iu again.

Fixes 9cb837480424 ("RDMA/rtrs: server: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +1 -2
index 6771fb4a1110..89e938a1a4fb 100644
@@ -651,7 +651,7 @@ static int map_cont_bufs(struct rtrs_srv_sess *sess)
 			if (!srv_mr->iu) {
 				err = -ENOMEM;
 				rtrs_err(ss, "rtrs_iu_alloc(), err: %d\n", err);
-				goto free_iu;
+				goto dereg_mr;
 			}
 		}
 		/* Eventually dma addr for each chunk can be cached */
@@ -667,7 +667,6 @@ static int map_cont_bufs(struct rtrs_srv_sess *sess)
 			srv_mr = &sess->mrs[mri];
 			sgt = &srv_mr->sgt;
 			mr = srv_mr->mr;
-free_iu:
 			rtrs_iu_free(srv_mr->iu, sess->s.dev->ib_dev, 1);
 dereg_mr:
 			ib_dereg_mr(mr);
Signature · 1 line
2.25.1
author
#9 lore raw

PATCH v2 7/19 RDMA/rtrs: Call kobject_put in the failure path

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

Per the comment of kobject_init_and_add, we need to free the memory by call kobject_put.

Fixes 215378b838df ("RDMA/rtrs: client: sysfs interface functions")
Fixes 91b11610af8d ("RDMA/rtrs: server: sysfs interface functions")
---
 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 2 ++
 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c +2 -0
index ba00f0de14ca..ad77659800cd 100644
@@ -408,6 +408,7 @@ int rtrs_clt_create_sess_files(struct rtrs_clt_sess *sess)
 				   "%s", str);
 	if (err) {
 		pr_err("kobject_init_and_add: %d\n", err);
+		kobject_put(&sess->kobj);
 		return err;
 	}
 	err = sysfs_create_group(&sess->kobj, &rtrs_clt_sess_attr_group);
@@ -419,6 +420,7 @@ int rtrs_clt_create_sess_files(struct rtrs_clt_sess *sess)
 				   &sess->kobj, "stats");
 	if (err) {
 		pr_err("kobject_init_and_add: %d\n", err);
+		kobject_put(&sess->stats->kobj_stats);
 		goto remove_group;
 	}
 
drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c +2 -1
index cca3a0acbabc..0a3886629cae 100644
@@ -236,6 +236,7 @@ static int rtrs_srv_create_stats_files(struct rtrs_srv_sess *sess)
 				   &sess->kobj, "stats");
 	if (err) {
 		rtrs_err(s, "kobject_init_and_add(): %d\n", err);
+		kobject_put(&sess->stats->kobj_stats);
 		return err;
 	}
 	err = sysfs_create_group(&sess->stats->kobj_stats,
@@ -292,8 +293,8 @@ int rtrs_srv_create_sess_files(struct rtrs_srv_sess *sess)
 	sysfs_remove_group(&sess->kobj, &rtrs_srv_sess_attr_group);
 put_kobj:
 	kobject_del(&sess->kobj);
-	kobject_put(&sess->kobj);
 destroy_root:
+	kobject_put(&sess->kobj);
 	rtrs_srv_destroy_once_sysfs_root_folders(sess);
 
 	return err;
Signature · 1 line
2.25.1
author
#10 lore raw

PATCH v2 8/19 RDMA/rtrs-clt: Consolidate rtrs_clt_destroy_sysfs_root_{folder,files}

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

Since the two functions are called together, let's consolidate them in a new function rtrs_clt_destroy_sysfs_root.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 9 +++------
 drivers/infiniband/ulp/rtrs/rtrs-clt.c       | 6 ++----
 drivers/infiniband/ulp/rtrs/rtrs-clt.h       | 3 +--
 3 files changed, 6 insertions(+), 12 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c +3 -6
index ad77659800cd..b6a0abf40589 100644
@@ -471,15 +471,12 @@ int rtrs_clt_create_sysfs_root_files(struct rtrs_clt *clt)
 	return sysfs_create_group(&clt->dev.kobj, &rtrs_clt_attr_group);
 }
 
-void rtrs_clt_destroy_sysfs_root_folders(struct rtrs_clt *clt)
+void rtrs_clt_destroy_sysfs_root(struct rtrs_clt *clt)
 {
+	sysfs_remove_group(&clt->dev.kobj, &rtrs_clt_attr_group);
+
 	if (clt->kobj_paths) {
 		kobject_del(clt->kobj_paths);
 		kobject_put(clt->kobj_paths);
 	}
 }
-
-void rtrs_clt_destroy_sysfs_root_files(struct rtrs_clt *clt)
-{
-	sysfs_remove_group(&clt->dev.kobj, &rtrs_clt_attr_group);
-}
drivers/infiniband/ulp/rtrs/rtrs-clt.c +2 -4
index b3fb5fb93815..99fc34950032 100644
@@ -2707,8 +2707,7 @@ struct rtrs_clt *rtrs_clt_open(struct rtrs_clt_ops *ops,
 		rtrs_clt_close_conns(sess, true);
 		kobject_put(&sess->kobj);
 	}
-	rtrs_clt_destroy_sysfs_root_files(clt);
-	rtrs_clt_destroy_sysfs_root_folders(clt);
+	rtrs_clt_destroy_sysfs_root(clt);
 	free_clt(clt);
 
 out:
@@ -2725,8 +2724,7 @@ void rtrs_clt_close(struct rtrs_clt *clt)
 	struct rtrs_clt_sess *sess, *tmp;
 
 	/* Firstly forbid sysfs access */
-	rtrs_clt_destroy_sysfs_root_files(clt);
-	rtrs_clt_destroy_sysfs_root_folders(clt);
+	rtrs_clt_destroy_sysfs_root(clt);
 
 	/* Now it is safe to iterate over all paths without locks */
 	list_for_each_entry_safe(sess, tmp, &clt->paths_list, s.entry) {
drivers/infiniband/ulp/rtrs/rtrs-clt.h +1 -2
index b8dbd701b3cb..a97a068c4c28 100644
@@ -243,8 +243,7 @@ ssize_t rtrs_clt_reset_all_help(struct rtrs_clt_stats *stats,
 /* rtrs-clt-sysfs.c */
 
 int rtrs_clt_create_sysfs_root_files(struct rtrs_clt *clt);
-void rtrs_clt_destroy_sysfs_root_folders(struct rtrs_clt *clt);
-void rtrs_clt_destroy_sysfs_root_files(struct rtrs_clt *clt);
+void rtrs_clt_destroy_sysfs_root(struct rtrs_clt *clt);
 
 int rtrs_clt_create_sess_files(struct rtrs_clt_sess *sess);
 void rtrs_clt_destroy_sess_files(struct rtrs_clt_sess *sess,
Signature · 1 line
2.25.1
author
#11 lore raw

PATCH v2 9/19 RDMA/rtrs-clt: Kill wait_for_inflight_permits

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

Let's wait the inflight permits before free it.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +6 -11
index 99fc34950032..6a5b72ad5ba1 100644
@@ -1318,6 +1318,12 @@ static int alloc_permits(struct rtrs_clt *clt)
 
 static void free_permits(struct rtrs_clt *clt)
 {
+	if (clt->permits_map) {
+		size_t sz = clt->queue_depth;
+
+		wait_event(clt->permits_wait,
+			   find_first_bit(clt->permits_map, sz) >= sz);
+	}
 	kfree(clt->permits_map);
 	clt->permits_map = NULL;
 	kfree(clt->permits);
@@ -2607,19 +2613,8 @@ static struct rtrs_clt *alloc_clt(const char *sessname, size_t paths_num,
 	return clt;
 }
 
-static void wait_for_inflight_permits(struct rtrs_clt *clt)
-{
-	if (clt->permits_map) {
-		size_t sz = clt->queue_depth;
-
-		wait_event(clt->permits_wait,
-			   find_first_bit(clt->permits_map, sz) >= sz);
-	}
-}
-
 static void free_clt(struct rtrs_clt *clt)
 {
-	wait_for_inflight_permits(clt);
 	free_permits(clt);
 	free_percpu(clt->pcpu_path);
 	mutex_destroy(&clt->paths_ev_mutex);
Signature · 1 line
2.25.1
author
#12 lore raw

PATCH v2 10/19 RDMA/rtrs-clt: Remove unnecessary 'goto out'

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

This is not needed since the label is just after the place.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 1 -
 1 file changed, 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +0 -1
index 6a5b72ad5ba1..d99fb1a1c194 100644
@@ -2434,7 +2434,6 @@ static int rtrs_send_sess_info(struct rtrs_clt_sess *sess)
 			err = -ECONNRESET;
 		else
 			err = -ETIMEDOUT;
-		goto out;
 	}
 
 out:
Signature · 1 line
2.25.1
author
#13 lore raw

PATCH v2 11/19 RDMA/rtrs-clt: Kill rtrs_clt_change_state

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

It is just a wrapper of rtrs_clt_change_state_get_old, and we can reuse rtrs_clt_change_state_get_old with add the checking of 'old_state' is valid or not.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 27 ++++++++++----------------
 1 file changed, 10 insertions(+), 17 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +10 -17
index d99fb1a1c194..39dc8423d7df 100644
@@ -1359,21 +1359,14 @@ static bool rtrs_clt_change_state_get_old(struct rtrs_clt_sess *sess,
 	bool changed;
 
 	spin_lock_irq(&sess->state_wq.lock);
-	*old_state = sess->state;
+	if (old_state)
+		*old_state = sess->state;
 	changed = __rtrs_clt_change_state(sess, new_state);
 	spin_unlock_irq(&sess->state_wq.lock);
 
 	return changed;
 }
 
-static bool rtrs_clt_change_state(struct rtrs_clt_sess *sess,
-				   enum rtrs_clt_state new_state)
-{
-	enum rtrs_clt_state old_state;
-
-	return rtrs_clt_change_state_get_old(sess, new_state, &old_state);
-}
-
 static void rtrs_clt_hb_err_handler(struct rtrs_con *c)
 {
 	struct rtrs_clt_con *con = container_of(c, typeof(*con), c);
@@ -1799,7 +1792,7 @@ static int rtrs_rdma_conn_rejected(struct rtrs_clt_con *con,
 
 static void rtrs_clt_close_conns(struct rtrs_clt_sess *sess, bool wait)
 {
-	if (rtrs_clt_change_state(sess, RTRS_CLT_CLOSING))
+	if (rtrs_clt_change_state_get_old(sess, RTRS_CLT_CLOSING, NULL))
 		queue_work(rtrs_wq, &sess->close_work);
 	if (wait)
 		flush_work(&sess->close_work);
@@ -2185,7 +2178,7 @@ static void rtrs_clt_close_work(struct work_struct *work)
 
 	cancel_delayed_work_sync(&sess->reconnect_dwork);
 	rtrs_clt_stop_and_destroy_conns(sess);
-	rtrs_clt_change_state(sess, RTRS_CLT_CLOSED);
+	rtrs_clt_change_state_get_old(sess, RTRS_CLT_CLOSED, NULL);
 }
 
 static int init_conns(struct rtrs_clt_sess *sess)
@@ -2237,7 +2230,7 @@ static int init_conns(struct rtrs_clt_sess *sess)
 	 * doing rdma_resolve_addr(), switch to CONNECTION_ERR state
 	 * manually to keep reconnecting.
 	 */
-	rtrs_clt_change_state(sess, RTRS_CLT_CONNECTING_ERR);
+	rtrs_clt_change_state_get_old(sess, RTRS_CLT_CONNECTING_ERR, NULL);
 
 	return err;
 }
@@ -2254,7 +2247,7 @@ static void rtrs_clt_info_req_done(struct ib_cq *cq, struct ib_wc *wc)
 	if (unlikely(wc->status != IB_WC_SUCCESS)) {
 		rtrs_err(sess->clt, "Sess info request send failed: %s\n",
 			  ib_wc_status_msg(wc->status));
-		rtrs_clt_change_state(sess, RTRS_CLT_CONNECTING_ERR);
+		rtrs_clt_change_state_get_old(sess, RTRS_CLT_CONNECTING_ERR, NULL);
 		return;
 	}
 
@@ -2378,7 +2371,7 @@ static void rtrs_clt_info_rsp_done(struct ib_cq *cq, struct ib_wc *wc)
 out:
 	rtrs_clt_update_wc_stats(con);
 	rtrs_iu_free(iu, sess->s.dev->ib_dev, 1);
-	rtrs_clt_change_state(sess, state);
+	rtrs_clt_change_state_get_old(sess, state, NULL);
 }
 
 static int rtrs_send_sess_info(struct rtrs_clt_sess *sess)
@@ -2443,7 +2436,7 @@ static int rtrs_send_sess_info(struct rtrs_clt_sess *sess)
 		rtrs_iu_free(rx_iu, sess->s.dev->ib_dev, 1);
 	if (unlikely(err))
 		/* If we've never taken async path because of malloc problems */
-		rtrs_clt_change_state(sess, RTRS_CLT_CONNECTING_ERR);
+		rtrs_clt_change_state_get_old(sess, RTRS_CLT_CONNECTING_ERR, NULL);
 
 	return err;
 }
@@ -2500,7 +2493,7 @@ static void rtrs_clt_reconnect_work(struct work_struct *work)
 	/* Stop everything */
 	rtrs_clt_stop_and_destroy_conns(sess);
 	msleep(RTRS_RECONNECT_BACKOFF);
-	if (rtrs_clt_change_state(sess, RTRS_CLT_CONNECTING)) {
+	if (rtrs_clt_change_state_get_old(sess, RTRS_CLT_CONNECTING, NULL)) {
 		err = init_sess(sess);
 		if (err)
 			goto reconnect_again;
@@ -2509,7 +2502,7 @@ static void rtrs_clt_reconnect_work(struct work_struct *work)
 	return;
 
 reconnect_again:
-	if (rtrs_clt_change_state(sess, RTRS_CLT_RECONNECTING)) {
+	if (rtrs_clt_change_state_get_old(sess, RTRS_CLT_RECONNECTING, NULL)) {
 		sess->stats->reconnects.fail_cnt++;
 		delay_ms = clt->reconnect_delay_sec * 1000;
 		queue_delayed_work(rtrs_wq, &sess->reconnect_dwork,
Signature · 1 line
2.25.1
author
#14 lore raw

PATCH v2 12/19 RDMA/rtrs-clt: Rename __rtrs_clt_change_state to rtrs_clt_change_state

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

Let's rename it to rtrs_clt_change_state since the previous one is killed.

Also update the comment to make it more clear.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +5 -5
index 39dc8423d7df..3c90718f668d 100644
@@ -178,18 +178,18 @@ struct rtrs_clt_con *rtrs_permit_to_clt_con(struct rtrs_clt_sess *sess,
 }
 
 /**
- * __rtrs_clt_change_state() - change the session state through session state
+ * rtrs_clt_change_state() - change the session state through session state
  * machine.
  *
  * @sess: client session to change the state of.
  * @new_state: state to change to.
  *
- * returns true if successful, false if the requested state can not be set.
+ * returns true if sess's state is changed to new state, otherwise return false.
  *
  * Locks:
  * state_wq lock must be hold.
  */
-static bool __rtrs_clt_change_state(struct rtrs_clt_sess *sess,
+static bool rtrs_clt_change_state(struct rtrs_clt_sess *sess,
 				     enum rtrs_clt_state new_state)
 {
 	enum rtrs_clt_state old_state;
@@ -286,7 +286,7 @@ static bool rtrs_clt_change_state_from_to(struct rtrs_clt_sess *sess,
 
 	spin_lock_irq(&sess->state_wq.lock);
 	if (sess->state == old_state)
-		changed = __rtrs_clt_change_state(sess, new_state);
+		changed = rtrs_clt_change_state(sess, new_state);
 	spin_unlock_irq(&sess->state_wq.lock);
 
 	return changed;
@@ -1361,7 +1361,7 @@ static bool rtrs_clt_change_state_get_old(struct rtrs_clt_sess *sess,
 	spin_lock_irq(&sess->state_wq.lock);
 	if (old_state)
 		*old_state = sess->state;
-	changed = __rtrs_clt_change_state(sess, new_state);
+	changed = rtrs_clt_change_state(sess, new_state);
 	spin_unlock_irq(&sess->state_wq.lock);
 
 	return changed;
Signature · 1 line
2.25.1
author
#15 lore raw

PATCH v2 13/19 RDMA/rtrs-srv: Fix missing wr_cqe

In reply to Jack Wang

We had a few places wr_cqe is not set, which could lead to NULL pointer deref or GPF in error case.

Fixes 9cb837480424 ("RDMA/rtrs: server: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +5 -0
index 89e938a1a4fb..465f20d6b89c 100644
@@ -267,6 +267,7 @@ static int rdma_write_sg(struct rtrs_srv_op *id)
 		WARN_ON_ONCE(rkey != wr->rkey);
 
 	wr->wr.opcode = IB_WR_RDMA_WRITE;
+	wr->wr.wr_cqe   = &io_comp_cqe;
 	wr->wr.ex.imm_data = 0;
 	wr->wr.send_flags  = 0;
 
@@ -294,6 +295,7 @@ static int rdma_write_sg(struct rtrs_srv_op *id)
 		inv_wr.sg_list = NULL;
 		inv_wr.num_sge = 0;
 		inv_wr.opcode = IB_WR_SEND_WITH_INV;
+		inv_wr.wr_cqe   = &io_comp_cqe;
 		inv_wr.send_flags = 0;
 		inv_wr.ex.invalidate_rkey = rkey;
 	}
@@ -304,6 +306,7 @@ static int rdma_write_sg(struct rtrs_srv_op *id)
 
 		srv_mr = &sess->mrs[id->msg_id];
 		rwr.wr.opcode = IB_WR_REG_MR;
+		rwr.wr.wr_cqe = &local_reg_cqe;
 		rwr.wr.num_sge = 0;
 		rwr.mr = srv_mr->mr;
 		rwr.wr.send_flags = 0;
@@ -379,6 +382,7 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id,
 
 		if (need_inval) {
 			if (likely(sg_cnt)) {
+				inv_wr.wr_cqe   = &io_comp_cqe;
 				inv_wr.sg_list = NULL;
 				inv_wr.num_sge = 0;
 				inv_wr.opcode = IB_WR_SEND_WITH_INV;
@@ -421,6 +425,7 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id,
 		srv_mr = &sess->mrs[id->msg_id];
 		rwr.wr.next = &imm_wr;
 		rwr.wr.opcode = IB_WR_REG_MR;
+		rwr.wr.wr_cqe = &local_reg_cqe;
 		rwr.wr.num_sge = 0;
 		rwr.wr.send_flags = 0;
 		rwr.mr = srv_mr->mr;
Signature · 1 line
2.25.1
author
#16 lore raw

PATCH v2 14/19 RDMA/rtrs-clt: Refactor the failure cases in alloc_clt

In reply to Jack Wang

From: Guoqing Jiang <[email protected]>

Make all failure cases go to the common path to avoid duplicate code. And some issued existed before.

1. clt need to be freed to avoid memory leak.

2. return ERR_PTR(-ENOMEM) if kobject_create_and_add fails, because rtrs_clt_open checks the return value of by call "IS_ERR(clt)".

Fixes 6a98d71daea1 ("RDMA/rtrs: client: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +12 -13
index 3c90718f668d..493f45a33b5e 100644
@@ -2568,11 +2568,8 @@ static struct rtrs_clt *alloc_clt(const char *sessname, size_t paths_num,
 	clt->dev.class = rtrs_clt_dev_class;
 	clt->dev.release = rtrs_clt_dev_release;
 	err = dev_set_name(&clt->dev, "%s", sessname);
-	if (err) {
-		free_percpu(clt->pcpu_path);
-		kfree(clt);
-		return ERR_PTR(err);
-	}
+	if (err)
+		goto err;
 	/*
 	 * Suppress user space notification until
 	 * sysfs files are created
@@ -2580,29 +2577,31 @@ static struct rtrs_clt *alloc_clt(const char *sessname, size_t paths_num,
 	dev_set_uevent_suppress(&clt->dev, true);
 	err = device_register(&clt->dev);
 	if (err) {
-		free_percpu(clt->pcpu_path);
 		put_device(&clt->dev);
-		return ERR_PTR(err);
+		goto err;
 	}
 
 	clt->kobj_paths = kobject_create_and_add("paths", &clt->dev.kobj);
 	if (!clt->kobj_paths) {
-		free_percpu(clt->pcpu_path);
-		device_unregister(&clt->dev);
-		return NULL;
+		err = -ENOMEM;
+		goto err_dev;
 	}
 	err = rtrs_clt_create_sysfs_root_files(clt);
 	if (err) {
-		free_percpu(clt->pcpu_path);
 		kobject_del(clt->kobj_paths);
 		kobject_put(clt->kobj_paths);
-		device_unregister(&clt->dev);
-		return ERR_PTR(err);
+		goto err_dev;
 	}
 	dev_set_uevent_suppress(&clt->dev, false);
 	kobject_uevent(&clt->dev.kobj, KOBJ_ADD);
 
 	return clt;
+err_dev:
+	device_unregister(&clt->dev);
+err:
+	free_percpu(clt->pcpu_path);
+	kfree(clt);
+	return ERR_PTR(err);
 }
 
 static void free_clt(struct rtrs_clt *clt)
Signature · 1 line
2.25.1
author
#17 lore raw

PATCH v2 15/19 RDMA/rtrs: Do not signal for heatbeat

In reply to Jack Wang

For HB, there is no need to generate signal for completion.

Also remove a comment accordingly.

Fixes c0894b3ea69d ("RDMA/rtrs: core: lib functions shared between client and server modules")
---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 1 -
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 1 -
 drivers/infiniband/ulp/rtrs/rtrs.c     | 4 ++--
 3 files changed, 2 insertions(+), 4 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +0 -1
index 493f45a33b5e..2053bf99418a 100644
@@ -664,7 +664,6 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc)
 	case IB_WC_RDMA_WRITE:
 		/*
 		 * post_send() RDMA write completions of IO reqs (read/write)
-		 * and hb
 		 */
 		break;
 
drivers/infiniband/ulp/rtrs/rtrs-srv.c +0 -1
index 465f20d6b89c..8ea1df6b4da0 100644
@@ -1242,7 +1242,6 @@ static void rtrs_srv_rdma_done(struct ib_cq *cq, struct ib_wc *wc)
 	case IB_WC_SEND:
 		/*
 		 * post_send() RDMA write completions of IO reqs (read/write)
-		 * and hb
 		 */
 		atomic_add(srv->queue_depth, &con->sq_wr_avail);
 
drivers/infiniband/ulp/rtrs/rtrs.c +2 -2
index df52427f1710..97af8f0bb806 100644
@@ -310,7 +310,7 @@ void rtrs_send_hb_ack(struct rtrs_sess *sess)
 
 	imm = rtrs_to_imm(RTRS_HB_ACK_IMM, 0);
 	err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm,
-					      IB_SEND_SIGNALED, NULL);
+					     0, NULL);
 	if (err) {
 		sess->hb_err_handler(usr_con);
 		return;
@@ -339,7 +339,7 @@ static void hb_work(struct work_struct *work)
 	}
 	imm = rtrs_to_imm(RTRS_HB_MSG_IMM, 0);
 	err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm,
-					      IB_SEND_SIGNALED, NULL);
+					     0, NULL);
 	if (err) {
 		sess->hb_err_handler(usr_con);
 		return;
Signature · 1 line
2.25.1
author
#18 lore raw

PATCH v2 16/19 RDMA/rtrs-clt: Use bitmask to check sess->flags

In reply to Jack Wang

We may want to add new flags, so it's better to use bitmask to check flags.

Fixes 6a98d71daea1 ("RDMA/rtrs: client: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs-clt.c +7 -7
index 2053bf99418a..7644c3f627ca 100644
@@ -494,7 +494,7 @@ static void rtrs_clt_recv_done(struct rtrs_clt_con *con, struct ib_wc *wc)
 	int err;
 	struct rtrs_clt_sess *sess = to_clt_sess(con->c.sess);
 
-	WARN_ON(sess->flags != RTRS_MSG_NEW_RKEY_F);
+	WARN_ON((sess->flags & RTRS_MSG_NEW_RKEY_F) == 0);
 	iu = container_of(wc->wr_cqe, struct rtrs_iu,
 			  cqe);
 	err = rtrs_iu_post_recv(&con->c, iu);
@@ -514,7 +514,7 @@ static void rtrs_clt_rkey_rsp_done(struct rtrs_clt_con *con, struct ib_wc *wc)
 	u32 buf_id;
 	int err;
 
-	WARN_ON(sess->flags != RTRS_MSG_NEW_RKEY_F);
+	WARN_ON((sess->flags & RTRS_MSG_NEW_RKEY_F) == 0);
 
 	iu = container_of(wc->wr_cqe, struct rtrs_iu, cqe);
 
@@ -621,12 +621,12 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc)
 		} else if (imm_type == RTRS_HB_MSG_IMM) {
 			WARN_ON(con->c.cid);
 			rtrs_send_hb_ack(&sess->s);
-			if (sess->flags == RTRS_MSG_NEW_RKEY_F)
+			if (sess->flags & RTRS_MSG_NEW_RKEY_F)
 				return  rtrs_clt_recv_done(con, wc);
 		} else if (imm_type == RTRS_HB_ACK_IMM) {
 			WARN_ON(con->c.cid);
 			sess->s.hb_missed_cnt = 0;
-			if (sess->flags == RTRS_MSG_NEW_RKEY_F)
+			if (sess->flags & RTRS_MSG_NEW_RKEY_F)
 				return  rtrs_clt_recv_done(con, wc);
 		} else {
 			rtrs_wrn(con->c.sess, "Unknown IMM type %u\n",
@@ -654,7 +654,7 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc)
 		WARN_ON(!(wc->wc_flags & IB_WC_WITH_INVALIDATE ||
 			  wc->wc_flags & IB_WC_WITH_IMM));
 		WARN_ON(wc->wr_cqe->done != rtrs_clt_rdma_done);
-		if (sess->flags == RTRS_MSG_NEW_RKEY_F) {
+		if (sess->flags & RTRS_MSG_NEW_RKEY_F) {
 			if (wc->wc_flags & IB_WC_WITH_INVALIDATE)
 				return  rtrs_clt_recv_done(con, wc);
 
@@ -679,7 +679,7 @@ static int post_recv_io(struct rtrs_clt_con *con, size_t q_size)
 	struct rtrs_clt_sess *sess = to_clt_sess(con->c.sess);
 
 	for (i = 0; i < q_size; i++) {
-		if (sess->flags == RTRS_MSG_NEW_RKEY_F) {
+		if (sess->flags & RTRS_MSG_NEW_RKEY_F) {
 			struct rtrs_iu *iu = &con->rsp_ius[i];
 
 			err = rtrs_iu_post_recv(&con->c, iu);
@@ -1563,7 +1563,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
 			      sess->queue_depth * 3 + 1);
 	}
 	/* alloc iu to recv new rkey reply when server reports flags set */
-	if (sess->flags == RTRS_MSG_NEW_RKEY_F || con->c.cid == 0) {
+	if (sess->flags & RTRS_MSG_NEW_RKEY_F || con->c.cid == 0) {
 		con->rsp_ius = rtrs_iu_alloc(max_recv_wr, sizeof(*rsp),
 					      GFP_KERNEL, sess->s.dev->ib_dev,
 					      DMA_FROM_DEVICE,
Signature · 1 line
2.25.1
author
#19 lore raw

PATCH v2 17/19 RDMA/rtrs-srv: Do not signal REG_MR

In reply to Jack Wang

We do not need to wait for REG_MR completion, so remove the SIGNAL flag.

Fixes 9cb837480424 ("RDMA/rtrs: server: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +1 -1
index 8ea1df6b4da0..2798c655d7ae 100644
@@ -818,7 +818,7 @@ static int process_info_req(struct rtrs_srv_con *con,
 		rwr[mri].wr.opcode = IB_WR_REG_MR;
 		rwr[mri].wr.wr_cqe = &local_reg_cqe;
 		rwr[mri].wr.num_sge = 0;
-		rwr[mri].wr.send_flags = mri ? 0 : IB_SEND_SIGNALED;
+		rwr[mri].wr.send_flags = 0;
 		rwr[mri].mr = mr;
 		rwr[mri].key = mr->rkey;
 		rwr[mri].access = (IB_ACCESS_LOCAL_WRITE |
Signature · 1 line
2.25.1
author
#20 lore raw

PATCH v2 18/19 RDMA/rtrs-srv: Init wr_cnt as 1

In reply to Jack Wang

Fix up wr_avail accounting. if wr_cnt is 0, then we do SIGNAL for first wr, in completion we add queue_depth back, which is not right in the sense of tracking for available wr.

So fix it by init wr_cnt to 1.

Fixes 9cb837480424 ("RDMA/rtrs: server: main functionality")
---
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-srv.c +1 -1
index 2798c655d7ae..54abbd9562d0 100644
@@ -1603,7 +1603,7 @@ static int create_con(struct rtrs_srv_sess *sess,
 	con->c.cm_id = cm_id;
 	con->c.sess = &sess->s;
 	con->c.cid = cid;
-	atomic_set(&con->wr_cnt, 0);
+	atomic_set(&con->wr_cnt, 1);
 
 	if (con->c.cid == 0) {
 		/*
Signature · 1 line
2.25.1
author
#21 lore raw

PATCH v2 19/19 RDMA/rtrs: Fix KASAN: stack-out-of-bounds bug

In reply to Jack Wang

When KASAN is enabled, we notice warning below:

[  483.436975] ==================================================================
[  483.437234] BUG: KASAN: stack-out-of-bounds in _mlx5_ib_post_send+0x188a/0x2560 [mlx5_ib]
[  483.437430] Read of size 4 at addr ffff88a195fd7d30 by task kworker/1:3/6954
[  483.437731] CPU: 1 PID: 6954 Comm: kworker/1:3 Kdump: loaded Tainted: G           O      5.4.82-pserver #5.4.82-1+feature+linux+5.4.y+dbg+20201210.1532+987e7a6~deb10
[  483.437976] Hardware name: Supermicro Super Server/X11DDW-L, BIOS 3.3 02/21/2020
[  483.438168] Workqueue: rtrs_server_wq hb_work [rtrs_core]
[  483.438323] Call Trace:
[  483.438486]  dump_stack+0x96/0xe0
[  483.438646]  ? _mlx5_ib_post_send+0x188a/0x2560 [mlx5_ib]
[  483.438802]  print_address_description.constprop.6+0x1b/0x220
[  483.438966]  ? _mlx5_ib_post_send+0x188a/0x2560 [mlx5_ib]
[  483.439133]  ? _mlx5_ib_post_send+0x188a/0x2560 [mlx5_ib]
[  483.439285]  __kasan_report.cold.9+0x1a/0x32
[  483.439444]  ? _mlx5_ib_post_send+0x188a/0x2560 [mlx5_ib]
[  483.439597]  kasan_report+0x10/0x20
[  483.439752]  _mlx5_ib_post_send+0x188a/0x2560 [mlx5_ib]
[  483.439910]  ? update_sd_lb_stats+0xfb1/0xfc0
[  483.440073]  ? set_reg_wr+0x520/0x520 [mlx5_ib]
[  483.440222]  ? update_group_capacity+0x340/0x340
[  483.440377]  ? find_busiest_group+0x314/0x870
[  483.440526]  ? update_sd_lb_stats+0xfc0/0xfc0
[  483.440683]  ? __bitmap_and+0x6f/0x100
[  483.440832]  ? __lock_acquire+0xa2/0x2150
[  483.440979]  ? __lock_acquire+0xa2/0x2150
[  483.441128]  ? __lock_acquire+0xa2/0x2150
[  483.441279]  ? debug_lockdep_rcu_enabled+0x23/0x60
[  483.441430]  ? lock_downgrade+0x390/0x390
[  483.441582]  ? __lock_acquire+0xa2/0x2150
[  483.441729]  ? __lock_acquire+0xa2/0x2150
[  483.441876]  ? newidle_balance+0x425/0x8f0
[  483.442024]  ? __lock_acquire+0xa2/0x2150
[  483.442172]  ? debug_lockdep_rcu_enabled+0x23/0x60
[  483.442330]  hb_work+0x15d/0x1d0 [rtrs_core]
[  483.442479]  ? schedule_hb+0x50/0x50 [rtrs_core]
[  483.442627]  ? lock_downgrade+0x390/0x390
[  483.442781]  ? process_one_work+0x40d/0xa50
[  483.442931]  process_one_work+0x4ee/0xa50
[  483.443082]  ? pwq_dec_nr_in_flight+0x110/0x110
[  483.443231]  ? do_raw_spin_lock+0x119/0x1d0
[  483.443383]  worker_thread+0x65/0x5c0
[  483.443532]  ? process_one_work+0xa50/0xa50
[  483.451839]  kthread+0x1e2/0x200
[  483.451983]  ? kthread_create_on_node+0xc0/0xc0
[  483.452139]  ret_from_fork+0x3a/0x50

The problem is we use wrong type when send wr, hw driver expect the type of IB_WR_RDMA_WRITE_WITH_IMM wr should be ib_rdma_wr, and doing container_of to access member. The fix is simple use ib_rdma_wr instread of ib_send_wr.

Fixes c0894b3ea69d ("RDMA/rtrs: core: lib functions shared between client and server modules")
---
 drivers/infiniband/ulp/rtrs/rtrs.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/infiniband/ulp/rtrs/rtrs.c +7 -7
index 97af8f0bb806..d13aff0aa816 100644
@@ -182,16 +182,16 @@ int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe,
 				    u32 imm_data, enum ib_send_flags flags,
 				    struct ib_send_wr *head)
 {
-	struct ib_send_wr wr;
+	struct ib_rdma_wr wr;
 
-	wr = (struct ib_send_wr) {
-		.wr_cqe	= cqe,
-		.send_flags	= flags,
-		.opcode	= IB_WR_RDMA_WRITE_WITH_IMM,
-		.ex.imm_data	= cpu_to_be32(imm_data),
+	wr = (struct ib_rdma_wr) {
+		.wr.wr_cqe	= cqe,
+		.wr.send_flags	= flags,
+		.wr.opcode	= IB_WR_RDMA_WRITE_WITH_IMM,
+		.wr.ex.imm_data	= cpu_to_be32(imm_data),
 	};
 
-	return rtrs_post_send(con->qp, head, &wr);
+	return rtrs_post_send(con->qp, head, &wr.wr);
 }
 EXPORT_SYMBOL_GPL(rtrs_post_rdma_write_imm_empty);
 
Signature · 1 line
2.25.1
author
#22 lore raw
In reply to Jack Wang
Quoted 15 lines · Jack Wang

Hi Jason, hi Doug,

Please consider to include following changes to the next merge window.

It contains a few bugfix and cleanup:
- Fix memory leak incase of failure in kobj_init_and_add in both clt/srv
- reduce memory footprint by set proper limit when create QP
- fix missing wr_cqe in a few cases on srv, it could lead to crash in error case.
- remove the SIGNAL flag for heartbeat, otherwise it could mess around the send_wr_awail accouting.
- flush on going session closing to release the memory presure on server.
- other misc fix and cleanup.

Quoted 1 line

The patches are created based on rdma/for-next.

Hi Jason,

ping, any comments or do you need a resend/rebase?

Thanks!

Quoted 45 lines

V2->V1
* more descprition for the patches above as requested by Jason, also include Fixes tag for bugfix.
* suppress the lockdep warning for PATCH 2 Occasionally flush ongoing session closing with comment.
* new bugfix PATCH 19 RDMA/rtrs: Fix KASAN: stack-out-of-bounds bug

Thanks!

Guoqing Jiang (8):
RDMA/rtrs-srv: Jump to dereg_mr label if allocate iu fails RDMA/rtrs: Call kobject_put in the failure path RDMA/rtrs-clt: Consolidate rtrs_clt_destroy_sysfs_root_{folder,files} RDMA/rtrs-clt: Kill wait_for_inflight_permits RDMA/rtrs-clt: Remove unnecessary 'goto out' RDMA/rtrs-clt: Kill rtrs_clt_change_state
RDMA/rtrs-clt: Rename __rtrs_clt_change_state to rtrs_clt_change_state RDMA/rtrs-clt: Refactor the failure cases in alloc_clt

Jack Wang (11):
RDMA/rtrs: Extend ibtrs_cq_qp_create RMDA/rtrs-srv: Occasionally flush ongoing session closing RDMA/rtrs-srv: Release lock before call into close_sess RDMA/rtrs-srv: Use sysfs_remove_file_self for disconnect RDMA/rtrs-clt: Set mininum limit when create QP RDMA/rtrs-srv: Fix missing wr_cqe
RDMA/rtrs: Do not signal for heatbeat RDMA/rtrs-clt: Use bitmask to check sess->flags RDMA/rtrs-srv: Do not signal REG_MR RDMA/rtrs-srv: Init wr_cnt as 1
RDMA/rtrs: Fix KASAN: stack-out-of-bounds bug

 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c |  11 +-
 drivers/infiniband/ulp/rtrs/rtrs-clt.c       | 120 +++++++++----------
 drivers/infiniband/ulp/rtrs/rtrs-clt.h       |   3 +-
 drivers/infiniband/ulp/rtrs/rtrs-pri.h       |   5 +-
 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c |   5 +-
 drivers/infiniband/ulp/rtrs/rtrs-srv.c       |  34 ++++--
 drivers/infiniband/ulp/rtrs/rtrs.c           |  32 ++---
 7 files changed, 110 insertions(+), 100 deletions(-)

--
2.25.1

maintainer Applied
#23 lore raw
In reply to Jack Wang
Quoted 35 lines · Jack Wang

Hi Jason, hi Doug,

Please consider to include following changes to the next merge window.

It contains a few bugfix and cleanup:
- Fix memory leak incase of failure in kobj_init_and_add in both clt/srv
- reduce memory footprint by set proper limit when create QP
- fix missing wr_cqe in a few cases on srv, it could lead to crash in error case.
- remove the SIGNAL flag for heartbeat, otherwise it could mess around the send_wr_awail accouting.
- flush on going session closing to release the memory presure on server.
- other misc fix and cleanup.

The patches are created based on rdma/for-next.

V2->V1
* more descprition for the patches above as requested by Jason, also include Fixes tag for bugfix.
* suppress the lockdep warning for PATCH 2 Occasionally flush ongoing session closing with comment.
* new bugfix PATCH 19 RDMA/rtrs: Fix KASAN: stack-out-of-bounds bug

Thanks!

Guoqing Jiang (8):
RDMA/rtrs-srv: Jump to dereg_mr label if allocate iu fails RDMA/rtrs: Call kobject_put in the failure path RDMA/rtrs-clt: Consolidate rtrs_clt_destroy_sysfs_root_{folder,files} RDMA/rtrs-clt: Kill wait_for_inflight_permits RDMA/rtrs-clt: Remove unnecessary 'goto out' RDMA/rtrs-clt: Kill rtrs_clt_change_state
RDMA/rtrs-clt: Rename __rtrs_clt_change_state to rtrs_clt_change_state RDMA/rtrs-clt: Refactor the failure cases in alloc_clt

Quoted 11 lines

Jack Wang (11):
RDMA/rtrs: Extend ibtrs_cq_qp_create RDMA/rtrs-srv: Release lock before call into close_sess RDMA/rtrs-srv: Use sysfs_remove_file_self for disconnect RDMA/rtrs-clt: Set mininum limit when create QP RDMA/rtrs-srv: Fix missing wr_cqe
RDMA/rtrs: Do not signal for heatbeat RDMA/rtrs-clt: Use bitmask to check sess->flags RDMA/rtrs-srv: Do not signal REG_MR RDMA/rtrs-srv: Init wr_cnt as 1
RDMA/rtrs: Fix KASAN: stack-out-of-bounds bug

Every one except:

Quoted 1 line

RMDA/rtrs-srv: Occasionally flush ongoing session closing

Applied to for-next, thanks

Jason