Skip to contentSkip to conversation

RDMA/rxe: Add workqueue support for tasks

PATCH Superseded Bob Pearson 2 participants 2 messages Mar 2, 2023 – Mar 27, 2023

Subsystems (2)

v6 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v6 1/1 RDMA/rxe: Add workqueue support for tasks

Replace tasklets by work queues for the three main rxe tasklets:
rxe_requester, rxe_completer and rxe_responder.

This patch is all that is left from an earlier patch series that also converted tasklets to workqueues with the same subject.

With this patch the rxe driver does not exhibit the soft lockups reported by Daisuke Matsuda in the link below.

This patch depends on an earlier patch series titled "RDMA/rxe: Correct qp reference counting" which corrects the same errors for the current tasklet version.

---
v6:
Fixed left over references to tasklets in the comments. Added WQ_UNBOUND to the parameters for alloc_workqueue(). This shows a significant performance improvement.
v5:
Based on corrected task logic for tasklets and simplified to only convert from tasklets to workqueues and not provide a flexible interface.

 drivers/infiniband/sw/rxe/rxe.c      |  9 ++-
 drivers/infiniband/sw/rxe/rxe_task.c | 84 ++++++++++++++++++----------
 drivers/infiniband/sw/rxe/rxe_task.h |  9 ++-
 3 files changed, 69 insertions(+), 33 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +8 -1
index 7a7e713de52d..54c723a6edda 100644
@@ -212,10 +212,16 @@ static int __init rxe_module_init(void)
 {
 	int err;
 
-	err = rxe_net_init();
+	err = rxe_alloc_wq();
 	if (err)
 		return err;
 
+	err = rxe_net_init();
+	if (err) {
+		rxe_destroy_wq();
+		return err;
+	}
+
 	rdma_link_register(&rxe_link_ops);
 	pr_info("loaded\n");
 	return 0;
@@ -226,6 +232,7 @@ static void __exit rxe_module_exit(void)
 	rdma_link_unregister(&rxe_link_ops);
 	ib_unregister_driver(RDMA_DRIVER_RXE);
 	rxe_net_exit();
+	rxe_destroy_wq();
 
 	pr_info("unloaded\n");
 }
drivers/infiniband/sw/rxe/rxe_task.c +54 -30

200 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_task.h +7 -2
index facb7c8e3729..4e937de9ebcf 100644
@@ -22,23 +22,28 @@ enum {
  * called again.
  */
 struct rxe_task {
-	struct tasklet_struct	tasklet;
+	struct work_struct	work;
 	int			state;
 	spinlock_t		lock;
 	struct rxe_qp		*qp;
+	const char		*name;
 	int			(*func)(struct rxe_qp *qp);
 	int			ret;
 	long			num_sched;
 	long			num_done;
 };
 
+int rxe_alloc_wq(void);
+
+void rxe_destroy_wq(void);
+
 /*
  * init rxe_task structure
  *	qp  => parameter to pass to func
  *	func => function to call until it returns != 0
  */
 int rxe_init_task(struct rxe_task *task, struct rxe_qp *qp,
-		  int (*func)(struct rxe_qp *));
+		  int (*func)(struct rxe_qp *), const char *name);
 
 /* cleanup task */
 void rxe_cleanup_task(struct rxe_task *task);

base-commit: 6a22f7fbde87336002886583d053bfa1cd8ff1d1

Signature · 1 line
2.37.2
#1 lore raw
Quoted 7 lines · Bob Pearson

Replace tasklets by work queues for the three main rxe tasklets:
rxe_requester, rxe_completer and rxe_responder.

This patch is all that is left from an earlier patch series that also converted tasklets to workqueues with the same subject.

Quoted 6 lines

With this patch the rxe driver does not exhibit the soft lockups reported by Daisuke Matsuda in the link below.

This patch depends on an earlier patch series titled "RDMA/rxe: Correct qp reference counting" which corrects the same errors for the current tasklet version.

Isn't it better to add the link to the patch series for easier reference?

Quoted 21 lines

Link:
https://lore.kernel.org/linux-rdma/TYCPR01MB845522FD536170D75068DD41E5099@TYCPR01MB8455.jpnprd01.prod.
outlook.com/

---
v6:
Fixed left over references to tasklets in the comments. Added WQ_UNBOUND to the parameters for alloc_workqueue(). This shows a significant performance improvement.
v5:
Based on corrected task logic for tasklets and simplified to only convert from tasklets to workqueues and not provide a flexible interface.

 drivers/infiniband/sw/rxe/rxe.c      |  9 ++-
 drivers/infiniband/sw/rxe/rxe_task.c | 84 ++++++++++++++++++----------
 drivers/infiniband/sw/rxe/rxe_task.h |  9 ++-
 3 files changed, 69 insertions(+), 33 deletions(-)

<...>

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_task.h +0 -1
index facb7c8e3729..4e937de9ebcf 100644
@@ -22,23 +22,28 @@ enum {
  * called again.
  */
 struct rxe_task {
-	struct tasklet_struct	tasklet;

drivers/infiniband/sw/rxe/rxe_task.h enum { · line 34

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_task.h +2 -0
+	struct work_struct	work;
 	int			state;
 	spinlock_t		lock;
 	struct rxe_qp		*qp;
+	const char		*name;
 	int			(*func)(struct rxe_qp *qp);
 	int			ret;
 	long			num_sched;
 	long			num_done;
 };

The member '*name' is newly added here but not used elsewhere. It seems we should just remove it.

drivers/infiniband/sw/rxe/rxe_task.h enum { · line 46

Quoted 13 lines
drivers/infiniband/sw/rxe/rxe_task.h +5 -1
+int rxe_alloc_wq(void);
+
+void rxe_destroy_wq(void);
+
 /*
  * init rxe_task structure
  *	qp  => parameter to pass to func
  *	func => function to call until it returns != 0
  */
 int rxe_init_task(struct rxe_task *task, struct rxe_qp *qp,
-		  int (*func)(struct rxe_qp *));
+		  int (*func)(struct rxe_qp *), const char *name);

This does not matches with the function definition. Kernel build fails here unless the 4th argument is removed. Additionally, the argument "*name" is not used in rxe_init_task().

Thanks,
Daisuke

Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_task.h +0 -0
 /* cleanup task */
 void rxe_cleanup_task(struct rxe_task *task);
 

base-commit: 6a22f7fbde87336002886583d053bfa1cd8ff1d1 --
2.37.2