Skip to contentSkip to conversation

NFS: Fix another 'check_flush_dependency' splat

RFC PATCH In review [email protected] 4 participants 17 messages Apr 29, 2024 – Jun 3, 2024

Subsystems (4)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

cover NFS: Fix another 'check_flush_dependency' splat

From: Chuck Lever <[email protected]>

Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process.

I've done some testing of this series, now looking for review comments.

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

 net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
 net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
 net/sunrpc/xprtrdma/transport.c |  20 +++-
 net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
 net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
 5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

Signature · 1 line
2.44.0
author maintainer
#1 lore raw

RFC PATCH 1/4 xprtrdma: Remove temp allocation of rpcrdma_rep objects

From: Chuck Lever <[email protected]>

The original code was designed so that most calls to rpcrdma_rep_create() would occur on the NUMA node that the device preferred. There are a few cases where that's not possible, so those reps are marked as temporary.

However, we have the device (and its preferred node) already in rpcrdma_rep_create(), so let's use that.

---
 net/sunrpc/xprtrdma/rpc_rdma.c  |  3 +-
 net/sunrpc/xprtrdma/verbs.c     | 57 ++++++++++++++-------------------
 net/sunrpc/xprtrdma/xprt_rdma.h |  3 +-
 3 files changed, 26 insertions(+), 37 deletions(-)
net/sunrpc/xprtrdma/rpc_rdma.c +1 -2
index 190a4de239c8..1478c41c7e9d 100644
@@ -1471,8 +1471,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
 		credits = 1;	/* don't deadlock */
 	else if (credits > r_xprt->rx_ep->re_max_requests)
 		credits = r_xprt->rx_ep->re_max_requests;
-	rpcrdma_post_recvs(r_xprt, credits + (buf->rb_bc_srv_max_requests << 1),
-			   false);
+	rpcrdma_post_recvs(r_xprt, credits + (buf->rb_bc_srv_max_requests << 1));
 	if (buf->rb_credits != credits)
 		rpcrdma_update_cwnd(r_xprt, credits);
 
net/sunrpc/xprtrdma/verbs.c +24 -33

146 lines not on this page · raw message

net/sunrpc/xprtrdma/xprt_rdma.h +1 -2
index da409450dfc0..08bda29ed953 100644
@@ -198,7 +198,6 @@ struct rpcrdma_rep {
 	__be32			rr_proc;
 	int			rr_wc_flags;
 	u32			rr_inv_rkey;
-	bool			rr_temp;
 	struct rpcrdma_regbuf	*rr_rdmabuf;
 	struct rpcrdma_xprt	*rr_rxprt;
 	struct rpc_rqst		*rr_rqst;
@@ -466,7 +465,7 @@ void rpcrdma_flush_disconnect(struct rpcrdma_xprt *r_xprt, struct ib_wc *wc);
 int rpcrdma_xprt_connect(struct rpcrdma_xprt *r_xprt);
 void rpcrdma_xprt_disconnect(struct rpcrdma_xprt *r_xprt);
 
-void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed, bool temp);
+void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed);
 
 /*
  * Buffer calls - xprtrdma/verbs.c
Signature · 1 line
2.44.0
author maintainer
#2 lore raw

RFC PATCH 2/4 xprtrdma: Clean up synopsis of frwr_mr_unmap()

In reply to [email protected]

From: Chuck Lever <[email protected]>

Commit 7a03aeb66c41 ("xprtrdma: Micro-optimize MR DMA-unmapping") removed the last use of the @r_xprt parameter in this function, but neglected to remove the parameter itself.

---
 net/sunrpc/xprtrdma/frwr_ops.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +3 -3
index ffbf99894970..6e508708d06d 100644
@@ -54,7 +54,7 @@ static void frwr_cid_init(struct rpcrdma_ep *ep,
 	cid->ci_completion_id = mr->mr_ibmr->res.id;
 }
 
-static void frwr_mr_unmap(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr)
+static void frwr_mr_unmap(struct rpcrdma_mr *mr)
 {
 	if (mr->mr_device) {
 		trace_xprtrdma_mr_unmap(mr);
@@ -73,7 +73,7 @@ void frwr_mr_release(struct rpcrdma_mr *mr)
 {
 	int rc;
 
-	frwr_mr_unmap(mr->mr_xprt, mr);
+	frwr_mr_unmap(mr);
 
 	rc = ib_dereg_mr(mr->mr_ibmr);
 	if (rc)
@@ -84,7 +84,7 @@ void frwr_mr_release(struct rpcrdma_mr *mr)
 
 static void frwr_mr_put(struct rpcrdma_mr *mr)
 {
-	frwr_mr_unmap(mr->mr_xprt, mr);
+	frwr_mr_unmap(mr);
 
 	/* The MR is returned to the req's MR free list instead
 	 * of to the xprt's MR free list. No spinlock is needed.
Signature · 1 line
2.44.0
author maintainer
#3 lore raw

RFC PATCH 3/4 xprtrdma: Delay releasing connection hardware resources

In reply to [email protected]

From: Chuck Lever <[email protected]>

xprtiod_workqueue is a MEM_RECLAIM-enabled workqueue. However, the RDMA core API functions are not memory reclaim-safe. This was partially accomplished by commit 6b1eb3b22272 ("SUNRPC: Replace the use of the xprtiod WQ in rpcrdma").

This commit addressed the issue in the connect path, but not in the disconnect path. Thus sometimes a transport disconnect results in this splat:

workqueue: WQ_MEM_RECLAIM xprtiod:xprt_autoclose [sunrpc] is flushing !WQ_MEM_RECLAIM events_highpri:rpcrdma_mr_refresh_worker [rpcrdma]

WARNING: CPU: 1 PID: 20378 at kernel/workqueue.c:3728 check_flush_dependency+0x101/0x120

 ? check_flush_dependency+0x101/0x120
 ? report_bug+0x175/0x1a0
 ? handle_bug+0x44/0x90
 ? exc_invalid_op+0x1c/0x70
 ? asm_exc_invalid_op+0x1f/0x30
 ? __pfx_rpcrdma_mr_refresh_worker+0x10/0x10 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac]
 ? check_flush_dependency+0x101/0x120
 __flush_work.isra.0+0x20a/0x290
 __cancel_work_sync+0x129/0x1c0
 cancel_work_sync+0x14/0x20
 rpcrdma_xprt_disconnect+0x229/0x3f0 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac]
 xprt_rdma_close+0x16/0x40 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac]
 xprt_autoclose+0x63/0x110 [sunrpc a04d701bce94b5a8fb541cafbe1a489d6b1ab5b3]
 process_one_work+0x19e/0x3f0
 worker_thread+0x340/0x510
 ? __pfx_worker_thread+0x10/0x10
 kthread+0xf7/0x130
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x41/0x60
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1a/0x30

Create a context in which it is safe to manage resources that are not memory reclaim-safe that can be invoked during transport disconnect. Essentially this means that releasing an rpcrdma_ep is now done completely asynchronously.

Subsequent patches will move the release of transport resources into this new context.

---
 net/sunrpc/xprtrdma/transport.c | 20 +++++++++++++-
 net/sunrpc/xprtrdma/verbs.c     | 46 ++++++++++++++++++++-------------
 net/sunrpc/xprtrdma/xprt_rdma.h |  5 +++-
 3 files changed, 51 insertions(+), 20 deletions(-)
net/sunrpc/xprtrdma/transport.c +19 -1
index 29b0562d62e7..237d78c1ec54 100644
@@ -761,8 +761,12 @@ static struct xprt_class xprt_rdma = {
 	.netid			= { "rdma", "rdma6", "" },
 };
 
+struct workqueue_struct *rpcrdma_release_wq __read_mostly;
+
 void xprt_rdma_cleanup(void)
 {
+	struct workqueue_struct *wq;
+
 #if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
 	if (sunrpc_table_header) {
 		unregister_sysctl_table(sunrpc_table_header);
@@ -772,18 +776,32 @@ void xprt_rdma_cleanup(void)
 
 	xprt_unregister_transport(&xprt_rdma);
 	xprt_unregister_transport(&xprt_rdma_bc);
+
+	wq = rpcrdma_release_wq;
+	rpcrdma_release_wq = NULL;
+	destroy_workqueue(wq);
 }
 
 int xprt_rdma_init(void)
 {
+	struct workqueue_struct *wq;
 	int rc;
 
+	/* provision a WQ that is always unbound and !mem_reclaim */
+	wq = alloc_workqueue("rpcrdma_release", WQ_UNBOUND, 0);
+	if (!wq)
+		return -ENOMEM;
+	rpcrdma_release_wq = wq;
+
 	rc = xprt_register_transport(&xprt_rdma);
-	if (rc)
+	if (rc) {
+		destroy_workqueue(wq);
 		return rc;
+	}
 
 	rc = xprt_register_transport(&xprt_rdma_bc);
 	if (rc) {
+		destroy_workqueue(wq);
 		xprt_unregister_transport(&xprt_rdma);
 		return rc;
 	}
net/sunrpc/xprtrdma/verbs.c +28 -18

110 lines not on this page · raw message

net/sunrpc/xprtrdma/xprt_rdma.h +4 -1
index 08bda29ed953..048d2e329384 100644
@@ -70,7 +70,6 @@
  */
 struct rpcrdma_mr;
 struct rpcrdma_ep {
-	struct kref		re_kref;
 	struct rdma_cm_id 	*re_id;
 	struct ib_pd		*re_pd;
 	unsigned int		re_max_rdma_segs;
@@ -100,6 +99,9 @@ struct rpcrdma_ep {
 	atomic_t		re_completion_ids;
 
 	char			re_write_pad[XDR_UNIT];
+
+	struct kref		re_kref;
+	struct work_struct	re_worker;
 };
 
 /* Pre-allocate extra Work Requests for handling reverse-direction
@@ -583,6 +585,7 @@ void xprt_rdma_format_addresses(struct rpc_xprt *xprt, struct sockaddr *sap);
 void xprt_rdma_free_addresses(struct rpc_xprt *xprt);
 void xprt_rdma_close(struct rpc_xprt *xprt);
 void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq);
+extern struct workqueue_struct *rpcrdma_release_wq;
 int xprt_rdma_init(void);
 void xprt_rdma_cleanup(void);
 
Signature · 1 line
2.44.0
author maintainer
#4 lore raw

RFC PATCH 4/4 xprtrdma: Move MRs to struct rpcrdma_ep

In reply to [email protected]

From: Chuck Lever <[email protected]>

MRs are a connection-specific hardware resource, thus they should be anchored in the EP and released with the other hardware resources.

---
 net/sunrpc/xprtrdma/frwr_ops.c  |  7 ++--
 net/sunrpc/xprtrdma/verbs.c     | 70 ++++++++++++++++-----------------
 net/sunrpc/xprtrdma/xprt_rdma.h | 13 +++---
 3 files changed, 45 insertions(+), 45 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +3 -4
index 6e508708d06d..7e918753eec4 100644
@@ -112,15 +112,14 @@ void frwr_reset(struct rpcrdma_req *req)
 
 /**
  * frwr_mr_init - Initialize one MR
- * @r_xprt: controlling transport instance
+ * @ep: controlling transport instance
  * @mr: generic MR to prepare for FRWR
  *
  * Returns zero if successful. Otherwise a negative errno
  * is returned.
  */
-int frwr_mr_init(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr)
+int frwr_mr_init(struct rpcrdma_ep *ep, struct rpcrdma_mr *mr)
 {
-	struct rpcrdma_ep *ep = r_xprt->rx_ep;
 	unsigned int depth = ep->re_max_fr_depth;
 	struct scatterlist *sg;
 	struct ib_mr *frmr;
@@ -134,7 +133,7 @@ int frwr_mr_init(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr)
 	if (IS_ERR(frmr))
 		goto out_mr_err;
 
-	mr->mr_xprt = r_xprt;
+	mr->mr_ep = ep;
 	mr->mr_ibmr = frmr;
 	mr->mr_device = NULL;
 	INIT_LIST_HEAD(&mr->mr_list);
net/sunrpc/xprtrdma/verbs.c +35 -35

171 lines not on this page · raw message

net/sunrpc/xprtrdma/xprt_rdma.h +7 -6
index 048d2e329384..ce703b6e3b86 100644
@@ -96,6 +96,11 @@ struct rpcrdma_ep {
 	unsigned int		re_inline_send;	/* negotiated */
 	unsigned int		re_inline_recv;	/* negotiated */
 
+	spinlock_t		re_mr_lock;
+	struct list_head	re_mrs;
+	struct list_head	re_all_mrs;
+	struct work_struct	re_refresh_worker;
+
 	atomic_t		re_completion_ids;
 
 	char			re_write_pad[XDR_UNIT];
@@ -253,7 +258,7 @@ struct rpcrdma_mr {
 		struct ib_reg_wr	mr_regwr;
 		struct ib_send_wr	mr_invwr;
 	};
-	struct rpcrdma_xprt	*mr_xprt;
+	struct rpcrdma_ep	*mr_ep;
 	u32			mr_handle;
 	u32			mr_length;
 	u64			mr_offset;
@@ -365,7 +370,6 @@ rpcrdma_mr_pop(struct list_head *list)
 struct rpcrdma_buffer {
 	spinlock_t		rb_lock;
 	struct list_head	rb_send_bufs;
-	struct list_head	rb_mrs;
 
 	unsigned long		rb_sc_head;
 	unsigned long		rb_sc_tail;
@@ -373,7 +377,6 @@ struct rpcrdma_buffer {
 	struct rpcrdma_sendctx	**rb_sc_ctxs;
 
 	struct list_head	rb_allreqs;
-	struct list_head	rb_all_mrs;
 	struct list_head	rb_all_reps;
 
 	struct llist_head	rb_free_reps;
@@ -383,8 +386,6 @@ struct rpcrdma_buffer {
 
 	u32			rb_bc_srv_max_requests;
 	u32			rb_bc_max_requests;
-
-	struct work_struct	rb_refresh_worker;
 };
 
 /*
@@ -533,7 +534,7 @@ rpcrdma_data_dir(bool writing)
  */
 void frwr_reset(struct rpcrdma_req *req);
 int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device);
-int frwr_mr_init(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr);
+int frwr_mr_init(struct rpcrdma_ep *ep, struct rpcrdma_mr *mr);
 void frwr_mr_release(struct rpcrdma_mr *mr);
 struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt,
 				struct rpcrdma_mr_seg *seg,
Signature · 1 line
2.44.0
#5 lore raw
In reply to [email protected]
Quoted 7 lines · On 29.04.24 17:25, [email protected] wrote:

From: Chuck Lever <[email protected]>

Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process.

I've done some testing of this series, now looking for review comments.

How to make tests with nfs && rdma? Can you provide some steps or tools? I am interested in nfs && rdma.

Thanks,
Zhu Yanjun

Quoted 16 lines

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

  net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
  net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
  net/sunrpc/xprtrdma/transport.c |  20 +++-
  net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
  net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
  5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

#6 lore raw
Quoted 12 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:

From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.

How to make tests with nfs && rdma? Can you provide some steps or tools?

We are building NFS tests into kdevops:

https://github.com/linux-kdevops/kdevops.git

and there is a config option to use soft iWARP instead of TCP.

kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.

Quoted 18 lines

I am interested in nfs && rdma.

Thanks,
Zhu Yanjun

Quoted 12 lines

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

 net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
 net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
 net/sunrpc/xprtrdma/transport.c |  20 +++-
 net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
 net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
 5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

--
Chuck Lever

#7 lore raw
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:

From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.

Quoted 6 lines
Quoted 1 line

How to make tests with nfs && rdma? Can you provide some steps or tools?

We are building NFS tests into kdevops:

https://github.com/linux-kdevops/kdevops.git

and there is a config option to use soft iWARP instead of TCP.

Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?

If yes, does nfs work well with RXE? I am just curious with nfs && RXE.

Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.

The popular RDMA implementation is RoCEv2 which is based on UDP protocol.

So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.

If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).

Best Regards,

Zhu Yanjun

Quoted 27 lines

kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.

Quoted 17 lines

I am interested in nfs && rdma.

Thanks,
Zhu Yanjun

Quoted 12 lines

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

  net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
  net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
  net/sunrpc/xprtrdma/transport.c |  20 +++-
  net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
  net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
  5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

--
Chuck Lever

Signature · 2 lines
Best Regards,
Yanjun.Zhu
#8 lore raw
Quoted 26 lines · Zhu Yanjun
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:

From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.

Quoted 6 lines
Quoted 1 line

How to make tests with nfs && rdma? Can you provide some steps or tools?

We are building NFS tests into kdevops:

https://github.com/linux-kdevops/kdevops.git

and there is a config option to use soft iWARP instead of TCP.

Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?

If yes, does nfs work well with RXE? I am just curious with nfs && RXE.

Quoted 7 lines

Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.

The popular RDMA implementation is RoCEv2 which is based on UDP protocol.

So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.

If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).

Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.

Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Quoted 26 lines

Best Regards,

Zhu Yanjun

Quoted 22 lines

kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.

Quoted 17 lines

I am interested in nfs && rdma.

Thanks,
Zhu Yanjun

Quoted 12 lines

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

 net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
 net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
 net/sunrpc/xprtrdma/transport.c |  20 +++-
 net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
 net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
 5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

Quoted 7 lines
Quoted 4 lines

--
Chuck Lever

Signature · 2 lines
Best Regards,
Yanjun.Zhu

--
Chuck Lever

#9 lore raw
Quoted 39 lines · Chuck Lever III
Quoted 26 lines · Zhu Yanjun
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:

From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.

Quoted 6 lines
Quoted 1 line

How to make tests with nfs && rdma? Can you provide some steps or tools?

We are building NFS tests into kdevops:

https://github.com/linux-kdevops/kdevops.git

and there is a config option to use soft iWARP instead of TCP.

Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?

If yes, does nfs work well with RXE? I am just curious with nfs && RXE.

Quoted 7 lines

Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.

The popular RDMA implementation is RoCEv2 which is based on UDP protocol.

So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.

If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).

Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.

Quoted 4 lines

Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.

Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device?

Quoted 4 lines

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

Best Regards,
Zhu Yanjun

Quoted 41 lines
Quoted 28 lines

Best Regards,

Zhu Yanjun

Quoted 22 lines

kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.

Quoted 17 lines

I am interested in nfs && rdma.

Thanks,
Zhu Yanjun

Quoted 12 lines

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

  net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
  net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
  net/sunrpc/xprtrdma/transport.c |  20 +++-
  net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
  net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
  5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

Quoted 7 lines
Quoted 4 lines

--
Chuck Lever

Signature · 2 lines
Best Regards,
Yanjun.Zhu

--
Chuck Lever

#10 lore raw
Quoted 45 lines · Zhu Yanjun
Quoted 31 lines · Chuck Lever III
Quoted 24 lines · Zhu Yanjun
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:

From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.

Quoted 6 lines
Quoted 1 line

How to make tests with nfs && rdma? Can you provide some steps or tools?

We are building NFS tests into kdevops:

https://github.com/linux-kdevops/kdevops.git

and there is a config option to use soft iWARP instead of TCP.

Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?

If yes, does nfs work well with RXE? I am just curious with nfs && RXE.

Quoted 7 lines

Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.

The popular RDMA implementation is RoCEv2 which is based on UDP protocol.

So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.

If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).

Quoted 8 lines

Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.
Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.

Quoted 1 line

Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device?

No, rxe cannot be configured to use tunnel devices, AFAIK.

Quoted 5 lines
Quoted 3 lines

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

No new tests are necessary. The only thing missing right now is the ability to set up rxe devices on all the test systems.

Quoted 38 lines

Best Regards,
Zhu Yanjun

Quoted 35 lines
Quoted 26 lines

Best Regards,

Zhu Yanjun

Quoted 22 lines

kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.

Quoted 17 lines

I am interested in nfs && rdma.

Thanks,
Zhu Yanjun

Quoted 12 lines

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

 net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
 net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
 net/sunrpc/xprtrdma/transport.c |  20 +++-
 net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
 net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
 5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

Quoted 7 lines
Quoted 4 lines

--
Chuck Lever

Signature · 2 lines
Best Regards,
Yanjun.Zhu

--
Chuck Lever

--
Chuck Lever

#11 lore raw
Quoted 45 lines · Chuck Lever III
Quoted 41 lines · Zhu Yanjun
Quoted 29 lines · Chuck Lever III
Quoted 22 lines · Zhu Yanjun
Quoted 8 lines · Chuck Lever III
Quoted 8 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:

From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.

Quoted 6 lines
Quoted 1 line

How to make tests with nfs && rdma? Can you provide some steps or tools?

We are building NFS tests into kdevops:

https://github.com/linux-kdevops/kdevops.git

and there is a config option to use soft iWARP instead of TCP.

Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?

If yes, does nfs work well with RXE? I am just curious with nfs && RXE.

Quoted 7 lines

Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.

The popular RDMA implementation is RoCEv2 which is based on UDP protocol.

So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.

If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).

Quoted 8 lines

Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.
Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.

Quoted 1 line

Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device?

No, rxe cannot be configured to use tunnel devices, AFAIK.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

No new tests are necessary. The only thing missing right now is the ability to set up rxe devices on all the test systems.

Got it. Thanks.

Zhu Yanjun

Quoted 45 lines
Quoted 40 lines

Best Regards,
Zhu Yanjun

Quoted 35 lines
Quoted 26 lines

Best Regards,

Zhu Yanjun

Quoted 22 lines

kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.

Quoted 17 lines

I am interested in nfs && rdma.

Thanks,
Zhu Yanjun

Quoted 12 lines

Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep

  net/sunrpc/xprtrdma/frwr_ops.c  |  13 ++-
  net/sunrpc/xprtrdma/rpc_rdma.c  |   3 +-
  net/sunrpc/xprtrdma/transport.c |  20 +++-
  net/sunrpc/xprtrdma/verbs.c     | 173 ++++++++++++++++----------------
  net/sunrpc/xprtrdma/xprt_rdma.h |  21 ++--
  5 files changed, 125 insertions(+), 105 deletions(-)

base-commit: e67572cd2204894179d89bd7b984072f19313b03

Quoted 7 lines
Quoted 4 lines

--
Chuck Lever

Signature · 2 lines
Best Regards,
Yanjun.Zhu

--
Chuck Lever

--
Chuck Lever

Signature · 2 lines
Best Regards,
Yanjun.Zhu
#12 lore raw
In reply to Zhu Yanjun
Quoted 9 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:

https://github.com/chucklever/kdevops/tree/add-rxe-support

But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...

--
Chuck Lever

#13 lore raw
Quoted 5 lines · Chuck Lever III
Quoted 5 lines · Zhu Yanjun

On 30.04.24 16:13, Chuck Lever III wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:

Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge. I hope rxe can work well in kdeops. So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.

Quoted 5 lines

https://github.com/chucklever/kdevops/tree/add-rxe-support

But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...

Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.

Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.
2. ping -c 3 server_ip on client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -a server_ip
rping -c -a server_ip -C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.

Zhu Yanjun

Quoted 5 lines

--
Chuck Lever

#14 lore raw
Quoted 16 lines · Zhu Yanjun
Quoted 11 lines · Chuck Lever III
Quoted 10 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

Quoted 7 lines
Quoted 2 lines

As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:

Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge.

I split this into four smaller patches, HTH.

Quoted 10 lines

I hope rxe can work well in kdeops.
So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.

Quoted 5 lines

https://github.com/chucklever/kdevops/tree/add-rxe-support

But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...

Quoted 8 lines

Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.

Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.

I don't have rxe set up on loopback, so I substituted the host's configured Ethernet IP.

The tests works on the NFS server, but the rping client hangs on the NFS client (both running v6.10-rc1).

I rebooted in to the Fedora 39 stock kernel, and the rping tests pass.

However, when I try to run fstests with NFS/RDMA using rxe, the client kernel reports a soft CPU lock-up, and top shows this:

    115 root      20   0       0      0      0 R  99.3   0.0   1:03.50 kworker/u8:5+rxe_wq

So I think this is enough to show that the Ansible parts of this change are working as expected. I can push this to kdevops now if there are no objections, and someone (maybe you, maybe me) can sort out the rxe specific issues later.

Quoted 10 lines

2. ping -c 3 server_ip on client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -a server_ip
rping -c -a server_ip -C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.

Quoted 7 lines

Zhu Yanjun

Quoted 5 lines

--
Chuck Lever

--
Chuck Lever

#15 lore raw
Quoted 53 lines · Chuck Lever III
Quoted 17 lines · Zhu Yanjun
Quoted 11 lines · Chuck Lever III
Quoted 10 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

Quoted 7 lines
Quoted 2 lines

As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:

Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge.

I split this into four smaller patches, HTH.

Quoted 10 lines

I hope rxe can work well in kdeops.
So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.

Quoted 5 lines

https://github.com/chucklever/kdevops/tree/add-rxe-support

But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...

Quoted 8 lines

Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.

Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.

I don't have rxe set up on loopback, so I substituted the host's configured Ethernet IP.

The tests works on the NFS server, but the rping client hangs on the NFS client (both running v6.10-rc1).

Quoted 7 lines

I rebooted in to the Fedora 39 stock kernel, and the rping tests pass.

However, when I try to run fstests with NFS/RDMA using rxe, the client kernel reports a soft CPU lock-up, and top shows this:

    115 root      20   0       0      0      0 R  99.3   0.0   1:03.50 kworker/u8:5+rxe_wq

rxe_wq is introduced in the commit 9b4b7c1f9f54 "RDMA/rxe: Add workqueue support for rxe tasks".
And this commit is merged into kernel v6.4-rc2-1-g9b4b7c1f9f54.

And the Fedora 39 stock kernel is kernel 6.5. So maybe some commits between 6.5 and 6.10 introduce this problem.

Quoted 5 lines

So I think this is enough to show that the Ansible parts of this change are working as expected. I can push this to kdevops now if there are no objections, and someone (maybe you, maybe me) can sort out the rxe specific issues later.

Thanks. After I can reproduce this problem in my local host, I am very glad to delve into this problem. Perhaps it will take me a long time since I do not have a good host to deploy kdeops.

To be honest, perhaps "git bisec" can find the commit that introduce this problem. If you can find the commit, we can fix this problem very quickly^_^

Thanks,
Zhu Yanjun

Quoted 24 lines
Quoted 12 lines

2. ping -c 3 server_ip on client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -a server_ip
rping -c -a server_ip -C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.

Quoted 7 lines

Zhu Yanjun

Quoted 5 lines

--
Chuck Lever

--
Chuck Lever

#16 lore raw
Quoted 64 lines · Zhu Yanjun
Quoted 59 lines · Chuck Lever III
Quoted 17 lines · Zhu Yanjun
Quoted 11 lines · Chuck Lever III
Quoted 10 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III

It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.

Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.

Quoted 7 lines
Quoted 2 lines

As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:

Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge.

I split this into four smaller patches, HTH.

Quoted 10 lines

I hope rxe can work well in kdeops.
So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.

Quoted 5 lines

https://github.com/chucklever/kdevops/tree/add-rxe-support

But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...

Quoted 8 lines

Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.

Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.

I don't have rxe set up on loopback, so I substituted the host's configured Ethernet IP.

The tests works on the NFS server, but the rping client hangs on the NFS client (both running v6.10-rc1).

I rebooted in to the Fedora 39 stock kernel, and the rping tests pass.

However, when I try to run fstests with NFS/RDMA using rxe, the client kernel reports a soft CPU lock-up, and top shows this:

Quoted 8 lines
Quoted 1 line
   115 root      20   0       0      0      0 R  99.3   0.0   1:03.50 kworker/u8:5+rxe_wq

rxe_wq is introduced in the commit 9b4b7c1f9f54 "RDMA/rxe: Add workqueue support for rxe tasks".
And this commit is merged into kernel v6.4-rc2-1-g9b4b7c1f9f54.

And the Fedora 39 stock kernel is kernel 6.5. So maybe some commits between 6.5 and 6.10 introduce this problem.

I couldn't get 6.10-rc1 working at all. This failure occurred with the stock Fedora 39 kernel and fstests with NFS v4.2 on RDMA.

Quoted 8 lines
Quoted 4 lines

So I think this is enough to show that the Ansible parts of this change are working as expected. I can push this to kdevops now if there are no objections, and someone (maybe you, maybe me) can sort out the rxe specific issues later.

Thanks. After I can reproduce this problem in my local host, I am very glad to delve into this problem. Perhaps it will take me a long time since I do not have a good host to deploy kdevops.

kdevops works on laptops too. The limiting factor seems to be memory for libvirt guests. Only two guests are needed for this test.

Quoted 3 lines

To be honest, perhaps "git bisec" can find the commit that introduce this problem. If you can find the commit, we can fix this problem very quickly^_^

Since this is the first time I've ever used rxe, I don't have a "good" commit to start from.

Quoted 25 lines

Thanks,
Zhu Yanjun

Quoted 22 lines
Quoted 12 lines

2. ping -c 3 server_ip on client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -a server_ip
rping -c -a server_ip -C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.

Quoted 7 lines

Zhu Yanjun

Quoted 5 lines

--
Chuck Lever

--
Chuck Lever

--
Chuck Lever