NFS: Fix another 'check_flush_dependency' splat
Subsystems (4)
[email protected]
maintainercover NFS: Fix another 'check_flush_dependency' splat
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process.
I've done some testing of this series, now looking for review comments.
Chuck Lever (4):
xprtrdma: Remove temp allocation of rpcrdma_rep objects
xprtrdma: Clean up synopsis of frwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to struct rpcrdma_ep
net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)
base-commit: e67572cd2204894179d89bd7b984072f19313b03
Signature · 1 line
2.44.0
[email protected]
maintainerRFC PATCH 1/4 xprtrdma: Remove temp allocation of rpcrdma_rep objects
From: Chuck Lever <[email protected]>
The original code was designed so that most calls to rpcrdma_rep_create() would occur on the NUMA node that the device preferred. There are a few cases where that's not possible, so those reps are marked as temporary.
However, we have the device (and its preferred node) already in rpcrdma_rep_create(), so let's use that.
--- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/verbs.c | 57 ++++++++++++++------------------- net/sunrpc/xprtrdma/xprt_rdma.h | 3 +- 3 files changed, 26 insertions(+), 37 deletions(-)
net/sunrpc/xprtrdma/rpc_rdma.c +1 -2
@@ -1471,8 +1471,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) credits =1 ;/* don't deadlock */ else if (credits > r_xprt->rx_ep->re_max_requests) credits = r_xprt->rx_ep->re_max_requests; -rpcrdma_post_recvs (r_xprt, credits + (buf->rb_bc_srv_max_requests <<1 ), - false); +rpcrdma_post_recvs (r_xprt, credits + (buf->rb_bc_srv_max_requests <<1 ));if (buf->rb_credits != credits)rpcrdma_update_cwnd (r_xprt, credits);
net/sunrpc/xprtrdma/verbs.c +24 -33
146 lines not on this page · raw message
net/sunrpc/xprtrdma/xprt_rdma.h +1 -2
@@ -198,7 +198,6 @@ struct rpcrdma_rep {__be32 rr_proc;int rr_wc_flags;u32 rr_inv_rkey; -bool rr_temp;struct rpcrdma_regbuf *rr_rdmabuf;struct rpcrdma_xprt *rr_rxprt;struct rpc_rqst *rr_rqst;
@@ -466,7 +465,7 @@ void rpcrdma_flush_disconnect(struct rpcrdma_xprt *r_xprt, struct ib_wc *wc);int rpcrdma_xprt_connect (struct rpcrdma_xprt *r_xprt);void rpcrdma_xprt_disconnect (struct rpcrdma_xprt *r_xprt); -void rpcrdma_post_recvs (struct rpcrdma_xprt *r_xprt,int needed,bool temp); +void rpcrdma_post_recvs (struct rpcrdma_xprt *r_xprt,int needed);/* * Buffer calls - xprtrdma/verbs.c
Signature · 1 line
2.44.0
[email protected]
maintainerRFC PATCH 2/4 xprtrdma: Clean up synopsis of frwr_mr_unmap()
In reply to [email protected]From: Chuck Lever <[email protected]>
Commit 7a03aeb66c41 ("xprtrdma: Micro-optimize MR DMA-unmapping") removed the last use of the @r_xprt parameter in this function, but neglected to remove the parameter itself.
--- net/sunrpc/xprtrdma/frwr_ops.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +3 -3
@@ -54,7 +54,7 @@ static void frwr_cid_init(struct rpcrdma_ep *ep, cid->ci_completion_id = mr->mr_ibmr->res.id; } -static void frwr_mr_unmap (struct rpcrdma_xprt *r_xprt,struct rpcrdma_mr *mr) +static void frwr_mr_unmap (struct rpcrdma_mr *mr) {if (mr->mr_device) {trace_xprtrdma_mr_unmap (mr);
@@ -73,7 +73,7 @@ void frwr_mr_release(struct rpcrdma_mr *mr) {int rc; -frwr_mr_unmap (mr->mr_xprt, mr); +frwr_mr_unmap (mr); rc =ib_dereg_mr (mr->mr_ibmr);if (rc)
@@ -84,7 +84,7 @@ void frwr_mr_release(struct rpcrdma_mr *mr)static void frwr_mr_put (struct rpcrdma_mr *mr) { -frwr_mr_unmap (mr->mr_xprt, mr); +frwr_mr_unmap (mr);/* The MR is returned to the req's MR free list instead * of to the xprt's MR free list. No spinlock is needed.
Signature · 1 line
2.44.0
[email protected]
maintainerRFC PATCH 3/4 xprtrdma: Delay releasing connection hardware resources
In reply to [email protected]From: Chuck Lever <[email protected]>
xprtiod_workqueue is a MEM_RECLAIM-enabled workqueue. However, the RDMA core API functions are not memory reclaim-safe. This was partially accomplished by commit 6b1eb3b22272 ("SUNRPC: Replace the use of the xprtiod WQ in rpcrdma").
This commit addressed the issue in the connect path, but not in the disconnect path. Thus sometimes a transport disconnect results in this splat:
workqueue: WQ_MEM_RECLAIM xprtiod:xprt_autoclose [sunrpc] is flushing !WQ_MEM_RECLAIM events_highpri:rpcrdma_mr_refresh_worker [rpcrdma]
WARNING: CPU: 1 PID: 20378 at kernel/workqueue.c:3728 check_flush_dependency+0x101/0x120 ? check_flush_dependency+0x101/0x120 ? report_bug+0x175/0x1a0 ? handle_bug+0x44/0x90 ? exc_invalid_op+0x1c/0x70 ? asm_exc_invalid_op+0x1f/0x30 ? __pfx_rpcrdma_mr_refresh_worker+0x10/0x10 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac] ? check_flush_dependency+0x101/0x120 __flush_work.isra.0+0x20a/0x290 __cancel_work_sync+0x129/0x1c0 cancel_work_sync+0x14/0x20 rpcrdma_xprt_disconnect+0x229/0x3f0 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac] xprt_rdma_close+0x16/0x40 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac] xprt_autoclose+0x63/0x110 [sunrpc a04d701bce94b5a8fb541cafbe1a489d6b1ab5b3] process_one_work+0x19e/0x3f0 worker_thread+0x340/0x510 ? __pfx_worker_thread+0x10/0x10 kthread+0xf7/0x130 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x41/0x60 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30
Create a context in which it is safe to manage resources that are not memory reclaim-safe that can be invoked during transport disconnect. Essentially this means that releasing an rpcrdma_ep is now done completely asynchronously.
Subsequent patches will move the release of transport resources into this new context.
--- net/sunrpc/xprtrdma/transport.c | 20 +++++++++++++- net/sunrpc/xprtrdma/verbs.c | 46 ++++++++++++++++++++------------- net/sunrpc/xprtrdma/xprt_rdma.h | 5 +++- 3 files changed, 51 insertions(+), 20 deletions(-)
net/sunrpc/xprtrdma/transport.c +19 -1
@@ -761,8 +761,12 @@ static struct xprt_class xprt_rdma = { .netid = { "rdma", "rdma6", "" }, }; +struct workqueue_struct *rpcrdma_release_wq__read_mostly ; +void xprt_rdma_cleanup (void ) { +struct workqueue_struct *wq; +#if IS_ENABLED (CONFIG_SUNRPC_DEBUG )if (sunrpc_table_header) {unregister_sysctl_table (sunrpc_table_header);
@@ -772,18 +776,32 @@ void xprt_rdma_cleanup(void)xprt_unregister_transport (&xprt_rdma);xprt_unregister_transport (&xprt_rdma_bc); + + wq = rpcrdma_release_wq; + rpcrdma_release_wq =NULL ; +destroy_workqueue (wq); }int xprt_rdma_init (void ) { +struct workqueue_struct *wq;int rc; +/* provision a WQ that is always unbound and !mem_reclaim */ + wq =alloc_workqueue ("rpcrdma_release" ,WQ_UNBOUND ,0 ); +if (!wq) +return -ENOMEM ; + rpcrdma_release_wq = wq; + rc =xprt_register_transport (&xprt_rdma); -if (rc) +if (rc) { +destroy_workqueue (wq);return rc; + } rc =xprt_register_transport (&xprt_rdma_bc);if (rc) { +destroy_workqueue (wq);xprt_unregister_transport (&xprt_rdma);return rc; }
net/sunrpc/xprtrdma/verbs.c +28 -18
110 lines not on this page · raw message
net/sunrpc/xprtrdma/xprt_rdma.h +4 -1
@@ -70,7 +70,6 @@*/ struct rpcrdma_mr ;struct rpcrdma_ep { -struct kref re_kref;struct rdma_cm_id *re_id;struct ib_pd *re_pd;unsigned int re_max_rdma_segs;
@@ -100,6 +99,9 @@ struct rpcrdma_ep {atomic_t re_completion_ids;char re_write_pad[XDR_UNIT ]; + +struct kref re_kref; +struct work_struct re_worker; };/* Pre-allocate extra Work Requests for handling reverse-direction
@@ -583,6 +585,7 @@ void xprt_rdma_format_addresses(struct rpc_xprt *xprt, struct sockaddr *sap);void xprt_rdma_free_addresses (struct rpc_xprt *xprt);void xprt_rdma_close (struct rpc_xprt *xprt);void xprt_rdma_print_stats (struct rpc_xprt *xprt,struct seq_file *seq); +extern struct workqueue_struct *rpcrdma_release_wq;int xprt_rdma_init (void );void xprt_rdma_cleanup (void );
Signature · 1 line
2.44.0
[email protected]
maintainerRFC PATCH 4/4 xprtrdma: Move MRs to struct rpcrdma_ep
In reply to [email protected]From: Chuck Lever <[email protected]>
MRs are a connection-specific hardware resource, thus they should be anchored in the EP and released with the other hardware resources.
--- net/sunrpc/xprtrdma/frwr_ops.c | 7 ++-- net/sunrpc/xprtrdma/verbs.c | 70 ++++++++++++++++----------------- net/sunrpc/xprtrdma/xprt_rdma.h | 13 +++--- 3 files changed, 45 insertions(+), 45 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +3 -4
@@ -112,15 +112,14 @@ void frwr_reset(struct rpcrdma_req *req)/** * frwr_mr_init - Initialize one MR -* @r_xprt: controlling transport instance +* @ep: controlling transport instance * @mr: generic MR to prepare for FRWR * * Returns zero if successful. Otherwise a negative errno * is returned. */ -int frwr_mr_init (struct rpcrdma_xprt *r_xprt,struct rpcrdma_mr *mr) +int frwr_mr_init (struct rpcrdma_ep *ep,struct rpcrdma_mr *mr) { -struct rpcrdma_ep *ep = r_xprt->rx_ep;unsigned int depth = ep->re_max_fr_depth;struct scatterlist *sg;struct ib_mr *frmr;
@@ -134,7 +133,7 @@ int frwr_mr_init(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr)if (IS_ERR (frmr)) goto out_mr_err; - mr->mr_xprt = r_xprt; + mr->mr_ep = ep; mr->mr_ibmr = frmr; mr->mr_device =NULL ;INIT_LIST_HEAD (&mr->mr_list);
net/sunrpc/xprtrdma/verbs.c +35 -35
171 lines not on this page · raw message
net/sunrpc/xprtrdma/xprt_rdma.h +7 -6
@@ -96,6 +96,11 @@ struct rpcrdma_ep {unsigned int re_inline_send;/* negotiated */ unsigned int re_inline_recv;/* negotiated */ +spinlock_t re_mr_lock; +struct list_head re_mrs; +struct list_head re_all_mrs; +struct work_struct re_refresh_worker; +atomic_t re_completion_ids;char re_write_pad[XDR_UNIT ];
@@ -253,7 +258,7 @@ struct rpcrdma_mr {struct ib_reg_wr mr_regwr;struct ib_send_wr mr_invwr; }; -struct rpcrdma_xprt *mr_xprt; +struct rpcrdma_ep *mr_ep;u32 mr_handle;u32 mr_length;u64 mr_offset;
@@ -365,7 +370,6 @@ rpcrdma_mr_pop(struct list_head *list)struct rpcrdma_buffer {spinlock_t rb_lock;struct list_head rb_send_bufs; -struct list_head rb_mrs;unsigned long rb_sc_head;unsigned long rb_sc_tail;
@@ -373,7 +377,6 @@ struct rpcrdma_buffer {struct rpcrdma_sendctx **rb_sc_ctxs;struct list_head rb_allreqs; -struct list_head rb_all_mrs;struct list_head rb_all_reps;struct llist_head rb_free_reps;
@@ -383,8 +386,6 @@ struct rpcrdma_buffer {u32 rb_bc_srv_max_requests;u32 rb_bc_max_requests; - -struct work_struct rb_refresh_worker; };/*
@@ -533,7 +534,7 @@ rpcrdma_data_dir(bool writing)*/ void frwr_reset (struct rpcrdma_req *req);int frwr_query_device (struct rpcrdma_ep *ep,const struct ib_device *device); -int frwr_mr_init (struct rpcrdma_xprt *r_xprt,struct rpcrdma_mr *mr); +int frwr_mr_init (struct rpcrdma_ep *ep,struct rpcrdma_mr *mr);void frwr_mr_release (struct rpcrdma_mr *mr); struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg,
Signature · 1 line
2.44.0
Quoted 7 lines · On 29.04.24 17:25, [email protected] wrote:
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process.
I've done some testing of this series, now looking for review comments.
How to make tests with nfs && rdma? Can you provide some steps or tools? I am interested in nfs && rdma.
Thanks,
Zhu Yanjun
Quoted 16 lines
Chuck Lever (4):
xprtrdma: Remove temp allocation ofrpcrdma_repobjects
xprtrdma: Clean up synopsis offrwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to structrpcrdma_epnet/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)base-commit:
e67572cd2204894179d89bd7b984072f19313b03
Quoted 12 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.How to make tests with nfs && rdma? Can you provide some steps or tools?
We are building NFS tests into kdevops:
https://github.com/linux-kdevops/kdevops.git
and there is a config option to use soft iWARP instead of TCP.
kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.
Quoted 18 lines
I am interested in nfs && rdma.
Thanks,
Zhu YanjunQuoted 12 lines
Chuck Lever (4):
xprtrdma: Remove temp allocation ofrpcrdma_repobjects
xprtrdma: Clean up synopsis offrwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to structrpcrdma_epnet/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)base-commit:
e67572cd2204894179d89bd7b984072f19313b03
--
Chuck Lever
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.
Quoted 6 lines
Quoted 1 line
How to make tests with nfs && rdma? Can you provide some steps or tools?
We are building NFS tests into kdevops:
https://github.com/linux-kdevops/kdevops.git
and there is a config option to use soft iWARP instead of TCP.
Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?
If yes, does nfs work well with RXE? I am just curious with nfs && RXE.
Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.
The popular RDMA implementation is RoCEv2 which is based on UDP protocol.
So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.
If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).
Best Regards,
Zhu Yanjun
Quoted 27 lines
kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.
Quoted 17 lines
I am interested in nfs && rdma.
Thanks,
Zhu YanjunQuoted 12 lines
Chuck Lever (4):
xprtrdma: Remove temp allocation ofrpcrdma_repobjects
xprtrdma: Clean up synopsis offrwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to structrpcrdma_epnet/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)base-commit:
e67572cd2204894179d89bd7b984072f19313b03--
Chuck Lever
Signature · 2 lines
Best Regards, Yanjun.Zhu
Quoted 26 lines · Zhu Yanjun
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.Quoted 6 lines
Quoted 1 line
How to make tests with nfs && rdma? Can you provide some steps or tools?
We are building NFS tests into kdevops:
https://github.com/linux-kdevops/kdevops.git
and there is a config option to use soft iWARP instead of TCP.
Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?
If yes, does nfs work well with RXE? I am just curious with nfs && RXE.
Quoted 7 lines
Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.
The popular RDMA implementation is RoCEv2 which is based on UDP protocol.
So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.
If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).
Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.
Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Quoted 26 lines
Best Regards,
Zhu Yanjun
Quoted 22 lines
kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.
Quoted 17 lines
I am interested in nfs && rdma.
Thanks,
Zhu YanjunQuoted 12 lines
Chuck Lever (4):
xprtrdma: Remove temp allocation ofrpcrdma_repobjects
xprtrdma: Clean up synopsis offrwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to structrpcrdma_epnet/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)base-commit:
e67572cd2204894179d89bd7b984072f19313b03
Quoted 7 lines
Quoted 4 lines
--
Chuck LeverSignature · 2 lines
Best Regards, Yanjun.Zhu
--
Chuck Lever
Quoted 39 lines · Chuck Lever III
Quoted 26 lines · Zhu Yanjun
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.Quoted 6 lines
Quoted 1 line
How to make tests with nfs && rdma? Can you provide some steps or tools?
We are building NFS tests into kdevops:
https://github.com/linux-kdevops/kdevops.git
and there is a config option to use soft iWARP instead of TCP.
Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?
If yes, does nfs work well with RXE? I am just curious with nfs && RXE.
Quoted 7 lines
Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.
The popular RDMA implementation is RoCEv2 which is based on UDP protocol.
So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.
If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).
Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.
Quoted 4 lines
Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.
Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device?
Quoted 4 lines
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
Best Regards,
Zhu Yanjun
Quoted 41 lines
Quoted 28 lines
Best Regards,
Zhu Yanjun
Quoted 22 lines
kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.
Quoted 17 lines
I am interested in nfs && rdma.
Thanks,
Zhu YanjunQuoted 12 lines
Chuck Lever (4):
xprtrdma: Remove temp allocation ofrpcrdma_repobjects
xprtrdma: Clean up synopsis offrwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to structrpcrdma_epnet/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)base-commit:
e67572cd2204894179d89bd7b984072f19313b03Quoted 7 lines
Quoted 4 lines
--
Chuck LeverSignature · 2 lines
Best Regards, Yanjun.Zhu--
Chuck Lever
Quoted 45 lines · Zhu Yanjun
Quoted 31 lines · Chuck Lever III
Quoted 24 lines · Zhu Yanjun
Quoted 9 lines · Chuck Lever III
Quoted 9 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.Quoted 6 lines
Quoted 1 line
How to make tests with nfs && rdma? Can you provide some steps or tools?
We are building NFS tests into kdevops:
https://github.com/linux-kdevops/kdevops.git
and there is a config option to use soft iWARP instead of TCP.
Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?
If yes, does nfs work well with RXE? I am just curious with nfs && RXE.
Quoted 7 lines
Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.
The popular RDMA implementation is RoCEv2 which is based on UDP protocol.
So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.
If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).
Quoted 8 lines
Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.
Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.
Quoted 1 line
Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device?
No, rxe cannot be configured to use tunnel devices, AFAIK.
Quoted 5 lines
Quoted 3 lines
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
No new tests are necessary. The only thing missing right now is the ability to set up rxe devices on all the test systems.
Quoted 38 lines
Best Regards,
Zhu YanjunQuoted 35 lines
Quoted 26 lines
Best Regards,
Zhu Yanjun
Quoted 22 lines
kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.
Quoted 17 lines
I am interested in nfs && rdma.
Thanks,
Zhu YanjunQuoted 12 lines
Chuck Lever (4):
xprtrdma: Remove temp allocation ofrpcrdma_repobjects
xprtrdma: Clean up synopsis offrwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to structrpcrdma_epnet/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)base-commit:
e67572cd2204894179d89bd7b984072f19313b03Quoted 7 lines
Quoted 4 lines
--
Chuck LeverSignature · 2 lines
Best Regards, Yanjun.Zhu--
Chuck Lever
--
Chuck Lever
Quoted 45 lines · Chuck Lever III
Quoted 41 lines · Zhu Yanjun
Quoted 29 lines · Chuck Lever III
Quoted 22 lines · Zhu Yanjun
Quoted 8 lines · Chuck Lever III
Quoted 8 lines · Zhu Yanjun
Quoted 5 lines · On 29.04.24 17:25, [email protected] wrote:
From: Chuck Lever <[email protected]>
Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments.Quoted 6 lines
Quoted 1 line
How to make tests with nfs && rdma? Can you provide some steps or tools?
We are building NFS tests into kdevops:
https://github.com/linux-kdevops/kdevops.git
and there is a config option to use soft iWARP instead of TCP.
Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP?
If yes, does nfs work well with RXE? I am just curious with nfs && RXE.
Quoted 7 lines
Normally nfs works with TCP. Now nfs will use RDMA instead of TCP.
The popular RDMA implementation is RoCEv2 which is based on UDP protocol.
So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not.
If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP).
Quoted 8 lines
Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE.
Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently.Quoted 1 line
Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device?
No, rxe cannot be configured to use tunnel devices, AFAIK.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
No new tests are necessary. The only thing missing right now is the ability to set up rxe devices on all the test systems.
Got it. Thanks.
Zhu Yanjun
Quoted 45 lines
Quoted 40 lines
Best Regards,
Zhu YanjunQuoted 35 lines
Quoted 26 lines
Best Regards,
Zhu Yanjun
Quoted 22 lines
kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations.
Quoted 17 lines
I am interested in nfs && rdma.
Thanks,
Zhu YanjunQuoted 12 lines
Chuck Lever (4):
xprtrdma: Remove temp allocation ofrpcrdma_repobjects
xprtrdma: Clean up synopsis offrwr_mr_unmap()
xprtrdma: Delay releasing connection hardware resources
xprtrdma: Move MRs to structrpcrdma_epnet/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-)base-commit:
e67572cd2204894179d89bd7b984072f19313b03Quoted 7 lines
Quoted 4 lines
--
Chuck LeverSignature · 2 lines
Best Regards, Yanjun.Zhu--
Chuck Lever--
Chuck Lever
Signature · 2 lines
Best Regards, Yanjun.Zhu
Quoted 9 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:
https://github.com/chucklever/kdevops/tree/add-rxe-support
But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...
--
Chuck Lever
Quoted 5 lines · Chuck Lever III
Quoted 5 lines · Zhu Yanjun
On 30.04.24 16:13, Chuck Lever III wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:
Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge. I hope rxe can work well in kdeops. So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.
Quoted 5 lines
https://github.com/chucklever/kdevops/tree/add-rxe-support
But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...
Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.
Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.
2. ping -c 3 server_ip on client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -a server_ip
rping -c -a server_ip -C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.
Zhu Yanjun
Quoted 5 lines
--
Chuck Lever
Quoted 16 lines · Zhu Yanjun
Quoted 11 lines · Chuck Lever III
Quoted 10 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
Quoted 7 lines
Quoted 2 lines
As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:
Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge.
I split this into four smaller patches, HTH.
Quoted 10 lines
I hope rxe can work well in kdeops.
So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.Quoted 5 lines
https://github.com/chucklever/kdevops/tree/add-rxe-support
But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...
Quoted 8 lines
Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.
Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.
I don't have rxe set up on loopback, so I substituted the host's configured Ethernet IP.
The tests works on the NFS server, but the rping client hangs on the NFS client (both running v6.10-rc1).
I rebooted in to the Fedora 39 stock kernel, and the rping tests pass.
However, when I try to run fstests with NFS/RDMA using rxe, the client kernel reports a soft CPU lock-up, and top shows this:
115 root 20 0 0 0 0 R 99.3 0.0 1:03.50 kworker/u8:5+rxe_wq
So I think this is enough to show that the Ansible parts of this change are working as expected. I can push this to kdevops now if there are no objections, and someone (maybe you, maybe me) can sort out the rxe specific issues later.
Quoted 10 lines
2. ping -c 3
server_ipon client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -aserver_ip
rping -c -aserver_ip-C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.
Quoted 7 lines
Zhu Yanjun
Quoted 5 lines
--
Chuck Lever
--
Chuck Lever
Quoted 53 lines · Chuck Lever III
Quoted 17 lines · Zhu Yanjun
Quoted 11 lines · Chuck Lever III
Quoted 10 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
Quoted 7 lines
Quoted 2 lines
As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:
Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge.I split this into four smaller patches, HTH.
Quoted 10 lines
I hope rxe can work well in kdeops.
So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.Quoted 5 lines
https://github.com/chucklever/kdevops/tree/add-rxe-support
But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...
Quoted 8 lines
Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.
Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.I don't have rxe set up on loopback, so I substituted the host's configured Ethernet IP.
The tests works on the NFS server, but the rping client hangs on the NFS client (both running v6.10-rc1).
Quoted 7 lines
I rebooted in to the Fedora 39 stock kernel, and the rping tests pass.
However, when I try to run fstests with NFS/RDMA using rxe, the client kernel reports a soft CPU lock-up, and top shows this:
115 root 20 0 0 0 0 R 99.3 0.0 1:03.50 kworker/u8:5+rxe_wq
rxe_wq is introduced in the commit 9b4b7c1f9f54 "RDMA/rxe: Add workqueue support for rxe tasks".
And this commit is merged into kernel v6.4-rc2-1-g9b4b7c1f9f54.
And the Fedora 39 stock kernel is kernel 6.5. So maybe some commits between 6.5 and 6.10 introduce this problem.
Quoted 5 lines
So I think this is enough to show that the Ansible parts of this change are working as expected. I can push this to kdevops now if there are no objections, and someone (maybe you, maybe me) can sort out the rxe specific issues later.
Thanks. After I can reproduce this problem in my local host, I am very glad to delve into this problem. Perhaps it will take me a long time since I do not have a good host to deploy kdeops.
To be honest, perhaps "git bisec" can find the commit that introduce this problem. If you can find the commit, we can fix this problem very quickly^_^
Thanks,
Zhu Yanjun
Quoted 24 lines
Quoted 12 lines
2. ping -c 3
server_ipon client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -aserver_ip
rping -c -aserver_ip-C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.Quoted 7 lines
Zhu Yanjun
Quoted 5 lines
--
Chuck Lever--
Chuck Lever
Quoted 64 lines · Zhu Yanjun
Quoted 59 lines · Chuck Lever III
Quoted 17 lines · Zhu Yanjun
Quoted 11 lines · Chuck Lever III
Quoted 10 lines · Zhu Yanjun
Quoted 3 lines · Chuck Lever III
It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous.
Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version.
Quoted 7 lines
Quoted 2 lines
As per our recent discussion, I have added rxe as a second software RDMA option in kdevops. Proof of concept:
Thanks a lot. I am very glad to know that rxe is treated as a second software RDMA option in kdeops.
And I also checked the commit related with this feature. It is very complicated and huge.I split this into four smaller patches, HTH.
Quoted 10 lines
I hope rxe can work well in kdeops.
So I can also use kdeops to verify rxe and rdma subsystems. Thanks a lot your efforts.Quoted 5 lines
https://github.com/chucklever/kdevops/tree/add-rxe-support
But basic rping testing is not working (with 6.10-rc1 kernels) in this set-up. It's missing something...
Quoted 8 lines
Just now I made tests with the latest rdma-core (rping is included in rdma-core) and 6.10-rc1 kernels. rping can work well.
Normally rping works as a basic tool to verify if rxe works well or not. If rping can not work well, normally I will do the followings:
1. rping -s -a 127.0.0.1
rping -c -a 127.0.0.1 -C 3 -d -v
This will verify whether rxe is configured correctly or not.I don't have rxe set up on loopback, so I substituted the host's configured Ethernet IP.
The tests works on the NFS server, but the rping client hangs on the NFS client (both running v6.10-rc1).
I rebooted in to the Fedora 39 stock kernel, and the rping tests pass.
However, when I try to run fstests with NFS/RDMA using rxe, the client kernel reports a soft CPU lock-up, and top shows this:
Quoted 8 lines
Quoted 1 line
115 root 20 0 0 0 0 R 99.3 0.0 1:03.50 kworker/u8:5+rxe_wq
rxe_wqis introduced in the commit9b4b7c1f9f54"RDMA/rxe: Add workqueue support for rxe tasks".
And this commit is merged into kernel v6.4-rc2-1-g9b4b7c1f9f54.And the Fedora 39 stock kernel is kernel 6.5. So maybe some commits between 6.5 and 6.10 introduce this problem.
I couldn't get 6.10-rc1 working at all. This failure occurred with the stock Fedora 39 kernel and fstests with NFS v4.2 on RDMA.
Quoted 8 lines
Quoted 4 lines
So I think this is enough to show that the Ansible parts of this change are working as expected. I can push this to kdevops now if there are no objections, and someone (maybe you, maybe me) can sort out the rxe specific issues later.
Thanks. After I can reproduce this problem in my local host, I am very glad to delve into this problem. Perhaps it will take me a long time since I do not have a good host to deploy kdevops.
kdevops works on laptops too. The limiting factor seems to be memory for libvirt guests. Only two guests are needed for this test.
Quoted 3 lines
To be honest, perhaps "git bisec" can find the commit that introduce this problem. If you can find the commit, we can fix this problem very quickly^_^
Since this is the first time I've ever used rxe, I don't have a "good" commit to start from.
Quoted 25 lines
Thanks,
Zhu YanjunQuoted 22 lines
Quoted 12 lines
2. ping -c 3
server_ipon client host.
This will verify whether the client host can connect to the server
host or not.
3. rping -s -aserver_ip
rping -c -aserver_ip-C 3 -d -v
1) shutdown firewall
2) tcpdump -ni xxxx to capture udp packets
Normally the above steps can find out the errors in rxe client/server. Hope the above can help to find out the errors.Quoted 7 lines
Zhu Yanjun
Quoted 5 lines
--
Chuck Lever--
Chuck Lever
--
Chuck Lever