Random NFS client lockups
Subsystems (3)
Hello,
I am experiencing random NFS client lockups after 1-2 days. Kernel reports
| nfs: server XXXXXXXXX not responding, timed out
processes are in D state and only a reboot helps.
The full log is available at https://pastebin.pl/view/7d0b345b
I can see one oddity there: shortly before the timeouts, log shows at 05:07:28:
| worker connecting xprt 0000000022aecad1 via tcp to XXXX:2001:1022:: (port 2049) | 0000000022aecad1 connect status 0 connected 0 sock state 8
All other connects go in EINPROGRESS first
| 0000000022aecad1 connect status 115 connected 0 sock state 2 | ... | state 8 conn 1 dead 0 zapped 1 sk_shutdown 1
After 'status 0', rpcdebug shows (around 05:07:43)
| --> nfs4_alloc_slot used_slots=03ff highest_used=9 max_slots=30 | <-- nfs4_alloc_slot used_slots=07ff highest_used=10 slotid=10 | ... | <-- nfs4_alloc_slot used_slots=fffffff highest_used=27 slotid=27 | --> nfs4_alloc_slot used_slots=fffffff highest_used=27 max_slots=30 | ... | --> nfs4_alloc_slot used_slots=3fffffff highest_used=29 max_slots=30 | <-- nfs4_alloc_slot used_slots=3fffffff highest_used=29 slotid=4294967295 | nfs41_sequence_process: Error 1 free the slot
and nfs server times out then.
The server reports nearly at this time
| Mar 16 05:02:40 kernel: rpc-srv/tcp: nfsd: got error -32 when sending 112 bytes - shutting down socket
Similar message (with other sizes and sometime error -104) appear frequently without a related client lockup.
How can I debug this further resp. solve it?
It happens (at least) with:
- a Fedora 35 client with kernel 5.16.7, kernel-5.17.0-0.rc7.20220310git3bf7edc84a9e.119.fc37.x86_64 and some other between them
- a Rocky Linux 8,5 server with kernel-4.18.0-348.12.2 and kernel-4.18.0-348.2.1
Problem started after a power outage were whole infrastructure rebooted. I ran the setup with kernel 5.16.7 on client and 4.18.0-348.2.1 on server without problems before the outage.
Issue affects a /home directory mounted with
| XXXX:/home /home nfs4 rw,seclabel,nosuid,nodev,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,soft,posix,proto=tcp6,timeo=600,retrans=2,sec=krb5p,clientaddr=XXXX:2001:...,local_lock=none,addr=XXXX:2001:1022:: 0 0
Happens also without the "soft" option.
There are applications like firefox or chromium running which held locks and access the filesystem frequently.
Logfile was created when rpcdebug enabled
| nfs proc xdr root callback client mount pnfs pnfs_ld state | rpc xprt call debug nfs auth bind sched trans svcsock svcdsp misc cache
Enrico
Quoted 3 lines · Enrico Scholz
I am experiencing random NFS client lockups after 1-2 days.
...
The full log is available at https://pastebin.pl/view/7d0b345b
Filtering only RPC messages[1] shows
-----
Mar 16 04:48:58.345357: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 04:48:58.345487: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2 Mar 16 04:48:58.345705: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 Mar 16 05:02:40.037783: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:02:40.037921: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:02:40.038076: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.038211: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.038293: RPC: xs_error_report client 0000000022aecad1, error=104... Mar 16 05:02:40.038449: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.039134: RPC: xs_connect scheduled xprt 0000000022aecad1 Mar 16 05:02:40.042991: RPC: xs_bind 0000:0000:0000:0000:0000:0000:0000:0000:883: ok (0) Mar 16 05:02:40.043072: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:02:40.043159: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2 Mar 16 05:02:40.043308: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 Mar 16 05:02:40.045883: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:02:40.046163: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:02:40.046410: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.046625: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.047428: RPC: xs_connect scheduled xprt 0000000022aecad1 Mar 16 05:02:40.050387: RPC: xs_bind 0000:0000:0000:0000:0000:0000:0000:0000:822: ok (0) Mar 16 05:02:40.050614: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:02:40.050662: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2 Mar 16 05:02:40.050788: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 Mar 16 05:02:40.051969: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:02:40.052067: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:02:40.052189: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.052243: RPC: xs_error_report client 0000000022aecad1, error=32... Mar 16 05:02:40.052367: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.052503: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.053201: RPC: xs_connect scheduled xprt 0000000022aecad1 Mar 16 05:02:40.055886: RPC: xs_bind 0000:0000:0000:0000:0000:0000:0000:0000:875: ok (0) __A__ 05:02:40.055947: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:02:40.055995: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2 Mar 16 05:02:40.056125: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 Mar 16 05:07:28.326605: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:07:28.326679: RPC: xs_connect scheduled xprt 0000000022aecad1 __B__ 05:07:28.326978: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:07:28.327050: RPC: 0000000022aecad1 connect status 0 connected 0 sock state 8 __C__ 05:07:28.327113: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:07:28.327229: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:07:28.327446: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3
-----
For me, it seems that 'xs_close' at position __C__ belongs to the connection at __A__ but is applied to the newly created connection __B__.
Because the close did not happened yet, __B__ operated on the old socket which is in state 8 (CLOSE_WAIT) and connect returned with 0 instead of -EINPROGRESS hence. There will not be generated a new connection in this case.
It seems to be some race where 'xs_connect()' is executed before a delayed 'xs_error_handle()'. Both functions are called in reaction of a 'CLOSE_WAIT'.
Enrico
Quoted 75 lines · Enrico Scholz
[You don't often get email from [email protected]. Learn why this is important at
http://aka.ms/LearnAboutSenderIdentification.]Quoted 3 lines · Enrico Scholz
I am experiencing random NFS client lockups after 1-2 days.
...
The full log is available at https://pastebin.pl/view/7d0b345bFiltering only RPC messages[1] shows
-----
Mar 16 04:48:58.345357: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 04:48:58.345487: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2 Mar 16 04:48:58.345705: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 Mar 16 05:02:40.037783: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:02:40.037921: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:02:40.038076: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.038211: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.038293: RPC: xs_error_report client 0000000022aecad1, error=104... Mar 16 05:02:40.038449: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.039134: RPC: xs_connect scheduled xprt 0000000022aecad1 Mar 16 05:02:40.042991: RPC: xs_bind 0000:0000:0000:0000:0000:0000:0000:0000:883: ok (0) Mar 16 05:02:40.043072: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:02:40.043159: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2 Mar 16 05:02:40.043308: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 Mar 16 05:02:40.045883: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:02:40.046163: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:02:40.046410: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.046625: RPC: state 9 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.047428: RPC: xs_connect scheduled xprt 0000000022aecad1 Mar 16 05:02:40.050387: RPC: xs_bind 0000:0000:0000:0000:0000:0000:0000:0000:822: ok (0) Mar 16 05:02:40.050614: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:02:40.050662: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2 Mar 16 05:02:40.050788: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 Mar 16 05:02:40.051969: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:02:40.052067: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:02:40.052189: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.052243: RPC: xs_error_report client 0000000022aecad1, error=32... Mar 16 05:02:40.052367: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.052503: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.053201: RPC: xs_connect scheduled xprt 0000000022aecad1 Mar 16 05:02:40.055886: RPC: xs_bind 0000:0000:0000:0000:0000:0000:0000:0000:875: ok (0) __A__ 05:02:40.055947: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:02:40.055995: RPC: 0000000022aecad1 connect status
Quoted 1 line
115 connected 0 sock state 2
Socket is in TCP state 2 == SYN_SENT
So the client requested to connect to the server
Quoted 2 lines
Mar 16 05:02:40.056125: RPC: state 1 conn 0 dead 0 zapped 1
sk_shutdown0
Socket is in TCP state 1 == TCP_ESTABLISHED
The connection to the server was accepted, and the connection is established.
Quoted 2 lines
Mar 16 05:07:28.326605: RPC: state 8 conn 1 dead 0 zapped 1
sk_shutdown1
Socket is now in TCP state 8 == TCP_CLOSE_WAIT...
That means the server sent a FIN to the client to request that the connection be closed.
(see
https://users.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf
for more information on the TCP state diagram)
Quoted 7 lines
Mar 16 05:07:28.326679: RPC: xs_connect scheduled xprt 0000000022aecad1 __B__ 05:07:28.326978: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:07:28.327050: RPC: 0000000022aecad1 connect status 0connected 0 sock state 8
__C__ 05:07:28.327113: RPC:xs_closexprt0000000022aecad1
Client closes the socket, which is still in TCP_CLOSE_WAIT
Quoted 2 lines
Mar 16 05:07:28.327229: RPC: state 9 conn 0 dead 0 zapped 1
sk_shutdown3
...and then transitions to TCP state 9 == TCP_LAST_ACK
Quoted 4 lines
Mar 16 05:07:28.327446: RPC: state 9 conn 0 dead 0 zapped 1
sk_shutdown3
-----
For more information on TCP states, see
https://users.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf
Basically, what the above means is that your server is initiating the close of the connection, not the client.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 11 lines · Trond Myklebust
Quoted 10 lines
Mar 16 05:02:40.051969: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1 Mar 16 05:02:40.052067: RPC: xs_close xprt 0000000022aecad1 Mar 16 05:02:40.052189: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.052243: RPC: xs_error_report client 0000000022aecad1, error=32... Mar 16 05:02:40.052367: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.052503: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 Mar 16 05:02:40.053201: RPC: xs_connect scheduled xprt 0000000022aecad1 Mar 16 05:02:40.055886: RPC: xs_bind 0000:0000:0000:0000:0000:0000:0000:0000:875: ok (0) __A__ 05:02:40.055947: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:02:40.055995: RPC: 0000000022aecad1 connect status 115 connected 0 sock state 2
Quoted 2 lines
Socket is in TCP state 2 ==
SYN_SENT
So the client requested to connect to the server
server closed the connection (state 8, CLOSE_WAIT), client cleaned up correctly and reconnected.
Quoted 6 lines
Quoted 1 line
Mar 16 05:07:28.326605: RPC: state 8 conn 1 dead 0 zapped 1 sk_shutdown 1Socket is now in TCP state 8 ==
TCP_CLOSE_WAIT...That means the server sent a FIN to the client to request that the connection be closed.
yes; the same situation like above
Quoted 6 lines
Quoted 4 lines
Mar 16 05:07:28.326679: RPC: xs_connect scheduled xprt 0000000022aecad1 __B__ 05:07:28.326978: RPC: worker connecting xprt 0000000022aecad1 via tcp to XXXXX:2001:1022:: (port 2049) Mar 16 05:07:28.327050: RPC: 0000000022aecad1 connect status 0 connected 0 sock state 8__C__ 05:07:28.327113: RPC:
xs_closexprt0000000022aecad1Client closes the socket, which is still in
TCP_CLOSE_WAIT
the 'xs_close' is very likely a reaction to the state change reported above and should happen before 'xs_connect'.
Quoted 2 lines
Basically, what the above means is that your server is initiating the close of the connection, not the client.
yes; but the client should reconnect (and does it in most cases). Sometimes there seems to be a race which prevents the reconnect and brings the client in a broken state.
Enrico
Quoted 6 lines · Enrico Scholz
[You don't often get email from [email protected]. Learn why this is important at
http://aka.ms/LearnAboutSenderIdentification.]Trond Myklebust <[email protected]> writes:
Quoted 8 lines
Quoted 3 lines
Basically, what the above means is that your server is initiating the
close of the connection, not the client.yes; but the client should reconnect (and does it in most cases). Sometimes there seems to be a race which prevents the reconnect and brings the client in a broken state.
I didn't realise that you were seeing a long term hang. Can you please try the attached patch and see if it helps?
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
From 85fd6aec7a1440b674548dd93bd3411a5486b49b Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Wed, 16 Mar 2022 19:10:43 -0400
Subject: [PATCH] SUNRPC: Don't call connect() more than once on a TCP socket
Avoid socket state races due to repeated calls to ->connect() using the same socket. If connect() returns 0 due to the connection having completed, but we are in fact in a closing state, then we may leave the XPRT_CONNECTING flag set on the transport.
--- include/linux/sunrpc/xprtsock.h | 1 + net/sunrpc/xprtsock.c | 22 ++++++++++++---------- 2 files changed, 13 insertions(+), 10 deletions(-)
include/linux/sunrpc/xprtsock.h +1 -0
@@ -89,5 +89,6 @@ struct sock_xprt {#define XPRT_SOCK_WAKE_WRITE (5)#define XPRT_SOCK_WAKE_PENDING (6)#define XPRT_SOCK_WAKE_DISCONNECT (7) +#define XPRT_SOCK_CONNECT_SENT (8)#endif /* _LINUX_SUNRPC_XPRTSOCK_H */
net/sunrpc/xprtsock.c +12 -10
@@ -2235,10 +2235,15 @@ static void xs_tcp_setup_socket(struct work_struct *work)if (atomic_read (&xprt->swapper)) current->flags |=PF_MEMALLOC ; -if (!sock) { - sock =xs_create_sock (xprt, transport, -xs_addr (xprt)->sa_family,SOCK_STREAM , -IPPROTO_TCP , true); + +if (xprt_connected (xprt)) + goto out; +if (test_and_clear_bit (XPRT_SOCK_CONNECT_SENT , + &transport->sock_state) || + !sock) { +xs_reset_transport (transport); + sock =xs_create_sock (xprt, transport,xs_addr (xprt)->sa_family, +SOCK_STREAM ,IPPROTO_TCP , true);if (IS_ERR (sock)) {xprt_wake_pending_tasks (xprt,PTR_ERR (sock)); goto out;
@@ -2262,6 +2267,7 @@ static void xs_tcp_setup_socket(struct work_struct *work) fallthrough;case -EINPROGRESS :/* SYN_SENT! */ +set_bit (XPRT_SOCK_CONNECT_SENT , &transport->sock_state);if (xprt->reestablish_timeout <XS_TCP_INIT_REEST_TO ) xprt->reestablish_timeout =XS_TCP_INIT_REEST_TO ; fallthrough;
@@ -2323,13 +2329,9 @@ static void xs_connect(struct rpc_xprt *xprt, struct rpc_task *task)WARN_ON_ONCE (!xprt_lock_connect (xprt, task, transport)); -if (transport->sock !=NULL && !xprt_connecting (xprt)) { +if (transport->sock !=NULL ) {dprintk ("RPC: xs_connect delayed xprt %p for %lu " -"seconds\n" , - xprt, xprt->reestablish_timeout /HZ ); - -/* Start by resetting any existing state */ -xs_reset_transport (transport); +"seconds\n" , xprt, xprt->reestablish_timeout /HZ ); delay =xprt_reconnect_delay (xprt);xprt_reconnect_backoff (xprt,XS_TCP_INIT_REEST_TO );
Signature · 1 line
2.35.1
Quoted 7 lines · Trond Myklebust
Quoted 4 lines
yes; but the client should reconnect (and does it in most cases). Sometimes there seems to be a race which prevents the reconnect and brings the client in a broken state.
I didn't realise that you were seeing a long term hang. Can you please try the attached patch and see if it helps?
Thanks; I applied the patch and it runs without problems so far. It fixes the connect-close race very likely. I hope, this was the reason for the lockups but it is too early to confirm as they usually take 1-2 days before they occur.
Enrico
Quoted 6 lines · Enrico Scholz
Quoted 1 line
Can you please try the attached patch and see if it helps?
Thanks; I applied the patch and it runs without problems so far. It fixes the connect-close race very likely. I hope, this was the reason for the lockups but it is too early to confirm as they usually take 1-2 days before they occur.
Client runs now for >10 days without seeing the lockups or other regressions.
Thanks again
Enrico