NFSv4.1 fix a kswap nfs4_state_manger race
Subsystems (2)
cover NFSv4.1 fix a kswap nfs4_state_manger race
From: Andy Adamson <[email protected]>
This is a three way race between the state manager, kswapd and sys_open. We are hitting this regularily in our long term testing. This patch should fix the race - but before we test with this patch, I'd like comments from the list.
The state manager is waiting in __rpc_wait_for_completion_task for a recovery OPEN to complete:
kernel: Call Trace:
kernel: [<ffffffff81054a39>] ? __wake_up_common+0x59/0x90
kernel: [<ffffffffa0358110>] ? rpc_wait_bit_killable+0x0/0xa0 [sunrpc]
kernel: [<ffffffffa0358152>] rpc_wait_bit_killable+0x42/0xa0 [sunrpc]
kernel: [<ffffffff8152914f>] __wait_on_bit+0x5f/0x90
kernel: [<ffffffffa0358110>] ? rpc_wait_bit_killable+0x0/0xa0 [sunrpc]
kernel: [<ffffffff815291f8>] out_of_line_wait_on_bit+0x78/0x90
kernel: [<ffffffff8109b520>] ? wake_bit_function+0x0/0x50
kernel: [<ffffffffa035810d>] __rpc_wait_for_completion_task+0x2d/0x30 [sunrpc]
kernel: [<ffffffffa040d44c>] nfs4_run_open_task+0x11c/0x160 [nfs]
kernel: [<ffffffffa04114d7>] nfs4_open_recover_helper+0x87/0x120 [nfs]
kernel: [<ffffffffa0411636>] nfs4_open_recover+0xc6/0x150 [nfs]
kernel: [<ffffffffa040cc6f>] ? nfs4_open_recoverdata_alloc+0x2f/0x60 [nfs]
kernel: [<ffffffffa041192d>] nfs4_open_reclaim+0xad/0x140 [nfs]
kernel: [<ffffffffa0421bfb>] nfs4_do_reclaim+0x15b/0x5e0 [nfs]
kernel: [<ffffffffa042afc3>] ? pnfs_destroy_layout+0x63/0x80 [nfs]
kernel: [<ffffffffa04224cb>] nfs4_run_state_manager+0x44b/0x620 [nfs]
kernel: [<ffffffffa0422080>] ? nfs4_run_state_manager+0x0/0x620 [nfs]
kernel: [<ffffffff8109b0f6>] kthread+0x96/0xa0
kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
kernel: [<ffffffff8109b060>] ? kthread+0x0/0xa0
kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Kswapd is shrinking the inode cache, and waiting for a layoutreturn:
kernel: Call Trace:
kernel: [<ffffffffa0358110>] ? rpc_wait_bit_killable+0x0/0xa0 [sunrpc]
kernel: [<ffffffffa0358152>] rpc_wait_bit_killable+0x42/0xa0 [sunrpc]
kernel: [<ffffffff8152914f>] __wait_on_bit+0x5f/0x90
kernel: [<ffffffff8152aacb>] ? _spin_unlock_bh+0x1b/0x20
kernel: [<ffffffffa0358110>] ? rpc_wait_bit_killable+0x0/0xa0 [sunrpc]
kernel: [<ffffffff815291f8>] out_of_line_wait_on_bit+0x78/0x90
kernel: [<ffffffff8109b520>] ? wake_bit_function+0x0/0x50
kernel: [<ffffffffa0357b90>] ? rpc_exit_task+0x0/0x60 [sunrpc]
kernel: [<ffffffffa0358695>] __rpc_execute+0xf5/0x350 [sunrpc]
kernel: [<ffffffff8109b327>] ? bit_waitqueue+0x17/0xd0
kernel: [<ffffffffa0358951>] rpc_execute+0x61/0xa0 [sunrpc]
kernel: [<ffffffffa034f3a5>] rpc_run_task+0x75/0x90 [sunrpc]
kernel: [<ffffffffa040b86c>] nfs4_proc_layoutreturn+0x9c/0x110 [nfs]
kernel: [<ffffffffa042b22e>] _pnfs_return_layout+0x11e/0x1e0 [nfs]
kernel: [<ffffffffa03f3ef4>] nfs4_clear_inode+0x44/0x70 [nfs]
kernel: [<ffffffff811a5c7c>] clear_inode+0xac/0x140
kernel: [<ffffffff811a5d50>] dispose_list+0x40/0x120
kernel: [<ffffffff811a60a4>] shrink_icache_memory+0x274/0x2e0
kernel: [<ffffffff81138cca>] shrink_slab+0x12a/0x1a0
kernel: [<ffffffff8113c10a>] balance_pgdat+0x59a/0x820
kernel: [<ffffffff8113c4c4>] kswapd+0x134/0x3b0
kernel: [<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40
kernel: [<ffffffff8113c390>] ? kswapd+0x0/0x3b0
kernel: [<ffffffff8109b0f6>] kthread+0x96/0xa0
kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
kernel: [<ffffffff8109b060>] ? kthread+0x0/0xa0
kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
The layoutreturn is on the cl_rpcwaitq waiting for the state manager to complete:
kernel: 14628 0a80 0 ffff88013c8a3a00 (null) 0 ffffffffa0430580 nfsv4 LAYOUTRETURN a:rpc_prepare_task q:NFS client
Meanwhile, a sys_open is waiting in __wait_on_freeing_inode for kswapd to complete the inode deletion. Note that this OPEN RPC has almost completed - it is stuck processing nfs4_opendata_to_nfs4_state, but it has yet to call nfs_release_seqid:
kernel: Call Trace:
kernel: [<ffffffff81224590>] ? user_match+0x0/0x20
kernel: [<ffffffff8109b7ce>] ? prepare_to_wait+0x4e/0x80
kernel: [<ffffffff811a55b8>] __wait_on_freeing_inode+0x98/0xc0
kernel: [<ffffffff8109b520>] ? wake_bit_function+0x0/0x50
kernel: [<ffffffffa03f3d80>] ? nfs_find_actor+0x0/0x90 [nfs]
kernel: [<ffffffff811a5764>] find_inode+0x64/0x90
kernel: [<ffffffffa03f3d80>] ? nfs_find_actor+0x0/0x90 [nfs]
kernel: [<ffffffff811a68ad>] ifind+0x4d/0xd0
kernel: [<ffffffffa03f3d80>] ? nfs_find_actor+0x0/0x90 [nfs]
kernel: [<ffffffff811a6d29>] iget5_locked+0x59/0x1b0
kernel: [<ffffffffa03f3280>] ? nfs_init_locked+0x0/0x40 [nfs]
kernel: [<ffffffffa03f54f6>] nfs_fhget+0xc6/0x6c0 [nfs]
kernel: [<ffffffffa040def1>] nfs4_opendata_to_nfs4_state+0x1c1/0x330 [nfs]
kernel: [<ffffffffa040ec3c>] _nfs4_do_open+0x21c/0x4f0 [nfs]
kernel: [<ffffffffa035ac05>] ? rpcauth_lookup_credcache+0xc5/0x260 [sunrpc]
kernel: [<ffffffffa040ef95>] nfs4_do_open+0x85/0x170 [nfs]
kernel: [<ffffffffa040f0a8>] nfs4_atomic_open+0x28/0x50 [nfs]
kernel: [<ffffffffa03ee9fd>] nfs_atomic_lookup+0x15d/0x310 [nfs]
kernel: [<ffffffff81198ae5>] do_lookup+0x1a5/0x230
kernel: [<ffffffff811993fc>] __link_path_walk+0x78c/0xfe0
kernel: [<ffffffff81121f20>] ? __generic_file_aio_write+0x260/0x490
kernel: [<ffffffffa0357d30>] ? rpc_do_put_task+0x30/0x40 [sunrpc]
kernel: [<ffffffff81199f1a>] path_walk+0x6a/0xe0
kernel: [<ffffffff8119a12b>] filename_lookup+0x6b/0xc0
kernel: [<ffffffff81226466>] ? security_file_alloc+0x16/0x20
kernel: [<ffffffff8119b5f4>] do_filp_open+0x104/0xd20
kernel: [<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40
kernel: [<ffffffff8118e9a4>] ? cp_new_stat+0xe4/0x100
kernel: [<ffffffff811a82b2>] ? alloc_fd+0x92/0x160
kernel: [<ffffffff81185f19>] do_sys_open+0x69/0x140
kernel: [<ffffffff81189a61>] ? sys_write+0x51/0x90
kernel: [<ffffffff81186030>] sys_open+0x20/0x30
kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
The OPEN from the state manager (this is an educated guess) is waiting for the above open to release the seqid - so it is waiting on the Seqid_waitqueue
kernel: 11683 0081 0 ffff880037827c00 (null) 0 ffffffffa0430180 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
Turning off error handling for layoutreturn calls that come from nfs4_evict_inode will prevent the race. It would be more accurate to only turn off this error handling when kswapd and the state manager are running, but that seemed too complicated to worry about as layoutreturn already passes in a NULL state to nfs4_async_handle_errors and so does not handle a good number errors.
Andy Adamson (1):
NFSv4.1 Don't handle layoutreturn errors when state manager is running
fs/nfs/nfs4proc.c | 6 ++++++ fs/nfs/pnfs.c | 5 ++++- include/linux/nfs_xdr.h | 1 + 3 files changed, 11 insertions(+), 1 deletion(-)
Signature · 1 line
1.8.3.1
PATCH 1/1 NFSv4.1 fix a kswap nfs4_state_manger race
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls evict_inode on an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client) cl_rpcwaitq.
At the same time an open is waiting for the inode deletion to complete in __wait_on_freeing_inode.
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the Seqid_waitqueue, then the state is deadlocked with kswapd.
Do not handle LAYOUTRETURN errors when called from nfs4_evict_inode.
--- fs/nfs/nfs4proc.c | 6 ++++++ fs/nfs/pnfs.c | 5 ++++- include/linux/nfs_xdr.h | 1 + 3 files changed, 11 insertions(+), 1 deletion(-)
fs/nfs/nfs4proc.c +6 -0
@@ -7596,6 +7596,12 @@ static void nfs4_layoutreturn_done(struct rpc_task *task, void *calldata)return ; server =NFS_SERVER (lrp->args.inode); + +/* Error handling can dead lock the state manager running open +* recovery when kswapd is also pruning inodes. */ +if (lrp->ino_freeing) +return ; +if (nfs4_async_handle_error (task, server,NULL ) == -EAGAIN ) {rpc_restart_call_prepare (task);return ;
fs/nfs/pnfs.c +4 -1
@@ -819,7 +819,7 @@ _pnfs_return_layout(struct inode *ino)LIST_HEAD (tmp_list);struct nfs4_layoutreturn *lrp;nfs4_stateid stateid; -int status =0 , empty; +int status =0 , empty, freeing =0 ;dprintk ("NFS: %s for inode %lu\n" , __func__, ino->i_ino);
@@ -844,6 +844,8 @@ _pnfs_return_layout(struct inode *ino) goto out; } lo->plh_block_lgets++; +if (ino->i_state &I_FREEING ) + freeing =1 ;spin_unlock (&ino->i_lock);pnfs_free_lseg_list (&tmp_list);
@@ -863,6 +865,7 @@ _pnfs_return_layout(struct inode *ino) lrp->args.layout = lo; lrp->clp =NFS_SERVER (ino)->nfs_client; lrp->cred = lo->plh_lc_cred; + lrp->ino_freeing = freeing; status =nfs4_proc_layoutreturn (lrp); out:
include/linux/nfs_xdr.h +1 -0
@@ -316,6 +316,7 @@ struct nfs4_layoutreturn {struct nfs4_layoutreturn_res res;struct rpc_cred *cred;struct nfs_client *clp; +int ino_freeing;int rpc_status; };
Signature · 1 line
1.8.3.1
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.
Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.
Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?
Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
www.netapp.com
Quoted 19 lines · Trond
Quoted 16 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.
Quoted 5 lines
Quoted 1 line
Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?
Yeah, I was thinking about this as well - perhaps recovering from session-level errors or grace/delay errors would be useful for the block client.
-->Andy
Quoted 12 lines
Cheers
Trond--
Trond Myklebust
Linux NFS client maintainerNetApp
[email protected]
www.netapp.com
Quoted 23 lines · Andy
Quoted 19 lines · Trond
Quoted 16 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Yeah, I was thinking about this as well - perhaps recovering from session-level errors or grace/delay errors would be useful for the block client.
NFS4ERR_DELAY, probably, yes.
NFS4ERR_GRACE, no… That’s a reboot situation
As for session level errors, I’d say that complicates things too much, since several of those can basically end up masking a NFS4ERR_STALE_CLIENTID error.
Either way, all the layout types (including blocks) should be able to continue on even if we miss a layout return or two. The server has to be coded to expect a forgetful client.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
www.netapp.com
Quoted 34 lines · Trond
Quoted 23 lines · Andy
Quoted 19 lines · Trond
Quoted 16 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Yeah, I was thinking about this as well - perhaps recovering from session-level errors or grace/delay errors would be useful for the block client.
Quoted 8 lines
NFS4ERR_DELAY, probably, yes.
NFS4ERR_GRACE, no… That’s a reboot situationAs for session level errors, I’d say that complicates things too much, since several of those can basically end up masking a
NFS4ERR_STALE_CLIENTIDerror.Either way, all the layout types (including blocks) should be able to continue on even if we miss a layout return or two. The server has to be coded to expect a forgetful client.
OK - I'll resend the patch.
-->Andy
Quoted 9 lines
--
Trond Myklebust
Linux NFS client maintainerNetApp
[email protected]
www.netapp.com
Quoted 19 lines · Trond
Quoted 16 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.
Quoted 5 lines
Quoted 1 line
Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode...
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
www.netapp.com
Quoted 22 lines · Trond
Quoted 19 lines · Trond
Quoted 16 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with sys_open and the state manager running.
With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
-->Andy
Quoted 9 lines
--
Trond Myklebust
Linux NFS client maintainerNetApp
[email protected]
www.netapp.com
Quoted 30 lines · Andy
Quoted 26 lines · Trond
Quoted 23 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in nfs4_evict_inode is asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.
Cheers
Trond
Quoted 35 lines · Trond
Quoted 32 lines · Andy
Quoted 28 lines · Trond
Quoted 24 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in
nfs4_evict_inodeis asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.
Except we just changed that to fix a different state manager hang:
commit 4a82fd7c4e78a1b7a224f9ae8bb7e1fd95f670e0
Author: Andy Adamson <[email protected]> Date: Fri Nov 15 16:36:16 2013 -0500
NFSv4 wait on recovery for async session errors
Quoted 3 lines
Cheers
Trond
Quoted 47 lines · Andy
Quoted 35 lines · Trond
Quoted 32 lines · Andy
Quoted 28 lines · Trond
Quoted 24 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in
nfs4_evict_inodeis asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.
Quoted 7 lines
Except we just changed that to fix a different state manager hang:
commit
4a82fd7c4e78a1b7a224f9ae8bb7e1fd95f670e0Author: Andy Adamson <[email protected]> Date: Fri Nov 15 16:36:16 2013 -0500NFSv4 wait on recovery for async session errors
Right, but that won’t prevent nfs4_evict_inode from completing, and hence the OPEN that is waiting in nfs_fhget() can also complete, and so there is no deadlock with the state manager thread.
Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
www.netapp.com
Quoted 52 lines · Trond
Quoted 49 lines · Andy
Quoted 35 lines · Trond
Quoted 32 lines · Andy
Quoted 28 lines · Trond
Quoted 24 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in
nfs4_evict_inodeis asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.Except we just changed that to fix a different state manager hang:
Quoted 7 lines
Quoted 5 lines
commit
4a82fd7c4e78a1b7a224f9ae8bb7e1fd95f670e0Author: Andy Adamson <[email protected]> Date: Fri Nov 15 16:36:16 2013 -0500NFSv4 wait on recovery for async session errors
Right, but that won’t prevent
nfs4_evict_inodefrom completing,
Ah - I was thinking of the synchronous handlers call to nfs4_wait_clnt_recover - so yes, no problem
-->Andy
Quoted 13 lines
and hence the OPEN that is waiting in
nfs_fhget()can also complete, and so there is no deadlock with the state manager thread.Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainerNetApp
[email protected]
www.netapp.com
Quoted 60 lines · Andy
Quoted 56 lines · Trond
Quoted 53 lines · Andy
Quoted 35 lines · Trond
Quoted 32 lines · Andy
Quoted 28 lines · Trond
Quoted 24 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in
nfs4_evict_inodeis asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.Except we just changed that to fix a different state manager hang:
commit
4a82fd7c4e78a1b7a224f9ae8bb7e1fd95f670e0Author: Andy Adamson <[email protected]> Date: Fri Nov 15 16:36:16 2013 -0500
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
NFSv4 wait on recovery for async session errors
Right, but that won’t prevent
nfs4_evict_inodefrom completing,Ah - I was thinking of the synchronous handlers call to
nfs4_wait_clnt_recover- so yes, no problem
In fact, this issue is NOT an upstream issue! RHEL6.5-pre has nfs4_proc_layoutreturn as as SYNC rpc call, and _that_ is the bug that is fixed upstream.
Really sorry for the confusion. I'll back port a solution for RHEL6.5
-->Andy
Quoted 17 lines
-->Andy
Quoted 13 lines
and hence the OPEN that is waiting in
nfs_fhget()can also complete, and so there is no deadlock with the state manager thread.Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainerNetApp
[email protected]
www.netapp.com
Quoted 66 lines · Andy
Quoted 62 lines · Andy
Quoted 58 lines · Trond
Quoted 54 lines · Andy
Quoted 35 lines · Trond
Quoted 32 lines · Andy
Quoted 28 lines · Trond
Quoted 24 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in
nfs4_evict_inodeis asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.Except we just changed that to fix a different state manager hang:
commit
4a82fd7c4e78a1b7a224f9ae8bb7e1fd95f670e0Author: Andy Adamson <[email protected]> Date: Fri Nov 15 16:36:16 2013 -0500NFSv4 wait on recovery for async session errors
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Right, but that won’t prevent
nfs4_evict_inodefrom completing,Ah - I was thinking of the synchronous handlers call to
nfs4_wait_clnt_recover- so yes, no problemIn fact, this issue is NOT an upstream issue! RHEL6.5-pre has
nfs4_proc_layoutreturnas as SYNC rpc call, and _that_ is the bug that is fixed upstream.Really sorry for the confusion. I'll back port a solution for RHEL6.5
please ignore this message.
Quoted 22 lines
-->Andy
Quoted 17 lines
-->Andy
Quoted 13 lines
and hence the OPEN that is waiting in
nfs_fhget()can also complete, and so there is no deadlock with the state manager thread.Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainerNetApp
[email protected]
www.netapp.com
Quoted 66 lines · Andy
Quoted 62 lines · Andy
Quoted 58 lines · Trond
Quoted 54 lines · Andy
Quoted 35 lines · Trond
Quoted 32 lines · Andy
Quoted 28 lines · Trond
Quoted 24 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in
nfs4_evict_inodeis asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.Except we just changed that to fix a different state manager hang:
commit
4a82fd7c4e78a1b7a224f9ae8bb7e1fd95f670e0Author: Andy Adamson <[email protected]> Date: Fri Nov 15 16:36:16 2013 -0500NFSv4 wait on recovery for async session errors
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Right, but that won’t prevent
nfs4_evict_inodefrom completing,Ah - I was thinking of the synchronous handlers call to
nfs4_wait_clnt_recover- so yes, no problemIn fact, this issue is NOT an upstream issue! RHEL6.5-pre has
nfs4_proc_layoutreturnas as SYNC rpc call, and _that_ is the bug that is fixed upstream.Really sorry for the confusion. I'll back port a solution for RHEL6.5
Are you sure? As far as I can tell, the upstream nfs4_proc_layoutreturn is also synchronous. I therefore suspect that we still have the same problem.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
www.netapp.com
Quoted 59 lines · Andy
Quoted 56 lines · Trond
Quoted 53 lines · Andy
Quoted 35 lines · Trond
Quoted 32 lines · Andy
Quoted 28 lines · Trond
Quoted 24 lines · Trond
Quoted 11 lines · On Nov 25, 2013, at 12:57, <[email protected]> <[email protected]> wrote:
From: Andy Adamson <[email protected]>
The state manager is recovering expired state and recovery OPENs are being processed. If kswapd is pruning inodes at the same time, a deadlock can occur when kswapd calls
evict_inodeon an NFSv4.1 inode with a layout, and the resultant layoutreturn gets an error that the state mangager is to handle, causing the layoutreturn to wait on the (NFS client)cl_rpcwaitq.At the same time an open is waiting for the inode deletion to complete in
__wait_on_freeing_inode.Quoted 6 lines
If the open is either the open called by the state manager, or an open from the same open owner that is holding the NFSv4.0 sequence id which causes the OPEN from the state manager to wait for the sequence id on the
Seqid_waitqueue, then the state is deadlocked with kswapd.Do not handle LAYOUTRETURN errors when called from
nfs4_evict_inode.Why are we waiting for recovery in LAYOUTRETURN at all? Layouts are automatically lost when the server reboots or when the lease is otherwise lost.
IOW: Is there any reason why we need to special-case
nfs4_evict_inode? Shouldn’t we just bail out on error in _all_ cases?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
BTW: Is it possible that we might have a similar problem with delegreturn? That too can be called from nfs4_evict_inode…
Yes, good point. kswapd could be waiting for a delegation to return which has an error along with the same scenario with
sys_openand the state manager running.With delegreturn, we most definately want to limit 'no error handling' to the evict inode case.
Ah… I forgot that the delegreturn in
nfs4_evict_inodeis asynchronous and doesn’t wait for completion, so it shouldn’t be a problem here.Except we just changed that to fix a different state manager hang:
commit
4a82fd7c4e78a1b7a224f9ae8bb7e1fd95f670e0Author: Andy Adamson <[email protected]> Date: Fri Nov 15 16:36:16 2013 -0500
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
NFSv4 wait on recovery for async session errors
Right, but that won’t prevent
nfs4_evict_inodefrom completing,Ah - I was thinking of the synchronous handlers call to
nfs4_wait_clnt_recover- so yes, no problem-->Andy
Quoted 1 line
and hence the OPEN that is waiting in
nfs_fhget()can also complete, and so there is no deadlock with the state manager thread.
How about something like the attached...
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
fs/nfs/nfs4proc.c +8 -1
@@ -7599,7 +7599,14 @@ static void nfs4_layoutreturn_done(struct rpc_task *task, void *calldata)return ; server =NFS_SERVER (lrp->args.inode); -if (nfs4_async_handle_error (task, server,NULL ) == -EAGAIN ) { +switch (task->tk_status) { +default : + task->tk_status =0 ; +case 0 : +break ; +case -NFS4ERR_DELAY : +if (nfs4_async_handle_error (task, server,NULL ) != -EAGAIN ) +break ;rpc_restart_call_prepare (task);return ; }