Fixing infinite state recovery loop due to failed delegation return
Subsystems (2)
PATCH 1/1 Fixing infinite state recovery loop due to failed delegation return
If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.
Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.
--- fs/nfs/delegation.c | 5 +++-- fs/nfs/nfs4proc.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +3 -2
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation err =nfs4_wait_clnt_recover (clp); }while (err ==0 ); -if (err) { +if (err && err != -EIO ) {nfs_abort_delegation_return (delegation, clp); goto out; }
@@ -458,7 +458,8 @@ restart:iput (inode);if (!err) goto restart; -set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state); +if (err != -EIO ) +set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state);return err; } }
fs/nfs/nfs4proc.c +1 -1
@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, structnfs_inode_find_state_and_recover (state->inode, stateid);nfs4_schedule_stateid_recovery (server, state); -return 0 ; +return -EIO ;case -NFS4ERR_DELAY :case -NFS4ERR_GRACE :set_bit (NFS_DELEGATED_STATE , &state->flags);
Signature · 1 line
1.7.1
Trond, can you please respond to the patch?
As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.
Please either accept this patch or let me know what needs to be fixed.
Thank you.
Quoted 60 lines · Olga Kornievskaia
If we get a bad-stateid-type of error when we send OPEN with
delegate_curto return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegationstate_idbecause we don't have anopen_stateidto be used for the LOCK operation. Thus, we should return an error from thenfs4_open_delegation_recall()in that case.Furthermore, if an error occurs the delegation code will call
nfs_abort_delegation_return()which sets again theNFS4CLNT_DELEGRETURNflags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/delegation.c | 5 +++-- fs/nfs/nfs4proc.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)fs/nfs/delegation.c +3
-2@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation err =nfs4_wait_clnt_recover (clp); }while (err ==0 ); -if (err) { +if (err && err != -EIO ) {nfs_abort_delegation_return (delegation, clp); goto out; }@@ -458,7 +458,8 @@ restart:iput (inode);if (!err) goto restart; -set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state); +if (err != -EIO ) +set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state);return err; } }fs/nfs/nfs4proc.c +1
-2@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, structnfs_inode_find_state_and_recover (state->inode, stateid);nfs4_schedule_stateid_recovery (server, state); -return 0 ; +return -EIO ;case -NFS4ERR_DELAY :case -NFS4ERR_GRACE :set_bit (NFS_DELEGATED_STATE , &state->flags); --1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Olga Kornievskaia
Trond, can you please respond to the patch?
As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.
Please either accept this patch or let me know what needs to be fixed.
Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).
Quoted 28 lines
Thank you.
Quoted 25 lines · Olga Kornievskaia
If we get a bad-stateid-type of error when we send OPEN with
delegate_curto return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegationstate_idbecause we don't have anopen_stateidto be used for the LOCK operation. Thus, we should return an error from thenfs4_open_delegation_recall()in that case.Furthermore, if an error occurs the delegation code will call
nfs_abort_delegation_return()which sets again theNFS4CLNT_DELEGRETURNflags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/delegation.c | 5 +++-- fs/nfs/nfs4proc.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)fs/nfs/delegation.c +0
-0@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation err =nfs4_wait_clnt_recover (clp); }while (err ==0 );
Quoted 3 lines
fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398
Quoted 3 lines
fs/nfs/delegation.c +1
-1-if (err) { +if (err && err != -EIO ) {nfs_abort_delegation_return (delegation, clp);
This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a FREE_STATEID, as required for the case of NFSv4.1.
Quoted 9 lines
fs/nfs/delegation.c restart: · line 462
Quoted 9 lines
fs/nfs/delegation.c +2
-1goto out; }@@ -458,7 +458,8 @@ restart:iput (inode);if (!err) goto restart; -set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state); +if (err != -EIO ) +set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state);
Please explain why this is needed. If we've cleared the bad delegation, then why should we not attempt to return any others that may be pending?
Quoted 23 lines
Quoted 23 lines
fs/nfs/delegation.c +0
-0return err; } }fs/nfs/nfs4proc.c +1
-2@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, structnfs_inode_find_state_and_recover (state->inode, stateid);nfs4_schedule_stateid_recovery (server, state); -return 0 ; +return -EIO ;case -NFS4ERR_DELAY :case -NFS4ERR_GRACE :set_bit (NFS_DELEGATED_STATE , &state->flags); --1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
Thank you for looking into this!
Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia
Trond, can you please respond to the patch?
As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.
Please either accept this patch or let me know what needs to be fixed.
Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).
Quoted 28 lines
Thank you.
Quoted 25 lines · Olga Kornievskaia
If we get a bad-stateid-type of error when we send OPEN with
delegate_curto return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegationstate_idbecause we don't have anopen_stateidto be used for the LOCK operation. Thus, we should return an error from thenfs4_open_delegation_recall()in that case.Furthermore, if an error occurs the delegation code will call
nfs_abort_delegation_return()which sets again theNFS4CLNT_DELEGRETURNflags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/delegation.c | 5 +++-- fs/nfs/nfs4proc.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)fs/nfs/delegation.c +0
-0@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation err =nfs4_wait_clnt_recover (clp); }while (err ==0 );
Quoted 7 lines
Quoted 3 lines
fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398
Quoted 3 lines
fs/nfs/delegation.c +1
-1-if (err) { +if (err && err != -EIO ) {nfs_abort_delegation_return (delegation, clp);This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a
FREE_STATEID, as required for the case of NFSv4.1.
I'd need to brush up on the FREE_STATEID op to confidently answer that. But can't this be taken care of in state_recovery when locks are marked lost?
Quoted 14 lines
Quoted 10 lines
fs/nfs/delegation.c restart: · line 462
Quoted 10 lines
fs/nfs/delegation.c +2
-1goto out; }@@ -458,7 +458,8 @@ restart:iput (inode);if (!err) goto restart; -set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state); +if (err != -EIO ) +set_bit (NFS4CLNT_DELEGRETURN , &clp->cl_state);Please explain why this is needed. If we've cleared the bad delegation, then why should we not attempt to return any others that may be pending?
You are right, I think this is unnecessary because after we return the delegation the bit can be set again for any other leftover delegations. I have tested my patch without that chuck and it still fixes the issues.
Quoted 33 lines
Quoted 20 lines
Quoted 20 lines
fs/nfs/delegation.c +0
-0return err; } }fs/nfs/nfs4proc.c +1
-2@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, structnfs_inode_find_state_and_recover (state->inode, stateid);nfs4_schedule_stateid_recovery (server, state); -return 0 ; +return -EIO ;case -NFS4ERR_DELAY :case -NFS4ERR_GRACE :set_bit (NFS_DELEGATED_STATE , &state->flags); --1.7.1
Quoted 4 lines
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
Trond MyklebustLinux NFS client maintainer, PrimaryData
Quoted 53 lines · Olga Kornievskaia
Thank you for looking into this!
Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia
Trond, can you please respond to the patch?
As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.
Please either accept this patch or let me know what needs to be fixed.
Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).
Quoted 28 lines
Thank you.
Quoted 25 lines · Olga Kornievskaia
If we get a bad-stateid-type of error when we send OPEN with
delegate_curto return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegationstate_idbecause we don't have anopen_stateidto be used for the LOCK operation. Thus, we should return an error from thenfs4_open_delegation_recall()in that case.Furthermore, if an error occurs the delegation code will call
nfs_abort_delegation_return()which sets again theNFS4CLNT_DELEGRETURNflags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/delegation.c | 5 +++-- fs/nfs/nfs4proc.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)fs/nfs/delegation.c +0
-0@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation err =nfs4_wait_clnt_recover (clp); }while (err ==0 );Quoted 7 lines
Quoted 3 lines
fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398
Quoted 3 lines
fs/nfs/delegation.c +1
-1-if (err) { +if (err && err != -EIO ) {nfs_abort_delegation_return (delegation, clp);This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a
FREE_STATEID, as required for the case of NFSv4.1.
Quoted 3 lines
I'd need to brush up on the
FREE_STATEIDop to confidently answer that. But can't this be taken care of instate_recoverywhen locks are marked lost?
No, because in the case of EIO we now fall through to nfs_detach_delegation() and nfs_do_return_delegation(). What we really want to do is call TEST_STATEID + FREE_STATEID (see RFC5661 sections 8.2.4, 8.5 and 18.38.3), which will now be done as part of state recovery.
Signature · 5 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
Quoted 58 lines · Trond Myklebust
Quoted 53 lines · Olga Kornievskaia
Thank you for looking into this!
Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia
Trond, can you please respond to the patch?
As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.
Please either accept this patch or let me know what needs to be fixed.
Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).
Quoted 28 lines
Thank you.
Quoted 25 lines · Olga Kornievskaia
If we get a bad-stateid-type of error when we send OPEN with
delegate_curto return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegationstate_idbecause we don't have anopen_stateidto be used for the LOCK operation. Thus, we should return an error from thenfs4_open_delegation_recall()in that case.Furthermore, if an error occurs the delegation code will call
nfs_abort_delegation_return()which sets again theNFS4CLNT_DELEGRETURNflags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/delegation.c | 5 +++-- fs/nfs/nfs4proc.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)fs/nfs/delegation.c +0
-0@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation err =nfs4_wait_clnt_recover (clp); }while (err ==0 );Quoted 7 lines
Quoted 3 lines
fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398
Quoted 3 lines
fs/nfs/delegation.c +1
-1-if (err) { +if (err && err != -EIO ) {nfs_abort_delegation_return (delegation, clp);This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a
FREE_STATEID, as required for the case of NFSv4.1.Quoted 3 lines
I'd need to brush up on the
FREE_STATEIDop to confidently answer that. But can't this be taken care of instate_recoverywhen locks are marked lost?
Quoted 5 lines
No, because in the case of EIO we now fall through to
nfs_detach_delegation()andnfs_do_return_delegation(). What we really want to do is callTEST_STATEID+FREE_STATEID(see RFC5661 sections 8.2.4, 8.5 and 18.38.3), which will now be done as part of state recovery.
I don't see it. :-/ Is it because there is a race between returning a delegation and state recovery that was initiated? Because as far as I can see, the stateid recovery initiated by the handle_delegation_recall_error() will call nfs4_test_stateid() via nfs41_open_expire().
Quoted 7 lines
--
Trond MyklebustLinux NFS client maintainer, PrimaryData
Quoted 66 lines · Olga Kornievskaia
Quoted 58 lines · Trond Myklebust
Quoted 53 lines · Olga Kornievskaia
Thank you for looking into this!
Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia
Trond, can you please respond to the patch?
As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.
Please either accept this patch or let me know what needs to be fixed.
Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).
Quoted 28 lines
Thank you.
Quoted 25 lines · Olga Kornievskaia
If we get a bad-stateid-type of error when we send OPEN with
delegate_curto return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegationstate_idbecause we don't have anopen_stateidto be used for the LOCK operation. Thus, we should return an error from thenfs4_open_delegation_recall()in that case.Furthermore, if an error occurs the delegation code will call
nfs_abort_delegation_return()which sets again theNFS4CLNT_DELEGRETURNflags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfs/delegation.c | 5 +++-- fs/nfs/nfs4proc.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)fs/nfs/delegation.c +0
-0@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation err =nfs4_wait_clnt_recover (clp); }while (err ==0 );Quoted 7 lines
Quoted 3 lines
fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398
Quoted 3 lines
fs/nfs/delegation.c +1
-1-if (err) { +if (err && err != -EIO ) {nfs_abort_delegation_return (delegation, clp);This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a
FREE_STATEID, as required for the case of NFSv4.1.Quoted 3 lines
I'd need to brush up on the
FREE_STATEIDop to confidently answer that. But can't this be taken care of instate_recoverywhen locks are marked lost?Quoted 5 lines
No, because in the case of EIO we now fall through to
nfs_detach_delegation()andnfs_do_return_delegation(). What we really want to do is callTEST_STATEID+FREE_STATEID(see RFC5661 sections 8.2.4, 8.5 and 18.38.3), which will now be done as part of state recovery.
Quoted 6 lines
I don't see it. :-/ Is it because there is a race between returning a delegation and state recovery that was initiated? Because as far as I can see, the stateid recovery initiated by the
handle_delegation_recall_error()will callnfs4_test_stateid()vianfs41_open_expire().
No. My point is that once we call nfs_detach_delegation(), then nobody will be able to find the delegation from either the struct nfs_inode or from the server->delegations list.
IOW: the question is how would the state recovery thread find it in the first place?
Signature · 5 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]