Skip to contentSkip to conversation

Fixing infinite state recovery loop due to failed delegation return

PATCH In review Olga Kornievskaia 3 participants 7 messages Nov 3, 2014 – Nov 6, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fixing infinite state recovery loop due to failed delegation return

If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.

Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.

---
 fs/nfs/delegation.c |    5 +++--
 fs/nfs/nfs4proc.c   |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +3 -2
index 5853f53..8016d89 100644
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation
 		err = nfs4_wait_clnt_recover(clp);
 	} while (err == 0);
 
-	if (err) {
+	if (err && err != -EIO) {
 		nfs_abort_delegation_return(delegation, clp);
 		goto out;
 	}
@@ -458,7 +458,8 @@ restart:
 			iput(inode);
 			if (!err)
 				goto restart;
-			set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);
+			if (err != -EIO)
+				set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);
 			return err;
 		}
 	}
fs/nfs/nfs4proc.c +1 -1
index 5aa55c1..6871055 100644
@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct
 			nfs_inode_find_state_and_recover(state->inode,
 					stateid);
 			nfs4_schedule_stateid_recovery(server, state);
-			return 0;
+			return -EIO;
 		case -NFS4ERR_DELAY:
 		case -NFS4ERR_GRACE:
 			set_bit(NFS_DELEGATED_STATE, &state->flags);
Signature · 1 line
1.7.1
#1 lore raw

Trond, can you please respond to the patch?

As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.

Please either accept this patch or let me know what needs to be fixed.

Thank you.

Quoted 60 lines · Olga Kornievskaia

If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.

Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.

---
 fs/nfs/delegation.c |    5 +++--
 fs/nfs/nfs4proc.c   |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +3 -2
index 5853f53..8016d89 100644
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation
                err = nfs4_wait_clnt_recover(clp);
        } while (err == 0);
 
-       if (err) {
+       if (err && err != -EIO) {
                nfs_abort_delegation_return(delegation, clp);
                goto out;
        }
@@ -458,7 +458,8 @@ restart:
                        iput(inode);
                        if (!err)
                                goto restart;
-                       set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);
+                       if (err != -EIO)
+                               set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);
                        return err;
                }
        }
fs/nfs/nfs4proc.c +1 -2
index 5aa55c1..6871055 100644
@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct
                        nfs_inode_find_state_and_recover(state->inode,
                                        stateid);
                        nfs4_schedule_stateid_recovery(server, state);
-                       return 0;
+                       return -EIO;
                case -NFS4ERR_DELAY:
                case -NFS4ERR_GRACE:
                        set_bit(NFS_DELEGATED_STATE, &state->flags);
--

1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 7 lines · Olga Kornievskaia

Trond, can you please respond to the patch?

As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.

Please either accept this patch or let me know what needs to be fixed.

Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).

Quoted 28 lines

Thank you.

Quoted 25 lines · Olga Kornievskaia

If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.

Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.

---
 fs/nfs/delegation.c |    5 +++--
 fs/nfs/nfs4proc.c   |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +0 -0
index 5853f53..8016d89 100644
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation
                err = nfs4_wait_clnt_recover(clp);
        } while (err == 0);
 
Quoted 3 lines

fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398

Quoted 3 lines
fs/nfs/delegation.c +1 -1
-       if (err) {
+       if (err && err != -EIO) {
                nfs_abort_delegation_return(delegation, clp);

This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a FREE_STATEID, as required for the case of NFSv4.1.

Quoted 9 lines

fs/nfs/delegation.c restart: · line 462

Quoted 9 lines
fs/nfs/delegation.c +2 -1
                goto out;
        }
@@ -458,7 +458,8 @@ restart:
                        iput(inode);
                        if (!err)
                                goto restart;
-                       set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);
+                       if (err != -EIO)
+                               set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);

Please explain why this is needed. If we've cleared the bad delegation, then why should we not attempt to return any others that may be pending?

Quoted 23 lines
Quoted 23 lines
fs/nfs/delegation.c +0 -0
                        return err;
                }
        }
fs/nfs/nfs4proc.c +1 -2
index 5aa55c1..6871055 100644
@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct
                        nfs_inode_find_state_and_recover(state->inode,
                                        stateid);
                        nfs4_schedule_stateid_recovery(server, state);
-                       return 0;
+                       return -EIO;
                case -NFS4ERR_DELAY:
                case -NFS4ERR_GRACE:
                        set_bit(NFS_DELEGATED_STATE, &state->flags);
--

1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]
#3 lore raw

Thank you for looking into this!

Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia

Trond, can you please respond to the patch?

As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.

Please either accept this patch or let me know what needs to be fixed.

Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).

Quoted 28 lines

Thank you.

Quoted 25 lines · Olga Kornievskaia

If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.

Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.

---
 fs/nfs/delegation.c |    5 +++--
 fs/nfs/nfs4proc.c   |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +0 -0
index 5853f53..8016d89 100644
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation
                err = nfs4_wait_clnt_recover(clp);
        } while (err == 0);
 
Quoted 7 lines
Quoted 3 lines

fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398

Quoted 3 lines
fs/nfs/delegation.c +1 -1
-       if (err) {
+       if (err && err != -EIO) {
                nfs_abort_delegation_return(delegation, clp);

This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a FREE_STATEID, as required for the case of NFSv4.1.

I'd need to brush up on the FREE_STATEID op to confidently answer that. But can't this be taken care of in state_recovery when locks are marked lost?

Quoted 14 lines
Quoted 10 lines

fs/nfs/delegation.c restart: · line 462

Quoted 10 lines
fs/nfs/delegation.c +2 -1
                goto out;
        }
@@ -458,7 +458,8 @@ restart:
                        iput(inode);
                        if (!err)
                                goto restart;
-                       set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);
+                       if (err != -EIO)
+                               set_bit(NFS4CLNT_DELEGRETURN, &clp->cl_state);

Please explain why this is needed. If we've cleared the bad delegation, then why should we not attempt to return any others that may be pending?

You are right, I think this is unnecessary because after we return the delegation the bit can be set again for any other leftover delegations. I have tested my patch without that chuck and it still fixes the issues.

Quoted 33 lines
Quoted 20 lines
Quoted 20 lines
fs/nfs/delegation.c +0 -0
                        return err;
                }
        }
fs/nfs/nfs4proc.c +1 -2
index 5aa55c1..6871055 100644
@@ -1655,7 +1655,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct
                        nfs_inode_find_state_and_recover(state->inode,
                                        stateid);
                        nfs4_schedule_stateid_recovery(server, state);
-                       return 0;
+                       return -EIO;
                case -NFS4ERR_DELAY:
                case -NFS4ERR_GRACE:
                        set_bit(NFS_DELEGATED_STATE, &state->flags);
--

1.7.1

Quoted 4 lines
Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]

#4 lore raw
Quoted 53 lines · Olga Kornievskaia

Thank you for looking into this!

Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia

Trond, can you please respond to the patch?

As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.

Please either accept this patch or let me know what needs to be fixed.

Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).

Quoted 28 lines

Thank you.

Quoted 25 lines · Olga Kornievskaia

If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.

Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.

---
 fs/nfs/delegation.c |    5 +++--
 fs/nfs/nfs4proc.c   |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +0 -0
index 5853f53..8016d89 100644
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation
                err = nfs4_wait_clnt_recover(clp);
        } while (err == 0);
 
Quoted 7 lines
Quoted 3 lines

fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398

Quoted 3 lines
fs/nfs/delegation.c +1 -1
-       if (err) {
+       if (err && err != -EIO) {
                nfs_abort_delegation_return(delegation, clp);

This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a FREE_STATEID, as required for the case of NFSv4.1.

Quoted 3 lines

I'd need to brush up on the FREE_STATEID op to confidently answer that. But can't this be taken care of in state_recovery when locks are marked lost?

No, because in the case of EIO we now fall through to nfs_detach_delegation() and nfs_do_return_delegation(). What we really want to do is call TEST_STATEID + FREE_STATEID (see RFC5661 sections 8.2.4, 8.5 and 18.38.3), which will now be done as part of state recovery.

Signature · 5 lines
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]
#5 lore raw
Quoted 58 lines · Trond Myklebust
Quoted 53 lines · Olga Kornievskaia

Thank you for looking into this!

Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia

Trond, can you please respond to the patch?

As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.

Please either accept this patch or let me know what needs to be fixed.

Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).

Quoted 28 lines

Thank you.

Quoted 25 lines · Olga Kornievskaia

If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.

Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.

---
 fs/nfs/delegation.c |    5 +++--
 fs/nfs/nfs4proc.c   |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +0 -0
index 5853f53..8016d89 100644
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation
                err = nfs4_wait_clnt_recover(clp);
        } while (err == 0);
 
Quoted 7 lines
Quoted 3 lines

fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398

Quoted 3 lines
fs/nfs/delegation.c +1 -1
-       if (err) {
+       if (err && err != -EIO) {
                nfs_abort_delegation_return(delegation, clp);

This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a FREE_STATEID, as required for the case of NFSv4.1.

Quoted 3 lines

I'd need to brush up on the FREE_STATEID op to confidently answer that. But can't this be taken care of in state_recovery when locks are marked lost?

Quoted 5 lines

No, because in the case of EIO we now fall through to nfs_detach_delegation() and nfs_do_return_delegation(). What we really want to do is call TEST_STATEID + FREE_STATEID (see RFC5661 sections 8.2.4, 8.5 and 18.38.3), which will now be done as part of state recovery.

I don't see it. :-/ Is it because there is a race between returning a delegation and state recovery that was initiated? Because as far as I can see, the stateid recovery initiated by the handle_delegation_recall_error() will call nfs4_test_stateid() via nfs41_open_expire().

Quoted 7 lines

--
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]

#6 lore raw
Quoted 66 lines · Olga Kornievskaia
Quoted 58 lines · Trond Myklebust
Quoted 53 lines · Olga Kornievskaia

Thank you for looking into this!

Quoted 41 lines · Trond Myklebust
Quoted 7 lines · Olga Kornievskaia

Trond, can you please respond to the patch?

As per earlier conversation, in this solution, state recovery is initiated which marks the locks lost.

Please either accept this patch or let me know what needs to be fixed.

Please see the 3 fixes I just sent out concerning delegation recovery w.r.t. NFSv4+NFSv4.1. In addition, we need to handle the case you patch attempts to address (however see the question I have below).

Quoted 28 lines

Thank you.

Quoted 25 lines · Olga Kornievskaia

If we get a bad-stateid-type of error when we send OPEN with delegate_cur to return currently held delegation, we shouldn't be trying to reclaim locks associated with that delegation state_id because we don't have an open_stateid to be used for the LOCK operation. Thus, we should return an error from the nfs4_open_delegation_recall() in that case.

Furthermore, if an error occurs the delegation code will call nfs_abort_delegation_return() which sets again the NFS4CLNT_DELEGRETURN flags in the state and it leads the state manager to into an infinite loop for trying to reclaim the delegated state.

---
 fs/nfs/delegation.c |    5 +++--
 fs/nfs/nfs4proc.c   |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/nfs/delegation.c +0 -0
index 5853f53..8016d89 100644
@@ -394,7 +394,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation
                err = nfs4_wait_clnt_recover(clp);
        } while (err == 0);
 
Quoted 7 lines
Quoted 3 lines

fs/nfs/delegation.c static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation · line 398

Quoted 3 lines
fs/nfs/delegation.c +1 -1
-       if (err) {
+       if (err && err != -EIO) {
                nfs_abort_delegation_return(delegation, clp);

This exception for EIO now has me worried. If we detach the delegation, then it looks to me as if we will never send a FREE_STATEID, as required for the case of NFSv4.1.

Quoted 3 lines

I'd need to brush up on the FREE_STATEID op to confidently answer that. But can't this be taken care of in state_recovery when locks are marked lost?

Quoted 5 lines

No, because in the case of EIO we now fall through to nfs_detach_delegation() and nfs_do_return_delegation(). What we really want to do is call TEST_STATEID + FREE_STATEID (see RFC5661 sections 8.2.4, 8.5 and 18.38.3), which will now be done as part of state recovery.

Quoted 6 lines

I don't see it. :-/ Is it because there is a race between returning a delegation and state recovery that was initiated? Because as far as I can see, the stateid recovery initiated by the handle_delegation_recall_error() will call nfs4_test_stateid() via nfs41_open_expire().

No. My point is that once we call nfs_detach_delegation(), then nobody will be able to find the delegation from either the struct nfs_inode or from the server->delegations list.
IOW: the question is how would the state recovery thread find it in the first place?

Signature · 5 lines
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]