Skip to contentSkip to conversation

NFSv4: Fix deadlock between nfs4_evict_inode() and nfs4_opendata_get_inode()

PATCH In review [email protected] 3 participants 5 messages Jun 1, 2021 – Jun 8, 2021

Subsystems (2)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

PATCH 1/2 NFSv4: Fix deadlock between nfs4_evict_inode() and nfs4_opendata_get_inode()

From: Trond Myklebust <[email protected]>

If the inode is being evicted, but has to return a delegation first, then it can cause a deadlock in the corner case where the server reboots before the delegreturn completes, but while the call to iget5_locked() in nfs4_opendata_get_inode() is waiting for the inode free to complete. Since the open call still holds a session slot, the reboot recovery cannot proceed.

In order to break the logjam, we can turn the delegation return into a privileged operation for the case where we're evicting the inode. We know that in that case, there can be no other state recovery operation that conflicts.

Fixes 5fcdfacc01f3 ("NFSv4: Return delegations synchronously in evict_inode")
---
 fs/nfs/nfs4_fs.h  |  1 +
 fs/nfs/nfs4proc.c | 12 +++++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)
fs/nfs/nfs4_fs.h +1 -0
index 065cb04222a1..543d916f79ab 100644
@@ -205,6 +205,7 @@ struct nfs4_exception {
 	struct inode *inode;
 	nfs4_stateid *stateid;
 	long timeout;
+	unsigned char task_is_privileged : 1;
 	unsigned char delay : 1,
 		      recovering : 1,
 		      retry : 1;
fs/nfs/nfs4proc.c +11 -1
index d671b2884d5a..673809644981 100644
@@ -589,6 +589,8 @@ int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_
 		goto out_retry;
 	}
 	if (exception->recovering) {
+		if (exception->task_is_privileged)
+			return -EDEADLOCK;
 		ret = nfs4_wait_clnt_recover(clp);
 		if (test_bit(NFS_MIG_FAILED, &server->mig_status))
 			return -EIO;
@@ -614,6 +616,8 @@ nfs4_async_handle_exception(struct rpc_task *task, struct nfs_server *server,
 		goto out_retry;
 	}
 	if (exception->recovering) {
+		if (exception->task_is_privileged)
+			return -EDEADLOCK;
 		rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
 		if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
 			rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
@@ -6417,6 +6421,7 @@ static void nfs4_delegreturn_done(struct rpc_task *task, void *calldata)
 	struct nfs4_exception exception = {
 		.inode = data->inode,
 		.stateid = &data->stateid,
+		.task_is_privileged = data->args.seq_args.sa_privileged,
 	};
 
 	if (!nfs4_sequence_done(task, &data->res.seq_res))
@@ -6540,7 +6545,6 @@ static int _nfs4_proc_delegreturn(struct inode *inode, const struct cred *cred,
 	data = kzalloc(sizeof(*data), GFP_NOFS);
 	if (data == NULL)
 		return -ENOMEM;
-	nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1, 0);
 
 	nfs4_state_protect(server->nfs_client,
 			NFS_SP4_MACH_CRED_CLEANUP,
@@ -6571,6 +6575,12 @@ static int _nfs4_proc_delegreturn(struct inode *inode, const struct cred *cred,
 		}
 	}
 
+	if (!data->inode)
+		nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1,
+				   1);
+	else
+		nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1,
+				   0);
 	task_setup_data.callback_data = data;
 	msg.rpc_argp = &data->args;
 	msg.rpc_resp = &data->res;
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH 2/2 NFSv4: Fix second deadlock in nfs4_evict_inode()

From: Trond Myklebust <[email protected]>

If the inode is being evicted but has to return a layout first, then that too can cause a deadlock in the corner case where the server reboots.

---
 fs/nfs/nfs4proc.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
fs/nfs/nfs4proc.c +7 -2
index 673809644981..e25c16257545 100644
@@ -9658,15 +9658,20 @@ int nfs4_proc_layoutreturn(struct nfs4_layoutreturn *lrp, bool sync)
 			&task_setup_data.rpc_client, &msg);
 
 	dprintk("--> %s\n", __func__);
+	lrp->inode = nfs_igrab_and_active(lrp->args.inode);
 	if (!sync) {
-		lrp->inode = nfs_igrab_and_active(lrp->args.inode);
 		if (!lrp->inode) {
 			nfs4_layoutreturn_release(lrp);
 			return -EAGAIN;
 		}
 		task_setup_data.flags |= RPC_TASK_ASYNC;
 	}
-	nfs4_init_sequence(&lrp->args.seq_args, &lrp->res.seq_res, 1, 0);
+	if (!lrp->inode)
+		nfs4_init_sequence(&lrp->args.seq_args, &lrp->res.seq_res, 1,
+				   1);
+	else
+		nfs4_init_sequence(&lrp->args.seq_args, &lrp->res.seq_res, 1,
+				   0);
 	task = rpc_run_task(&task_setup_data);
 	if (IS_ERR(task))
 		return PTR_ERR(task);
Signature · 1 line
2.31.1
#2 lore raw
In reply to [email protected]
Quoted 9 lines · 在 2021/6/2 1:36, [email protected] 写道:

From: Trond Myklebust <[email protected]>

If the inode is being evicted, but has to return a delegation first, then it can cause a deadlock in the corner case where the server reboots before the delegreturn completes, but while the call to iget5_locked() in nfs4_opendata_get_inode() is waiting for the inode free to complete. Since the open call still holds a session slot, the reboot recovery cannot proceed.

Quoted 5 lines

In order to break the logjam, we can turn the delegation return into a privileged operation for the case where we're evicting the inode. We know that in that case, there can be no other state recovery operation that conflicts.

it's looks good to me.

but i have another confuse, how to ensure no writeback when evict nfs inode? because flush writes to server when close?
but not all close will flush writes to server.

Quoted 72 lines
Fixes 5fcdfacc01f3 ("NFSv4: Return delegations synchronously in evict_inode")
---
  fs/nfs/nfs4_fs.h  |  1 +
  fs/nfs/nfs4proc.c | 12 +++++++++++-
  2 files changed, 12 insertions(+), 1 deletion(-)
fs/nfs/nfs4_fs.h +1 -0
index 065cb04222a1..543d916f79ab 100644
@@ -205,6 +205,7 @@ struct nfs4_exception {
  	struct inode *inode;
  	nfs4_stateid *stateid;
  	long timeout;
+	unsigned char task_is_privileged : 1;
  	unsigned char delay : 1,
  		      recovering : 1,
  		      retry : 1;
fs/nfs/nfs4proc.c +11 -1
index d671b2884d5a..673809644981 100644
@@ -589,6 +589,8 @@ int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_
  		goto out_retry;
  	}
  	if (exception->recovering) {
+		if (exception->task_is_privileged)
+			return -EDEADLOCK;
  		ret = nfs4_wait_clnt_recover(clp);
  		if (test_bit(NFS_MIG_FAILED, &server->mig_status))
  			return -EIO;
@@ -614,6 +616,8 @@ nfs4_async_handle_exception(struct rpc_task *task, struct nfs_server *server,
  		goto out_retry;
  	}
  	if (exception->recovering) {
+		if (exception->task_is_privileged)
+			return -EDEADLOCK;
  		rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
  		if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
  			rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
@@ -6417,6 +6421,7 @@ static void nfs4_delegreturn_done(struct rpc_task *task, void *calldata)
  	struct nfs4_exception exception = {
  		.inode = data->inode,
  		.stateid = &data->stateid,
+		.task_is_privileged = data->args.seq_args.sa_privileged,
  	};
 
  	if (!nfs4_sequence_done(task, &data->res.seq_res))
@@ -6540,7 +6545,6 @@ static int _nfs4_proc_delegreturn(struct inode *inode, const struct cred *cred,
  	data = kzalloc(sizeof(*data), GFP_NOFS);
  	if (data == NULL)
  		return -ENOMEM;
-	nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1, 0);
 
  	nfs4_state_protect(server->nfs_client,
  			NFS_SP4_MACH_CRED_CLEANUP,
@@ -6571,6 +6575,12 @@ static int _nfs4_proc_delegreturn(struct inode *inode, const struct cred *cred,
  		}
  	}
 
+	if (!data->inode)
+		nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1,
+				   1);
+	else
+		nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1,
+				   0);
  	task_setup_data.callback_data = data;
  	msg.rpc_argp = &data->args;
  	msg.rpc_resp = &data->res;
#3 lore raw
Quoted 24 lines · zhangxiaoxu (A)
Quoted 13 lines · 在 2021/6/2 1:36, [email protected] 写道:

From: Trond Myklebust <[email protected]>

If the inode is being evicted, but has to return a delegation first,
then it can cause a deadlock in the corner case where the server reboots
before the delegreturn completes, but while the call to iget5_locked() in
nfs4_opendata_get_inode() is waiting for the inode free to complete.
Since the open call still holds a session slot, the reboot recovery cannot proceed.

Quoted 8 lines

In order to break the logjam, we can turn the delegation return into a
privileged operation for the case where we're evicting the inode. We
know that in that case, there can be no other state recovery operation
that conflicts.

Quoted 6 lines

it's looks good to me.

but i have another confuse, how to ensure no writeback when evict nfs inode?
because flush writes to server when close?
but not all close will flush writes to server.

The struct nfs_open_context holds a reference to the dentry (which holds a reference to the inode) and to the superblock. The struct nfs_page that is tracking page dirtiness then holds a reference to the nfs_open_context.

That mechanism ensures the inode cannot be evicted until all dirty pages have been either flushed or cancelled. The only thing we need to worry about is the delegation and the pNFS layout since neither one is allowed to reference the inode in any way (because otherwise they would prevent the memory reclaim mechanisms from working).

Quoted 84 lines
Quoted 84 lines
Fixes 5fcdfacc01f3 ("NFSv4: Return delegations synchronously in

evict_inode")

---
  fs/nfs/nfs4_fs.h  |  1 +
  fs/nfs/nfs4proc.c | 12 +++++++++++-

  2 files changed, 12 insertions(+), 1 deletion(-)
fs/nfs/nfs4_fs.h +1 -0
index 065cb04222a1..543d916f79ab 100644
@@ -205,6 +205,7 @@ struct nfs4_exception {
         struct inode *inode;
         nfs4_stateid *stateid;
         long timeout;
+       unsigned char task_is_privileged : 1;
         unsigned char delay : 1,
                       recovering : 1,
                       retry : 1;
fs/nfs/nfs4proc.c +2 -0
index d671b2884d5a..673809644981 100644
@@ -589,6 +589,8 @@ int nfs4_handle_exception(struct nfs_server
 *server, int errorcode, struct nfs4_
                 goto out_retry;
         }
         if (exception->recovering) {
+               if (exception->task_is_privileged)
+                       return -EDEADLOCK;
                 ret = nfs4_wait_clnt_recover(clp);
                 if (test_bit(NFS_MIG_FAILED, &server->mig_status))

                        return -EIO;

fs/nfs/nfs4proc.c +2 -0
@@ -614,6 +616,8 @@ nfs4_async_handle_exception(struct rpc_task
 *task, struct nfs_server *server,
                 goto out_retry;
         }
         if (exception->recovering) {
+               if (exception->task_is_privileged)
+                       return -EDEADLOCK;
                 rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);

                if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp-

Quoted 1 line

cl_state) == 0)

                        rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);

fs/nfs/nfs4proc.c +1 -0
@@ -6417,6 +6421,7 @@ static void nfs4_delegreturn_done(struct
 rpc_task *task, void *calldata)
         struct nfs4_exception exception = {
                 .inode = data->inode,
                 .stateid = &data->stateid,
+               .task_is_privileged = data-
 >args.seq_args.sa_privileged,
         };


        if (!nfs4_sequence_done(task, &data->res.seq_res))

fs/nfs/nfs4proc.c +0 -1
@@ -6540,7 +6545,6 @@ static int _nfs4_proc_delegreturn(struct
 inode *inode, const struct cred *cred,
         data = kzalloc(sizeof(*data), GFP_NOFS);
         if (data == NULL)
                 return -ENOMEM;
-       nfs4_init_sequence(&data->args.seq_args, &data-
 >res.seq_res, 1, 0);


        nfs4_state_protect(server->nfs_client, NFS_SP4_MACH_CRED_CLEANUP,

fs/nfs/nfs4proc.c +0 -0
@@ -6571,6 +6575,12 @@ static int _nfs4_proc_delegreturn(struct
 inode *inode, const struct cred *cred,
                 }
         }


+       if (!data->inode)
+               nfs4_init_sequence(&data->args.seq_args, &data-

Quoted 1 line

res.seq_res, 1,

+                                  1);
+       else
+               nfs4_init_sequence(&data->args.seq_args, &data-

Quoted 1 line

res.seq_res, 1,

+                                  0);
        task_setup_data.callback_data = data;
        msg.rpc_argp = &data->args;
        msg.rpc_resp = &data->res;

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#4 lore raw
Quoted 33 lines · Trond Myklebust
Quoted 22 lines · 在 2021/6/2 1:36,[email protected]  写道:
Quoted 13 lines

From: Trond Myklebust<[email protected]>

If the inode is being evicted, but has to return a delegation first,
then it can cause a deadlock in the corner case where the server reboots
before the delegreturn completes, but while the call to iget5_locked() in
nfs4_opendata_get_inode() is waiting for the inode free to complete.
Since the open call still holds a session slot, the reboot recovery cannot proceed.

Quoted 8 lines

In order to break the logjam, we can turn the delegation return into a
privileged operation for the case where we're evicting the inode. We
know that in that case, there can be no other state recovery operation
that conflicts.

Quoted 6 lines

it's looks good to me.

but i have another confuse, how to ensure no writeback when evict nfs inode?
because flush writes to server when close?
but not all close will flush writes to server.

The struct nfs_open_context holds a reference to the dentry (which holds a reference to the inode) and to the superblock. The struct nfs_page that is tracking page dirtiness then holds a reference to the nfs_open_context.

Quoted 6 lines

That mechanism ensures the inode cannot be evicted until all dirty pages have been either flushed or cancelled. The only thing we need to worry about is the delegation and the pNFS layout since neither one is allowed to reference the inode in any way (because otherwise they would prevent the memory reclaim mechanisms from working).

Yes, it is.
Thank you very much.