Skip to contentSkip to conversation

nfs: fix NULL deference in nfs4_get_valid_delegation

PATCH In review J. Bruce Fields 3 participants 9 messages May 8, 2020 – May 11, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nfs: fix NULL deference in nfs4_get_valid_delegation

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.

That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode.

Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4state.c +1 -1
index ac93715c05a4..a8dc25ce48bb 100644
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add stateless
 		 * and read-only stateids to the end of the list */
Signature · 1 line
2.26.2
Reviewed-by Seiichi Ikarashi Tested-by Daisuke Matsuda Tested-by on v1 1/1
#1 lore raw
Quoted 25 lines · J. Bruce Fields

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.

That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode.

Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4state.c +0 -0
index ac93715c05a4..a8dc25ce48bb 100644

fs/nfs/nfs4state.c nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) · line 742

Quoted 10 lines
fs/nfs/nfs4state.c +1 -1
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add stateless
 		 * and read-only stateids to the end of the list */

Thank you for posting the patch! It works for our box. Please feel free to add:

Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:

   BUG: unable to handle page fault for address: ffffffffffffffb0
   ...
   RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4]
   ...
   Call Trace:
    nfs4_open_prepare+0x80/0x1c0 [nfsv4]
    __rpc_execute+0x75/0x390 [sunrpc]
    ? finish_task_switch+0x75/0x260
    rpc_async_schedule+0x29/0x40 [sunrpc]
    process_one_work+0x1ad/0x370
    worker_thread+0x30/0x390
    ? create_worker+0x1a0/0x1a0
    kthread+0x10c/0x130
    ? kthread_park+0x80/0x80
    ret_from_fork+0x22/0x30

After applied the patch, the panic is gone.

Thanks!
Masa

#2 lore raw

Thanks, applying.--b.

Quoted 67 lines · Masayoshi Mizuma
Quoted 25 lines · J. Bruce Fields

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.

That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode.

Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4state.c +0 -0
index ac93715c05a4..a8dc25ce48bb 100644

fs/nfs/nfs4state.c nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) · line 742

Quoted 10 lines
fs/nfs/nfs4state.c +1 -1
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add stateless
 		 * and read-only stateids to the end of the list */

Thank you for posting the patch! It works for our box. Please feel free to add:

Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:

   BUG: unable to handle page fault for address: ffffffffffffffb0
   ...
   RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4]
   ...
   Call Trace:
    nfs4_open_prepare+0x80/0x1c0 [nfsv4]
    __rpc_execute+0x75/0x390 [sunrpc]
    ? finish_task_switch+0x75/0x260
    rpc_async_schedule+0x29/0x40 [sunrpc]
    process_one_work+0x1ad/0x370
    worker_thread+0x30/0x390
    ? create_worker+0x1a0/0x1a0
    kthread+0x10c/0x130
    ? kthread_park+0x80/0x80
    ret_from_fork+0x22/0x30

After applied the patch, the panic is gone.

Thanks!
Masa

#3 lore raw
Quoted 2 lines · J. Bruce Fields

Thanks, applying.--b.

You're applying? So should I remove it from the NFS client bugfixes?

Quoted 73 lines · Masayoshi Mizuma
Quoted 72 lines · J. Bruce Fields
Quoted 37 lines

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing
it.

That wasn't a problem when all the readers were also taking the i_lock
(as we do here), but since we switched to RCU, there's now a possibility
that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode.

Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4state.c +1 -1
index ac93715c05a4..a8dc25ce48bb 100644
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode,
 struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add
Quoted 3 lines

stateless
* and read-only stateids to the end of the list */

Thank you for posting the patch! It works for our box. Please feel free to add:

Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:

   BUG: unable to handle page fault for address: ffffffffffffffb0
   ...
   RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4]
   ...
   Call Trace:
    nfs4_open_prepare+0x80/0x1c0 [nfsv4]
    __rpc_execute+0x75/0x390 [sunrpc]
    ? finish_task_switch+0x75/0x260
    rpc_async_schedule+0x29/0x40 [sunrpc]
    process_one_work+0x1ad/0x370
    worker_thread+0x30/0x390
    ? create_worker+0x1a0/0x1a0
    kthread+0x10c/0x130
    ? kthread_park+0x80/0x80
    ret_from_fork+0x22/0x30

After applied the patch, the panic is gone.

Thanks!
Masa

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#4 lore raw
Quoted 5 lines · Trond Myklebust
Quoted 2 lines · J. Bruce Fields

Thanks, applying.--b.

You're applying? So should I remove it from the NFS client bugfixes?

No. Sorry, I responded to the wrong email!

--b.

Quoted 80 lines
Quoted 74 lines · Masayoshi Mizuma
Quoted 72 lines · J. Bruce Fields
Quoted 37 lines

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing
it.

That wasn't a problem when all the readers were also taking the i_lock
(as we do here), but since we switched to RCU, there's now a possibility
that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode.

Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4state.c +1 -1
index ac93715c05a4..a8dc25ce48bb 100644
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode,
 struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add
Quoted 3 lines

stateless
* and read-only stateids to the end of the list */

Thank you for posting the patch! It works for our box. Please feel free to add:

Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:

   BUG: unable to handle page fault for address: ffffffffffffffb0
   ...
   RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4]
   ...
   Call Trace:
    nfs4_open_prepare+0x80/0x1c0 [nfsv4]
    __rpc_execute+0x75/0x390 [sunrpc]
    ? finish_task_switch+0x75/0x260
    rpc_async_schedule+0x29/0x40 [sunrpc]
    process_one_work+0x1ad/0x370
    worker_thread+0x30/0x390
    ? create_worker+0x1a0/0x1a0
    kthread+0x10c/0x130
    ? kthread_park+0x80/0x80
    ret_from_fork+0x22/0x30

After applied the patch, the panic is gone.

Thanks!
Masa

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#5 lore raw
Quoted 8 lines · On Mon, May 11, 2020 at 09:57:45AM -0400, [email protected] wrote:
Quoted 5 lines · Trond Myklebust
Quoted 2 lines · J. Bruce Fields

Thanks, applying.--b.

You're applying? So should I remove it from the NFS client bugfixes?

No. Sorry, I responded to the wrong email!

I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.

#6 lore raw

PATCH 1/1 nfs: fix NULL deference in nfs4_get_valid_delegation

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.

That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode, resulting in an oops like:

	BUG: unable to handle page fault for address: ffffffffffffffb0 ...
	RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ...
	Call Trace:
	 nfs4_open_prepare+0x80/0x1c0 [nfsv4]
	 __rpc_execute+0x75/0x390 [sunrpc]
	 ? finish_task_switch+0x75/0x260
	 rpc_async_schedule+0x29/0x40 [sunrpc]
	 process_one_work+0x1ad/0x370
	 worker_thread+0x30/0x390
	 ? create_worker+0x1a0/0x1a0
	 kthread+0x10c/0x130
	 ? kthread_park+0x80/0x80
	 ret_from_fork+0x22/0x30
Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 2 lines

I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.

Here you go.

fs/nfs/nfs4state.c +1 -1
index f7723d221945..459c7fb5d103 100644
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add stateless
 		 * and read-only stateids to the end of the list */
Signature · 1 line
2.26.2
#7 lore raw

Hello,

How about adding Cc: [email protected] in the sign-off area? 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU" was merged to v4.20 and some stable tree have 9ae075fdd190.

Quoted 27 lines · On Mon, May 11, 2020 at 10:02:48AM -0400, [email protected] wrote:

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.

That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode, resulting in an oops like:

	BUG: unable to handle page fault for address: ffffffffffffffb0 ...
	RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ...
	Call Trace:
	 nfs4_open_prepare+0x80/0x1c0 [nfsv4]
	 __rpc_execute+0x75/0x390 [sunrpc]
	 ? finish_task_switch+0x75/0x260
	 rpc_async_schedule+0x29/0x40 [sunrpc]
	 process_one_work+0x1ad/0x370
	 worker_thread+0x30/0x390
	 ? create_worker+0x1a0/0x1a0
	 kthread+0x10c/0x130
	 ? kthread_park+0x80/0x80
	 ret_from_fork+0x22/0x30
Quoted 1 line
Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
Cc [email protected] # 4.20.x-

Thanks!
Masa

Quoted 31 lines
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 2 lines

I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.

Here you go.

fs/nfs/nfs4state.c +1 -1
index f7723d221945..459c7fb5d103 100644
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add stateless
 		 * and read-only stateids to the end of the list */
Signature · 1 line
2.26.2
#8 lore raw
Quoted 4 lines · Masayoshi Mizuma

Hello,

How about adding Cc: [email protected] in the sign-off area? 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU" was merged

Makes sense to me. I'll leave it to Trond.--b.

Quoted 68 lines

to v4.20 and some stable tree have 9ae075fdd190.

Quoted 27 lines · On Mon, May 11, 2020 at 10:02:48AM -0400, [email protected] wrote:

From: "J. Bruce Fields" <[email protected]>

We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.

That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.

Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode, resulting in an oops like:

	BUG: unable to handle page fault for address: ffffffffffffffb0 ...
	RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ...
	Call Trace:
	 nfs4_open_prepare+0x80/0x1c0 [nfsv4]
	 __rpc_execute+0x75/0x390 [sunrpc]
	 ? finish_task_switch+0x75/0x260
	 rpc_async_schedule+0x29/0x40 [sunrpc]
	 process_one_work+0x1ad/0x370
	 worker_thread+0x30/0x390
	 ? create_worker+0x1a0/0x1a0
	 kthread+0x10c/0x130
	 ? kthread_park+0x80/0x80
	 ret_from_fork+0x22/0x30
Quoted 1 line
Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
Cc [email protected] # 4.20.x-

Thanks!
Masa

Quoted 31 lines
---
 fs/nfs/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 2 lines

I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.

Here you go.

fs/nfs/nfs4state.c +1 -1
index f7723d221945..459c7fb5d103 100644
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner)
 		state = new;
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
-		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		ihold(inode);
 		state->inode = inode;
+		list_add_rcu(&state->inode_states, &nfsi->open_states);
 		spin_unlock(&inode->i_lock);
 		/* Note: The reclaim code dictates that we add stateless
 		 * and read-only stateids to the end of the list */
Signature · 1 line
2.26.2