nfs: fix NULL deference in nfs4_get_valid_delegation
Subsystems (2)
PATCH 1/1 nfs: fix NULL deference in nfs4_get_valid_delegation
From: "J. Bruce Fields" <[email protected]>
We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.
That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.
Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode.
--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4state.c +1 -1
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add stateless * and read-only stateids to the end of the list */
Signature · 1 line
2.26.2
Quoted 25 lines · J. Bruce Fields
From: "J. Bruce Fields" <[email protected]>
We add the new state to the
nfsi->open_stateslist, making it potentially visible to other threads, before we've finished initializing it.That wasn't a problem when all the readers were also taking the
i_lock(as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.Symptoms observed were a crash when another thread called
nfs4_get_valid_delegation()on a NULL inode.Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"Signed-off-by J. Bruce Fields <[email protected]>--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4state.c +0
-0
fs/nfs/nfs4state.c nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) · line 742
Quoted 10 lines
fs/nfs/nfs4state.c +1
-1@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add stateless * and read-only stateids to the end of the list */
Thank you for posting the patch! It works for our box. Please feel free to add:
Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:
BUG: unable to handle page fault for address: ffffffffffffffb0
...
RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4]
...
Call Trace:
nfs4_open_prepare+0x80/0x1c0 [nfsv4]
__rpc_execute+0x75/0x390 [sunrpc]
? finish_task_switch+0x75/0x260
rpc_async_schedule+0x29/0x40 [sunrpc]
process_one_work+0x1ad/0x370
worker_thread+0x30/0x390
? create_worker+0x1a0/0x1a0
kthread+0x10c/0x130
? kthread_park+0x80/0x80
ret_from_fork+0x22/0x30After applied the patch, the panic is gone.
Thanks!
Masa
Thanks, applying.--b.
Quoted 67 lines · Masayoshi Mizuma
Quoted 25 lines · J. Bruce Fields
From: "J. Bruce Fields" <[email protected]>
We add the new state to the
nfsi->open_stateslist, making it potentially visible to other threads, before we've finished initializing it.That wasn't a problem when all the readers were also taking the
i_lock(as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.Symptoms observed were a crash when another thread called
nfs4_get_valid_delegation()on a NULL inode.Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"Signed-off-by J. Bruce Fields <[email protected]>--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4state.c +0
-0fs/nfs/nfs4state.c nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) · line 742
Quoted 10 lines
fs/nfs/nfs4state.c +1
-1@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add stateless * and read-only stateids to the end of the list */ Thank you for posting the patch! It works for our box. Please feel free to add:
Reviewed-by Seiichi Ikarashi <[email protected]>Tested-by Daisuke Matsuda <[email protected]>Tested-by Masayoshi Mizuma <[email protected]>Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:
BUG: unable to handle page fault for address: ffffffffffffffb0 ... RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ... Call Trace: nfs4_open_prepare+0x80/0x1c0 [nfsv4] __rpc_execute+0x75/0x390 [sunrpc] ? finish_task_switch+0x75/0x260 rpc_async_schedule+0x29/0x40 [sunrpc] process_one_work+0x1ad/0x370 worker_thread+0x30/0x390 ? create_worker+0x1a0/0x1a0 kthread+0x10c/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x22/0x30After applied the patch, the panic is gone.
Thanks!
Masa
Quoted 2 lines · J. Bruce Fields
Thanks, applying.--b.
You're applying? So should I remove it from the NFS client bugfixes?
Quoted 73 lines · Masayoshi Mizuma
Quoted 72 lines · J. Bruce Fields
Quoted 37 lines
From: "J. Bruce Fields" <[email protected]>
We add the new state to the
nfsi->open_stateslist, making it potentially visible to other threads, before we've finished initializing
it.That wasn't a problem when all the readers were also taking the
i_lock
(as we do here), but since we switched to RCU, there's now a possibility
that a reader could see the partially initialized state.Symptoms observed were a crash when another thread called
nfs4_get_valid_delegation()on a NULL inode.Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"Signed-off-by J. Bruce Fields <[email protected]>--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4state.c +1
-1@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add Quoted 3 lines
stateless
* and read-only stateids to the end of the list */Thank you for posting the patch! It works for our box. Please feel free to add:
Reviewed-by Seiichi Ikarashi <[email protected]>Tested-by Daisuke Matsuda <[email protected]>Tested-by Masayoshi Mizuma <[email protected]>Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:
BUG: unable to handle page fault for address: ffffffffffffffb0 ... RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ... Call Trace: nfs4_open_prepare+0x80/0x1c0 [nfsv4] __rpc_execute+0x75/0x390 [sunrpc] ? finish_task_switch+0x75/0x260 rpc_async_schedule+0x29/0x40 [sunrpc] process_one_work+0x1ad/0x370 worker_thread+0x30/0x390 ? create_worker+0x1a0/0x1a0 kthread+0x10c/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x22/0x30After applied the patch, the panic is gone.
Thanks!
Masa
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 5 lines · Trond Myklebust
Quoted 2 lines · J. Bruce Fields
Thanks, applying.--b.
You're applying? So should I remove it from the NFS client bugfixes?
No. Sorry, I responded to the wrong email!
--b.
Quoted 80 lines
Quoted 74 lines · Masayoshi Mizuma
Quoted 72 lines · J. Bruce Fields
Quoted 37 lines
From: "J. Bruce Fields" <[email protected]>
We add the new state to the
nfsi->open_stateslist, making it potentially visible to other threads, before we've finished initializing
it.That wasn't a problem when all the readers were also taking the
i_lock
(as we do here), but since we switched to RCU, there's now a possibility
that a reader could see the partially initialized state.Symptoms observed were a crash when another thread called
nfs4_get_valid_delegation()on a NULL inode.Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"Signed-off-by J. Bruce Fields <[email protected]>--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/nfs/nfs4state.c +1
-1@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add Quoted 3 lines
stateless
* and read-only stateids to the end of the list */Thank you for posting the patch! It works for our box. Please feel free to add:
Reviewed-by Seiichi Ikarashi <[email protected]>Tested-by Daisuke Matsuda <[email protected]>Tested-by Masayoshi Mizuma <[email protected]>Without the patch, the system which is a NFSv4 client has been crashed randomly. The panic log is such as:
BUG: unable to handle page fault for address: ffffffffffffffb0 ... RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ... Call Trace: nfs4_open_prepare+0x80/0x1c0 [nfsv4] __rpc_execute+0x75/0x390 [sunrpc] ? finish_task_switch+0x75/0x260 rpc_async_schedule+0x29/0x40 [sunrpc] process_one_work+0x1ad/0x370 worker_thread+0x30/0x390 ? create_worker+0x1a0/0x1a0 kthread+0x10c/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x22/0x30After applied the patch, the panic is gone.
Thanks!
MasaSignature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 8 lines · On Mon, May 11, 2020 at 09:57:45AM -0400, [email protected] wrote:
Quoted 5 lines · Trond Myklebust
Quoted 2 lines · J. Bruce Fields
Thanks, applying.--b.
You're applying? So should I remove it from the NFS client bugfixes?
No. Sorry, I responded to the wrong email!
I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.
PATCH 1/1 nfs: fix NULL deference in nfs4_get_valid_delegation
From: "J. Bruce Fields" <[email protected]>
We add the new state to the nfsi->open_states list, making it potentially visible to other threads, before we've finished initializing it.
That wasn't a problem when all the readers were also taking the i_lock (as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.
Symptoms observed were a crash when another thread called nfs4_get_valid_delegation() on a NULL inode, resulting in an oops like:
BUG: unable to handle page fault for address: ffffffffffffffb0 ... RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ... Call Trace: nfs4_open_prepare+0x80/0x1c0 [nfsv4] __rpc_execute+0x75/0x390 [sunrpc] ? finish_task_switch+0x75/0x260 rpc_async_schedule+0x29/0x40 [sunrpc] process_one_work+0x1ad/0x370 worker_thread+0x30/0x390 ? create_worker+0x1a0/0x1a0 kthread+0x10c/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x22/0x30
--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 2 lines
I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.
Here you go.
fs/nfs/nfs4state.c +1 -1
@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add stateless * and read-only stateids to the end of the list */
Signature · 1 line
2.26.2
Hello,
How about adding Cc: [email protected] in the sign-off area? 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU" was merged to v4.20 and some stable tree have 9ae075fdd190.
Quoted 27 lines · On Mon, May 11, 2020 at 10:02:48AM -0400, [email protected] wrote:
From: "J. Bruce Fields" <[email protected]>
We add the new state to the
nfsi->open_stateslist, making it potentially visible to other threads, before we've finished initializing it.That wasn't a problem when all the readers were also taking the
i_lock(as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.Symptoms observed were a crash when another thread called
nfs4_get_valid_delegation()on a NULL inode, resulting in an oops like:BUG: unable to handle page fault for address: ffffffffffffffb0 ... RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ... Call Trace: nfs4_open_prepare+0x80/0x1c0 [nfsv4] __rpc_execute+0x75/0x390 [sunrpc] ? finish_task_switch+0x75/0x260 rpc_async_schedule+0x29/0x40 [sunrpc] process_one_work+0x1ad/0x370 worker_thread+0x30/0x390 ? create_worker+0x1a0/0x1a0 kthread+0x10c/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x22/0x30
Quoted 1 line
Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"
Thanks!
Masa
Quoted 31 lines
Reviewed-by Seiichi Ikarashi <[email protected]>Tested-by Daisuke Matsuda <[email protected]>Tested-by Masayoshi Mizuma <[email protected]>Signed-off-by J. Bruce Fields <[email protected]>--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 2 lines
I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.
Here you go.
fs/nfs/nfs4state.c +1
-1@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add stateless * and read-only stateids to the end of the list */ Signature · 1 line
2.26.2
Quoted 4 lines · Masayoshi Mizuma
Hello,
How about adding Cc: [email protected] in the sign-off area?
9ae075fdd190"NFSv4: Convert open state lookup to use RCU" was merged
Makes sense to me. I'll leave it to Trond.--b.
Quoted 68 lines
to v4.20 and some stable tree have
9ae075fdd190.Quoted 27 lines · On Mon, May 11, 2020 at 10:02:48AM -0400, [email protected] wrote:
From: "J. Bruce Fields" <[email protected]>
We add the new state to the
nfsi->open_stateslist, making it potentially visible to other threads, before we've finished initializing it.That wasn't a problem when all the readers were also taking the
i_lock(as we do here), but since we switched to RCU, there's now a possibility that a reader could see the partially initialized state.Symptoms observed were a crash when another thread called
nfs4_get_valid_delegation()on a NULL inode, resulting in an oops like:BUG: unable to handle page fault for address: ffffffffffffffb0 ... RIP: 0010:nfs4_get_valid_delegation+0x6/0x30 [nfsv4] ... Call Trace: nfs4_open_prepare+0x80/0x1c0 [nfsv4] __rpc_execute+0x75/0x390 [sunrpc] ? finish_task_switch+0x75/0x260 rpc_async_schedule+0x29/0x40 [sunrpc] process_one_work+0x1ad/0x370 worker_thread+0x30/0x390 ? create_worker+0x1a0/0x1a0 kthread+0x10c/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x22/0x30Quoted 1 line
Fixes 9ae075fdd190 "NFSv4: Convert open state lookup to use RCU"Cc [email protected] # 4.20.x-Thanks!
MasaQuoted 31 lines
Reviewed-by Seiichi Ikarashi <[email protected]>Tested-by Daisuke Matsuda <[email protected]>Tested-by Masayoshi Mizuma <[email protected]>Signed-off-by J. Bruce Fields <[email protected]>--- fs/nfs/nfs4state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 2 lines
I do have a patch including the tags and oops provided by Masayoshi Mizuma, if you'd like to take that instead. See followup.--b.
Here you go.
fs/nfs/nfs4state.c +1
-1@@ -734,9 +734,9 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state = new; state->owner = owner;atomic_inc (&owner->so_count); -list_add_rcu (&state->inode_states, &nfsi->open_states);ihold (inode); state->inode = inode; +list_add_rcu (&state->inode_states, &nfsi->open_states);spin_unlock (&inode->i_lock);/* Note: The reclaim code dictates that we add stateless * and read-only stateids to the end of the list */ Signature · 1 line
2.26.2