nfs4_state_manager() vs. nfs_server_remove_lists()
Subsystems (2)
Hello,
I've been seeing a panic where nfs4_state_manager() ends up processing an v3 nfs client pointer.
The panic happens at the top of nfs4_state_manager() because clp->cl_mvops == NULL;
Looking at the pointer (via crash) it becomes obvious it is a V3 client point (AKA rpc_ops = nfs_v3_clientop)
Now the reason we are in the state manager code is a NFSv4 mount doing server discovery so it is waking the client list in nfs41_walk_client_list()
Now looking at the at the entire stack with crash, the only time that v3 client pointer appears is after nfs41_walk_client_list() has been called so I'm 99% sure the pointer is coming from the cl_share_link list.
So the question is how is that v3 client pointer on that list, in non NFS_CS_READY state.
Well, simultaneously a V3 mount is happening. In nfs_fs_mount_common() it notices there is already a existing supper block sit decides to free its server pointer so nfs_server_remove_lists() is called.
What nfs_server_remove_lists() and nfs41_walk_client_list() have in common is the nfs_client_lock spin lock.
Also the client pointer in the server pointer being freed is in a non NFS_CS_READY state
To answer the question, the v3 client pointer, in a non NFS_CS_READY state, is found by nfs41_walk_client_list() because it beat nfs_server_remove_lists() to the nfs_client_lock spin lock.
nfs41_walk_client_list() finds the uninitialized client pointer nfs_server_remove_lists() is trying to free and processes it and then fall over...
Note this was very hard to reproduce since a very large client (many cores) is needed and a very fast server and a few hours...
Question, since both v3 and v4 clients are on the cl_share_link list should there be a check in nfs41_walk_client_list() to process only v4 clients?
steved.
Quoted 42 lines · Steve Dickson
Hello,
I've been seeing a panic where
nfs4_state_manager()ends up processing an v3 nfs client pointer.The panic happens at the top of
nfs4_state_manager()becauseclp->cl_mvops== NULL;Looking at the pointer (via crash) it becomes obvious it is a V3 client point (AKA
rpc_ops=nfs_v3_clientop)Now the reason we are in the state manager code is a NFSv4 mount doing server discovery so it is waking the client list in
nfs41_walk_client_list()Now looking at the at the entire stack with crash, the only time that v3 client pointer appears is after
nfs41_walk_client_list()has been called so I'm 99% sure the pointer is coming from thecl_share_linklist.So the question is how is that v3 client pointer on that list, in non
NFS_CS_READYstate.Well, simultaneously a V3 mount is happening. In
nfs_fs_mount_common()it notices there is already a existing supper block sit decides to free its server pointer sonfs_server_remove_lists()is called.What
nfs_server_remove_lists()andnfs41_walk_client_list()have in common is thenfs_client_lockspin lock.Also the client pointer in the server pointer being freed is in a non
NFS_CS_READYstateTo answer the question, the v3 client pointer, in a non
NFS_CS_READYstate, is found bynfs41_walk_client_list()because it beatnfs_server_remove_lists()to thenfs_client_lockspin lock.
nfs41_walk_client_list()finds the uninitialized client pointernfs_server_remove_lists()is trying to free and processes it and then fall over...
Quoted 8 lines
Note this was very hard to reproduce since a very large client (many cores) is needed and a very fast server and a few hours...
Question, since both v3 and v4 clients are on the
cl_share_linklist should there be a check innfs41_walk_client_list()to process only v4 clients?
Hi Steve,
Let's just move up the test for "pos->rpc_ops != new->rpc_ops", "pos->cl_minorversion != new->cl_minorversion" and "pos->cl_proto != new->cl_proto" so that they all happen before we try to test the value of cl_cons_state.
As far as I can tell, all those values are guaranteed to be set as part of the struct nfs_client allocators, before we ever put the result on the cl_share_link list.
Cheers
Trond
Signature · 5 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
Quoted 7 lines · Trond Myklebust
Let's just move up the test for "
pos->rpc_ops!=new->rpc_ops", "pos->cl_minorversion!=new->cl_minorversion" and "pos->cl_proto!=new->cl_proto" so that they all happen before we try to test the value ofcl_cons_state.
As far as I can tell, all those values are guaranteed to be set as part of the structnfs_clientallocators, before we ever put the result on thecl_share_linklist.
The check for
if (pos->cl_cons_state > NFS_CS_READY)
then right after that check is:
if (pos->cl_cons_state != NFS_CS_READY)
continue;
confuses me... Is the second check even needed?
steved.
Quoted 12 lines · Steve Dickson
Quoted 7 lines · Trond Myklebust
Let's just move up the test for "
pos->rpc_ops!=new->rpc_ops", "pos->cl_minorversion!=new->cl_minorversion" and "pos->cl_proto!=new->cl_proto" so that they all happen before we try to test the value ofcl_cons_state.
As far as I can tell, all those values are guaranteed to be set as part of the structnfs_clientallocators, before we ever put the result on thecl_share_linklist.The check for
if (pos->cl_cons_state>NFS_CS_READY)
Quoted 8 lines
then right after that check is:
if (pos->cl_cons_state != NFS_CS_READY) continue;confuses me... Is the second check even needed?
steved.
Yes. The result of the lease_recovery could be that the nfs_client is left in a state of error if, say, we get a NFS4ERR_CLID_INUSE beastie.
Cheers
Trond
Signature · 5 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
A message that is not in the archive (sent off-list or not yet received): CADnza45-ytG-0GcfS3Q0SczekP9M+F9z5EJA0dMsbhD3V=d2Gg@mail.gmail.com
Quoted 2 lines · Fred Isaman
Was a patch ever submitted for this? I'm seeing something similar but can't find the fix upstream.
As far as I can tell, the answer is no. If you are seeing the bug, then could you please post the discussed fix?
Cheers
TRond
Quoted 46 lines
Fred
Quoted 39 lines · Trond Myklebust
Quoted 12 lines · Steve Dickson
Quoted 7 lines · Trond Myklebust
Let's just move up the test for "
pos->rpc_ops!=new->rpc_ops", "pos->cl_minorversion!=new->cl_minorversion" and "pos->cl_proto!=new->cl_proto" so that they all happen before we try to test the value ofcl_cons_state.
As far as I can tell, all those values are guaranteed to be set as part of the structnfs_clientallocators, before we ever put the result on thecl_share_linklist.The check for
if (pos->cl_cons_state>NFS_CS_READY)Quoted 8 lines
then right after that check is:
if (pos->cl_cons_state != NFS_CS_READY) continue;confuses me... Is the second check even needed?
steved.
Yes. The result of the
lease_recoverycould be that thenfs_clientis left in a state of error if, say, we get aNFS4ERR_CLID_INUSEbeastie.Cheers
Trond--
Trond MyklebustLinux NFS client maintainer, PrimaryData
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
Hello,
Quoted 5 lines · Trond Myklebust
Quoted 2 lines · Fred Isaman
Quoted 2 lines
Was a patch ever submitted for this? I'm seeing something similar but can't find the fix upstream.
As far as I can tell, the answer is no. If you are seeing the bug, then could you please post the discussed fix?
It looks like I dropped the ball... I too am see this problem and can not find the posted patch... Working on it....
steved.