asynchronous destroy messages
Subsystems (2)
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
--b.
Quoted 7 lines · bfields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
In other words--when does the task that's created to send the destroy message get freed, and if that doesn't happen till after the rpc client and associated objects are freed, is there a risk of someone trying to access those objects through fields in that task?
--b.
Quoted 7 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
--b.
Quoted 9 lines · J. Bruce Fields
Quoted 7 lines · bfields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Quoted 6 lines
In other words--when does the task that's created to send the destroy message get freed, and if that doesn't happen till after the rpc client and associated objects are freed, is there a risk of someone trying to access those objects through fields in that task?
--b.
Are you talking about the rpc_task? That gets destroyed in the normal fashion, and since the collection of all rpc_tasks should hold a reference to the rpc_client, there shouldn't normally be any ordering problems there.
Note however that we do some magic in 'rpc_free_auth' to extend the natural lifetime of the rpc_client beyond that of the NFS 'shutdown'.
Quoted 6 lines
Quoted 6 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
--b.
Is the new code touching the destroy path? If so, how, and what are the new assumptions that are being made?
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
A message that is not in the archive (sent off-list or not yet received): 1205882854.15178.17.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · J. Bruce Fields
Quoted 7 lines · bfields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
In other words--when does the task that's created to send the destroy message get freed, and if that doesn't happen till after the rpc client and associated objects are freed, is there a risk of someone trying to access those objects through fields in that task?
Quoted 6 lines
Quoted 1 line
--b.
Are you talking about the
rpc_task? That gets destroyed in the normal fashion, and since the collection of allrpc_tasksshould hold a reference to therpc_client, there shouldn't normally be any ordering problems there.
Oh, OK, got it. Like Ogla says, we were getting oopses due to references to an rpc_stats structure that was freed on the assumption it was safe to do so after rpc_shutdown_client() returned.
(In the case of the nfs client, the rpc_stats structure is declared statically. But isn't that memory freed when the module is removed?)
--b.
Quoted 21 lines
Note however that we do some magic in '
rpc_free_auth' to extend the natural lifetime of therpc_clientbeyond that of the NFS 'shutdown'.Quoted 6 lines
Quoted 6 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
--b.
Is the new code touching the destroy path? If so, how, and what are the new assumptions that are being made?
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 8 lines · J. Bruce Fields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Quoted 6 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.
put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.
Quoted 21 lines · Olga Kornievskaia
Quoted 8 lines · J. Bruce Fields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Quoted 6 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
What I'm seeing is that
nfs4_clientstructure goes away while anrpc_clientis still active.nfs4_clientandrpc_clientshare a pointer to therpc_statstructure. so whennfs4_clientmemory goes away, therpc_clientoopses trying to dereference something withincl_stats.
Quoted 6 lines
put_nfs4_client()causesrpc_shutdown_client()causes an async destroy context message. that message shares therpc_statsmemory with thenfs4_clientmemory that is currently being released. since the task is asynchronous,put_nfs4_client()completes and memory goes away. the task that's handling destroy context message wakes up (usually incall_timeoutorcall_refresh) and tries to dereferencecl_stats.
clnt->cl_stats is supposed to point to a statically allocated structure (in this case 'nfs_rpcstat'). While that can, in theory, disappear if the user removes the NFS module, in practice that is very unlikely. I therefore think you are rather seeing some sort of memory corruption issue that is affecting the rpc_client.
Are you seeing this with the 'devel' branch from my git tree, or does it only affect your patched kernel. If the latter, may we see your patches?
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
A message that is not in the archive (sent off-list or not yet received): 1205937584.8388.40.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 19 lines · Trond Myklebust
Quoted 17 lines · Olga Kornievskaia
Quoted 15 lines · J. Bruce Fields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
Quoted 7 lines
Quoted 4 lines
What I'm seeing is that
nfs4_clientstructure goes away while anrpc_clientis still active.nfs4_clientandrpc_clientshare a pointer to therpc_statstructure. so whennfs4_clientmemory goes away, therpc_clientoopses trying to dereference something withincl_stats.
put_nfs4_client()causesrpc_shutdown_client()causes an async destroy context message. that message shares therpc_statsmemory with thenfs4_clientmemory that is currently being released. since the task is asynchronous,put_nfs4_client()completes and memory goes away. the task that's handling destroy context message wakes up (usually incall_timeoutorcall_refresh) and tries to dereferencecl_stats.
clnt->cl_statsis supposed to point to a statically allocated structure
that is not true for the callback. look at do_probe_callback().
Quoted 10 lines
(in this case '
nfs_rpcstat'). While that can, in theory, disappear if the user removes the NFS module, in practice that is very unlikely. I therefore think you are rather seeing some sort of memory corruption issue that is affecting therpc_client.Are you seeing this with the 'devel' branch from my git tree, or does it only affect your patched kernel. If the latter, may we see your patches?
Quoted 34 lines · Olga Kornievskaia
Quoted 32 lines · Trond Myklebust
Quoted 29 lines · Olga Kornievskaia
Quoted 8 lines · J. Bruce Fields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Quoted 7 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
What I'm seeing is that
nfs4_clientstructure goes away while anrpc_clientis still active.nfs4_clientandrpc_clientshare a pointer to therpc_statstructure. so whennfs4_clientmemory goes away, therpc_clientoopses trying to dereference something withincl_stats.
put_nfs4_client()causesrpc_shutdown_client()causes an async destroy context message. that message shares therpc_statsmemory with thenfs4_clientmemory that is currently being released. since the task is asynchronous,put_nfs4_client()completes and memory goes away. the task that's handling destroy context message wakes up (usually incall_timeoutorcall_refresh) and tries to dereferencecl_stats.
Quoted 3 lines
Quoted 2 lines
clnt->cl_statsis supposed to point to a statically allocated structurethat is not true for the callback. look at
do_probe_callback().
I can see why the rpc_program may needs to change, since the client doesn't know a priori what program number the client will choose, but why is there a need for a dynamically allocated cb->cb_stat?
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
A message that is not in the archive (sent off-list or not yet received): 1205941115.8388.81.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 36 lines · Trond Myklebust
Quoted 34 lines · Olga Kornievskaia
Quoted 32 lines · Trond Myklebust
Quoted 29 lines · Olga Kornievskaia
Quoted 8 lines · J. Bruce Fields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Quoted 7 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
What I'm seeing is that
nfs4_clientstructure goes away while anrpc_clientis still active.nfs4_clientandrpc_clientshare a pointer to therpc_statstructure. so whennfs4_clientmemory goes away, therpc_clientoopses trying to dereference something withincl_stats.
put_nfs4_client()causesrpc_shutdown_client()causes an async destroy context message. that message shares therpc_statsmemory with thenfs4_clientmemory that is currently being released. since the task is asynchronous,put_nfs4_client()completes and memory goes away. the task that's handling destroy context message wakes up (usually incall_timeoutorcall_refresh) and tries to dereferencecl_stats.
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
clnt->cl_statsis supposed to point to a statically allocated structurethat is not true for the callback. look at
do_probe_callback().I can see why the
rpc_programmay needs to change, since the client doesn't know a priori what program number the client will choose, but why is there a need for a dynamically allocatedcb->cb_stat?
I agree--it's not necessary. I was delaying changing that just because I was confused as to why that memory was still being used after rpc_shutdown_client() returned. Now I understand--thanks.
But are you sure the module-unload race isn't a concern? Unloading the nfs module soon after the last nfs mount might not be so rare, and those tasks could hang around for quite a while, right?
--b.
Quoted 8 lines · J. Bruce Fields
When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.
If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?
Quoted 5 lines
Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.
What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.
put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.