Skip to contentSkip to conversation

asynchronous destroy messages

DISCUSSION J. Bruce Fields 3 participants 10 messages Mar 19, 2008 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

--b.

#1 lore raw
Quoted 7 lines · bfields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

In other words--when does the task that's created to send the destroy message get freed, and if that doesn't happen till after the rpc client and associated objects are freed, is there a risk of someone trying to access those objects through fields in that task?

--b.

Quoted 7 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

--b.

#2 lore raw
Quoted 9 lines · J. Bruce Fields
Quoted 7 lines · bfields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Quoted 6 lines

In other words--when does the task that's created to send the destroy message get freed, and if that doesn't happen till after the rpc client and associated objects are freed, is there a risk of someone trying to access those objects through fields in that task?

--b.

Are you talking about the rpc_task? That gets destroyed in the normal fashion, and since the collection of all rpc_tasks should hold a reference to the rpc_client, there shouldn't normally be any ordering problems there.

Note however that we do some magic in 'rpc_free_auth' to extend the natural lifetime of the rpc_client beyond that of the NFS 'shutdown'.

Quoted 6 lines
Quoted 6 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

--b.

Is the new code touching the destroy path? If so, how, and what are the new assumptions that are being made?

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

A message that is not in the archive (sent off-list or not yet received): 1205882854.15178.17.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#4 lore raw
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · J. Bruce Fields
Quoted 7 lines · bfields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

In other words--when does the task that's created to send the destroy message get freed, and if that doesn't happen till after the rpc client and associated objects are freed, is there a risk of someone trying to access those objects through fields in that task?

Quoted 6 lines
Quoted 1 line

--b.

Are you talking about the rpc_task? That gets destroyed in the normal fashion, and since the collection of all rpc_tasks should hold a reference to the rpc_client, there shouldn't normally be any ordering problems there.

Oh, OK, got it. Like Ogla says, we were getting oopses due to references to an rpc_stats structure that was freed on the assumption it was safe to do so after rpc_shutdown_client() returned.

(In the case of the nfs client, the rpc_stats structure is declared statically. But isn't that memory freed when the module is removed?)

--b.

Quoted 21 lines

Note however that we do some magic in 'rpc_free_auth' to extend the natural lifetime of the rpc_client beyond that of the NFS 'shutdown'.

Quoted 6 lines
Quoted 6 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

--b.

Is the new code touching the destroy path? If so, how, and what are the new assumptions that are being made?

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#5 lore raw
In reply to J. Bruce Fields
Quoted 8 lines · J. Bruce Fields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Quoted 6 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.

put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.

#6 lore raw
Quoted 21 lines · Olga Kornievskaia
Quoted 8 lines · J. Bruce Fields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Quoted 6 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.

Quoted 6 lines

put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.

clnt->cl_stats is supposed to point to a statically allocated structure (in this case 'nfs_rpcstat'). While that can, in theory, disappear if the user removes the NFS module, in practice that is very unlikely. I therefore think you are rather seeing some sort of memory corruption issue that is affecting the rpc_client.

Are you seeing this with the 'devel' branch from my git tree, or does it only affect your patched kernel. If the latter, may we see your patches?

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

A message that is not in the archive (sent off-list or not yet received): 1205937584.8388.40.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#8 lore raw
Quoted 19 lines · Trond Myklebust
Quoted 17 lines · Olga Kornievskaia
Quoted 15 lines · J. Bruce Fields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

Quoted 7 lines
Quoted 4 lines

What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.

put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.

clnt->cl_stats is supposed to point to a statically allocated structure

that is not true for the callback. look at do_probe_callback().

Quoted 10 lines

(in this case 'nfs_rpcstat'). While that can, in theory, disappear if the user removes the NFS module, in practice that is very unlikely. I therefore think you are rather seeing some sort of memory corruption issue that is affecting the rpc_client.

Are you seeing this with the 'devel' branch from my git tree, or does it only affect your patched kernel. If the latter, may we see your patches?

#9 lore raw
Quoted 34 lines · Olga Kornievskaia
Quoted 32 lines · Trond Myklebust
Quoted 29 lines · Olga Kornievskaia
Quoted 8 lines · J. Bruce Fields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Quoted 7 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.

put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.

Quoted 3 lines
Quoted 2 lines

clnt->cl_stats is supposed to point to a statically allocated structure

that is not true for the callback. look at do_probe_callback().

I can see why the rpc_program may needs to change, since the client doesn't know a priori what program number the client will choose, but why is there a need for a dynamically allocated cb->cb_stat?

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

A message that is not in the archive (sent off-list or not yet received): 1205941115.8388.81.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#11 lore raw
Quoted 36 lines · Trond Myklebust
Quoted 34 lines · Olga Kornievskaia
Quoted 32 lines · Trond Myklebust
Quoted 29 lines · Olga Kornievskaia
Quoted 8 lines · J. Bruce Fields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Quoted 7 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.

put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

clnt->cl_stats is supposed to point to a statically allocated structure

that is not true for the callback. look at do_probe_callback().

I can see why the rpc_program may needs to change, since the client doesn't know a priori what program number the client will choose, but why is there a need for a dynamically allocated cb->cb_stat?

I agree--it's not necessary. I was delaying changing that just because I was confused as to why that memory was still being used after rpc_shutdown_client() returned. Now I understand--thanks.

But are you sure the module-unload race isn't a concern? Unloading the nfs module soon after the last nfs mount might not be so rare, and those tasks could hang around for quite a while, right?

--b.

#12 lore raw
In reply to J. Bruce Fields
Quoted 8 lines · J. Bruce Fields

When an rpc client is shut down, gss destroy messages are sent out asynchronously, and nobody waits for them.

If those rpc messages arrive after the client is completely shut down, I assume they just get dropped by the networking code? Is it possible for them to arrive while we're still in the process of shutting down, and if so, what makes this safe?

Quoted 5 lines

Olga's seeing some odd oopses on shutdown after testing our gss callback code. And of course it's probably our callback patches at fault, but I was starting to wonder if there was a problem with those destroy messages arriving at the wrong moment. Any pointers welcomed.

What I'm seeing is that nfs4_client structure goes away while an rpc_client is still active. nfs4_client and rpc_client share a pointer to the rpc_stat structure. so when nfs4_client memory goes away, the rpc_client oopses trying to dereference something within cl_stats.

put_nfs4_client() causes rpc_shutdown_client() causes an async destroy context message. that message shares the rpc_stats memory with the nfs4_client memory that is currently being released. since the task is asynchronous, put_nfs4_client() completes and memory goes away. the task that's handling destroy context message wakes up (usually in call_timeout or call_refresh) and tries to dereference cl_stats.