nfsd leaks 3 sunrpc module references

PATCH In review Stefan Hajnoczi 3 participants 6 messages Sep 11, 2015 – Jan 5, 2016

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Stefan Hajnoczi

author

2015-09-11 10:45 UTC #0 lore raw

I'm having a little trouble debugging this sunrpc module refcnt leak in nfsd. Perhaps a fresh pair of eyes can help.

nfsd leaks 3 references to the sunrpc module here:

  # echo -n "asdf 1234" >/proc/fs/nfsd/portlist
  bash: echo: write error: Protocol not supported

Now stop nfsd and try unloading the kernel modules:

  # systemctl stop nfs-server
  # systemctl stop nfs
  # systemctl stop proc-fs-nfsd.mount
  # systemctl stop var-lib-nfs-rpc_pipefs.mount
  # rmmod nfsd
  # rmmod nfs_acl
  # rmmod lockd
  # rmmod auth_rpcgss
  # rmmod sunrpc
  rmmod: ERROR: Module sunrpc is in use
  # lsmod | grep rpc
  sunrpc                315392  3

I have reproduced this from source with a tree based on Linux 4.2-rc3 and also with Linux 4.1.6-200.fc22.x86_64.

The following unmatched module_get() calls are recorded by perf record -e module:module_get -e module:module_put. Perhaps the backtrace is a little mangled because I don't see xdr_reserve_space() calls in svc_create_pooled() (even indirectly):

nfsd 4291 [002] 87.914752: module:module_get: sunrpc call_site=0xffffffffa031c86as refcnt=7

                  318d93 try_module_get (vmlinux)
                    98da xs_tcp_setup_socket ([sunrpc])
                    5ca5 xprt_create_transport ([sunrpc])
                    34ad rpc_create ([sunrpc])
                   1892a rpcb_getport_async ([sunrpc])
                   196ea xdr_reserve_space ([sunrpc])
                   128dc svc_create_pooled ([sunrpc])
                     686 nfsd_create_serv ([nfsd])
                    1be9 write_ports ([nfsd])
                    182c nfsctl_transaction_write ([nfsd])
                  410d87 __vfs_write (vmlinux)
                  411706 vfs_write (vmlinux)
                  4123f5 sys_write (vmlinux)
                  95ed95 tracesys_phase2 (vmlinux)
                   f2ff0 __GI___libc_write (/usr/lib64/libc-2.21.so)

nfsd 4291 [002] 87.914775: module:module_get: sunrpc call_site=0xffffffffa03228a5s refcnt=8

                  318d93 try_module_get (vmlinux)
                    f915 rpcauth_cache_do_shrink ([sunrpc])
                    2bbd rpc_new_client ([sunrpc])
                    337a rpc_create_xprt ([sunrpc])
                    34d6 rpc_create ([sunrpc])
                   1892a rpcb_getport_async ([sunrpc])
                   196ea xdr_reserve_space ([sunrpc])
                   128dc svc_create_pooled ([sunrpc])
                     686 nfsd_create_serv ([nfsd])
                    1be9 write_ports ([nfsd])
                    182c nfsctl_transaction_write ([nfsd])
                  410d87 __vfs_write (vmlinux)
                  411706 vfs_write (vmlinux)
                  4123f5 sys_write (vmlinux)
                  95ed95 tracesys_phase2 (vmlinux)
                   f2ff0 __GI___libc_write (/usr/lib64/libc-2.21.so)

nfsd 4291 [002] 87.914936: module:module_get: sunrpc call_site=0xffffffffa03228a5s refcnt=9

                  318d93 try_module_get (vmlinux)
                    f915 rpcauth_cache_do_shrink ([sunrpc])
                    2bbd rpc_new_client ([sunrpc])
                    2ead __rpc_clone_client ([sunrpc])
                    35e7 rpc_bind_new_program ([sunrpc])
                   18949 rpcb_getport_async ([sunrpc])
                   196ea xdr_reserve_space ([sunrpc])
                   128dc svc_create_pooled ([sunrpc])
                     686 nfsd_create_serv ([nfsd])
                    1be9 write_ports ([nfsd])
                    182c nfsctl_transaction_write ([nfsd])
                  410d87 __vfs_write (vmlinux)
                  411706 vfs_write (vmlinux)
                  4123f5 sys_write (vmlinux)
                  95ed95 tracesys_phase2 (vmlinux)
                   f2ff0 __GI___libc_write (/usr/lib64/libc-2.21.so)

In this trace I modified nfs-utils nfsd to write to portlist but you get the same result with echo from a shell (as shown above).

Any ideas?

Stefan

Kinglong Mee

2016-01-04 03:15 UTC #1 lore raw

PATCH 1/1 nfsd: Fix nfsd leaks sunrpc module references

Stefan Hajnoczi reports,
nfsd leaks 3 references to the sunrpc module here:

  # echo -n "asdf 1234" >/proc/fs/nfsd/portlist
  bash: echo: write error: Protocol not supported

Now stop nfsd and try unloading the kernel modules:

  # systemctl stop nfs-server
  # systemctl stop nfs
  # systemctl stop proc-fs-nfsd.mount
  # systemctl stop var-lib-nfs-rpc_pipefs.mount
  # rmmod nfsd
  # rmmod nfs_acl
  # rmmod lockd
  # rmmod auth_rpcgss
  # rmmod sunrpc
  rmmod: ERROR: Module sunrpc is in use
  # lsmod | grep rpc
  sunrpc                315392  3

It is caused by nfsd don't cleanup rpcb program for nfsd when destroying svc service after creating xprt fail.

Reported-by Stefan Hajnoczi <[email protected]>

Signed-off-by Kinglong Mee <[email protected]>

---
 fs/nfsd/nfssvc.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

fs/nfsd/nfssvc.c +3 -4

index ad4e237..543de5f 100644
@@ -314,14 +314,13 @@ static void nfsd_last_thread(struct svc_serv *serv, struct net *net)
 	 * write_ports can create the server without actually starting
 	 * any threads--if we get shut down before any threads are
 	 * started, then nfsd_last_thread will be run before any of this
-	 * other initialization has been done.
+	 * other initialization has been done except the rpcb information.
 	 */
+	svc_rpcb_cleanup(serv, net);
 	if (!nn->nfsd_net_up)
 		return;
-	nfsd_shutdown_net(net);
-
-	svc_rpcb_cleanup(serv, net);
 
+	nfsd_shutdown_net(net);
 	printk(KERN_WARNING "nfsd: last server has exited, flushing export "
 			    "cache\n");
 	nfsd_export_flush(net);

Signature · 1 line

2.5.0

Stefan Hajnoczi

author Tested-by on v1 1/1

2016-01-04 09:03 UTC #2 lore raw

Re: [PATCH] nfsd: Fix nfsd leaks sunrpc module references

Quoted 21 lines · Kinglong Mee

Stefan Hajnoczi reports,
nfsd leaks 3 references to the sunrpc module here:

  # echo -n "asdf 1234" >/proc/fs/nfsd/portlist
  bash: echo: write error: Protocol not supported

Now stop nfsd and try unloading the kernel modules:

  # systemctl stop nfs-server
  # systemctl stop nfs
  # systemctl stop proc-fs-nfsd.mount
  # systemctl stop var-lib-nfs-rpc_pipefs.mount
  # rmmod nfsd
  # rmmod nfs_acl
  # rmmod lockd
  # rmmod auth_rpcgss
  # rmmod sunrpc
  rmmod: ERROR: Module sunrpc is in use
  # lsmod | grep rpc
  sunrpc                315392  3

Quoted 5 lines

It is caused by nfsd don't cleanup rpcb program for nfsd when destroying svc service after creating xprt fail.
Reported-by Stefan Hajnoczi <[email protected]>
Signed-off-by Kinglong Mee <[email protected]>

Thanks for looking into this!

Tested-by Stefan Hajnoczi <[email protected]>

J. Bruce Fields

2016-01-04 22:39 UTC #3 lore raw

Re: [PATCH] nfsd: Fix nfsd leaks sunrpc module references

In reply to Kinglong Mee

Quoted 35 lines · Kinglong Mee

Stefan Hajnoczi reports,
nfsd leaks 3 references to the sunrpc module here:
  # echo -n "asdf 1234" >/proc/fs/nfsd/portlist
  bash: echo: write error: Protocol not supported
Now stop nfsd and try unloading the kernel modules:
  # systemctl stop nfs-server
  # systemctl stop nfs
  # systemctl stop proc-fs-nfsd.mount
  # systemctl stop var-lib-nfs-rpc_pipefs.mount
  # rmmod nfsd
  # rmmod nfs_acl
  # rmmod lockd
  # rmmod auth_rpcgss
  # rmmod sunrpc
  rmmod: ERROR: Module sunrpc is in use
  # lsmod | grep rpc
  sunrpc                315392  3
It is caused by nfsd don't cleanup rpcb program for nfsd when destroying svc service after creating xprt fail.
Reported-by Stefan Hajnoczi <[email protected]>
Signed-off-by Kinglong Mee <[email protected]>
---
 fs/nfsd/nfssvc.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
fs/nfsd/nfssvc.c +0 -0
index ad4e237..543de5f 100644

fs/nfsd/nfssvc.c static void nfsd_last_thread(struct svc_serv *serv, struct net *net) · line 319

Quoted 7 lines

fs/nfsd/nfssvc.c +2 -1

@@ -314,14 +314,13 @@ static void nfsd_last_thread(struct svc_serv *serv, struct net *net)
 	 * write_ports can create the server without actually starting
 	 * any threads--if we get shut down before any threads are
 	 * started, then nfsd_last_thread will be run before any of this
-	 * other initialization has been done.
+	 * other initialization has been done except the rpcb information.
 	 */
+	svc_rpcb_cleanup(serv, net);

Are you sure this is right?

For example, nfsd_create_serv calls svc_destroy on svc_bind failure, is it OK to call svc_rpcb_cleanup in that case?

--b.

Quoted 12 lines

fs/nfsd/nfssvc.c +1 -3

 	if (!nn->nfsd_net_up)
 		return;
-	nfsd_shutdown_net(net);
-
-	svc_rpcb_cleanup(serv, net);
 
+	nfsd_shutdown_net(net);
 	printk(KERN_WARNING "nfsd: last server has exited, flushing export "
 			    "cache\n");
 	nfsd_export_flush(net);

Signature · 1 line

2.5.0

Kinglong Mee

2016-01-05 00:49 UTC #4 lore raw

Re: [PATCH] nfsd: Fix nfsd leaks sunrpc module references

Quoted 46 lines · J. Bruce Fields

Quoted 35 lines · Kinglong Mee
Stefan Hajnoczi reports,
nfsd leaks 3 references to the sunrpc module here:
  # echo -n "asdf 1234" >/proc/fs/nfsd/portlist
  bash: echo: write error: Protocol not supported
Now stop nfsd and try unloading the kernel modules:
  # systemctl stop nfs-server
  # systemctl stop nfs
  # systemctl stop proc-fs-nfsd.mount
  # systemctl stop var-lib-nfs-rpc_pipefs.mount
  # rmmod nfsd
  # rmmod nfs_acl
  # rmmod lockd
  # rmmod auth_rpcgss
  # rmmod sunrpc
  rmmod: ERROR: Module sunrpc is in use
  # lsmod | grep rpc
  sunrpc                315392  3
It is caused by nfsd don't cleanup rpcb program for nfsd when destroying svc service after creating xprt fail.
Reported-by Stefan Hajnoczi <[email protected]>
Signed-off-by Kinglong Mee <[email protected]>
---
 fs/nfsd/nfssvc.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
fs/nfsd/nfssvc.c +0 -0
index ad4e237..543de5f 100644
fs/nfsd/nfssvc.c static void nfsd_last_thread(struct svc_serv *serv, struct net *net) · line 319
Quoted 7 lines
fs/nfsd/nfssvc.c +2 -1
@@ -314,14 +314,13 @@ static void nfsd_last_thread(struct svc_serv *serv, struct net *net)
 	 * write_ports can create the server without actually starting
 	 * any threads--if we get shut down before any threads are
 	 * started, then nfsd_last_thread will be run before any of this
-	 * other initialization has been done.
+	 * other initialization has been done except the rpcb information.
 	 */
+	svc_rpcb_cleanup(serv, net);
Are you sure this is right?

Quoted 2 lines

For example, nfsd_create_serv calls svc_destroy on svc_bind failure, is it OK to call svc_rpcb_cleanup in that case?

The svc_rpcb_cleanup in nfsd_last_thread is called in svc_shutdown_net, svc_destroy never call it.

svc_shutdown_net is called in nfsd_destroy only nfsd_serv->sv_nrthreads == 1.

So that, it's ok calling svc_rpcb_cleanup in nfsd_last_thread always.

Is there anything I missing?

thanks,
Kinglong Mee

Quoted 16 lines

--b.

Quoted 12 lines

fs/nfsd/nfssvc.c +1 -3

 	if (!nn->nfsd_net_up)
 		return;
-	nfsd_shutdown_net(net);
-
-	svc_rpcb_cleanup(serv, net);
 
+	nfsd_shutdown_net(net);
 	printk(KERN_WARNING "nfsd: last server has exited, flushing export "
 			    "cache\n");
 	nfsd_export_flush(net);

Signature · 1 line

2.5.0

J. Bruce Fields

2016-01-05 22:44 UTC #5 lore raw

Re: [PATCH] nfsd: Fix nfsd leaks sunrpc module references

Quoted 50 lines · Kinglong Mee

Quoted 46 lines · J. Bruce Fields
Quoted 35 lines · Kinglong Mee
Stefan Hajnoczi reports,
nfsd leaks 3 references to the sunrpc module here:
  # echo -n "asdf 1234" >/proc/fs/nfsd/portlist
  bash: echo: write error: Protocol not supported
Now stop nfsd and try unloading the kernel modules:
  # systemctl stop nfs-server
  # systemctl stop nfs
  # systemctl stop proc-fs-nfsd.mount
  # systemctl stop var-lib-nfs-rpc_pipefs.mount
  # rmmod nfsd
  # rmmod nfs_acl
  # rmmod lockd
  # rmmod auth_rpcgss
  # rmmod sunrpc
  rmmod: ERROR: Module sunrpc is in use
  # lsmod | grep rpc
  sunrpc                315392  3
It is caused by nfsd don't cleanup rpcb program for nfsd when destroying svc service after creating xprt fail.
Reported-by Stefan Hajnoczi <[email protected]>
Signed-off-by Kinglong Mee <[email protected]>
---
 fs/nfsd/nfssvc.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
fs/nfsd/nfssvc.c +0 -0
index ad4e237..543de5f 100644
fs/nfsd/nfssvc.c static void nfsd_last_thread(struct svc_serv *serv, struct net *net) · line 319
Quoted 7 lines
fs/nfsd/nfssvc.c +2 -1
@@ -314,14 +314,13 @@ static void nfsd_last_thread(struct svc_serv *serv, struct net *net)
 	 * write_ports can create the server without actually starting
 	 * any threads--if we get shut down before any threads are
 	 * started, then nfsd_last_thread will be run before any of this
-	 * other initialization has been done.
+	 * other initialization has been done except the rpcb information.
 	 */
+	svc_rpcb_cleanup(serv, net);
Are you sure this is right?
Quoted 2 lines
For example, nfsd_create_serv calls svc_destroy on svc_bind failure, is it OK to call svc_rpcb_cleanup in that case?

Quoted 8 lines

The svc_rpcb_cleanup in nfsd_last_thread is called in svc_shutdown_net, svc_destroy never call it.
svc_shutdown_net is called in nfsd_destroy only nfsd_serv->sv_nrthreads == 1.
So that, it's ok calling svc_rpcb_cleanup in nfsd_last_thread always.
Is there anything I missing?

Oops, somehow I got nfsd_destroy and svc_destroy confused.

Applying. Thanks for your patience!

--b.

Quoted 20 lines

thanks,
Kinglong Mee

Quoted 16 lines

--b.

Quoted 12 lines

fs/nfsd/nfssvc.c +1 -3

 	if (!nn->nfsd_net_up)
 		return;
-	nfsd_shutdown_net(net);
-
-	svc_rpcb_cleanup(serv, net);
 
+	nfsd_shutdown_net(net);
 	printk(KERN_WARNING "nfsd: last server has exited, flushing export "
 			    "cache\n");
 	nfsd_export_flush(net);

Signature · 1 line

2.5.0