deadlock in lockd
Subsystems (2)
Hi all,
I just debugged a deadlock in lockd in SLES8 (i.e 2.4.21), but I think the same problem exists in 2.6.
Here's the backtrace courtesy of sysrq:
lockd D 00000000 3648 791 1 792 795 783 (L-TLB)
Call Trace: [do_schedule+338/608] (36) [__down+131/224] (52) [__down_failed+8/12] (16) [.text.lock.svclock+5/150] (04) [vsnprintf+519/1120] (24) [nlm_traverse_files+324/368] (32) [nlmsvc_mark_resources+32/64] (12) [nlm_gc_hosts+69/384] (28) [nlm_lookup_host+139/800] (48) [nlmsvc_lookup_host+48/64] (20) [nlmsvc_create_block+145/352] (16) [posix_test_lock+132/160] (20) [nlmsvc_lock+222/784] (28) [nlm4svc_retrieve_args+199/288] (40) [nlm4svc_proc_lock+172/256] (44) [svc_process+827/1392] (56) [lockd+426/720] (40) [arch_kernel_thread+46/64] (08) [lockd+0/720] (04)
What happens is that nlmsvc_lock takes the f_sema on the file the client wishes to lock, then calls posix_test_lock, which finds there's a blocking lock. So it calls nlmsvc_create_block, which calls nlmsvc_lookup_host - and the host code decides to do a garbage collection pass.
We call nlm_traverse_files, that hits the file we're just trying to lock, and invokes nlmsvc_traverse_blocks, which will try to down f_sema once more. And there it hangs...
The attched (untested) patch changes the way we do garbage collection passes, instead of doing it inside nlm_lookup_host, nlm_gc_hosts is called from the top-level service loop in lockd now, where we don'T hold any locks. It looks saner anyway.
Comments?
Olaf
Signature · 2 lines
Olaf Kirch | --- o --- Nous sommes du soleil we love when we play [email protected] | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax
fs/lockd/host.c +12 -6
@@ -34,7 +34,7 @@ static int nrhosts;static DECLARE_MUTEX (nlm_host_sema); -static void nlm_gc_hosts (void ); +static void __nlm_gc_hosts (void );/* * Find an NLM server handle in the cache. If there is none, create it.
@@ -94,9 +94,6 @@ nlm_lookup_host(struct svc_client *clnt,/* Lock hash table */ down (&nlm_host_sema); -if (time_after_eq (jiffies, next_gc)) -nlm_gc_hosts (); -for (hp = &nlm_hosts[hash]; (host = *hp); hp = &host->h_next) {if (proto && host->h_proto != proto)continue ;
@@ -273,7 +270,7 @@ nlm_shutdown_hosts(void) }/* Then, perform a garbage collection pass */ -nlm_gc_hosts (); +__nlm_gc_hosts ();up (&nlm_host_sema);/* complain if any hosts are left */
@@ -296,7 +293,7 @@ nlm_shutdown_hosts(void)* mark & sweep for resources held by remote clients. */ static void -nlm_gc_hosts (void ) +__nlm_gc_hosts (void ) {struct nlm_host **q, *host;struct rpc_clnt *clnt;
@@ -341,3 +338,12 @@ nlm_gc_hosts(void) next_gc = jiffies +NLM_HOST_COLLECT ; } +void +nlm_gc_hosts () +{ +down (&nlm_host_sema); +if (time_after_eq (jiffies, next_gc)) +__nlm_gc_hosts (); +up (&nlm_host_sema); +} +
fs/lockd/svc.c +3 -0
@@ -154,6 +154,9 @@ lockd(struct svc_rqst *rqstp)break ; } +/* Perform hosts cache garbage collection */ +nlm_gc_hosts (); +dprintk ("lockd: request from %08x\n" , (unsigned )ntohl (rqstp->rq_addr.sin_addr.s_addr));
include/linux/lockd/lockd.h +1 -0
@@ -150,6 +150,7 @@ void nlm_rebind_host(struct nlm_hoststruct nlm_host *nlm_get_host (struct nlm_host *);void nlm_release_host (struct nlm_host *);void nlm_shutdown_hosts (void ); +void nlm_gc_hosts (void );/* * Server-side lock handling
Olaf,
Quoted 4 lines
Hi all,
I just debugged a deadlock in lockd in SLES8 (i.e 2.4.21), but I think the same problem exists in 2.6.
I found this ages ago.
Neil Brown submitted a patch on 12 Mar 2003:
Subject: [NFS] [PATCH] kNFSd - 3 of 6 - Fix deadlock problem in lockd.
Which fixes this. Apparently this made it into 2.5, but not 2.4.
He submitted the same patch on 29 Oct 2003:
Subject: [NFS] [PATCH] Fix deadlock problem in lockd.
With the comment: "(This was fixed in 2.5 8 months ago)".
You may want to check if this ever made it into the 2.4 source.
Here is that e-mail:
---
nlmsvc_lock calls nlmsvc_create_block with file->f_sema held.nlmsvc_create_block calls nlmclnt_lookup_host which might call nlm_gc_hosts which might, eventually, try to claim file->f_sema for the same file -> deadlock.
nlmsvc_create_block does not need any protection under any lock as lockd is single-threaded and _create_block only plays with internal data structures.
So we release the f_sema before calling in, and make sure it gets claimed again afterwards.
(This was fixed in 2.5 8 months ago)
fs/lockd/svclock.c renamed from fs/lockd/svclock.c~current~ +6 -2
@@ -317,8 +317,6 @@ nlmsvc_lock(struct svc_rqst *rqstp, stru (long long )lock->fl.fl_end, wait); -/* Lock file against concurrent access */ -down (&file->f_sema);/* Get existing block (in case client is busy-waiting) */ block =nlmsvc_lookup_block (file, lock,0 );
@@ -326,6 +324,9 @@ nlmsvc_lock(struct svc_rqst *rqstp, stru lock->fl.fl_flags |=FL_LOCKD ; again: +/* Lock file against concurrent access */ +down (&file->f_sema); +if (!(conflock =posix_test_lock (&file->f_file, &lock->fl))) { error =posix_lock_file (&file->f_file, &lock->fl,0 );
@@ -358,7 +359,10 @@ again:/* If we don't have a block, create and initialize it. Then * retry because we may have slept in kmalloc. */ +/* We have to release f_sema as nlmsvc_create_block may try to +* claim it while doing host garbage collection */ if (block ==NULL ) { +up (&file->f_sema);dprintk ("lockd: blocking on this lock (allocating).\n" );if (!(block =nlmsvc_create_block (rqstp, file, lock, cookie)))return nlm_lck_denied_nolocks;
Signature · 13 lines
Daniel K. Forrest Laboratory for Molecular and [email protected] Computational Genomics University of Wisconsin, Madison ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line · Daniel Forrest
Subject: [NFS] [PATCH] kNFSd - 3 of 6 - Fix deadlock problem in lockd.
I see... I would have been surprised if I had been the first one to run into this.
Still I think it's cleaner to move the nlm_gc_hosts calls out of nlm_lookup_host.
Olaf
Signature · 12 lines
Olaf Kirch | --- o --- Nous sommes du soleil we love when we play [email protected] | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs