From: Andrew W Elble Date: Mon, 12 Oct 2015 15:15:14 -0400 Subject: [Cluster-devel] [GFS2 PATCH 06/15] GFS2: Prevent gl_delete work for re-used inodes In-Reply-To: (Andrew W. Elble's message of "Tue, 6 Oct 2015 15:33:41 -0400") References: <1444059397-4705-1-git-send-email-rpeterso@redhat.com> <1444059397-4705-7-git-send-email-rpeterso@redhat.com> <1092374767.41518211.1444158139643.JavaMail.zimbra@redhat.com> Message-ID: List-Id: To: cluster-devel.redhat.com MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Bob, The deadlock is slightly different - but still occurs with your patches in place. I'm pretty sure this is what's happening: nfsd (unlink) has i_mutex fs/nfsd/vfs.c:nfsd_unlink() -> fh_lock_nested(fhp, I_MUTEX_PARENT); nfsd (lookup) gets a glock fs/gfs2/inode.c:gfs2_inode_lookup() -> error = gfs2_glock_nq_init(io_gl, LM_ST_SHARED, GL_EXACT, &ip->i_iopen_gh); nfsd (unlink) needs conflicting glock fs/gfs2/super.c:gfs2_evict_inode() -> if (ip->i_iopen_gh.gh_gl && test_bit(HIF_HOLDER, &ip->i_iopen_gh.gh_iflags)) { gfs2_glock_dq_wait(&ip->i_iopen_gh); client (lookup) waiting on i_mutex fs/nfsd/vfs.c:nfsd_lookup_dentry() -> fh_lock_nested(fhp, I_MUTEX_PARENT); -> deadlock G: s:EX n:2/2f4935a f:yIqob t:EX d:EX/0 a:0 v:0 r:10 m:200 H: s:EX f:H e:0 p:34415 [nfsd] gfs2_evict_inode+0x160/0x4d0 [gfs2] I: n:329967/49582938 t:8 f:0x00 d:0x00000000 s:500 G: s:SH n:5/2f4935a f:DIqob t:SH d:UN/3833211000 a:0 v:0 r:4 m:200 H: s:SH f:EH e:0 p:34414 [nfsd] gfs2_inode_lookup+0xee/0x1f0 [gfs2] nfs client host a: 313189 11:47:26.499858000 x.y.z.a -> x.y.z.q NFS 342 V4 Call REMOVE DH: 0x46fbb746/0353fd0043cc75dd8203b16b5bd4c197-cache-mod_custom-e2acfa1435db9601a6b9645e9f8be86f.php nfs client host b: 539106 11:49:28.390748000 x.y.z.b -> x.y.z.q NFS 362 V4 Call LOOKUP DH: 0x46fbb746/0353fd0043cc75dd8203b16b5bd4c197-cache-mod_custom-ea521049a8a64b325300eab10b4ac871.php crash> bt PID: 34414 TASK: ffff881fdc7428b0 CPU: 38 COMMAND: "nfsd" #0 [ffff881f2da57b70] __schedule at ffffffff8165bbc4 #1 [ffff881f2da57bc0] schedule at ffffffff8165c267 #2 [ffff881f2da57be0] schedule_preempt_disabled at ffffffff8165c59e #3 [ffff881f2da57bf0] __mutex_lock_slowpath at ffffffff8165e0d5 #4 [ffff881f2da57c50] mutex_lock at ffffffff8165e173 #5 [ffff881f2da57c70] nfsd_lookup_dentry at ffffffffa035454f [nfsd] #6 [ffff881f2da57cf0] nfsd_lookup at ffffffffa0354989 [nfsd] #7 [ffff881f2da57d40] nfsd4_lookup at ffffffffa0361a2a [nfsd] #8 [ffff881f2da57d50] nfsd4_proc_compound at ffffffffa0363d57 [nfsd] #9 [ffff881f2da57db0] nfsd_dispatch at ffffffffa034ff83 [nfsd] #10 [ffff881f2da57df0] svc_process_common at ffffffffa0188260 [sunrpc] #11 [ffff881f2da57e60] svc_process at ffffffffa0188603 [sunrpc] #12 [ffff881f2da57e90] nfsd at ffffffffa034f98f [nfsd] #13 [ffff881f2da57ec0] kthread at ffffffff81096989 #14 [ffff881f2da57f50] ret_from_fork at ffffffff81660462 crash> bt PID: 34415 TASK: ffff881fec1a6c80 CPU: 24 COMMAND: "nfsd" #0 [ffff881f2db779f0] __schedule at ffffffff8165bbc4 #1 [ffff881f2db77a40] schedule at ffffffff8165c267 #2 [ffff881f2db77a60] bit_wait at ffffffff8165ca7c #3 [ffff881f2db77a70] __wait_on_bit at ffffffff8165c705 #4 [ffff881f2db77ac0] out_of_line_wait_on_bit at ffffffff8165c7a2 #5 [ffff881f2db77b30] gfs2_glock_dq_wait at ffffffffa0850553 [gfs2] #6 [ffff881f2db77b50] gfs2_evict_inode at ffffffffa08697d5 [gfs2] #7 [ffff881f2db77bf0] evict at ffffffff811fcbcb #8 [ffff881f2db77c20] iput at ffffffff811fd52b #9 [ffff881f2db77c50] d_delete at ffffffff811f8e38 #10 [ffff881f2db77c80] vfs_unlink at ffffffff811edf79 #11 [ffff881f2db77cd0] nfsd_unlink at ffffffffa0355dcf [nfsd] #12 [ffff881f2db77d10] nfsd4_remove at ffffffffa0362ebd [nfsd] #13 [ffff881f2db77d50] nfsd4_proc_compound at ffffffffa0363d57 [nfsd] #14 [ffff881f2db77db0] nfsd_dispatch at ffffffffa034ff83 [nfsd] #15 [ffff881f2db77df0] svc_process_common at ffffffffa0188260 [sunrpc] #16 [ffff881f2db77e60] svc_process at ffffffffa0188603 [sunrpc] #17 [ffff881f2db77e90] nfsd at ffffffffa034f98f [nfsd] #18 [ffff881f2db77ec0] kthread at ffffffff81096989 #19 [ffff881f2db77f50] ret_from_fork at ffffffff81660462 Thanks, Andy -- Andrew W. Elble aweits at discipline.rit.edu Infrastructure Engineer, Communications Technical Lead Rochester Institute of Technology PGP: BFAD 8461 4CCF DC95 DA2C B0EB 965B 082E 863E C912