linux-next: JFFS2 deadlock

Just tested the current next-20130226 on a custom AM335X board, and I received the JFFS2 deadlock shown below.

Regards
Mark JACKSON
---

[    3.250349] jffs2: notice: (1) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[    3.268364] VFS: Mounted root (jffs2 filesystem) readonly on device 31:6.
[    3.277233] devtmpfs: mounted
[    3.280982] Freeing init memory: 332K
[    3.706697]
[    3.708306] ======================================================
[    3.714804] [ INFO: possible circular locking dependency detected ]
[    3.721398] 3.8.0-next-20130226-dirty #10 Not tainted
[    3.726708] -------------------------------------------------------
[    3.733297] rcS/686 is trying to acquire lock:
[    3.737969]  (&mm->mmap_sem){++++++}, at: [<c00f0af4>] might_fault+0x3c/0x90
[    3.745437]
[    3.745437] but task is already holding lock:
[    3.751569]  (&f->sem){+.+.+.}, at: [<c023d128>] jffs2_readdir+0x44/0x1a8
[    3.758748]
[    3.758748] which lock already depends on the new lock.
[    3.758748]
[    3.767348]
[    3.767348] the existing dependency chain (in reverse order) is:
[    3.775215]
-> #1 (&f->sem){+.+.+.}:
[    3.779184]        [<c0092df0>] lock_acquire+0x9c/0x104
[    3.784701]        [<c04b76e4>] mutex_lock_nested+0x3c/0x334
[    3.790666]        [<c023d950>] jffs2_readpage+0x20/0x44
[    3.796261]        [<c00d9d38>] __do_page_cache_readahead+0x2a0/0x2cc
[    3.803050]        [<c00da004>] ra_submit+0x28/0x30
[    3.808187]        [<c00d179c>] filemap_fault+0x304/0x458
[    3.813884]        [<c00f0c58>] __do_fault+0x6c/0x490
[    3.819203]        [<c00f3c5c>] handle_pte_fault+0xb0/0x6f0
[    3.825071]        [<c00f433c>] handle_mm_fault+0xa0/0xd4
[    3.830755]        [<c04bbdcc>] do_page_fault+0x2a0/0x3d4
[    3.836449]        [<c000845c>] do_DataAbort+0x30/0x9c
[    3.841861]        [<c04ba2a4>] __dabt_svc+0x44/0x80
[    3.847089]        [<c0289c34>] __clear_user_std+0x1c/0x64
[    3.852877]
-> #0 (&mm->mmap_sem){++++++}:
[    3.857393]        [<c00927ec>] __lock_acquire+0x1d70/0x1de0
[    3.863353]        [<c0092df0>] lock_acquire+0x9c/0x104
[    3.868855]        [<c00f0b18>] might_fault+0x60/0x90
[    3.874174]        [<c011bc3c>] filldir+0x5c/0x158
[    3.879230]        [<c023d1c0>] jffs2_readdir+0xdc/0x1a8
[    3.884823]        [<c011becc>] vfs_readdir+0x98/0xb4
[    3.890144]        [<c011bfcc>] sys_getdents+0x74/0xd0
[    3.895554]        [<c0013820>] ret_fast_syscall+0x0/0x3c
[    3.901251]
[    3.901251] other info that might help us debug this:
[    3.901251]
[    3.909668]  Possible unsafe locking scenario:
[    3.909668]
[    3.915892]        CPU0                    CPU1
[    3.920652]        ----                    ----
[    3.925411]   lock(&f->sem);
[    3.928451]                                lock(&mm->mmap_sem);
[    3.934688]                                lock(&f->sem);
[    3.940376]   lock(&mm->mmap_sem);
[    3.943965]
[    3.943965]  *** DEADLOCK ***
[    3.943965]
[    3.950196] 2 locks held by rcS/686:
[    3.953952]  #0:  (&type->i_mutex_dir_key){+.+.+.}, at: [<c011be90>] vfs_readdir+0x5c/0xb4
[    3.962686]  #1:  (&f->sem){+.+.+.}, at: [<c023d128>] jffs2_readdir+0x44/0x1a8
[    3.970320]
[    3.970320] stack backtrace:
[    3.974930] [<c001b158>] (unwind_backtrace+0x0/0xf0) from [<c008f29c>] (print_circular_bug+0x1d0/0x2dc)
[    3.984815] [<c008f29c>] (print_circular_bug+0x1d0/0x2dc) from [<c00927ec>] (__lock_acquire+0x1d70/0x1de0)
[    3.994975] [<c00927ec>] (__lock_acquire+0x1d70/0x1de0) from [<c0092df0>] (lock_acquire+0x9c/0x104)
[    4.004494] [<c0092df0>] (lock_acquire+0x9c/0x104) from [<c00f0b18>] (might_fault+0x60/0x90)
[    4.013376] [<c00f0b18>] (might_fault+0x60/0x90) from [<c011bc3c>] (filldir+0x5c/0x158)
[    4.021802] [<c011bc3c>] (filldir+0x5c/0x158) from [<c023d1c0>] (jffs2_readdir+0xdc/0x1a8)
[    4.030502] [<c023d1c0>] (jffs2_readdir+0xdc/0x1a8) from [<c011becc>] (vfs_readdir+0x98/0xb4)
[    4.039477] [<c011becc>] (vfs_readdir+0x98/0xb4) from [<c011bfcc>] (sys_getdents+0x74/0xd0)
[    4.048270] [<c011bfcc>] (sys_getdents+0x74/0xd0) from [<c0013820>] (ret_fast_syscall+0x0/0x3c)

Hi Mark,

Is this new today? is it reproducible? Does if ail for Linus' tree?

Al, could this be something to do with the new stuff in the vfs tree?

Cheers,
Stephen Rothwell                    [email protected]

Very unlikely. jffs2_readdir() does, indeed, grab ->sem on the inode in question and then calls the callback (which might fault, grabbing ->mmap_sem). Had been doing that all along. And if the userland area we are doing getdents(2) into had been mmapped from jffs2 file, jffs2_readpage() would be called, which would grab ->sem on the inode of file mmaped in there. Again, that had been going on all along. Unlike the situation with ->i_mutex, this one is probably a false positive - ->sem on directories nests outside of ->mmap_sem, ->sem on non-directories - inside. But that false positive shouldn't be something new; hell, both paths are present in 2.6.12 with the lock orders as in the current tree (except that ->sem used to be a semaphore instead of a mutex).

Almost identical on Linus' tree:-

[    3.685186]
[    3.686803] ======================================================
[    3.693333] [ INFO: possible circular locking dependency detected ]
[    3.699962] 3.8.0-09426-g986876f-dirty #133 Not tainted
[    3.705482] -------------------------------------------------------
[    3.712105] rcS/682 is trying to acquire lock:
[    3.716801]  (&mm->mmap_sem){++++++}, at: [<c00f1258>] might_fault+0x3c/0x90
[    3.724307]
[    3.724307] but task is already holding lock:
[    3.730470]  (&f->sem){+.+.+.}, at: [<c023cb30>] jffs2_readdir+0x44/0x1a8
[    3.737686]
[    3.737686] which lock already depends on the new lock.
[    3.737686]
[    3.746331]
[    3.746331] the existing dependency chain (in reverse order) is:
[    3.754239]
-> #1 (&f->sem){+.+.+.}:
[    3.758229]        [<c0093614>] lock_acquire+0x9c/0x104
[    3.763775]        [<c04b3b4c>] mutex_lock_nested+0x3c/0x334
[    3.769769]        [<c023d358>] jffs2_readpage+0x20/0x44
[    3.775392]        [<c00da510>] __do_page_cache_readahead+0x2a0/0x2cc
[    3.782217]        [<c00da7dc>] ra_submit+0x28/0x30
[    3.787380]        [<c00d2010>] filemap_fault+0x304/0x458
[    3.793094]        [<c00f13bc>] __do_fault+0x6c/0x490
[    3.798439]        [<c00f43a4>] handle_pte_fault+0xb0/0x6f0
[    3.804337]        [<c00f4a84>] handle_mm_fault+0xa0/0xd4
[    3.810050]        [<c04b80e8>] do_page_fault+0x2a0/0x3d4
[    3.815773]        [<c000845c>] do_DataAbort+0x30/0x9c
[    3.821212]        [<c04b65e4>] __dabt_svc+0x44/0x80
[    3.826467]        [<c0288d10>] __clear_user_std+0x1c/0x64
[    3.832285]
-> #0 (&mm->mmap_sem){++++++}:
[    3.836824]        [<c0093010>] __lock_acquire+0x1d70/0x1de0
[    3.842815]        [<c0093614>] lock_acquire+0x9c/0x104
[    3.848345]        [<c00f127c>] might_fault+0x60/0x90
[    3.853690]        [<c011c504>] filldir+0x5c/0x158
[    3.858762]        [<c023cbc8>] jffs2_readdir+0xdc/0x1a8
[    3.864384]        [<c011c794>] vfs_readdir+0x98/0xb4
[    3.869729]        [<c011c894>] sys_getdents+0x74/0xd0
[    3.875166]        [<c0013800>] ret_fast_syscall+0x0/0x3c
[    3.880890]
[    3.880890] other info that might help us debug this:
[    3.880890]
[    3.889350]  Possible unsafe locking scenario:
[    3.889350]
[    3.895604]        CPU0                    CPU1
[    3.900388]        ----                    ----
[    3.905171]   lock(&f->sem);
[    3.908227]                                lock(&mm->mmap_sem);
[    3.914494]                                lock(&f->sem);
[    3.920210]   lock(&mm->mmap_sem);
[    3.923816]
[    3.923816]  *** DEADLOCK ***
[    3.923816]
[    3.930077] 2 locks held by rcS/682:
[    3.933851]  #0:  (&type->i_mutex_dir_key){+.+.+.}, at: [<c011c758>] vfs_readdir+0x5c/0xb4
[    3.942625]  #1:  (&f->sem){+.+.+.}, at: [<c023cb30>] jffs2_readdir+0x44/0x1a8
[    3.950298]
[    3.950298] stack backtrace:
[    3.954930] [<c001b11c>] (unwind_backtrace+0x0/0xf0) from [<c008fac0>] (print_circular_bug+0x1d0/0x2dc)
[    3.964870] [<c008fac0>] (print_circular_bug+0x1d0/0x2dc) from [<c0093010>] (__lock_acquire+0x1d70/0x1de0)
[    3.975081] [<c0093010>] (__lock_acquire+0x1d70/0x1de0) from [<c0093614>] (lock_acquire+0x9c/0x104)
[    3.984650] [<c0093614>] (lock_acquire+0x9c/0x104) from [<c00f127c>] (might_fault+0x60/0x90)
[    3.993573] [<c00f127c>] (might_fault+0x60/0x90) from [<c011c504>] (filldir+0x5c/0x158)
[    4.002039] [<c011c504>] (filldir+0x5c/0x158) from [<c023cbc8>] (jffs2_readdir+0xdc/0x1a8)
[    4.010781] [<c023cbc8>] (jffs2_readdir+0xdc/0x1a8) from [<c011c794>] (vfs_readdir+0x98/0xb4)
[    4.019797] [<c011c794>] (vfs_readdir+0x98/0xb4) from [<c011c894>] (sys_getdents+0x74/0xd0)
[    4.028629] [<c011c894>] (sys_getdents+0x74/0xd0) from [<c0013800>] (ret_fast_syscall+0x0/0x3c)