generic/232 hangs on XFS DAX mount
Hi all,
I hit a generic/232 hang on XFS DAX mount, kernel is 4.10-rc4, 4.9 kernel also could reproduce, so at least it's not a 4.10 regression. I haven't tested other kernel versions yet.
I reproduced it by running generic/232 in a tight loop, it usually hangs within 50 iterations. test dev is 15G brd ramdisk.
I append detailed host info and test config at the end. If you need more info please let me know.
Thanks,
Eryu
TEST_DEV=/dev/ram0
SCRATCH_DEV=/dev/ram1
TEST_DIR=/mnt/test
SCRATCH_MNT=/mnt/scratch
MKFS_OPTIONS="-f -m crc=1 -b size=4k"
MOUNT_OPTIONS="-o dax"
[ 4489.611370] sysrq: SysRq : Show Blocked State [ 4489.632865] task PC stack pid father [ 4489.661722] xfs_quota D 0 30771 30468 0x00000080 [ 4489.689072] Call Trace: [ 4489.701239] __schedule+0x21c/0x6b0 [ 4489.717525] ? __free_pages+0x19/0x30 [ 4489.734712] schedule+0x36/0x80 [ 4489.749458] schedule_timeout+0x21c/0x3a0 [ 4489.768249] ? __slab_free+0x9b/0x2c0 [ 4489.785477] wait_for_completion+0x11c/0x180 [ 4489.805688] ? wake_up_q+0x80/0x80 [ 4489.821670] xfs_qm_dqpurge+0x6b/0x1b0 [xfs] [ 4489.841764] ? xfs_qm_need_dqattach+0x80/0x80 [xfs] [ 4489.864823] xfs_qm_dquot_walk.isra.10+0xfb/0x190 [xfs] [ 4489.889610] xfs_qm_dqpurge_all+0x5c/0x80 [xfs] [ 4489.910959] xfs_qm_scall_quotaoff+0x127/0x380 [xfs] [ 4489.934407] xfs_quota_disable+0x3d/0x50 [xfs] [ 4489.955286] SyS_quotactl+0x391/0x850 [ 4489.972522] ? __audit_syscall_entry+0xaf/0x100 [ 4489.993523] ? syscall_trace_enter+0x1d0/0x2b0 [ 4490.014486] ? __audit_syscall_exit+0x209/0x290 [ 4490.035763] do_syscall_64+0x67/0x180 [ 4490.053266] entry_SYSCALL64_slow_path+0x25/0x25 [ 4490.074955] RIP: 0033:0x7eff755e2fea [ 4490.091854] RSP: 002b:00007ffe012179d8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b3 [ 4490.127517] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007eff755e2fea [ 4490.162364] RDX: 0000000000000000 RSI: 00000000012ee2f0 RDI: 0000000000580200 [ 4490.196833] RBP: 00000000012ee310 R08: 00007ffe012179ec R09: 0000000000000000 [ 4490.231139] R10: 00007ffe012179ec R11: 0000000000000206 R12: 00000000012ee2d0 [ 4490.264475] R13: 00000000012ed0a0 R14: 0000000000000000 R15: 00000000012ed080
[root@hp-dl360g9-15 xfstests]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Stepping: 2
CPU MHz: 2400.000
BogoMIPS: 4801.94
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-3,8-11
NUMA node1 CPU(s): 4-7,12-15
[root@hp-dl360g9-15 xfstests]# free -m
total used free shared buff/cache available
Mem: 64305 761 63382 9 161 63071
Swap: 16379 0 16379[root@hp-dl360g9-15 xfstests]# uname -a
Linux hp-dl360g9-15 4.10.0-rc4 #18 SMP Wed Jan 18 11:39:05 CST 2017 x86_64 x86_64 x86_64 GNU/Linux
Quoted 6 lines · Eryu Guan
Hi all,
I hit a generic/232 hang on XFS DAX mount, kernel is 4.10-rc4, 4.9 kernel also could reproduce, so at least it's not a 4.10 regression. I haven't tested other kernel versions yet.
Quoted 6 lines
I reproduced it by running generic/232 in a tight loop, it usually hangs within 50 iterations. test dev is 15G brd ramdisk.
I append detailed host info and test config at the end. If you need more info please let me know.
FYI, this doesn't appear to be related to DAX. I could reproduce on a current kernel using defaults on a ramdisk. I also saw the following assert before the hang:
[ 935.549559] XFS: Assertion failed: !(bp->b_flags & _XBF_DELWRI_Q), file: fs/xfs/xfs_buf.c, line: 1010 [ 935.558779] ------------[ cut here ]------------ [ 935.563516] WARNING: CPU: 7 PID: 29192 at fs/xfs/xfs_message.c:105 asswarn+0x31/0x40 [xfs] [ 935.571778] Modules linked in: xfs brd ip6t_REJECT nf_reject_ipv6 xt_conntrack ip6t_rpfilter ip_set nfnetlink ebtable_broute bridge stp llc ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_e [ 935.642928] CPU: 7 PID: 29192 Comm: 232 Not tainted 4.10.0-rc4 #3 [ 935.649005] Hardware name: Dell Inc. PowerEdge R715/0G2DP3, BIOS 2.3.0 10/18/2011 [ 935.656468] Call Trace: [ 935.658912] dump_stack+0x63/0x86 [ 935.662217] __warn+0xcb/0xf0 [ 935.665210] ? xfs_buftarg_shrink_scan+0x8d/0xc0 [xfs] [ 935.670335] warn_slowpath_null+0x1d/0x20 [ 935.674365] asswarn+0x31/0x40 [xfs] [ 935.677962] xfs_buf_rele+0x399/0x3e0 [xfs] [ 935.682163] xfs_buftarg_shrink_scan+0x8d/0xc0 [xfs] [ 935.687117] shrink_slab.part.40+0x1f5/0x420 [ 935.691376] shrink_slab+0x29/0x30 [ 935.694768] drop_slab_node+0x31/0x60 [ 935.698419] drop_slab+0x3f/0x70 [ 935.701638] drop_caches_sysctl_handler+0x71/0xc0 [ 935.706329] proc_sys_call_handler+0xea/0x110 [ 935.710675] proc_sys_write+0x14/0x20 [ 935.714327] __vfs_write+0x37/0x160 [ 935.717806] ? selinux_file_permission+0xd7/0x110 [ 935.722499] ? security_file_permission+0x3b/0xc0 [ 935.727192] vfs_write+0xb5/0x1a0 [ 935.730496] SyS_write+0x55/0xc0 [ 935.733717] entry_SYSCALL_64_fastpath+0x1a/0xa9 [ 935.738322] RIP: 0033:0x7f5351f92c20 [ 935.741888] RSP: 002b:00007fff469bc9a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 935.749440] RAX: ffffffffffffffda RBX: 00007f535225c5e0 RCX: 00007f5351f92c20 [ 935.756559] RDX: 0000000000000002 RSI: 000055e935a2ba90 RDI: 0000000000000001 [ 935.763677] RBP: 0000000000000001 R08: 00007f535225d740 R09: 00007f53528aa700 [ 935.770794] R10: 0000000000000073 R11: 0000000000000246 R12: 000055e935c14780 [ 935.777912] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001 [ 935.785072] ---[ end trace 43d9a50105018845 ]---
... which suggests we've freed a buffer while on a delwri submit queue (presumably via an AIL push). Indeed, after the fact tracing shows the AIL spinning on some flush locked dquots.
The problem didn't reproduce with the drop_caches removed from _check_quota_usage() in xfstests/common/quota. It's possible for that to be a false negative, but otherwise I'd suspect we have some kind of a buffer tracking problem between LRU reclaim (i.e., the above shrinker) and the AIL which results in the flush lock being acquired and never released (I/O asociated w/ flush never submitted perhaps?). The quotactl() call below is probably stuck waiting on the flush lock as a result.
Brian
Quoted 75 lines
Thanks,
EryuTEST_DEV=/dev/ram0
SCRATCH_DEV=/dev/ram1
TEST_DIR=/mnt/test
SCRATCH_MNT=/mnt/scratch
MKFS_OPTIONS="-f -m crc=1 -b size=4k"
MOUNT_OPTIONS="-o dax"[ 4489.611370] sysrq: SysRq : Show Blocked State [ 4489.632865] task PC stack pid father [ 4489.661722] xfs_quota D 0 30771 30468 0x00000080 [ 4489.689072] Call Trace: [ 4489.701239] __schedule+0x21c/0x6b0 [ 4489.717525] ? __free_pages+0x19/0x30 [ 4489.734712] schedule+0x36/0x80 [ 4489.749458] schedule_timeout+0x21c/0x3a0 [ 4489.768249] ? __slab_free+0x9b/0x2c0 [ 4489.785477] wait_for_completion+0x11c/0x180 [ 4489.805688] ? wake_up_q+0x80/0x80 [ 4489.821670] xfs_qm_dqpurge+0x6b/0x1b0 [xfs] [ 4489.841764] ? xfs_qm_need_dqattach+0x80/0x80 [xfs] [ 4489.864823] xfs_qm_dquot_walk.isra.10+0xfb/0x190 [xfs] [ 4489.889610] xfs_qm_dqpurge_all+0x5c/0x80 [xfs] [ 4489.910959] xfs_qm_scall_quotaoff+0x127/0x380 [xfs] [ 4489.934407] xfs_quota_disable+0x3d/0x50 [xfs] [ 4489.955286] SyS_quotactl+0x391/0x850 [ 4489.972522] ? __audit_syscall_entry+0xaf/0x100 [ 4489.993523] ? syscall_trace_enter+0x1d0/0x2b0 [ 4490.014486] ? __audit_syscall_exit+0x209/0x290 [ 4490.035763] do_syscall_64+0x67/0x180 [ 4490.053266] entry_SYSCALL64_slow_path+0x25/0x25 [ 4490.074955] RIP: 0033:0x7eff755e2fea [ 4490.091854] RSP: 002b:00007ffe012179d8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b3 [ 4490.127517] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007eff755e2fea [ 4490.162364] RDX: 0000000000000000 RSI: 00000000012ee2f0 RDI: 0000000000580200 [ 4490.196833] RBP: 00000000012ee310 R08: 00007ffe012179ec R09: 0000000000000000 [ 4490.231139] R10: 00007ffe012179ec R11: 0000000000000206 R12: 00000000012ee2d0 [ 4490.264475] R13: 00000000012ed0a0 R14: 0000000000000000 R15: 00000000012ed080[root@hp-dl360g9-15 xfstests]# lscpu
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Stepping: 2 CPU MHz: 2400.000 BogoMIPS: 4801.94 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0-3,8-11 NUMA node1 CPU(s): 4-7,12-15 [root@hp-dl360g9-15 xfstests]# free -m total used free shared buff/cache available Mem: 64305 761 63382 9 161 63071 Swap: 16379 0 16379[root@hp-dl360g9-15 xfstests]# uname -a
Linux hp-dl360g9-15 4.10.0-rc4 #18 SMP Wed Jan 18 11:39:05 CST 2017x86_64x86_64x86_64GNU/Linux --
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html