New: Deleting a folder with 500 000 small files causes deadlock in start_this_handle.irsa.7
https://bugzilla.kernel.org/show_bug.cgi?id=43305
Summary: Deleting a folder with 500 000 small files causes
deadlock in start_this_handle.irsa.7
Product: File System
Version: 2.5
Kernel Version: 3.2.0-24-generic-pae
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: ext4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No
As part of unrelated software experiment I created a folder with 500 000 small files (10-15 bytes of content in each) and when I tried to then delete this folder I ran into trouble - when I ran rm against files in that folder (either the whole folder or individually or in batches) one of the rm commands would invariably hang in Uninterruptible sleep in function start_this_handle.irsa.7 and this would block any other writes to that filesystem causing messages like this in the kernel logs:
[ 480.617348] INFO: task bounce:4006 blocked for more than 120 seconds. [ 480.617350] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 480.617353] bounce D ef39dec0 0 4006 1685 0x00000000 [ 480.617357] ef39df10 00000086 00000000 ef39dec0 c10f6526 f7570ca0 c1930e00 c1930e00 [ 480.617365] cd507544 00000052 f78c7e00 df428000 f7570ca0 ef39df2c c10f65c5 00000001 [ 480.617372] 0000000e 00000001 d5af5a68 ef39dee0 ef39dee4 00000001 ef39deec c1036578 [ 480.617379] Call Trace: [ 480.617382] [<c10f6526>] ? wait_on_page_bit+0x86/0x90 [ 480.617386] [<c10f65c5>] ? filemap_fdatawait_range+0x95/0x160 [ 480.617391] [<c1036578>] ? default_spin_lock_flags+0x8/0x10 [ 480.617395] [<c15a819d>] ? _raw_spin_lock_irqsave+0x2d/0x40 [ 480.617399] [<c15a65a5>] schedule+0x35/0x50 [ 480.617403] [<c12175e5>] jbd2_log_wait_commit+0x95/0x100 [ 480.617408] [<c1079e90>] ? add_wait_queue+0x50/0x50 [ 480.617412] [<c11c81f5>] ext4_sync_file+0x1f5/0x2b0 [ 480.617416] [<c11c8000>] ? ext4_flush_completed_IO+0xa0/0xa0 [ 480.617421] [<c116cf83>] vfs_fsync+0x33/0x50 [ 480.617425] [<c116d2d6>] sys_fsync+0x26/0x50 [ 480.617429] [<c15af35f>] sysenter_do_call+0x12/0x28
If a signal of SIGKILL or SIGTERM is sent to the specific rm process that is in that function, then after up to 30 seconds the process would die and the filesystem would continue function and all hung operations would complete. The rm process does delete some files before the deadlock occurs, but after the wchan of the rm process changes to start_this_handle.irsa.7 no more files are deleted.
Rebooting the machine and forcing a fsck does not change the outcome.
This is on a PCIX type SSD drive.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 43305] Deleting a folder with 500 000 small files causes deadlock in start_this_handle.irsa.7
https://bugzilla.kernel.org/show_bug.cgi?id=43305
Jan Kara <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #1 from Jan Kara <[email protected]> 2012-05-29 14:24:12 --- Hmm, can you make the problem happen, then run 'echo w >/proc/sysrq-trigger' and post here output from dmesg? Thanks!
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 43305] Deleting a folder with 500 000 small files causes deadlock in start_this_handle.irsa.7
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=43305
--- Comment #2 from Aigars Mahinovs <[email protected]> 2012-05-29 15:38:47 --- Created an attachment (id=73458)
--> (https://bugzilla.kernel.org/attachment.cgi?id=73458)
Kernel log after pressing SysRq+w twice during the lockup
Note that I first pressed the Sysrq+w on the keyboard and after that ran the command to also execute it via /proc
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 43305] Deleting a folder with 500 000 small files causes deadlock in start_this_handle.irsa.7
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=43305
--- Comment #3 from Aigars Mahinovs <[email protected]> 2012-05-29 15:43:16 --- I reproduced the problem like this (in an empty folder):
for i in {0..500000}; do echo 1234567890 > $i; done; ls | xargs -n 100 -P 100 rm
Got the lock-up on the third try.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 43305] Deleting a folder with 500 000 small files causes deadlock in start_this_handle.irsa.7
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=43305
--- Comment #4 from Jan Kara <[email protected]> 2012-05-29 20:34:50 --- Actually, this does not seem like deadlock as such. Rather the storm of discard requests we send to the drive probably causes a headache to its FTL. Can you try mounting the filesystem with 'nodiscard' mount option?
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 43305] Deleting a folder with 500 000 small files causes deadlock in start_this_handle.irsa.7
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=43305
--- Comment #5 from Jan Kara <[email protected]> 2012-06-04 10:22:48 --- Any result with nodiscard mount option? I'll just close this bug if there's no reply...
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 43305] Deleting a folder with 500 000 small files causes deadlock in start_this_handle.irsa.7
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=43305
Alan <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
CC| |[email protected]
Resolution| |INSUFFICIENT_DATASignature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.