New: directory access hangs without error

Bug 42763 [email protected] 3 participants 15 messages Feb 13, 2012 – Aug 30, 2012

Share & tools

[email protected]

author

2012-02-13 03:41 UTC #0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=42763

Summary: directory access hangs without error
Product: File System
Version: 2.5
Kernel Version: 3.2.5
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: ext4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Kernel 3.2.5
ext4 over RAID-6

I have a specific directory that freezes all processes that try to getdents() or open() a new file. In some cases, the kernel gives me "blocked for more than 120 seconds" messages, often reporting that the process is stuck in ext4_getblk.

I have found no errors, however. The stuck processes stay stuck forever (at least hours), and dmesg shows no complaints about RAID, filesystem, or anything else. Full reads of the RAID devices work without hanging, and drive self-tests pass.

Other processes are able to access other parts of the filesystem normally; this is not a system-wide or fs-wide hang.

Rebooting and fscking made the directory accessible again, but now there is a different directory exhibiting the problem (unknown whether it had the problem before the fsck/reboot)

This particular problem exhibited itself for the first time after tests of my multi-threaded 'du' that beat the filesystem with a few dozen threads simultaneously.

Given that this seems to very reproducible, I have many opportunities to poke at a hung process, query the filesystem, or recompile the kernel in any way that would be helpful; I just don't know to approach it from here.

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-13 15:55 UTC #1 lore raw

[Bug 42763] directory access hangs without error

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #1 from Eric Buddington <[email protected]> 2012-02-13 15:55:11 --- More info:

After a reboot (and forced but uneventful fsck), the problem is gone until I run my program again. The program consists of multiple threads traversing and stat()ing different directory trees simultaneously.

The program will actually work fine for several iterations before getting stuck,
but once it gets stuck, that directory is a trap for all processes until reboot.

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-13 16:02 UTC #2 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #2 from Eric Buddington <[email protected]> 2012-02-13 16:02:51 --- Created an attachment (id=72366)
--> (https://bugzilla.kernel.org/attachment.cgi?id=72366)
C program to trigger the bug

compile with --std=gnu99 -lpthread -lrt

invocation is ./edu <directory> <number of threads>

triggers the bug after about a dozen iteratons. Program will stall showing "1a", indicating one still-active thread, and fail to exit.

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-13 16:56 UTC #3 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

Eric Sandeen <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from Eric Sandeen <[email protected]> 2012-02-13 16:56:54 --- Did fsck ever find any problems with the filesystem?

I'm not clear on whether a simple reboot will clear it up, or if it requires a fsck to get things happy again.

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-13 17:57 UTC #4 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #4 from Eric Buddington <[email protected]> 2012-02-13 17:57:16 --- fsck did not find any problems with the filesystem.

I rebooted without any fsck, and was found the same symptoms; a couple dozen successful runs, then a hang (this time, it seems to be on two directories simultaneously).

I have been dropping caches between runs. I haven't tried to trigger the bug without dropping caches.

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-13 18:21 UTC #5 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #5 from Eric Sandeen <[email protected]> 2012-02-13 18:21:23 --- Ok, I'm traveling w/ limited time to try to reproduce but thanks for all the info.

A sysrq-W or sysrq-T when it's hung might show us where the threads are at.

-Eric

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-13 18:30 UTC #6 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #6 from Eric Buddington <[email protected]> 2012-02-13 18:30:27 --- The stuck threads look like this:

edu D c023a2f4 0 9912 1 0x00000004
f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000 c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0 000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246

Call Trace:
[<c023a2f4>] ? ext4_getblk+0x8b/0x13d
[<c023f172>] ? search_dirblock+0x76/0xaf
[<c01558a1>] ? arch_local_irq_save+0xf/0x14
[<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
[<c01c2cc3>] ? inode_wait+0x5/0x8
[<c0650c36>] ? __wait_on_bit+0x2f/0x54
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c014532b>] ? autoremove_wake_function+0x2f/0x2f
[<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
[<c01c3c8b>] ? iget_locked+0x42/0xc5
[<c023aad8>] ? ext4_iget+0x24/0x5be
[<c01bad90>] ? do_lookup+0x1e4/0x224
[<c01243a6>] ? should_resched+0x5/0x1e
[<c06506de>] ? _cond_resched+0x5/0x18
[<c01a61c9>] ? slab_pre_alloc_hook.isra.62+0x20/0x23
[<c01a6d76>] ? kmem_cache_alloc+0x1c/0xb1
[<c0240b61>] ? ext4_lookup.part.29+0x50/0xc6
[<c01c1cfc>] ? __d_alloc+0xec/0xfb
[<c01ba3b0>] ? d_alloc_and_lookup+0x2c/0x49
[<c01bacf9>] ? do_lookup+0x14d/0x224
[<c01bae26>] ? walk_component+0x56/0x10c
[<c01baf15>] ? lookup_last+0x39/0x3f
[<c01bb9c3>] ? path_lookupat+0x74/0x238
[<c01243a6>] ? should_resched+0x5/0x1e
[<c06506de>] ? _cond_resched+0x5/0x18
[<c01bbba1>] ? do_path_lookup+0x1a/0x4f
[<c01bcd77>] ? user_path_at_empty+0x3d/0x69
[<c01b6065>] ? cp_new_stat64+0xec/0xfe
[<c01bcdbb>] ? user_path_at+0x18/0x1d
[<c01b6410>] ? vfs_fstatat+0x3f/0x67
[<c01b644e>] ? vfs_lstat+0x16/0x18
[<c01b6645>] ? sys_lstat64+0xe/0x21
[<c01b81a4>] ? flush_old_exec+0x29/0x81
[<c0655f63>] ? sysenter_do_call+0x12/0x2c

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-13 20:29 UTC #9 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #7 from Eric Buddington <[email protected]> 2012-02-13 20:29:13 --- If I run the multi-threaded 'du' *without* flushing caches between, I don't seem to trigger the problem (after ~250 iterations, whereas ~20 trigger it otherwise)

The command I use that triggers the bug (/packages is about 17G of ordinary software installs):

n=1; while [ $n -lt 128 ]; do sync; echo 3 > /proc/sys/vm/drop_caches; ./edu /packages $n; n=$[$n+1]; done

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-14 05:07 UTC #10 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #8 from Eric Buddington <[email protected]> 2012-02-14 05:07:39 --- I just discovered that one RAID drive has large and increasing SMART values for Raw_Read_Error_Rate and Hardware_ECC_Recovered.

Apparently, any disk read errors are being recovered, since I'm not seeing any errors in dmesg or /proc/mdstat - so I still think there's an ext4 problem (though perhaps it takes a flooded queue and a very-slow-response disk to trigger...)

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Jan Kara

2012-02-14 14:22 UTC #7 lore raw

Re: [Bug 42763] directory access hangs without error

In reply to [email protected]

Quoted 8 lines · On Mon 13-02-12 18:30:28, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #6 from Eric Buddington <[email protected]> 2012-02-13 18:30:27 --- The stuck threads look like this:
edu D c023a2f4 0 9912 1 0x00000004
f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000 c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0 000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246

Quoted 14 lines

Call Trace:
[<c023a2f4>] ? ext4_getblk+0x8b/0x13d
[<c023f172>] ? search_dirblock+0x76/0xaf
[<c01558a1>] ? arch_local_irq_save+0xf/0x14
[<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
[<c01c2cc3>] ? inode_wait+0x5/0x8
[<c0650c36>] ? __wait_on_bit+0x2f/0x54
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c014532b>] ? autoremove_wake_function+0x2f/0x2f
[<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
[<c01c3c8b>] ? iget_locked+0x42/0xc5
[<c023aad8>] ? ext4_iget+0x24/0x5be

...
Interesting. So this isn't ext4 related at all. Rather it's a generic bug
in VFS's I_NEW handling introduced by 250df6ed (adding Dave and Al to CC). That commit removed wake_up_inode() (in particular a memory barrier before wake_up_bit()) on the basis that i_state transitions are protected by i_lock. That would be fine if all the readers of i_state were using i_lock as well. But they don't - in particular wait_on_inode() from include/linux/writeback.h does not. So that commit opened a reordering possibility where __I_NEW can be cleared *after* wake_up_bit() in unlock_new_inode() happens and so wait_on_bit() in wait_on_inode() goes to sleep indefinitely.

It seems to me the intent was that wait_on_inode() should use i_lock as well so it would opencode bit waiting similarly to __wait_on_freeing_inode(). Am I right? Alternatively, we'd have to back out changes of unlock_new_inode() and wake_up_inode()...

Honza

Signature · 2 lines

Jan Kara <[email protected]>
SUSE Labs, CR

Dave Chinner

2012-02-17 05:33 UTC #8 lore raw

Re: [Bug 42763] directory access hangs without error

Quoted 23 lines · Jan Kara

Quoted 8 lines · On Mon 13-02-12 18:30:28, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #6 from Eric Buddington <[email protected]> 2012-02-13 18:30:27 --- The stuck threads look like this:
edu D c023a2f4 0 9912 1 0x00000004
f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000 c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0 000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246
Quoted 14 lines
Call Trace:
[<c023a2f4>] ? ext4_getblk+0x8b/0x13d
[<c023f172>] ? search_dirblock+0x76/0xaf
[<c01558a1>] ? arch_local_irq_save+0xf/0x14
[<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
[<c01c2cc3>] ? inode_wait+0x5/0x8
[<c0650c36>] ? __wait_on_bit+0x2f/0x54
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c014532b>] ? autoremove_wake_function+0x2f/0x2f
[<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
[<c01c3c8b>] ? iget_locked+0x42/0xc5
[<c023aad8>] ? ext4_iget+0x24/0x5be

Quoted 7 lines

...
Interesting. So this isn't ext4 related at all. Rather it's a generic bug
in VFS's I_NEW handling introduced by 250df6ed (adding Dave and Al to CC). That commit removed wake_up_inode() (in particular a memory barrier before wake_up_bit()) on the basis that i_state transitions are protected by i_lock. That would be fine if all the readers of i_state were using i_lock as well.

Hmmmm. I guess I missed that one.

Quoted 9 lines

But they don't - in particular wait_on_inode() from include/linux/writeback.h does not. So that commit opened a reordering possibility where __I_NEW can be cleared *after* wake_up_bit() in unlock_new_inode() happens and so wait_on_bit() in wait_on_inode() goes to sleep indefinitely.
It seems to me the intent was that wait_on_inode() should use i_lock as well so it would opencode bit waiting similarly to __wait_on_freeing_inode().

Yeah, more like inode_wait_for_writeback() rather than __wait_on_freeing_inode(), though, as we should loop until the bit is cleared.

Cheers,

Dave.

Signature · 2 lines

Dave Chinner
[email protected]

[email protected]

author

2012-02-20 17:31 UTC #11 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

Jan Kara <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #9 from Jan Kara <[email protected]> 2012-02-20 17:31:29 --- Hmm, it seems mirroring of emails and bugzilla is broken. Anyway, thanks for the traces. It's a generic VFS bug. I'll attach here a fix in a moment.

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-20 17:32 UTC #12 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #10 from Jan Kara <[email protected]> 2012-02-20 17:32:36 --- Created an attachment (id=72448)
--> (https://bugzilla.kernel.org/attachment.cgi?id=72448)
Fix missed wakeup on I_NEW

Can you please try this patch? Thanks.

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-02-24 02:22 UTC #13 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

--- Comment #11 from Eric Buddington <[email protected]> 2012-02-24 02:22:15 --- Applied to 3.2.7, and it seems to fix it; at least I ran the same test for about 4x as long as before without the problem exhibiting itself.

Thanks!

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[email protected]

author

2012-08-30 14:35 UTC #14 lore raw

[Bug 42763] directory access hangs without error

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

Alan <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
                 CC|                            |[email protected]
         Resolution|                            |CODE_FIX

Signature · 3 lines

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.