XFS related Oops

Hi,

after resume from suspend with 2.6.23.1, I got the following Oops:

BUG: unable to handle kernel paging request at virtual address 3e0d204c
 printing eip:
c022807f
*pde = 00000000
Oops: 0000 [#1]
SMP 
Modules linked in: dvb_usb_cinergyT2 i915 drm cpufreq_stats usblp
firewire_ohci firewire_core dvb_usb crc_itu_t evdev snd_hda_intel
rtc_cmos sky2 dvb_core applesmc led_class coretemp hwmon
CPU:    0
EIP:    0060:[<c022807f>]    Not tainted VLI
EFLAGS: 00010202   (2.6.23.1 #4)

EIP is at xfs_iget_core+0x6f/0x690
eax: 0033f000 ebx: 3e0d2010 ecx: f7826000 edx: 0033f000
esi: 001346d7 edi: 00000000 ebp: f7525214 esp: f68b7cb4
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process squid (pid: 2967, ti=f68b6000 task=f6833030 task.ti=f68b6000)

Stack: c024f1b1 00000001 f6a3d6c0 001346d7 f789ce00 c18993d8 c04fb280
f7525218 
       00000000 00000000 f7826000 e652c240 00000000 3e0d2010 e652c25c
e652c240 
       001346d7 f7826000 c0228774 001346d7 00000000 00000000 00000000
f68b7d80 
Call Trace:
 [<c024f1b1>] kmem_zone_alloc+0x51/0xb0
 [<c0228774>] xfs_iget+0xd4/0x160
 [<c024474b>] xfs_dir_lookup_int+0x9b/0x100
 [<c0247ec5>] xfs_lookup+0x75/0xa0
 [<c0256874>] xfs_vn_lookup+0x54/0x90
 [<c0177a32>] do_lookup+0x122/0x1a0
 [<c01794d4>] __link_path_walk+0x784/0xd80
 [<c0273072>] __next_cpu+0x12/0x30
 [<c0121dbd>] find_busiest_group+0x19d/0x6a0
 [<c0179b15>] link_path_walk+0x45/0xc0
 [<c0131350>] process_timeout+0x0/0x10
 [<c016eb62>] get_unused_fd_flags+0x52/0xc0
 [<c0179db3>] do_path_lookup+0x73/0x1b0
 [<c01719b8>] get_empty_filp+0x58/0x120
 [<c017a9d1>] __path_lookup_intent_open+0x51/0xa0
 [<c017aab0>] path_lookup_open+0x20/0x30
 [<c017abb6>] open_namei+0x66/0x640
 [<c0131697>] lock_timer_base+0x27/0x60
 [<c0131715>] try_to_del_timer_sync+0x45/0x50
 [<c016ee7e>] do_filp_open+0x2e/0x60
 [<c0131350>] process_timeout+0x0/0x10
 [<c016eb62>] get_unused_fd_flags+0x52/0xc0
 [<c016eefc>] do_sys_open+0x4c/0xe0
 [<c016efcc>] sys_open+0x1c/0x20
 [<c010428e>] sysenter_past_esp+0x5f/0x85
 =======================
Code: 44 24 1c 8b 44 24 1c e8 60 92 1b 00 8b 5d 00 85 db 89 5c 24 34 75

14 e9 90 00 00 00 8b 5b 04 85 db 89 5c 24 34 0f 84 81 00 00 00 <8b> 53 3c 8b 43 38 31 fa 31 f0 09 c2 75 e3 8d 83 dc 00 00 00 e8
EIP: [<c022807f>] xfs_iget_core+0x6f/0x690 SS:ESP 0068:f68b7cb4

Does this ring any bells?

No. I'd say something got screwed up during suspend/resume. Is it reproducable?

On Tue, Nov 13, 2007 at 09:27:20 +1100, David Chinner wrote:

[...]

Quoted 2 lines

No. I'd say something got screwed up during suspend/resume. Is it reproducable?

No. I often use suspend to RAM, and usually it works without such failures. I restart squid during the resume prosecure, and the above Oops lead to a squid in D state.

Quoted 7 lines · Tino Keitel

On Tue, Nov 13, 2007 at 09:27:20 +1100, David Chinner wrote:

[...]

Quoted 2 lines

No. I'd say something got screwed up during suspend/resume. Is it reproducable?

Quoted 3 lines

No. I often use suspend to RAM, and usually it works without such failures. I restart squid during the resume prosecure, and the above Oops lead to a squid in D state.

Ok. Sounds like there's not much we can debug at this point. Thanks for the report, though.

Quoted 7 lines · Tino Keitel

On Tue, Nov 13, 2007 at 09:27:20 +1100, David Chinner wrote:

[...]

Quoted 2 lines

No. I'd say something got screwed up during suspend/resume. Is it reproducable?

Quoted 3 lines

No. I often use suspend to RAM, and usually it works without such failures. I restart squid during the resume prosecure, and the above Oops lead to a squid in D state.

Ok. Sounds like there's not much we can debug at this point. Thanks for the report, though.

Quoted 12 lines · David Chinner

Quoted 10 lines · Tino Keitel

On Tue, Nov 13, 2007 at 09:27:20 +1100, David Chinner wrote:

[...]

Quoted 2 lines

No. I'd say something got screwed up during suspend/resume. Is it reproducable?

No. I often use suspend to RAM, and usually it works without such failures. I restart squid during the resume prosecure, and the above Oops lead to a squid in D state.

Quoted 2 lines

Ok. Sounds like there's not much we can debug at this point. Thanks for the report, though.

I got a similar Oops again:

xfs_iget_core: ambiguous vns: vp/0xc00700c0, invp/0xcb5a1680

Quoted 13 lines · Tino Keitel

Quoted 12 lines · David Chinner

Quoted 10 lines · Tino Keitel

On Tue, Nov 13, 2007 at 09:27:20 +1100, David Chinner wrote:

[...]

Quoted 2 lines

No. I'd say something got screwed up during suspend/resume. Is it reproducable?

No. I often use suspend to RAM, and usually it works without such failures. I restart squid during the resume prosecure, and the above Oops lead to a squid in D state.

Quoted 6 lines

Quoted 2 lines

Ok. Sounds like there's not much we can debug at this point. Thanks for the report, though.

I got a similar Oops again:

xfs_iget_core: ambiguous vns: vp/0xc00700c0, invp/0xcb5a1680

Now there's a message that I haven't seen in about 3 years.

It indicates that the linux inode connected to the xfs_inode is not the correct one. i.e. that the linux inode cache is out of step with the XFS inode cache.

Basically, that is not supposed to happen. I suspect that the way threads are frozen is resulting in an inode lookup racing with a reclaim. The reclaim thread gets stopped after any use threads, and so we could have the situation that a process blocked in lookup has the XFS inode reclaimed and reused before it gets unblocked.

The question is why is it happening now when none of that code in XFS has changed?

Rafael, when are threads frozen? Only when they schedule or call try_to_freeze()?

Did the freezer mechanism change in 2.6.23 (this is on 2.6.23.1)?

Is there some way of getting a stack trace of all the processes in the system once the machine is frozen and about to suspend so we can see if we blocked in a lookup?

On 2.6.23.1 you can test the freezer alone by doing

# echo testproc > /sys/power/disk
# echo disk > /sys/power/state

On Mon, Nov 26, 2007 at 23:07:56 +0100, Rafael J. Wysocki wrote:

[...]

Quoted 4 lines

On 2.6.23.1 you can test the freezer alone by doing

# echo testproc > /sys/power/disk
# echo disk > /sys/power/state

This is suspend to RAM, not to disk.

Quoted 33 lines · David Chinner

Quoted 13 lines · Tino Keitel

Quoted 12 lines · David Chinner

Quoted 10 lines · Tino Keitel

On Tue, Nov 13, 2007 at 09:27:20 +1100, David Chinner wrote:

[...]

Quoted 2 lines

No. I'd say something got screwed up during suspend/resume. Is it reproducable?

No. I often use suspend to RAM, and usually it works without such failures. I restart squid during the resume prosecure, and the above Oops lead to a squid in D state.

Quoted 6 lines

Quoted 2 lines

Ok. Sounds like there's not much we can debug at this point. Thanks for the report, though.

I got a similar Oops again:

xfs_iget_core: ambiguous vns: vp/0xc00700c0, invp/0xcb5a1680

Now there's a message that I haven't seen in about 3 years.

It indicates that the linux inode connected to the xfs_inode is not the correct one. i.e. that the linux inode cache is out of step with the XFS inode cache.

Basically, that is not supposed to happen. I suspect that the way threads are frozen is resulting in an inode lookup racing with a reclaim. The reclaim thread gets stopped after any use threads, and so we could have the situation that a process blocked in lookup has the XFS inode reclaimed and reused before it gets unblocked.

Quoted 5 lines

The question is why is it happening now when none of that code in XFS has changed?

Rafael, when are threads frozen? Only when they schedule or call try_to_freeze()?

Kernel threads freeze only when they call try_to_freeze(). User space tasks freeze while executing the signals handling code.

Quoted 1 line

Did the freezer mechanism change in 2.6.23 (this is on 2.6.23.1)?

Yes. Kernel threads are not sent fake signals by the freezer any more.

Quoted 22 lines · Rafael J. Wysocki

Quoted 12 lines · David Chinner

Now there's a message that I haven't seen in about 3 years.

It indicates that the linux inode connected to the xfs_inode is not the correct one. i.e. that the linux inode cache is out of step with the XFS inode cache.

Basically, that is not supposed to happen. I suspect that the way threads are frozen is resulting in an inode lookup racing with a reclaim. The reclaim thread gets stopped after any use threads, and so we could have the situation that a process blocked in lookup has the XFS inode reclaimed and reused before it gets unblocked.

Quoted 5 lines

The question is why is it happening now when none of that code in XFS has changed?

Rafael, when are threads frozen? Only when they schedule or call try_to_freeze()?

Kernel threads freeze only when they call try_to_freeze(). User space tasks freeze while executing the signals handling code.

Quoted 3 lines

Quoted 1 line

Did the freezer mechanism change in 2.6.23 (this is on 2.6.23.1)?

Yes. Kernel threads are not sent fake signals by the freezer any more.

Ah, sorry, this change has been merged after 2.6.23. However, before 2.6.23 we had another important change that caused all kernel threads to have PF_NOFREEZE set by default, unless they call set_freezable() explicitly.

Quoted 25 lines · Rafael J. Wysocki

Quoted 24 lines · Rafael J. Wysocki

Quoted 12 lines · David Chinner

Now there's a message that I haven't seen in about 3 years.

It indicates that the linux inode connected to the xfs_inode is not the correct one. i.e. that the linux inode cache is out of step with the XFS inode cache.

Basically, that is not supposed to happen. I suspect that the way threads are frozen is resulting in an inode lookup racing with a reclaim. The reclaim thread gets stopped after any use threads, and so we could have the situation that a process blocked in lookup has the XFS inode reclaimed and reused before it gets unblocked.

Quoted 5 lines

The question is why is it happening now when none of that code in XFS has changed?

Rafael, when are threads frozen? Only when they schedule or call try_to_freeze()?

Kernel threads freeze only when they call try_to_freeze(). User space tasks freeze while executing the signals handling code.

Quoted 1 line

Did the freezer mechanism change in 2.6.23 (this is on 2.6.23.1)?

Quoted 5 lines

Quoted 1 line

Yes. Kernel threads are not sent fake signals by the freezer any more.

Ah, sorry, this change has been merged after 2.6.23. However, before 2.6.23 we had another important change that caused all kernel threads to have PF_NOFREEZE set by default, unless they call set_freezable() explicitly.

So try_to_freeze() will never freeze a thread if it has not been set_freezable()? And xfsbufd will never be frozen?

Quoted 26 lines · David Chinner

Quoted 25 lines · Rafael J. Wysocki

Quoted 24 lines · Rafael J. Wysocki

Quoted 12 lines · David Chinner

Now there's a message that I haven't seen in about 3 years.

It indicates that the linux inode connected to the xfs_inode is not the correct one. i.e. that the linux inode cache is out of step with the XFS inode cache.

Basically, that is not supposed to happen. I suspect that the way threads are frozen is resulting in an inode lookup racing with a reclaim. The reclaim thread gets stopped after any use threads, and so we could have the situation that a process blocked in lookup has the XFS inode reclaimed and reused before it gets unblocked.

Quoted 5 lines

The question is why is it happening now when none of that code in XFS has changed?

Rafael, when are threads frozen? Only when they schedule or call try_to_freeze()?

Kernel threads freeze only when they call try_to_freeze(). User space tasks freeze while executing the signals handling code.

Quoted 1 line

Did the freezer mechanism change in 2.6.23 (this is on 2.6.23.1)?

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

Yes. Kernel threads are not sent fake signals by the freezer any more.

Ah, sorry, this change has been merged after 2.6.23. However, before 2.6.23 we had another important change that caused all kernel threads to have PF_NOFREEZE set by default, unless they call set_freezable() explicitly.

So try_to_freeze() will never freeze a thread if it has not been set_freezable()? And xfsbufd will never be frozen?

No, it won't.

I must have overlooked it, probably because it calls refrigerator() directly and not try_to_freeze() ...

I think something like the appended patch will help, then.

---
Fix breakage caused by commit 831441862956fffa17b9801db37e6ea1650b0f69 that did not introduce the necessary call to set_freezable() in xfs/linux-2.6/xfs_buf.c .

Signed-off-by Rafael J. Wysocki <[email protected]>

---
 fs/xfs/linux-2.6/xfs_buf.c |    2 ++
 1 file changed, 2 insertions(+)

Index: linux-2.6/fs/xfs/linux-2.6/xfs_buf.c
===================================================================

fs/xfs/linux-2.6/xfs_buf.c +2 -1

@@ -1750,6 +1750,8 @@ xfsbufd(
 
 	current->flags |= PF_MEMALLOC;
 
+	set_freezable();
+
 	do {
 		if (unlikely(freezing(current))) {
 			set_bit(XBT_FORCE_SLEEP, &target->bt_flags);
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Tino, can you check if this patch helps, please?

On Thu, Nov 29, 2007 at 22:05:24 +0100, Rafael J. Wysocki wrote:

[...]

Quoted 1 line

Tino, can you check if this patch helps, please?

Not really. I suspend one to several times a day, and in most cases resume works. I thing checking the patch is hard when I have no real procedure to reproduce the resume failure.