Skip to contentSkip to conversation

Panic with XFS on RHEL5 (2.6.18-8.1.8.el5)

DISCUSSION Chris Boot 4 participants 8 messages Aug 18, 2007 – Aug 20, 2007

lore
Share & tools
lore

Chris Boot

author
#0 lore raw

All,

I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.

Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?

Many thanks,
Chris Boot

BUG: unable to handle kernel paging request at virtual address b8af9d60
printing eip:
c0415974
*pde = 00000000
Oops: 0000 [#1]
SMP last sysfs file: /block/loop7/dev
Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U) autofs4 hidp nls_utf8 cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbs i2c_ec button battery asus_acpi ac lp st sg floppy serio_raw intel_rng pcspkr e100 mii e7xxx_edac i2c_i801 edac_mc i2c_core e1000 r8169 ide_cd cdrom parport_pc parport dm_snapshot dm_zero dm_mirror dm_mod cciss mptspi mptscsih scsi_transport_spi sd_mod scsi_mod mptbase ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    1
EIP:    0060:[<c0415974>]    Tainted: P      VLI
EFLAGS: 00010046   (2.6.18-8.1.8.el5 #1) EIP is at smp_send_reschedule+0x3/0x53
eax: c213f000   ebx: c213f000   ecx: eef84000   edx: c213f000
esi: 00001086   edi: f668c000   ebp: f4f2fce8   esp: f4f2fc8c
ds: 007b   es: 007b   ss: 0068
Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000)
Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000 0000000f 
      00000000 00000001 00000001 c200c6e0 00000100 00000000 00000069 
00000180 
      018fc500 c200d240 00000003 00000292 f601efc0 f6027e00 00000000 
00000050 Call Trace:
[<c041dc23>] try_to_wake_up+0x351/0x37b
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs]
[<c04572f9>] shrink_slab+0x56/0x13c
[<c0457c0c>] try_to_free_pages+0x162/0x23e
[<c0454064>] __alloc_pages+0x18d/0x27e
[<c045214e>] find_or_create_page+0x53/0x8c
[<c046c7b1>] __getblk+0x162/0x270
[<c0475be0>] do_lookup+0x53/0x157
[<f889138f>] ext3_getblk+0x7c/0x233 [ext3]
[<f88913fe>] ext3_getblk+0xeb/0x233 [ext3]
[<c048215c>] mntput_no_expire+0x11/0x6a
[<f889226e>] ext3_bread+0x13/0x69 [ext3]
[<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3]
[<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3]
[<c047828b>] do_path_lookup+0x20e/0x25f
[<c046b987>] get_empty_filp+0x99/0x15e
[<f889d611>] ext3_permission+0x0/0xa [ext3]
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3]
[<c047a0dd>] filldir+0x0/0xb9
[<c0472973>] sys_fstat64+0x1e/0x23
[<c047a1f9>] vfs_readdir+0x63/0x8d
[<c047a0dd>] filldir+0x0/0xb9
[<c047a447>] sys_getdents+0x5f/0x9c
[<c0403eff>] syscall_call+0x7/0xb
=======================
Code: 5d c3 b9 01 00 00 00 31 d2 6a 00 b8 f0 5a 41 c0 e8 2a ff ff ff fa e8 52 16 00 00 fb 58 c3 b8 54 3a 66 c0 e9 8e 6b 1e 00 53 89 c3 <0f> a3 05 60 1f 6d c0 19 c0 85 c0 75 27 e8 bf db 00 00 50 68 55 EIP: [<c0415974>] smp_send_reschedule+0x3/0x53 SS:ESP 0068:f4f2fc8c

<0>Kernel panic - not syncing: Fatal exception

#1 lore raw
Quoted 55 lines · Chris Boot

All,

I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.

Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?

BUG: unable to handle kernel paging request at virtual address b8af9d60
printing eip:
c0415974
*pde = 00000000
Oops: 0000 [#1]
SMP last sysfs file: /block/loop7/dev
Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U)

autofs4 hidp nls_utf8 cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbs i2c_ec button battery asus_acpi ac lp st sg floppy serio_raw intel_rng pcspkr e100 mii e7xxx_edac i2c_i801 edac_mc i2c_core e1000 r8169 ide_cd cdrom parport_pc parport dm_snapshot dm_zero dm_mirror dm_mod cciss mptspi mptscsih scsi_transport_spi sd_mod scsi_mod mptbase ext3 jbd ehci_hcd ohci_hcd uhci_hcd

CPU:    1
EIP:    0060:[<c0415974>]    Tainted: P      VLI
EFLAGS: 00010046   (2.6.18-8.1.8.el5 #1) EIP is at
smp_send_reschedule+0x3/0x53
eax: c213f000   ebx: c213f000   ecx: eef84000   edx: c213f000
esi: 00001086   edi: f668c000   ebp: f4f2fce8   esp: f4f2fc8c
ds: 007b   es: 007b   ss: 0068
Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000)
Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000
0000000f     00000000 00000001 00000001 c200c6e0 00000100 00000000
00000069 00000180     018fc500 c200d240 00000003 00000292 f601efc0
f6027e00 00000000 00000050 Call Trace:
[<c041dc23>] try_to_wake_up+0x351/0x37b
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs]
[<c04572f9>] shrink_slab+0x56/0x13c
[<c0457c0c>] try_to_free_pages+0x162/0x23e
[<c0454064>] __alloc_pages+0x18d/0x27e
[<c045214e>] find_or_create_page+0x53/0x8c
[<c046c7b1>] __getblk+0x162/0x270
[<c0475be0>] do_lookup+0x53/0x157
[<f889138f>] ext3_getblk+0x7c/0x233 [ext3]
[<f88913fe>] ext3_getblk+0xeb/0x233 [ext3]
[<c048215c>] mntput_no_expire+0x11/0x6a
[<f889226e>] ext3_bread+0x13/0x69 [ext3]
[<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3]
[<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3]
[<c047828b>] do_path_lookup+0x20e/0x25f
[<c046b987>] get_empty_filp+0x99/0x15e
[<f889d611>] ext3_permission+0x0/0xa [ext3]
Quoted 8 lines
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3]
[<c047a0dd>] filldir+0x0/0xb9
[<c0472973>] sys_fstat64+0x1e/0x23
[<c047a1f9>] vfs_readdir+0x63/0x8d
[<c047a0dd>] filldir+0x0/0xb9
[<c047a447>] sys_getdents+0x5f/0x9c
[<c0403eff>] syscall_call+0x7/0xb
=======================

Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.

Signature · 2 lines
Måns Rullgård
[email protected]
#2 lore raw
Quoted 30 lines · Måns Rullgård
Quoted 30 lines
BUG: unable to handle kernel paging request at virtual address b8af9d60
printing eip:
c0415974
*pde = 00000000
Oops: 0000 [#1]
SMP last sysfs file: /block/loop7/dev
Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U)

autofs4 hidp nls_utf8 cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbs i2c_ec button battery asus_acpi ac lp st sg floppy serio_raw intel_rng pcspkr e100 mii e7xxx_edac i2c_i801 edac_mc i2c_core e1000 r8169 ide_cd cdrom parport_pc parport dm_snapshot dm_zero dm_mirror dm_mod cciss mptspi mptscsih scsi_transport_spi sd_mod scsi_mod mptbase ext3 jbd ehci_hcd ohci_hcd uhci_hcd

CPU:    1
EIP:    0060:[<c0415974>]    Tainted: P      VLI
EFLAGS: 00010046   (2.6.18-8.1.8.el5 #1) EIP is at
smp_send_reschedule+0x3/0x53
eax: c213f000   ebx: c213f000   ecx: eef84000   edx: c213f000
esi: 00001086   edi: f668c000   ebp: f4f2fce8   esp: f4f2fc8c
ds: 007b   es: 007b   ss: 0068
Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000)
Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000
0000000f     00000000 00000001 00000001 c200c6e0 00000100 00000000
00000069 00000180     018fc500 c200d240 00000003 00000292 f601efc0
f6027e00 00000000 00000050 Call Trace:
[<c041dc23>] try_to_wake_up+0x351/0x37b
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs]
[<c04572f9>] shrink_slab+0x56/0x13c

[...]

Quoted 3 lines

Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.

I think we should include the vermagic string in oopses too, so that the flags SMP, PREEMPT, RT, 4KSTACKS, mod_unload, etc. are shown and the situation is a bit more apparent.

Jan

#3 lore raw
In reply to Måns Rullgård
Quoted 68 lines · Måns Rullgård
Quoted 55 lines · Chris Boot

All,

I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.

Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?

BUG: unable to handle kernel paging request at virtual address b8af9d60
printing eip:
c0415974
*pde = 00000000
Oops: 0000 [#1]
SMP last sysfs file: /block/loop7/dev
Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U)

autofs4 hidp nls_utf8 cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbs i2c_ec button battery asus_acpi ac lp st sg floppy serio_raw intel_rng pcspkr e100 mii e7xxx_edac i2c_i801 edac_mc i2c_core e1000 r8169 ide_cd cdrom parport_pc parport dm_snapshot dm_zero dm_mirror dm_mod cciss mptspi mptscsih scsi_transport_spi sd_mod scsi_mod mptbase ext3 jbd ehci_hcd ohci_hcd uhci_hcd

CPU:    1
EIP:    0060:[<c0415974>]    Tainted: P      VLI
EFLAGS: 00010046   (2.6.18-8.1.8.el5 #1) EIP is at
smp_send_reschedule+0x3/0x53
eax: c213f000   ebx: c213f000   ecx: eef84000   edx: c213f000
esi: 00001086   edi: f668c000   ebp: f4f2fce8   esp: f4f2fc8c
ds: 007b   es: 007b   ss: 0068
Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000)
Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000
0000000f     00000000 00000001 00000001 c200c6e0 00000100 00000000
00000069 00000180     018fc500 c200d240 00000003 00000292 f601efc0
f6027e00 00000000 00000050 Call Trace:
[<c041dc23>] try_to_wake_up+0x351/0x37b
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs]
[<c04572f9>] shrink_slab+0x56/0x13c
[<c0457c0c>] try_to_free_pages+0x162/0x23e
[<c0454064>] __alloc_pages+0x18d/0x27e
[<c045214e>] find_or_create_page+0x53/0x8c
[<c046c7b1>] __getblk+0x162/0x270
[<c0475be0>] do_lookup+0x53/0x157
[<f889138f>] ext3_getblk+0x7c/0x233 [ext3]
[<f88913fe>] ext3_getblk+0xeb/0x233 [ext3]
[<c048215c>] mntput_no_expire+0x11/0x6a
[<f889226e>] ext3_bread+0x13/0x69 [ext3]
[<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3]
[<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3]
[<c047828b>] do_path_lookup+0x20e/0x25f
[<c046b987>] get_empty_filp+0x99/0x15e
[<f889d611>] ext3_permission+0x0/0xa [ext3]
Quoted 9 lines
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3]
[<c047a0dd>] filldir+0x0/0xb9
[<c0472973>] sys_fstat64+0x1e/0x23
[<c047a1f9>] vfs_readdir+0x63/0x8d
[<c047a0dd>] filldir+0x0/0xb9
[<c047a447>] sys_getdents+0x5f/0x9c
[<c0403eff>] syscall_call+0x7/0xb
=======================
Quoted 3 lines

Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.

Thanks, that explains a lot. However, I don't have any XFS filesystems mounted over loop devices on ext3. Earlier in the day I had iso9660 on loop on xfs, could that have caused the issue? It was unmounted and deleted when this panic occurred.

I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P

Many thanks,
Chris

#4 lore raw
Quoted 22 lines · Chris Boot
Quoted 21 lines · Måns Rullgård
Quoted 18 lines · Chris Boot

All,

I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.

Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?

BUG: unable to handle kernel paging request at virtual address b8af9d60
printing eip:
c0415974
*pde = 00000000
Oops: 0000 [#1]

SMP last sysfs file: /block/loop7/dev

[...]

Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs]
[<c04572f9>] shrink_slab+0x56/0x13c
[<c0457c0c>] try_to_free_pages+0x162/0x23e
[<c0454064>] __alloc_pages+0x18d/0x27e
[<c045214e>] find_or_create_page+0x53/0x8c
[<c046c7b1>] __getblk+0x162/0x270
[<c0475be0>] do_lookup+0x53/0x157
[<f889138f>] ext3_getblk+0x7c/0x233 [ext3]
[<f88913fe>] ext3_getblk+0xeb/0x233 [ext3]
[<c048215c>] mntput_no_expire+0x11/0x6a
[<f889226e>] ext3_bread+0x13/0x69 [ext3]
[<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3]
[<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3]
[<c047828b>] do_path_lookup+0x20e/0x25f
[<c046b987>] get_empty_filp+0x99/0x15e
[<f889d611>] ext3_permission+0x0/0xa [ext3]
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3]
[<c047a0dd>] filldir+0x0/0xb9
[<c0472973>] sys_fstat64+0x1e/0x23
[<c047a1f9>] vfs_readdir+0x63/0x8d
[<c047a0dd>] filldir+0x0/0xb9
[<c047a447>] sys_getdents+0x5f/0x9c
[<c0403eff>] syscall_call+0x7/0xb
=======================
Quoted 7 lines
Quoted 3 lines

Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.

Thanks, that explains a lot. However, I don't have any XFS filesystems mounted over loop devices on ext3. Earlier in the day I had iso9660 on loop on xfs, could that have caused the issue? It was unmounted and deleted when this panic occurred.

The mention of /block/loop7/dev and the presence both XFS and ext3 function in the call stack suggested to me that you might have an ext3 filesystem in a loop device on XFS. I see no other explanation for that call stack other than a stack overflow, but then we're still back at the same root cause.

Are you using device-mapper and/or md? They too are known to blow 4k stacks when used with XFS.

Quoted 2 lines

I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P

Please report how this works out.

Signature · 2 lines
Måns Rullgård
[email protected]
#5 lore raw
Quoted 66 lines · Måns Rullgård
Quoted 25 lines · Chris Boot
Quoted 23 lines · Måns Rullgård
Quoted 19 lines · Chris Boot

All,

I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.

Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?

BUG: unable to handle kernel paging request at virtual address b8af9d60
printing eip:
c0415974
*pde = 00000000
Oops: 0000 [#1]

SMP last sysfs file: /block/loop7/dev

[...]

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs]
[<c04572f9>] shrink_slab+0x56/0x13c
[<c0457c0c>] try_to_free_pages+0x162/0x23e
[<c0454064>] __alloc_pages+0x18d/0x27e
[<c045214e>] find_or_create_page+0x53/0x8c
[<c046c7b1>] __getblk+0x162/0x270
[<c0475be0>] do_lookup+0x53/0x157
[<f889138f>] ext3_getblk+0x7c/0x233 [ext3]
[<f88913fe>] ext3_getblk+0xeb/0x233 [ext3]
[<c048215c>] mntput_no_expire+0x11/0x6a
[<f889226e>] ext3_bread+0x13/0x69 [ext3]
[<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3]
[<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3]
[<c047828b>] do_path_lookup+0x20e/0x25f
[<c046b987>] get_empty_filp+0x99/0x15e
[<f889d611>] ext3_permission+0x0/0xa [ext3]
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3]
[<c047a0dd>] filldir+0x0/0xb9
[<c0472973>] sys_fstat64+0x1e/0x23
[<c047a1f9>] vfs_readdir+0x63/0x8d
[<c047a0dd>] filldir+0x0/0xb9
[<c047a447>] sys_getdents+0x5f/0x9c
[<c0403eff>] syscall_call+0x7/0xb
=======================
Quoted 9 lines
Quoted 4 lines

Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.

Thanks, that explains a lot. However, I don't have any XFS filesystems mounted over loop devices on ext3. Earlier in the day I had iso9660 on loop on xfs, could that have caused the issue? It was unmounted and deleted when this panic occurred.

Quoted 9 lines

The mention of /block/loop7/dev and the presence both XFS and ext3 function in the call stack suggested to me that you might have an ext3 filesystem in a loop device on XFS. I see no other explanation for that call stack other than a stack overflow, but then we're still back at the same root cause.

Are you using device-mapper and/or md? They too are known to blow 4k stacks when used with XFS.

I am. The situation was earlier on was iso9660 on loop on xfs on lvm on cciss. I guess that might have smashed the stack undetectably and induced corruption encountered later on? When I experienced this panic the machine would have probably been performing a backup, which was simply a load of ext3/xfs filesystems on lvm on the HP cciss controller. None of the loop devices would have been mounted.

I have a few machines now with 4k stacks and using lvm + md + xfs and have no trouble at all, but none are Red Hat (all Debian) and none use cciss either. Maybe it's a deadly combination.

Quoted 6 lines
Quoted 3 lines

I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P

Please report how this works out.

I will. This will probably be on Monday now, since the machine isn't accepting SysRq requests over the serial console. :-(

Many thanks,
Chris

#6 lore raw
Quoted 3 lines · Chris Boot

I will. This will probably be on Monday now, since the machine isn't accepting SysRq requests over the serial console. :-(

Ah yeah, stupid null-modem cables!
You can also trigger sysrq from /proc/sysrq-trigger (well, as long as the system lives)

Jan

#7 lore raw
In reply to Chris Boot
Quoted 8 lines · Chris Boot
Quoted 6 lines
Quoted 3 lines

I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P

Please report how this works out.

I will. This will probably be on Monday now, since the machine isn't accepting SysRq requests over the serial console. :-(

OK, with the recompiled kernel this appears to work just fine now. I've been pounding the box all day with rsyncs, VMware VMs, plenty of web serving (inc. SVN) and so far it's holding up just fine. Cheers for the diagnosis.

Many thanks,
Chris