Panic with XFS on RHEL5 (2.6.18-8.1.8.el5)
All,
I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.
Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?
Many thanks,
Chris Boot
BUG: unable to handle kernel paging request at virtual address b8af9d60
printing eip:
c0415974
*pde = 00000000
Oops: 0000 [#1]
SMP last sysfs file: /block/loop7/dev
Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U) autofs4 hidp nls_utf8 cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbs i2c_ec button battery asus_acpi ac lp st sg floppy serio_raw intel_rng pcspkr e100 mii e7xxx_edac i2c_i801 edac_mc i2c_core e1000 r8169 ide_cd cdrom parport_pc parport dm_snapshot dm_zero dm_mirror dm_mod cciss mptspi mptscsih scsi_transport_spi sd_mod scsi_mod mptbase ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU: 1
EIP: 0060:[<c0415974>] Tainted: P VLI
EFLAGS: 00010046 (2.6.18-8.1.8.el5 #1) EIP is at smp_send_reschedule+0x3/0x53
eax: c213f000 ebx: c213f000 ecx: eef84000 edx: c213f000
esi: 00001086 edi: f668c000 ebp: f4f2fce8 esp: f4f2fc8c
ds: 007b es: 007b ss: 0068
Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000)
Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000 0000000f
00000000 00000001 00000001 c200c6e0 00000100 00000000 00000069
00000180
018fc500 c200d240 00000003 00000292 f601efc0 f6027e00 00000000
00000050 Call Trace:
[<c041dc23>] try_to_wake_up+0x351/0x37b
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs]
[<c04572f9>] shrink_slab+0x56/0x13c
[<c0457c0c>] try_to_free_pages+0x162/0x23e
[<c0454064>] __alloc_pages+0x18d/0x27e
[<c045214e>] find_or_create_page+0x53/0x8c
[<c046c7b1>] __getblk+0x162/0x270
[<c0475be0>] do_lookup+0x53/0x157
[<f889138f>] ext3_getblk+0x7c/0x233 [ext3]
[<f88913fe>] ext3_getblk+0xeb/0x233 [ext3]
[<c048215c>] mntput_no_expire+0x11/0x6a
[<f889226e>] ext3_bread+0x13/0x69 [ext3]
[<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3]
[<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3]
[<c047828b>] do_path_lookup+0x20e/0x25f
[<c046b987>] get_empty_filp+0x99/0x15e
[<f889d611>] ext3_permission+0x0/0xa [ext3]
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3]
[<c047a0dd>] filldir+0x0/0xb9
[<c0472973>] sys_fstat64+0x1e/0x23
[<c047a1f9>] vfs_readdir+0x63/0x8d
[<c047a0dd>] filldir+0x0/0xb9
[<c047a447>] sys_getdents+0x5f/0x9c
[<c0403eff>] syscall_call+0x7/0xb
=======================
Code: 5d c3 b9 01 00 00 00 31 d2 6a 00 b8 f0 5a 41 c0 e8 2a ff ff ff fa e8 52 16 00 00 fb 58 c3 b8 54 3a 66 c0 e9 8e 6b 1e 00 53 89 c3 <0f> a3 05 60 1f 6d c0 19 c0 85 c0 75 27 e8 bf db 00 00 50 68 55 EIP: [<c0415974>] smp_send_reschedule+0x3/0x53 SS:ESP 0068:f4f2fc8c<0>Kernel panic - not syncing: Fatal exception
Quoted 55 lines · Chris Boot
All,
I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.
Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?
BUG: unable to handle kernel paging request at virtual address b8af9d60 printing eip: c0415974 *pde = 00000000 Oops: 0000 [#1] SMP last sysfs file: /block/loop7/dev Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U)autofs4 hidp
nls_utf8cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbsi2c_ecbutton batteryasus_acpiac lp st sg floppyserio_rawintel_rngpcspkr e100 miie7xxx_edaci2c_i801edac_mci2c_coree1000 r8169ide_cdcdromparport_pcparportdm_snapshotdm_zerodm_mirrordm_modcciss mptspi mptscsihscsi_transport_spisd_modscsi_modmptbase ext3 jbdehci_hcdohci_hcduhci_hcdCPU: 1 EIP: 0060:[<c0415974>] Tainted: P VLI EFLAGS: 00010046 (2.6.18-8.1.8.el5 #1) EIP is at smp_send_reschedule+0x3/0x53 eax: c213f000 ebx: c213f000 ecx: eef84000 edx: c213f000 esi: 00001086 edi: f668c000 ebp: f4f2fce8 esp: f4f2fc8c ds: 007b es: 007b ss: 0068 Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000) Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000 0000000f 00000000 00000001 00000001 c200c6e0 00000100 00000000 00000069 00000180 018fc500 c200d240 00000003 00000292 f601efc0 f6027e00 00000000 00000050 Call Trace: [<c041dc23>] try_to_wake_up+0x351/0x37b [<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs] [<c04572f9>] shrink_slab+0x56/0x13c [<c0457c0c>] try_to_free_pages+0x162/0x23e [<c0454064>] __alloc_pages+0x18d/0x27e [<c045214e>] find_or_create_page+0x53/0x8c [<c046c7b1>] __getblk+0x162/0x270 [<c0475be0>] do_lookup+0x53/0x157 [<f889138f>] ext3_getblk+0x7c/0x233 [ext3] [<f88913fe>] ext3_getblk+0xeb/0x233 [ext3] [<c048215c>] mntput_no_expire+0x11/0x6a [<f889226e>] ext3_bread+0x13/0x69 [ext3] [<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3] [<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3] [<c047828b>] do_path_lookup+0x20e/0x25f [<c046b987>] get_empty_filp+0x99/0x15e [<f889d611>] ext3_permission+0x0/0xa [ext3]
Quoted 8 lines
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3] [<c047a0dd>] filldir+0x0/0xb9 [<c0472973>] sys_fstat64+0x1e/0x23 [<c047a1f9>] vfs_readdir+0x63/0x8d [<c047a0dd>] filldir+0x0/0xb9 [<c047a447>] sys_getdents+0x5f/0x9c [<c0403eff>] syscall_call+0x7/0xb =======================
Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 30 lines · Måns Rullgård
Quoted 30 lines
BUG: unable to handle kernel paging request at virtual address b8af9d60 printing eip: c0415974 *pde = 00000000 Oops: 0000 [#1] SMP last sysfs file: /block/loop7/dev Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U)autofs4 hidp
nls_utf8cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbsi2c_ecbutton batteryasus_acpiac lp st sg floppyserio_rawintel_rngpcspkr e100 miie7xxx_edaci2c_i801edac_mci2c_coree1000 r8169ide_cdcdromparport_pcparportdm_snapshotdm_zerodm_mirrordm_modcciss mptspi mptscsihscsi_transport_spisd_modscsi_modmptbase ext3 jbdehci_hcdohci_hcduhci_hcdCPU: 1 EIP: 0060:[<c0415974>] Tainted: P VLI EFLAGS: 00010046 (2.6.18-8.1.8.el5 #1) EIP is at smp_send_reschedule+0x3/0x53 eax: c213f000 ebx: c213f000 ecx: eef84000 edx: c213f000 esi: 00001086 edi: f668c000 ebp: f4f2fce8 esp: f4f2fc8c ds: 007b es: 007b ss: 0068 Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000) Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000 0000000f 00000000 00000001 00000001 c200c6e0 00000100 00000000 00000069 00000180 018fc500 c200d240 00000003 00000292 f601efc0 f6027e00 00000000 00000050 Call Trace: [<c041dc23>] try_to_wake_up+0x351/0x37b [<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs] [<c04572f9>] shrink_slab+0x56/0x13c
[...]
Quoted 3 lines
Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.
I think we should include the vermagic string in oopses too, so that the flags SMP, PREEMPT, RT, 4KSTACKS, mod_unload, etc. are shown and the situation is a bit more apparent.
Jan
Quoted 68 lines · Måns Rullgård
Quoted 55 lines · Chris Boot
All,
I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.
Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?
BUG: unable to handle kernel paging request at virtual address b8af9d60 printing eip: c0415974 *pde = 00000000 Oops: 0000 [#1] SMP last sysfs file: /block/loop7/dev Modules linked in: loop nfsd exportfs lockd nfs_acl iscsi_trgt(U)autofs4 hidp
nls_utf8cifs ppdev rfcomm l2cap bluetooth vmnet(U) vmmon(U) sunrpc ipv6 xfs(U) video sbsi2c_ecbutton batteryasus_acpiac lp st sg floppyserio_rawintel_rngpcspkr e100 miie7xxx_edaci2c_i801edac_mci2c_coree1000 r8169ide_cdcdromparport_pcparportdm_snapshotdm_zerodm_mirrordm_modcciss mptspi mptscsihscsi_transport_spisd_modscsi_modmptbase ext3 jbdehci_hcdohci_hcduhci_hcdCPU: 1 EIP: 0060:[<c0415974>] Tainted: P VLI EFLAGS: 00010046 (2.6.18-8.1.8.el5 #1) EIP is at smp_send_reschedule+0x3/0x53 eax: c213f000 ebx: c213f000 ecx: eef84000 edx: c213f000 esi: 00001086 edi: f668c000 ebp: f4f2fce8 esp: f4f2fc8c ds: 007b es: 007b ss: 0068 Process crond (pid: 3146, ti=f4f2f000 task=f51faaa0 task.ti=f4f2f000) Stack: 66d66b89 c041dc23 00000000 a9afbb0e fffffea5 01904500 00000000 0000000f 00000000 00000001 00000001 c200c6e0 00000100 00000000 00000069 00000180 018fc500 c200d240 00000003 00000292 f601efc0 f6027e00 00000000 00000050 Call Trace: [<c041dc23>] try_to_wake_up+0x351/0x37b [<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs] [<c04572f9>] shrink_slab+0x56/0x13c [<c0457c0c>] try_to_free_pages+0x162/0x23e [<c0454064>] __alloc_pages+0x18d/0x27e [<c045214e>] find_or_create_page+0x53/0x8c [<c046c7b1>] __getblk+0x162/0x270 [<c0475be0>] do_lookup+0x53/0x157 [<f889138f>] ext3_getblk+0x7c/0x233 [ext3] [<f88913fe>] ext3_getblk+0xeb/0x233 [ext3] [<c048215c>] mntput_no_expire+0x11/0x6a [<f889226e>] ext3_bread+0x13/0x69 [ext3] [<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3] [<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3] [<c047828b>] do_path_lookup+0x20e/0x25f [<c046b987>] get_empty_filp+0x99/0x15e [<f889d611>] ext3_permission+0x0/0xa [ext3]Quoted 9 lines
[<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3] [<c047a0dd>] filldir+0x0/0xb9 [<c0472973>] sys_fstat64+0x1e/0x23 [<c047a1f9>] vfs_readdir+0x63/0x8d [<c047a0dd>] filldir+0x0/0xb9 [<c047a447>] sys_getdents+0x5f/0x9c [<c0403eff>] syscall_call+0x7/0xb =======================
Quoted 3 lines
Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.
Thanks, that explains a lot. However, I don't have any XFS filesystems mounted over loop devices on ext3. Earlier in the day I had iso9660 on loop on xfs, could that have caused the issue? It was unmounted and deleted when this panic occurred.
I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P
Many thanks,
Chris
Quoted 22 lines · Chris Boot
Quoted 21 lines · Måns Rullgård
Quoted 18 lines · Chris Boot
All,
I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.
Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?
BUG: unable to handle kernel paging request at virtual address b8af9d60 printing eip: c0415974 *pde = 00000000 Oops: 0000 [#1]SMP last sysfs file: /block/loop7/dev
[...]
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs] [<c04572f9>] shrink_slab+0x56/0x13c [<c0457c0c>] try_to_free_pages+0x162/0x23e [<c0454064>] __alloc_pages+0x18d/0x27e [<c045214e>] find_or_create_page+0x53/0x8c [<c046c7b1>] __getblk+0x162/0x270 [<c0475be0>] do_lookup+0x53/0x157 [<f889138f>] ext3_getblk+0x7c/0x233 [ext3] [<f88913fe>] ext3_getblk+0xeb/0x233 [ext3] [<c048215c>] mntput_no_expire+0x11/0x6a [<f889226e>] ext3_bread+0x13/0x69 [ext3] [<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3] [<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3] [<c047828b>] do_path_lookup+0x20e/0x25f [<c046b987>] get_empty_filp+0x99/0x15e [<f889d611>] ext3_permission+0x0/0xa [ext3] [<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3] [<c047a0dd>] filldir+0x0/0xb9 [<c0472973>] sys_fstat64+0x1e/0x23 [<c047a1f9>] vfs_readdir+0x63/0x8d [<c047a0dd>] filldir+0x0/0xb9 [<c047a447>] sys_getdents+0x5f/0x9c [<c0403eff>] syscall_call+0x7/0xb =======================
Quoted 7 lines
Quoted 3 lines
Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.
Thanks, that explains a lot. However, I don't have any XFS filesystems mounted over loop devices on ext3. Earlier in the day I had iso9660 on loop on xfs, could that have caused the issue? It was unmounted and deleted when this panic occurred.
The mention of /block/loop7/dev and the presence both XFS and ext3 function in the call stack suggested to me that you might have an ext3 filesystem in a loop device on XFS. I see no other explanation for that call stack other than a stack overflow, but then we're still back at the same root cause.
Are you using device-mapper and/or md? They too are known to blow 4k stacks when used with XFS.
Quoted 2 lines
I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P
Please report how this works out.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 66 lines · Måns Rullgård
Quoted 25 lines · Chris Boot
Quoted 23 lines · Måns Rullgård
Quoted 19 lines · Chris Boot
All,
I've got a box running RHEL5 and haven't been impressed by ext3 performance on it (running of a 1.5TB HP MSA20 using the cciss driver). I compiled XFS as a module and tried it out since I'm used to using it on Debian, which runs much more efficiently. However, every so often the kernel panics as below. Apologies for the tainted kernel, but we run VMware Server on the box as well.
Does anyone have any hits/tips for using XFS on Red Hat? What's causing the panic below, and is there a way around this?
BUG: unable to handle kernel paging request at virtual address b8af9d60 printing eip: c0415974 *pde = 00000000 Oops: 0000 [#1]SMP last sysfs file: /block/loop7/dev
[...]
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
[<f936884e>] xfsbufd_wakeup+0x28/0x49 [xfs] [<c04572f9>] shrink_slab+0x56/0x13c [<c0457c0c>] try_to_free_pages+0x162/0x23e [<c0454064>] __alloc_pages+0x18d/0x27e [<c045214e>] find_or_create_page+0x53/0x8c [<c046c7b1>] __getblk+0x162/0x270 [<c0475be0>] do_lookup+0x53/0x157 [<f889138f>] ext3_getblk+0x7c/0x233 [ext3] [<f88913fe>] ext3_getblk+0xeb/0x233 [ext3] [<c048215c>] mntput_no_expire+0x11/0x6a [<f889226e>] ext3_bread+0x13/0x69 [ext3] [<f8895606>] htree_dirblock_to_tree+0x22/0x113 [ext3] [<f889574f>] ext3_htree_fill_tree+0x58/0x1a0 [ext3] [<c047828b>] do_path_lookup+0x20e/0x25f [<c046b987>] get_empty_filp+0x99/0x15e [<f889d611>] ext3_permission+0x0/0xa [ext3] [<f888eaa3>] ext3_readdir+0x1ce/0x59b [ext3] [<c047a0dd>] filldir+0x0/0xb9 [<c0472973>] sys_fstat64+0x1e/0x23 [<c047a1f9>] vfs_readdir+0x63/0x8d [<c047a0dd>] filldir+0x0/0xb9 [<c047a447>] sys_getdents+0x5f/0x9c [<c0403eff>] syscall_call+0x7/0xb =======================Quoted 9 lines
Quoted 4 lines
Your Redhat kernel is probably built with 4k stacks and XFS+loop+ext3 seems to be enough to overflow it.
Thanks, that explains a lot. However, I don't have any XFS filesystems mounted over loop devices on ext3. Earlier in the day I had iso9660 on loop on xfs, could that have caused the issue? It was unmounted and deleted when this panic occurred.
Quoted 9 lines
The mention of /block/loop7/dev and the presence both XFS and ext3 function in the call stack suggested to me that you might have an ext3 filesystem in a loop device on XFS. I see no other explanation for that call stack other than a stack overflow, but then we're still back at the same root cause.
Are you using device-mapper and/or md? They too are known to blow 4k stacks when used with XFS.
I am. The situation was earlier on was iso9660 on loop on xfs on lvm on cciss. I guess that might have smashed the stack undetectably and induced corruption encountered later on? When I experienced this panic the machine would have probably been performing a backup, which was simply a load of ext3/xfs filesystems on lvm on the HP cciss controller. None of the loop devices would have been mounted.
I have a few machines now with 4k stacks and using lvm + md + xfs and have no trouble at all, but none are Red Hat (all Debian) and none use cciss either. Maybe it's a deadly combination.
Quoted 6 lines
Quoted 3 lines
I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P
Please report how this works out.
I will. This will probably be on Monday now, since the machine isn't accepting SysRq requests over the serial console. :-(
Many thanks,
Chris
Quoted 3 lines · Chris Boot
I will. This will probably be on Monday now, since the machine isn't accepting SysRq requests over the serial console. :-(
Ah yeah, stupid null-modem cables!
You can also trigger sysrq from /proc/sysrq-trigger (well, as long as the system lives)
Jan
Quoted 8 lines · Chris Boot
Quoted 6 lines
Quoted 3 lines
I'll probably just try and recompile the kernel with 8k stacks and see how it goes. Screw the support, we're unlikely to get it anyway. :-P
Please report how this works out.
I will. This will probably be on Monday now, since the machine isn't accepting SysRq requests over the serial console. :-(
OK, with the recompiled kernel this appears to work just fine now. I've been pounding the box all day with rsyncs, VMware VMs, plenty of web serving (inc. SVN) and so far it's holding up just fine. Cheers for the diagnosis.
Many thanks,
Chris