Skip to contentSkip to conversation

GPF oops on 2.6.18-1.2200.fc5 and repeated DWARF2 unwinder XFS errors under 2.6.18-1.2239.fc5

DISCUSSION [email protected] 2 participants 2 messages Nov 15, 2006 – Nov 16, 2006

lore
Share & tools
lore
#0 lore raw

Hi,

I just started up a new box yesterday with Fedora Core 5. Its running with 2 dual core AMD Opteron 2220 SE's and 24Gb of memory and an Adaptec SCSI card and I've had a number of errors which I can't seem to find solutions for. I'd had no end of problems with spinlock issues in the aacraid driver in the 2.6.17 series on another dual opteron box, but on hitting 2.6.18-1.2200 these went away, so I started the new box off with 2.6.18-1.2200 as well. As I understand it, this is 2.6.18.1 as compiled by Redhat/Fedora and includes various DWARD2 unwinder fixes.

Well this caused a GPF and the following trace:

-----------

general protection fault: 0000 [1] SMP
last sysfs file: /class/net/sit0/address
CPU 1
Modules linked in: nls_utf8 ipv6 ip_conntrack_ftp ip_conntrack_netbios_ns ipt_owner ipt_LOG xt_limit ipt_REJECT xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables xfs dm_mod video sbs i2c_ec button battery asus_acpi ac lp parport_pc parport ide_cd cdrom sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core forcedeth serio_raw k8_edac edac_mc shpchp pcspkr ext3 jbd sata_nv libata aacraid sd_mod scsi_mod
Pid: 1093, comm: gawk Not tainted 2.6.18-1.2200.fc5 #1
RIP: 0010:[<ffffffff8826b4c5>]  [<ffffffff8826b4c5>]
:xfs:xfs_bmap_search_extents+0x1c/0xcb
RSP: 0018:ffff8105fd653b40  EFLAGS: 00010202
RAX: ffffffff806785a0 RBX: ffff8105fd653d28 RCX: ffff8105fd653d70
RDX: 0000000000000000 RSI: 00000000000033ce RDI: ffff8102fe801080
RBP: ffff8105fd653b40 R08: ffff8105fd653d6c R09: ffff8105fd653d28
R10: ffff8105fd653d70 R11: ffff8102f4655250 R12: ffff8105fd653d6c
R13: ffff8105ff04d800 R14: 0007ffffffffcc32 R15: ffff8105fd653de8
FS:  00002aaaab093e00(0000) GS:ffff8102ffc3b1c0(0000)

knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaaae4a020 CR3: 0000000000201000 CR4: 00000000000006e0 Process gawk (pid: 1093, threadinfo ffff8105fd652000, task ffff8105fd4f4810)

Stack:  ffff8102fe801080 0000000000000005 0000000000000000 ffff8105ff04d800
 ffffffff8826b972 ffff8105fd653d08 0000000000000007 0000000000000048
 0000000000000000 000000000000029b 0000000000100000 ffff8105fd653c18
Call Trace:
 [<ffffffff8826b972>] :xfs:xfs_bmapi+0x2d2/0x1b66
 [<ffffffff8829dfba>] :xfs:xfs_inactive_free_eofblocks+0xa3/0x1ec
 [<ffffffff882a13cc>] :xfs:xfs_release+0x97/0xc8
 [<ffffffff882a820e>] :xfs:xfs_file_release+0x1a/0x1e
 [<ffffffff8021239b>] __fput+0xbf/0x1aa
 [<ffffffff8021a4de>] remove_vma+0x4e/0x75
 [<ffffffff8023a035>] exit_mmap+0xcf/0xf3
 [<ffffffff8023c1c1>] mmput+0x41/0x96
 [<ffffffff802150e2>] do_exit+0x28c/0x8c3
 [<ffffffff80247d0e>] cpuset_exit+0x0/0x6c
 [<00002aaaab089888>]


Code: 18 4c 8b 4c 24 40 65 8b 0c 25 2c 00 00 00 48 63 c9 48 8b 0c
RIP  [<ffffffff8826b4c5>] :xfs:xfs_bmap_search_extents+0x1c/0xcb
 RSP <ffff8105fd653b40>
 <1>Fixing recursive fault but reboot is needed!

-----------

At the time the box was sitting there doing nothing but running openssh. (This gawk process seems to be from anacron kicking in 'makewhatis'). The machine didn't die but didn't seem happy. I searching I discovered a number of people with the same message "general protection fault: 0000 [1] SMP" on lots of different processes so I assumed that it wasn't related to the XFS drivers directly, but to a problem somewhere else which is being triggered by the dual-core opterons (could heat be a factor as its just sitting on a desk in the office not in a machine room?).

Anyway since this had happened I decided to upgrade to the next Fedora kernel 2.6.18-1.2239.fc5 which appears to be 2.6.18.2 + some redhat/fedora patches (mostly for Xen, which I'm not running). This sit there for a few hours and hadn't thrown an error so I decided to upload some data to it overnight ready for the morning. As soon as I did I started getting traces for:

-----------
Filesystem "sda5": XFS internal error xfs_btree_check_sblock at line 334 of file fs/xfs/xfs_btree.c. Caller 0xffffffff8825e203

Call Trace:
 [<ffffffff802691d9>] show_trace+0x34/0x47
 [<ffffffff802691fe>] dump_stack+0x12/0x17
 [<ffffffff88272bb4>] :xfs:xfs_btree_check_sblock+0xbc/0xcc
 [<ffffffff8825e203>] :xfs:xfs_alloc_lookup+0x14f/0x39a
 [<ffffffff8825bed3>] :xfs:xfs_alloc_ag_vextent+0x74/0xf61
 [<ffffffff8825d116>] :xfs:xfs_alloc_fix_freelist+0x356/0x410
 [<ffffffff8825d54a>] :xfs:xfs_alloc_vextent+0x2ae/0x400
 [<ffffffff8826b578>] :xfs:xfs_bmapi+0xed6/0x1b66
 [<ffffffff8828ba33>] :xfs:xfs_iomap_write_allocate+0x257/0x3fc
 [<ffffffff8828aa3a>] :xfs:xfs_iomap+0x31a/0x521
 [<ffffffff882a38f0>] :xfs:xfs_map_blocks+0x2f/0x5f
 [<ffffffff882a3c46>] :xfs:xfs_page_state_convert+0x2b7/0xb63
 [<ffffffff882a4724>] :xfs:xfs_vm_writepage+0xa7/0xde
 [<ffffffff8021c78f>] mpage_writepages+0x1d0/0x395
 [<ffffffff80259e0f>] do_writepages+0x23/0x32
 [<ffffffff8024e2b8>] __filemap_fdatawrite_range+0x54/0x5e
 [<ffffffff882a779d>] :xfs:fs_flush_pages+0x4b/0x64
 [<ffffffff882a71ec>] :xfs:xfs_file_close+0x2a/0x2e
 [<ffffffff80223b7f>] filp_close+0x36/0x64
 [<ffffffff8021d873>] sys_close+0x8f/0xaa
 [<ffffffff8025c181>] tracesys+0xd1/0xdc
DWARF2 unwinder stuck at tracesys+0xd1/0xdc

Leftover inexact backtrace:
-----------

I first booted into 2.6.18-1.2239.fc5 in single user mode and forced a check of the disk with xfs_repair and I'm using xfs-progs-2.8.11 as I discovered on my other system that the 2.6.17 XFS kernel driver bugs were breaking the FS in a way that the xfs-progs-2.7.x code didn't fix.

These XFS bugs seem to be the same problems that were cropping up in the 2.6.17 series which were resolved in 2.6.18.1 (2.6.18-1.2200.fc5).

Any suggestions are greatly appreciated. Also please let me know if more details are required.

Should I just simply go back to ext3? I'd prefer not to because of the fsck'ing time on a 1Tb array, but if it means that the kernel doesn't throw a hissy fit then I'll be more than happy to do that.

Regards,
Colin.

thor# uname -a
Linux thor 2.6.18-1.2239.fc5 #1 SMP Fri Nov 10 12:51:06 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

thor# cat /proc/cmdline
ro root=LABEL=/

Adaptec aacraid driver (1.1-5[2409]-mh2)

processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2220 SE
stepping : 2
cpu MHz : 2800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
bogomips : 5639.77
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

Signature · 2 lines
 "Developers are like artists; they produce their best work if they
  have the freedom to do so" - Werner Vogels, CTO Amazon.com
#1 lore raw
Quoted 103 lines · On Wed, Nov 15, 2006 at 03:06:16PM +0000, [email protected] wrote:

Hi,

I just started up a new box yesterday with Fedora Core 5. Its running with 2 dual core AMD Opteron 2220 SE's and 24Gb of memory and an Adaptec SCSI card and I've had a number of errors which I can't seem to find solutions for. I'd had no end of problems with spinlock issues in the aacraid driver in the 2.6.17 series on another dual opteron box, but on hitting 2.6.18-1.2200 these went away, so I started the new box off with 2.6.18-1.2200 as well. As I understand it, this is 2.6.18.1 as compiled by Redhat/Fedora and includes various DWARD2 unwinder fixes.

Well this caused a GPF and the following trace:

-----------

general protection fault: 0000 [1] SMP
last sysfs file: /class/net/sit0/address
CPU 1
Modules linked in: nls_utf8 ipv6 ip_conntrack_ftp ip_conntrack_netbios_ns ipt_owner ipt_LOG xt_limit ipt_REJECT xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables xfs dm_mod video sbs i2c_ec button battery asus_acpi ac lp parport_pc parport ide_cd cdrom sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core forcedeth serio_raw k8_edac edac_mc shpchp pcspkr ext3 jbd sata_nv libata aacraid sd_mod scsi_mod
Pid: 1093, comm: gawk Not tainted 2.6.18-1.2200.fc5 #1
RIP: 0010:[<ffffffff8826b4c5>]  [<ffffffff8826b4c5>]
:xfs:xfs_bmap_search_extents+0x1c/0xcb
RSP: 0018:ffff8105fd653b40  EFLAGS: 00010202
RAX: ffffffff806785a0 RBX: ffff8105fd653d28 RCX: ffff8105fd653d70
RDX: 0000000000000000 RSI: 00000000000033ce RDI: ffff8102fe801080
RBP: ffff8105fd653b40 R08: ffff8105fd653d6c R09: ffff8105fd653d28
R10: ffff8105fd653d70 R11: ffff8102f4655250 R12: ffff8105fd653d6c
R13: ffff8105ff04d800 R14: 0007ffffffffcc32 R15: ffff8105fd653de8
FS:  00002aaaab093e00(0000) GS:ffff8102ffc3b1c0(0000)

knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaaae4a020 CR3: 0000000000201000 CR4: 00000000000006e0 Process gawk (pid: 1093, threadinfo ffff8105fd652000, task ffff8105fd4f4810)

Stack:  ffff8102fe801080 0000000000000005 0000000000000000 ffff8105ff04d800
 ffffffff8826b972 ffff8105fd653d08 0000000000000007 0000000000000048
 0000000000000000 000000000000029b 0000000000100000 ffff8105fd653c18
Call Trace:
 [<ffffffff8826b972>] :xfs:xfs_bmapi+0x2d2/0x1b66
 [<ffffffff8829dfba>] :xfs:xfs_inactive_free_eofblocks+0xa3/0x1ec
 [<ffffffff882a13cc>] :xfs:xfs_release+0x97/0xc8
 [<ffffffff882a820e>] :xfs:xfs_file_release+0x1a/0x1e
 [<ffffffff8021239b>] __fput+0xbf/0x1aa
 [<ffffffff8021a4de>] remove_vma+0x4e/0x75
 [<ffffffff8023a035>] exit_mmap+0xcf/0xf3
 [<ffffffff8023c1c1>] mmput+0x41/0x96
 [<ffffffff802150e2>] do_exit+0x28c/0x8c3
 [<ffffffff80247d0e>] cpuset_exit+0x0/0x6c
 [<00002aaaab089888>]


Code: 18 4c 8b 4c 24 40 65 8b 0c 25 2c 00 00 00 48 63 c9 48 8b 0c
RIP  [<ffffffff8826b4c5>] :xfs:xfs_bmap_search_extents+0x1c/0xcb
 RSP <ffff8105fd653b40>
 <1>Fixing recursive fault but reboot is needed!

-----------

At the time the box was sitting there doing nothing but running openssh. (This gawk process seems to be from anacron kicking in 'makewhatis'). The machine didn't die but didn't seem happy. I searching I discovered a number of people with the same message "general protection fault: 0000 [1] SMP" on lots of different processes so I assumed that it wasn't related to the XFS drivers directly, but to a problem somewhere else which is being triggered by the dual-core opterons (could heat be a factor as its just sitting on a desk in the office not in a machine room?).

Anyway since this had happened I decided to upgrade to the next Fedora kernel 2.6.18-1.2239.fc5 which appears to be 2.6.18.2 + some redhat/fedora patches (mostly for Xen, which I'm not running). This sit there for a few hours and hadn't thrown an error so I decided to upload some data to it overnight ready for the morning. As soon as I did I started getting traces for:

-----------
Filesystem "sda5": XFS internal error xfs_btree_check_sblock at line 334 of file fs/xfs/xfs_btree.c. Caller 0xffffffff8825e203

Call Trace:
 [<ffffffff802691d9>] show_trace+0x34/0x47
 [<ffffffff802691fe>] dump_stack+0x12/0x17
 [<ffffffff88272bb4>] :xfs:xfs_btree_check_sblock+0xbc/0xcc
 [<ffffffff8825e203>] :xfs:xfs_alloc_lookup+0x14f/0x39a
 [<ffffffff8825bed3>] :xfs:xfs_alloc_ag_vextent+0x74/0xf61
 [<ffffffff8825d116>] :xfs:xfs_alloc_fix_freelist+0x356/0x410
 [<ffffffff8825d54a>] :xfs:xfs_alloc_vextent+0x2ae/0x400
 [<ffffffff8826b578>] :xfs:xfs_bmapi+0xed6/0x1b66
 [<ffffffff8828ba33>] :xfs:xfs_iomap_write_allocate+0x257/0x3fc
 [<ffffffff8828aa3a>] :xfs:xfs_iomap+0x31a/0x521
 [<ffffffff882a38f0>] :xfs:xfs_map_blocks+0x2f/0x5f
 [<ffffffff882a3c46>] :xfs:xfs_page_state_convert+0x2b7/0xb63
 [<ffffffff882a4724>] :xfs:xfs_vm_writepage+0xa7/0xde
 [<ffffffff8021c78f>] mpage_writepages+0x1d0/0x395
 [<ffffffff80259e0f>] do_writepages+0x23/0x32
 [<ffffffff8024e2b8>] __filemap_fdatawrite_range+0x54/0x5e
 [<ffffffff882a779d>] :xfs:fs_flush_pages+0x4b/0x64
 [<ffffffff882a71ec>] :xfs:xfs_file_close+0x2a/0x2e
 [<ffffffff80223b7f>] filp_close+0x36/0x64
 [<ffffffff8021d873>] sys_close+0x8f/0xaa
 [<ffffffff8025c181>] tracesys+0xd1/0xdc
DWARF2 unwinder stuck at tracesys+0xd1/0xdc
Quoted 2 lines

Leftover inexact backtrace:
-----------

You've got a corrupt freelist btree block. how were you uploading files to the machine?

Can you cc bug reports involving XFS to the [email protected] list in future? (added to this reply)

Quoted 10 lines

I first booted into 2.6.18-1.2239.fc5 in single user mode and forced a check of the disk with xfs_repair and I'm using xfs-progs-2.8.11 as I discovered on my other system that the 2.6.17 XFS kernel driver bugs were breaking the FS in a way that the xfs-progs-2.7.x code didn't fix.

These XFS bugs seem to be the same problems that were cropping up in the 2.6.17 series which were resolved in 2.6.18.1 (2.6.18-1.2200.fc5).

Any suggestions are greatly appreciated. Also please let me know if more details are required.

The 2.6.17 problems can leave on disk corruption that is not tripped over until some time later on - even after a kernel upgrade.

Running the latest repair over all your XFS filesystems that were in use on 2.6.17.x (x <= 6) really needs to be done regardless of whether you've tripped over corruption or not.

However, this could be a result of the problems you've been having with the aacraid driver, and not an XFS problem at all....

Cheers,

Dave.

Quoted 53 lines

Should I just simply go back to ext3? I'd prefer not to because of the fsck'ing time on a 1Tb array, but if it means that the kernel doesn't throw a hissy fit then I'll be more than happy to do that.

Regards,
Colin.

thor# uname -a
Linux thor 2.6.18-1.2239.fc5 #1 SMP Fri Nov 10 12:51:06 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

thor# cat /proc/cmdline
ro root=LABEL=/

Adaptec aacraid driver (1.1-5[2409]-mh2)

processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2220 SE
stepping : 2
cpu MHz : 2800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
bogomips : 5639.77
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

Signature · 7 lines
 "Developers are like artists; they produce their best work if they
  have the freedom to do so" - Werner Vogels, CTO Amazon.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group