Skip to contentSkip to conversation

Critical xfs bug in 2.6.17.11?

DISCUSSION Christian Schmid 3 participants 9 messages Sep 10, 2006 – Sep 11, 2006

lore
Share & tools
lore
#0 lore raw

Hello.

Instead of a tmpfs, I use a raid 10 softraid. Unfortunately it crashed after 10 hours of extreme activities (read/block-writes with up to 250 streams/deletes)

12 gb memory-test successful. 2 cpu xeon smp system.

Tell me if this helps you:

Sep  9 18:08:49 inode430 kernel: [87433.143498] 0x0: 58 41 47 46 00 00 00 01 00 00 00 00 00 04 34 a0 
Sep  9 18:08:49 inode430 kernel: [87433.143672] Filesystem "md5": XFS internal error xfs_alloc_read_agf at line 2176 of file fs/xfs/xfs_alloc.c.  Caller 0xfffffff

f80314069

Sep  9 18:08:49 inode430 kernel: [87433.143904] 
Sep  9 18:08:49 inode430 kernel: [87433.143905] Call Trace: <ffffffff8033c909>{xfs_corruption_error+244}
Sep  9 18:08:49 inode430 kernel: [87433.143995]        <ffffffff80346efa>{xfs_iext_insert+65} <ffffffff803588d1>{xfs_trans_read_buf+203}
Sep  9 18:08:49 inode430 kernel: [87433.144353]        <ffffffff803121d9>{xfs_alloc_read_agf+281} <ffffffff80314069>{xfs_alloc_fix_freelist+356}
Sep  9 18:08:49 inode430 kernel: [87433.144628] <ffffffff80314069>{xfs_alloc_fix_freelist+356} <ffffffff80515a1f>{__down_read+18} 
Sep  9 18:08:49 inode430 kernel: [87433.144855]        <ffffffff8031450f>{xfs_alloc_vextent+289} <ffffffff80323945>{xfs_bmapi+4061}
Sep  9 18:08:49 inode430 kernel: [87433.145091] <ffffffff803215ba>{xfs_bmap_search_multi_extents+175} 
Sep  9 18:08:49 inode430 kernel: [87433.145226] <ffffffff80349aad>{xfs_iomap_write_allocate+675} <ffffffff80348c29>{xfs_iomap+701} 
Sep  9 18:08:49 inode430 kernel: [87433.145473]        <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.145846] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} 
Sep  9 18:08:49 inode430 kernel: [87433.146079]        <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.146330]        <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.146583]        <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.146847]        <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.147230]        <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.147463]        <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.147825]        <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.148142]        <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.148420]        <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.148775]        <ffffffff8020a98a>{child_rip+0} 
Sep  9 18:08:49 inode430 kernel: [87433.149105] Filesystem "md5": XFS internal error xfs_trans_cancel at line 1150 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff8

0349bf8

Sep  9 18:08:49 inode430 kernel: [87433.149262] 
Sep  9 18:08:49 inode430 kernel: [87433.149263] Call Trace: <ffffffff803574e9>{xfs_trans_cancel+111} 
Sep  9 18:08:49 inode430 kernel: [87433.149348] <ffffffff80349bf8>{xfs_iomap_write_allocate+1006} <ffffffff80348c29>{xfs_iomap+701} 
Sep  9 18:08:49 inode430 kernel: [87433.149568]        <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.149847] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} 
Sep  9 18:08:49 inode430 kernel: [87433.150169]        <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.150435]        <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.150593]        <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.150807]        <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.151042]        <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.151271]        <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.151439]        <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.151680]        <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.151922]        <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.152086]        <ffffffff8020a98a>{child_rip+0} 
Sep  9 18:08:49 inode430 kernel: [87433.152489] xfs_force_shutdown(md5,0x8) called from line 1151 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff80357507
Sep  9 18:08:49 inode430 kernel: [87433.168623] Filesystem "md5": Corruption of in-memory data detected.  Shutting down filesystem: md5
Sep  9 18:08:49 inode430 kernel: [87433.168903] Please umount the filesystem, and rectify the problem(s)
#1 lore raw

I hope this is not a repeat of 2.6.17 -> 2.6.17.6..

$ grep -i xfs ChangeLog-2.6.17.*
ChangeLog-2.6.17.7:    XFS: corruption fix
ChangeLog-2.6.17.7:    check in xfs_dir2_leafn_remove() fails every time and xfs_dir2_shrink_inode()

It appears the only changes to the XFS code though went into 2.6.17.7 so I am not sure what you are seeing there, had you fixed your filesystem from the 2.6.17 -> .17.6 bug?

Justin.

Quoted 40 lines · Christian Schmid

Hello.

Instead of a tmpfs, I use a raid 10 softraid. Unfortunately it crashed after 10 hours of extreme activities (read/block-writes with up to 250 streams/deletes)

12 gb memory-test successful. 2 cpu xeon smp system.

Tell me if this helps you:

Sep  9 18:08:49 inode430 kernel: [87433.143498] 0x0: 58 41 47 46 00 00 00 01 00 00 00 00 00 04 34 a0 Sep  9 18:08:49 inode430 kernel: [87433.143672] Filesystem "md5": XFS internal error xfs_alloc_read_agf at line 2176 of file fs/xfs/xfs_alloc.c. Caller 0xfffffff
f80314069 Sep  9 18:08:49 inode430 kernel: [87433.143904] Sep  9 18:08:49 inode430 kernel: [87433.143905] Call Trace: <ffffffff8033c909>{xfs_corruption_error+244}
Sep  9 18:08:49 inode430 kernel: [87433.143995] <ffffffff80346efa>{xfs_iext_insert+65} <ffffffff803588d1>{xfs_trans_read_buf+203}
Sep  9 18:08:49 inode430 kernel: [87433.144353] <ffffffff803121d9>{xfs_alloc_read_agf+281} <ffffffff80314069>{xfs_alloc_fix_freelist+356}
Sep  9 18:08:49 inode430 kernel: [87433.144628] <ffffffff80314069>{xfs_alloc_fix_freelist+356} <ffffffff80515a1f>{__down_read+18} Sep  9 18:08:49 inode430 kernel: [87433.144855] <ffffffff8031450f>{xfs_alloc_vextent+289} <ffffffff80323945>{xfs_bmapi+4061}
Sep  9 18:08:49 inode430 kernel: [87433.145091] <ffffffff803215ba>{xfs_bmap_search_multi_extents+175} Sep  9 18:08:49 inode430 kernel: [87433.145226] <ffffffff80349aad>{xfs_iomap_write_allocate+675} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.145473] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.145846] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.146079] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.146330] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.146583] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.146847] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.147230] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.147463] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.147825] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.148142] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.148420] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.148775] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.149105] Filesystem "md5": XFS internal error xfs_trans_cancel at line 1150 of file fs/xfs/xfs_trans.c. Caller 0xffffffff8
0349bf8 Sep  9 18:08:49 inode430 kernel: [87433.149262] Sep  9 18:08:49 inode430 kernel: [87433.149263] Call Trace: <ffffffff803574e9>{xfs_trans_cancel+111} Sep  9 18:08:49 inode430 kernel: [87433.149348] <ffffffff80349bf8>{xfs_iomap_write_allocate+1006} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.149568] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.149847] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.150169] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.150435] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.150593] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.150807] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.151042] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.151271] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.151439] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.151680] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.151922] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.152086] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.152489] xfs_force_shutdown(md5,0x8) called from line 1151 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff80357507
Sep  9 18:08:49 inode430 kernel: [87433.168623] Filesystem "md5": Corruption of in-memory data detected.  Shutting down filesystem: md5
Sep  9 18:08:49 inode430 kernel: [87433.168903] Please umount the filesystem, and rectify the problem(s)
#2 lore raw

This file-system was created 2 days before with the same kernel.

Quoted 54 lines · Justin Piszcz

I hope this is not a repeat of 2.6.17 -> 2.6.17.6..

$ grep -i xfs ChangeLog-2.6.17.*
ChangeLog-2.6.17.7:    XFS: corruption fix
ChangeLog-2.6.17.7:    check in xfs_dir2_leafn_remove() fails every time and xfs_dir2_shrink_inode()

It appears the only changes to the XFS code though went into 2.6.17.7 so I am not sure what you are seeing there, had you fixed your filesystem from the 2.6.17 -> .17.6 bug?

Justin.

Quoted 40 lines · Christian Schmid

Hello.

Instead of a tmpfs, I use a raid 10 softraid. Unfortunately it crashed after 10 hours of extreme activities (read/block-writes with up to 250 streams/deletes)

12 gb memory-test successful. 2 cpu xeon smp system.

Tell me if this helps you:

Sep  9 18:08:49 inode430 kernel: [87433.143498] 0x0: 58 41 47 46 00 00 00 01 00 00 00 00 00 04 34 a0 Sep  9 18:08:49 inode430 kernel: [87433.143672] Filesystem "md5": XFS internal error xfs_alloc_read_agf at line 2176 of file fs/xfs/xfs_alloc.c. Caller 0xfffffff
f80314069 Sep  9 18:08:49 inode430 kernel: [87433.143904] Sep  9 18:08:49 inode430 kernel: [87433.143905] Call Trace: <ffffffff8033c909>{xfs_corruption_error+244}
Sep  9 18:08:49 inode430 kernel: [87433.143995] <ffffffff80346efa>{xfs_iext_insert+65} <ffffffff803588d1>{xfs_trans_read_buf+203}
Sep  9 18:08:49 inode430 kernel: [87433.144353] <ffffffff803121d9>{xfs_alloc_read_agf+281} <ffffffff80314069>{xfs_alloc_fix_freelist+356}
Sep  9 18:08:49 inode430 kernel: [87433.144628] <ffffffff80314069>{xfs_alloc_fix_freelist+356} <ffffffff80515a1f>{__down_read+18} Sep  9 18:08:49 inode430 kernel: [87433.144855] <ffffffff8031450f>{xfs_alloc_vextent+289} <ffffffff80323945>{xfs_bmapi+4061}
Sep  9 18:08:49 inode430 kernel: [87433.145091] <ffffffff803215ba>{xfs_bmap_search_multi_extents+175} Sep  9 18:08:49 inode430 kernel: [87433.145226] <ffffffff80349aad>{xfs_iomap_write_allocate+675} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.145473] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.145846] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.146079] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.146330] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.146583] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.146847] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.147230] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.147463] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.147825] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.148142] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.148420] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.148775] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.149105] Filesystem "md5": XFS internal error xfs_trans_cancel at line 1150 of file fs/xfs/xfs_trans.c. Caller 0xffffffff8
0349bf8 Sep  9 18:08:49 inode430 kernel: [87433.149262] Sep  9 18:08:49 inode430 kernel: [87433.149263] Call Trace: <ffffffff803574e9>{xfs_trans_cancel+111} Sep  9 18:08:49 inode430 kernel: [87433.149348] <ffffffff80349bf8>{xfs_iomap_write_allocate+1006} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.149568] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.149847] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.150169] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.150435] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.150593] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.150807] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.151042] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.151271] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.151439] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.151680] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.151922] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.152086] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.152489] xfs_force_shutdown(md5,0x8) called from line 1151 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff80357507
Sep  9 18:08:49 inode430 kernel: [87433.168623] Filesystem "md5": Corruption of in-memory data detected.  Shutting down filesystem: md5
Sep  9 18:08:49 inode430 kernel: [87433.168903] Please umount the filesystem, and rectify the problem(s)
#3 lore raw

ACK, scary, will wait for Nathan Scott's/other SGI members reply on this one...

I have not had that happen to me yet, what were you doing that caused the problem? Is it repeatable? Have you checked the XFS FAQ for the FS fix for 2.6.17-17.6? just to check if there is indeed any problems (basically an xfs check on your FS), if you do it, dont use knoppix 5.0.2 (contains the 2.6.17 XFS corruption bug), use 4.0.2.

Justin.

Quoted 59 lines · Christian Schmid

This file-system was created 2 days before with the same kernel.

Quoted 54 lines · Justin Piszcz

I hope this is not a repeat of 2.6.17 -> 2.6.17.6..

$ grep -i xfs ChangeLog-2.6.17.*
ChangeLog-2.6.17.7:    XFS: corruption fix
ChangeLog-2.6.17.7:    check in xfs_dir2_leafn_remove() fails every time and xfs_dir2_shrink_inode()

It appears the only changes to the XFS code though went into 2.6.17.7 so I am not sure what you are seeing there, had you fixed your filesystem from the 2.6.17 -> .17.6 bug?

Justin.

Quoted 40 lines · Christian Schmid

Hello.

Instead of a tmpfs, I use a raid 10 softraid. Unfortunately it crashed after 10 hours of extreme activities (read/block-writes with up to 250 streams/deletes)

12 gb memory-test successful. 2 cpu xeon smp system.

Tell me if this helps you:

Sep  9 18:08:49 inode430 kernel: [87433.143498] 0x0: 58 41 47 46 00 00 00 01 00 00 00 00 00 04 34 a0 Sep  9 18:08:49 inode430 kernel: [87433.143672] Filesystem "md5": XFS internal error xfs_alloc_read_agf at line 2176 of file fs/xfs/xfs_alloc.c. Caller 0xfffffff
f80314069 Sep  9 18:08:49 inode430 kernel: [87433.143904] Sep  9 18:08:49 inode430 kernel: [87433.143905] Call Trace: <ffffffff8033c909>{xfs_corruption_error+244}
Sep  9 18:08:49 inode430 kernel: [87433.143995] <ffffffff80346efa>{xfs_iext_insert+65} <ffffffff803588d1>{xfs_trans_read_buf+203}
Sep  9 18:08:49 inode430 kernel: [87433.144353] <ffffffff803121d9>{xfs_alloc_read_agf+281} <ffffffff80314069>{xfs_alloc_fix_freelist+356}
Sep  9 18:08:49 inode430 kernel: [87433.144628] <ffffffff80314069>{xfs_alloc_fix_freelist+356} <ffffffff80515a1f>{__down_read+18} Sep  9 18:08:49 inode430 kernel: [87433.144855] <ffffffff8031450f>{xfs_alloc_vextent+289} <ffffffff80323945>{xfs_bmapi+4061}
Sep  9 18:08:49 inode430 kernel: [87433.145091] <ffffffff803215ba>{xfs_bmap_search_multi_extents+175} Sep  9 18:08:49 inode430 kernel: [87433.145226] <ffffffff80349aad>{xfs_iomap_write_allocate+675} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.145473] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.145846] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.146079] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.146330] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.146583] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.146847] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.147230] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.147463] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.147825] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.148142] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.148420] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.148775] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.149105] Filesystem "md5": XFS internal error xfs_trans_cancel at line 1150 of file fs/xfs/xfs_trans.c. Caller 0xffffffff8
0349bf8 Sep  9 18:08:49 inode430 kernel: [87433.149262] Sep  9 18:08:49 inode430 kernel: [87433.149263] Call Trace: <ffffffff803574e9>{xfs_trans_cancel+111} Sep  9 18:08:49 inode430 kernel: [87433.149348] <ffffffff80349bf8>{xfs_iomap_write_allocate+1006} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.149568] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.149847] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.150169] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.150435] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.150593] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.150807] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.151042] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.151271] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.151439] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.151680] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.151922] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.152086] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.152489] xfs_force_shutdown(md5,0x8) called from line 1151 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff80357507
Sep  9 18:08:49 inode430 kernel: [87433.168623] Filesystem "md5": Corruption of in-memory data detected.  Shutting down filesystem: md5
Sep  9 18:08:49 inode430 kernel: [87433.168903] Please umount the filesystem, and rectify the problem(s)
#4 lore raw

I am using www.linuxfromscratch.org. Kernel is vanilla 2.6.17.11 from kernel.org, xfsprogs is 2.7.3, libc is 2.3.3. I was not doing anything special. In fact its a heavy-duty tmpfs with up to 300 write-streams at once, reads and deletes. So basically a heavy stress-test on a SMP. Maybe a race-condition? Pure speculations from my side. But memory is ok. Memory-test with ECC disabled ran through 12 hours without any errors. ECC is on now of course, so the possibility of a simple hardware problem is eliminated from my side.

Quoted 70 lines · Justin Piszcz

ACK, scary, will wait for Nathan Scott's/other SGI members reply on this one...

I have not had that happen to me yet, what were you doing that caused the problem? Is it repeatable? Have you checked the XFS FAQ for the FS fix for 2.6.17-17.6? just to check if there is indeed any problems (basically an xfs check on your FS), if you do it, dont use knoppix 5.0.2 (contains the 2.6.17 XFS corruption bug), use 4.0.2.

Justin.

Quoted 60 lines · Christian Schmid

This file-system was created 2 days before with the same kernel.

Quoted 54 lines · Justin Piszcz

I hope this is not a repeat of 2.6.17 -> 2.6.17.6..

$ grep -i xfs ChangeLog-2.6.17.*
ChangeLog-2.6.17.7:    XFS: corruption fix
ChangeLog-2.6.17.7:    check in xfs_dir2_leafn_remove() fails every time and xfs_dir2_shrink_inode()

It appears the only changes to the XFS code though went into 2.6.17.7 so I am not sure what you are seeing there, had you fixed your filesystem from the 2.6.17 -> .17.6 bug?

Justin.

Quoted 40 lines · Christian Schmid

Hello.

Instead of a tmpfs, I use a raid 10 softraid. Unfortunately it crashed after 10 hours of extreme activities (read/block-writes with up to 250 streams/deletes)

12 gb memory-test successful. 2 cpu xeon smp system.

Tell me if this helps you:

Sep  9 18:08:49 inode430 kernel: [87433.143498] 0x0: 58 41 47 46 00 00 00 01 00 00 00 00 00 04 34 a0 Sep  9 18:08:49 inode430 kernel: [87433.143672] Filesystem "md5": XFS internal error xfs_alloc_read_agf at line 2176 of file fs/xfs/xfs_alloc.c. Caller 0xfffffff
f80314069 Sep  9 18:08:49 inode430 kernel: [87433.143904] Sep  9 18:08:49 inode430 kernel: [87433.143905] Call Trace: <ffffffff8033c909>{xfs_corruption_error+244}
Sep  9 18:08:49 inode430 kernel: [87433.143995] <ffffffff80346efa>{xfs_iext_insert+65} <ffffffff803588d1>{xfs_trans_read_buf+203}
Sep  9 18:08:49 inode430 kernel: [87433.144353] <ffffffff803121d9>{xfs_alloc_read_agf+281} <ffffffff80314069>{xfs_alloc_fix_freelist+356}
Sep  9 18:08:49 inode430 kernel: [87433.144628] <ffffffff80314069>{xfs_alloc_fix_freelist+356} <ffffffff80515a1f>{__down_read+18} Sep  9 18:08:49 inode430 kernel: [87433.144855] <ffffffff8031450f>{xfs_alloc_vextent+289} <ffffffff80323945>{xfs_bmapi+4061}
Sep  9 18:08:49 inode430 kernel: [87433.145091] <ffffffff803215ba>{xfs_bmap_search_multi_extents+175} Sep  9 18:08:49 inode430 kernel: [87433.145226] <ffffffff80349aad>{xfs_iomap_write_allocate+675} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.145473] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.145846] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.146079] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.146330] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.146583] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.146847] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.147230] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.147463] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.147825] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.148142] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.148420] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.148775] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.149105] Filesystem "md5": XFS internal error xfs_trans_cancel at line 1150 of file fs/xfs/xfs_trans.c. Caller 0xffffffff8
0349bf8 Sep  9 18:08:49 inode430 kernel: [87433.149262] Sep  9 18:08:49 inode430 kernel: [87433.149263] Call Trace: <ffffffff803574e9>{xfs_trans_cancel+111} Sep  9 18:08:49 inode430 kernel: [87433.149348] <ffffffff80349bf8>{xfs_iomap_write_allocate+1006} <ffffffff80348c29>{xfs_iomap+701} Sep  9 18:08:49 inode430 kernel: [87433.149568] <ffffffff803797f1>{generic_make_request+515} <ffffffff803632f3>{xfs_map_blocks+67}
Sep  9 18:08:49 inode430 kernel: [87433.149847] <ffffffff80363a09>{xfs_page_state_convert+722} <ffffffff80364344>{xfs_vm_writepage+179} Sep  9 18:08:49 inode430 kernel: [87433.150169] <ffffffff802986ed>{mpage_writepages+459} <ffffffff80364291>{xfs_vm_writepage+0}
Sep  9 18:08:49 inode430 kernel: [87433.150435] <ffffffff802553c1>{do_writepages+41} <ffffffff80296de0>{__writeback_single_inode+559}
Sep  9 18:08:49 inode430 kernel: [87433.150593] <ffffffff8022a82c>{default_wake_function+0} <ffffffff8022a82c>{default_wake_function+0}
Sep  9 18:08:49 inode430 kernel: [87433.150807] <ffffffff80357f56>{xfs_trans_first_ail+28} <ffffffff802974be>{sync_sb_inodes+501}
Sep  9 18:08:49 inode430 kernel: [87433.151042] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff802977cd>{writeback_inodes+144}
Sep  9 18:08:49 inode430 kernel: [87433.151271] <ffffffff802551fc>{wb_kupdate+148} <ffffffff80255bfd>{pdflush+313}
Sep  9 18:08:49 inode430 kernel: [87433.151439] <ffffffff80255168>{wb_kupdate+0} <ffffffff80255ac4>{pdflush+0}
Sep  9 18:08:49 inode430 kernel: [87433.151680] <ffffffff80244479>{kthread+218} <ffffffff8020a992>{child_rip+8}
Sep  9 18:08:49 inode430 kernel: [87433.151922] <ffffffff802444b4>{keventd_create_kthread+0} <ffffffff8024439f>{kthread+0}
Sep  9 18:08:49 inode430 kernel: [87433.152086] <ffffffff8020a98a>{child_rip+0} Sep  9 18:08:49 inode430 kernel: [87433.152489] xfs_force_shutdown(md5,0x8) called from line 1151 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff80357507
Sep  9 18:08:49 inode430 kernel: [87433.168623] Filesystem "md5": Corruption of in-memory data detected.  Shutting down filesystem: md5
Sep  9 18:08:49 inode430 kernel: [87433.168903] Please umount the filesystem, and rectify the problem(s)
#7 lore raw
In reply to Christian Schmid
Quoted 8 lines · Christian Schmid

Hello.

Instead of a tmpfs, I use a raid 10 softraid. Unfortunately it crashed after 10 hours of extreme activities (read/block-writes with up to 250 streams/deletes)

12 gb memory-test successful. 2 cpu xeon smp system.

Quoted 7 lines

Tell me if this helps you:

Sep  9 18:08:49 inode430 kernel: [87433.143498] 0x0: 58 41 47 46 00 00 00 
01 00 00 00 00 00 04 34 a0 
Sep  9 18:08:49 inode430 kernel: [87433.143672] Filesystem "md5": XFS 

internal error xfs_alloc_read_agf at line 2176 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff80314069

Hmm - bad read of the AGF during allocation, which causes a shutdown due to cancelling a dirty transaction.

If you run "xfs_check -n /dev/md5" and "xfs_repair -n /dev/md5" do they warn about any AGF related corruption?

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#5 lore raw
In reply to Christian Schmid
Quoted 3 lines · Christian Schmid

Memory-test with ECC disabled ran through 12 hours without any errors. ECC is on now of course, so the possibility of a simple hardware problem is eliminated from my side.

A _memory error_ can be ruled out, but what about a bad disk, bad disk controller, bad PCI bus interface, a bad driver, etc. Memory is just one piece of hardware that can result in bad data being read from or written to disk. Is there any indication of disk or driver errors in your syslog?

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#8 lore raw
In reply to David Chinner

Unfortunately after this error I did a mkfs.xfs on this device, because its just a /tmp fs.

Quoted 22 lines · David Chinner
Quoted 9 lines · Christian Schmid

Hello.

Instead of a tmpfs, I use a raid 10 softraid. Unfortunately it crashed after 10 hours of extreme activities (read/block-writes with up to 250 streams/deletes)

12 gb memory-test successful. 2 cpu xeon smp system.

Tell me if this helps you:

Sep  9 18:08:49 inode430 kernel: [87433.143498] 0x0: 58 41 47 46 00 00 00 01 00 00 00 00 00 04 34 a0 Sep  9 18:08:49 inode430 kernel: [87433.143672] Filesystem "md5": XFS internal error xfs_alloc_read_agf at line 2176 of file fs/xfs/xfs_alloc.c.  Caller 0xffffffff80314069 

Hmm - bad read of the AGF during allocation, which causes a shutdown due to cancelling a dirty transaction.

If you run "xfs_check -n /dev/md5" and "xfs_repair -n /dev/md5" do they warn about any AGF related corruption?

Cheers,

Dave.

#6 lore raw
In reply to David Chinner

I am not sure. Its a linux software-raid and as far as I know there is a crc-check in the drives so if they cant read the data, they give an error and the raid gets the data from the other drive and marks this drive as broken. As far as the log says, the drives are ok.

Quoted 14 lines · David Chinner
Quoted 1 line · Christian Schmid

Memory-test with ECC disabled ran through 12 hours without any errors. ECC is on now of course, so the possibility of a simple hardware problem is eliminated from my side.

A _memory error_ can be ruled out, but what about a bad disk, bad disk controller, bad PCI bus interface, a bad driver, etc. Memory is just one piece of hardware that can result in bad data being read from or written to disk. Is there any indication of disk or driver errors in your syslog?

Cheers,

Dave.