XFS filesystem corruption
Hello lists,
[resend with correct email address for LKML]
[Please CC all answers from linux-xfs to me, since I'm not subscribed on that list]
Yesterday I encountered an on-the-fly corruption of my /home filesystem. It worked perfectly one second, the next I hit these nice errors:
Nov 21 16:37:22 precious kernel: 0x0: 31 9e ce 63 cf ff 9c cf ff 31 61 63 ff ff ff ff Nov 21 16:37:23 precious kernel: Filesystem "hda5": XFS internal error xfs_da_do_buf(2) at line 2273 of file fs/xfs/xfs_da_btree.c. Caller 0xc01fb908 Nov 21 16:37:23 precious kernel: [xfs_da_do_buf+905/2160] xfs_da_do_buf+0x389/0x870 Nov 21 16:37:23 precious kernel: [xfs_da_read_buf+88/96] xfs_da_read_buf+0x58/0x60 Nov 21 16:37:23 precious last message repeated 2 times Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_lookup_int+386/720] xfs_dir2_leaf_lookup_int+0x182/0x2d0 Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_lookup_int+386/720] xfs_dir2_leaf_lookup_int+0x182/0x2d0 Nov 21 16:37:23 precious kernel: [__wake_up_common+65/112] __wake_up_common+0x41/0x70 Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_lookup+55/224] xfs_dir2_leaf_lookup+0x37/0xe0 Nov 21 16:37:23 precious kernel: [xfs_dir2_lookup+298/352] xfs_dir2_lookup+0x12a/0x160 Nov 21 16:37:23 precious kernel: [xfs_dir_lookup_int+76/304] xfs_dir_lookup_int+0x4c/0x130 Nov 21 16:37:23 precious kernel: [xfs_lookup+80/144] xfs_lookup+0x50/0x90 Nov 21 16:37:23 precious kernel: [linvfs_lookup+82/144] linvfs_lookup+0x52/0x90 Nov 21 16:37:23 precious kernel: [real_lookup+193/240] real_lookup+0xc1/0xf0 Nov 21 16:37:23 precious kernel: [do_lookup+150/176] do_lookup+0x96/0xb0 Nov 21 16:37:23 precious kernel: [link_path_walk+1732/3424] link_path_walk+0x6c4/0xd60 Nov 21 16:37:23 precious kernel: [link_path_walk+2603/3424] link_path_walk+0xa2b/0xd60 Nov 21 16:37:23 precious kernel: [cp_new_stat64+248/272] cp_new_stat64+0xf8/0x110 Nov 21 16:37:23 precious kernel: [path_lookup+124/320] path_lookup+0x7c/0x140 Nov 21 16:37:23 precious kernel: [__user_walk+51/96] __user_walk+0x33/0x60 Nov 21 16:37:23 precious kernel: [dput+51/544] dput+0x33/0x220 Nov 21 16:37:23 precious kernel: [sys_access+133/336] sys_access+0x85/0x150 Nov 21 16:37:23 precious kernel: [path_release+21/80] path_release+0x15/0x50 Nov 21 16:37:23 precious kernel: [syscall_call+7/11] syscall_call+0x7/0xb Nov 21 16:37:23 precious kernel: 0x0: 31 9e ce 63 cf ff 9c cf ff 31 61 63 ff ff ff ff Nov 21 16:37:23 precious kernel: Filesystem "hda5": XFS internal error xfs_da_do_buf(2) at line 2273 of file fs/xfs/xfs_da_btree.c. Caller 0xc01fb908 Nov 21 16:37:23 precious kernel: [xfs_da_do_buf+905/2160] xfs_da_do_buf+0x389/0x870 Nov 21 16:37:23 precious kernel: [xfs_da_read_buf+88/96] xfs_da_read_buf+0x58/0x60 Nov 21 16:37:23 precious last message repeated 2 times Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_lookup_int+386/720] xfs_dir2_leaf_lookup_int+0x182/0x2d0 Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_lookup_int+386/720] xfs_dir2_leaf_lookup_int+0x182/0x2d0 Nov 21 16:37:23 precious kernel: [__wake_up_common+65/112] __wake_up_common+0x41/0x70 Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_lookup+55/224] xfs_dir2_leaf_lookup+0x37/0xe0 Nov 21 16:37:23 precious kernel: [xfs_dir2_lookup+298/352] xfs_dir2_lookup+0x12a/0x160 Nov 21 16:37:23 precious kernel: [xfs_dir_lookup_int+76/304] xfs_dir_lookup_int+0x4c/0x130 Nov 21 16:37:23 precious kernel: [xfs_lookup+80/144] xfs_lookup+0x50/0x90 Nov 21 16:37:23 precious kernel: [linvfs_lookup+82/144] linvfs_lookup+0x52/0x90 Nov 21 16:37:23 precious kernel: [real_lookup+193/240] real_lookup+0xc1/0xf0 Nov 21 16:37:23 precious kernel: [do_lookup+150/176] do_lookup+0x96/0xb0 Nov 21 16:37:23 precious kernel: [link_path_walk+1732/3424] link_path_walk+0x6c4/0xd60 Nov 21 16:37:23 precious kernel: [link_path_walk+2603/3424] link_path_walk+0xa2b/0xd60 Nov 21 16:37:23 precious kernel: [cp_new_stat64+248/272] cp_new_stat64+0xf8/0x110 Nov 21 16:37:23 precious kernel: [path_lookup+124/320] path_lookup+0x7c/0x140 Nov 21 16:37:23 precious kernel: [__user_walk+51/96] __user_walk+0x33/0x60 Nov 21 16:37:23 precious kernel: [sys_access+133/336] sys_access+0x85/0x150 Nov 21 16:37:23 precious kernel: [path_release+21/80] path_release+0x15/0x50 Nov 21 16:37:23 precious kernel: [syscall_call+7/11] syscall_call+0x7/0xb Nov 21 16:37:23 precious kernel: 0x0: 10 10 10 10 10 10 00 00 21 10 10 10 10 10 10 10 Nov 21 16:37:23 precious kernel: Filesystem "hda5": XFS internal error xfs_da_do_buf(2) at line 2273 of file fs/xfs/xfs_da_btree.c. Caller 0xc01fb908 Nov 21 16:37:23 precious kernel: [xfs_da_do_buf+905/2160] xfs_da_do_buf+0x389/0x870 Nov 21 16:37:23 precious kernel: [xfs_da_read_buf+88/96] xfs_da_read_buf+0x58/0x60 Nov 21 16:37:23 precious kernel: [xfs_da_read_buf+88/96] xfs_da_read_buf+0x58/0x60 Nov 21 16:37:23 precious kernel: [xfs_initialize_vnode+780/800] xfs_initialize_vnode+0x30c/0x320 Nov 21 16:37:23 precious kernel: [xfs_da_read_buf+88/96] xfs_da_read_buf+0x58/0x60 Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_addname+875/2688] xfs_dir2_leaf_addname+0x36b/0xa80 Nov 21 16:37:23 precious kernel: [xfs_dir2_leaf_addname+875/2688] xfs_dir2_leaf_addname+0x36b/0xa80 Nov 21 16:37:23 precious kernel: [xfs_bmap_last_offset+197/304] xfs_bmap_last_offset+0xc5/0x130 Nov 21 16:37:23 precious kernel: [xfs_dir2_isleaf+44/112] xfs_dir2_isleaf+0x2c/0x70 Nov 21 16:37:23 precious kernel: [xfs_dir2_createname+341/384] xfs_dir2_createname+0x155/0x180 Nov 21 16:37:23 precious kernel: [xfs_dir_ialloc+145/736] xfs_dir_ialloc+0x91/0x2e0 Nov 21 16:37:23 precious kernel: [xfs_trans_ijoin+53/144] xfs_trans_ijoin+0x35/0x90 Nov 21 16:37:23 precious kernel: [xfs_create+1115/1888] xfs_create+0x45b/0x760 Nov 21 16:37:23 precious kernel: [linvfs_mknod+475/576] linvfs_mknod+0x1db/0x240 Nov 21 16:37:23 precious kernel: [xfs_dir2_lookup+298/352] xfs_dir2_lookup+0x12a/0x160 Nov 21 16:37:23 precious kernel: [real_lookup+193/240] real_lookup+0xc1/0xf0 Nov 21 16:37:23 precious kernel: [dput+51/544] dput+0x33/0x220 Nov 21 16:37:23 precious kernel: [xfs_dir_lookup_int+76/304] xfs_dir_lookup_int+0x4c/0x130 Nov 21 16:37:23 precious kernel: [permission+53/96] permission+0x35/0x60 Nov 21 16:37:23 precious kernel: [vfs_create+121/224] vfs_create+0x79/0xe0 Nov 21 16:37:23 precious kernel: [open_namei+1462/1552] open_namei+0x5b6/0x610 Nov 21 16:37:23 precious kernel: [filp_open+62/112] filp_open+0x3e/0x70 Nov 21 16:37:23 precious kernel: [get_unused_fd+57/224] get_unused_fd+0x39/0xe0 Nov 21 16:37:23 precious kernel: [sys_open+73/144] sys_open+0x49/0x90 Nov 21 16:37:23 precious kernel: [syscall_call+7/11] syscall_call+0x7/0xb Nov 21 16:37:23 precious kernel: xfs_force_shutdown(hda5,0x8) called from line 1091 of file fs/xfs/xfs_trans.c. Return address = 0xc0244e4b Nov 21 16:37:23 precious kernel: Filesystem "hda5": Corruption of in-memory data detected. Shutting down filesystem: hda5 Nov 21 16:37:23 precious kernel: Please umount the filesystem, and rectify the problem(s)
Doing an unmount/remount didn't solve it, i had to run xfs_repair on the filesystem to get it back to 'work', which caused me to lose a lot of stuff. (thank <deity> for backups.)
Any idea what can cause this?
Thanks,
Jan
Signature · 1 line
What's the MATTER Sid? ... Is your BEVERAGE unsatisfactory?
Quoted 6 lines · Jan De Luyck
[resend with correct email address for LKML]
[Please CC all answers from linux-xfs to me, since I'm not subscribed on that list]
Yesterday I encountered an on-the-fly corruption of my /home filesystem. It worked perfectly one second, the next I hit these nice errors:Nov 21 16:37:22 precious kernel: 0x0: 31 9e ce 63 cf ff 9c cf ff 31 61 63 ff ff ff ff Nov 21 16:37:23 precious kernel: Filesystem "hda5": XFS internal error xfs_da_do_buf(2) at line 2273 of file fs/xfs/xfs_da_btree.c. Caller 0xc01fb908 Nov 21 16:37:23 precious kernel: [xfs_da_do_buf+905/2160] xfs_da_do_buf+0x389/0x870
I don't have any ideas at the moment, but please cc: me also. I'd like to watch for issues I do understand as this bug's nature is clarified.
-- wli
The trigger was a bad magic number related to directories... hard to say what happened in the first place. Can you send the output from xfs_repair, that might offer some hints.
Thanks,
-Eric
Quoted 3 lines · Eric Sandeen
The trigger was a bad magic number related to directories... hard to say what happened in the first place. Can you send the output from
xfs_repair, that might offer some hints.
Sorry, but as a repair was very urgent, I didn't really think of saving the xfs_repair output.. My bad I guess.
Jan
Signature · 2 lines
The seven year itch comes from fooling around during the fourth, fifth, and sixth years.
Quoted 11 lines · William Lee Irwin III
Quoted 5 lines · Jan De Luyck
[resend with correct email address for LKML]
[Please CC all answers from linux-xfs to me, since I'm not subscribed on that list]
Yesterday I encountered an on-the-fly corruption of my /home filesystem. It worked perfectly one second, the next I hit these nice errors:Nov 21 16:37:22 precious kernel: 0x0: 31 9e ce 63 cf ff 9c cf ff 31 61 63 ff ff ff ff Nov 21 16:37:23 precious kernel: Filesystem "hda5": XFS internal error xfs_da_do_buf(2) at line 2273 of file fs/xfs/xfs_da_btree.c. Caller 0xc01fb908 Nov 21 16:37:23 precious kernel: [xfs_da_do_buf+905/2160] xfs_da_do_buf+0x389/0x870I don't have any ideas at the moment, but please cc: me also. I'd like to watch for issues I do understand as this bug's nature is clarified.
While we are at it: Is xfs known to be broken while preempt is on? (Esp using ck's preemp big kernel lock?) I got following using a raid0 setup with xfs. I thought it would be a driver issue, but reformatting to ext3 the stripe array runs now w/o probs for a few days. (xfs crapped out after a few hours on heavy disk activity.)
Nov 21 10:10:15 tachyon ata2: command 0x25 timeout, stat 0xd0 host_stat 0x61
Nov 21 10:10:15 tachyon ata2: status=0xd0 { Busy }
Nov 21 10:10:15 tachyon SCSI error : <1 0 0 0> return code = 0x8000002
Nov 21 10:10:15 tachyon Current sdb: sense = 70 10
Nov 21 10:10:15 tachyon end_request: I/O error, dev sdb, sector 10480847
Nov 21 10:10:15 tachyon ATA: abnormal status 0xD0 on port 0xF08060C7
Nov 21 10:10:15 tachyon ATA: abnormal status 0xD0 on port 0xF08060C7
Nov 21 10:10:15 tachyon ATA: abnormal status 0xD0 on port 0xF08060C7
Nov 21 10:10:45 tachyon ata2: command 0x25 timeout, stat 0xd0 host_stat 0x61
Nov 21 10:10:45 tachyon ata2: status=0xd0 { Busy }
Nov 21 10:10:45 tachyon SCSI error : <1 0 0 0> return code = 0x8000002
Nov 21 10:10:45 tachyon Current sdb: sense = 70 10
Nov 21 10:10:45 tachyon end_request: I/O error, dev sdb, sector 10480855
Nov 21 10:10:45 tachyon I/O error in filesystem ("md0") meta-data dev md0 block 0x13fd990 ("xfs_trans_read_buf") error 5 buf count 8192etc...
If you need more infos (dmesg, .config, etc) let me know.
Prakash
Quoted 2 lines · Prakash K. Cheemplavam
Is xfs known to be broken while preempt is on? (Esp using ck's preemp big kernel lock?)
Minor nitpick: Ingo wrote the preempt BKL code, not Con.
Lee
Quoted 6 lines · Lee Revell
Quoted 1 line · Prakash K. Cheemplavam
Is xfs known to be broken while preempt is on? (Esp using ck's preemp big kernel lock?)
Minor nitpick: Ingo wrote the preempt BKL code, not Con.
Oh yes, I know, but it is in Con's kernel, and IIRC he merged nto all of his patches, so I thought this would be more precise. ;-)
Prakash
Quoted 2 lines · Prakash K. Cheemplavam
While we are at it: Is xfs known to be broken while preempt is on? (Esp
Nope.
Quoted 8 lines
using ck's preemp big kernel lock?) I got following using a raid0 setup with xfs. I thought it would be a driver issue, but reformatting to ext3 the stripe array runs now w/o probs for a few days. (xfs crapped out after a few hours on heavy disk activity.)
...Nov 21 10:10:45 tachyon end_request: I/O error, dev sdb, sector 10480855 Nov 21 10:10:45 tachyon I/O error in filesystem ("md0") meta-data devmd0 block
0x13fd990("xfs_trans_read_buf") error 5 buf count 8192
This looks like your driver passed an error back up to the filesystem while it was doing metadata IO and XFS chose to shut it down to prevent further damage. It's unlikely to be a preempt/xfs problem. Possibly hardware. Did you see any of those device errors since switching to ext3?
cheers.
Signature · 1 line
Nathan
Quoted 7 lines · Nathan Scott
Quoted 4 lines · Prakash K. Cheemplavam
using ck's preemp big kernel lock?) I got following using a raid0 setup with xfs. I thought it would be a driver issue, but reformatting to ext3 the stripe array runs now w/o probs for a few days. (xfs crapped out after a few hours on heavy disk activity.)
...Nov 21 10:10:45 tachyon end_request: I/O error, dev sdb, sector 10480855 Nov 21 10:10:45 tachyon I/O error in filesystem ("md0") meta-data dev md0 block 0x13fd990 ("xfs_trans_read_buf") error 5 buf count 8192
Quoted 5 lines
This looks like your driver passed an error back up to the filesystem while it was doing metadata IO and XFS chose to shut it down to prevent further damage. It's unlikely to be a preempt/xfs problem. Possibly hardware. Did you see any of those device errors since switching to ext3?
No. That's why I am wondering. I read about such errors like I got before in lkml and usually they were not fs related but libata siimage driver related. It could be just a coincidence that it came up with xfs, but till now (I guess 5 days now, though not 24/7 running) ext3 is behaving nicely.
bye,
Prakash
Quoted 7 lines · Prakash K. Cheemplavam
Quoted 2 lines · Nathan Scott
Did you see
any of those device errors since switching to ext3?No. That's why I am wondering. I read about such errors like I got before in lkml and usually they were not fs related but libata siimage driver related. It could be just a coincidence that it came up with xfs, but till now (I guess 5 days now, though not 24/7 running) ext3 is behaving nicely.
It's almost certainly not a filesystem problem, but an IO layer problem.
Maybe you only see it with xfs due to different disk IO patterns with
xfs vs. ext3... the two will certainly be allocating & writing to the disk in different ways.
-Eric
Quoted 9 lines · Eric Sandeen
Quoted 7 lines · Prakash K. Cheemplavam
Quoted 2 lines · Nathan Scott
Did you see
any of those device errors since switching to ext3?
Quoted 6 lines
Quoted 1 line
No. That's why I am wondering. I read about such errors like I got before in lkml and usually they were not fs related but libata siimage driver related. It could be just a coincidence that it came up with xfs, but till now (I guess 5 days now, though not 24/7 running) ext3 is behaving nicely.
It's almost certainly not a filesystem problem, but an IO layer problem.
Maybe you only see it with xfs due to different disk IO patterns with
xfs vs. ext3... the two will certainly be allocating & writing to the disk in different ways.
Hmm, OK. When I have some hd space again. I might try to reproduce this error. Whom should I bug then if it reappears?
Cheers,
Prakash