XFS breakage in 2.6.18-rc1
Hi friends,
I upgraded to 2.6.18-rc1 on sunday, with the following results (taken from my /var/log/kern.log), which ultimately led me to reinstall my system:
Jul 17 07:10:12 pulsar kernel: klogd 1.4.1#18, log source = /proc/kmsg started. Jul 17 07:10:12 pulsar kernel: Linux version 2.6.18-rc1 (torsten@pulsar) (gcc version 4.1.2 20060630 (prerelease) (Debian 4.1.1-6)) #18 SMP PREEMPT Fri Jul 14 07:58:49 CEST 2006 ... Jul 17 07:10:32 pulsar kernel: agpgart: Putting AGP V3 device at 0000:03:00.0 into 4x mode Jul 17 07:10:32 pulsar kernel: [drm] Setting GART location based on new memory map Jul 17 07:10:32 pulsar kernel: [drm] Loading R200 Microcode Jul 17 07:10:32 pulsar kernel: [drm] writeback test succeeded in 1 usecs Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538 Jul 17 07:33:53 pulsar kernel: Filesystem "dm-6": XFS internal error xfs_da_do_buf(1) at line 1992 of file fs/xfs/xfs_da_btree.c. Caller 0xf8a837d0 Jul 17 07:33:53 pulsar kernel: [<f8a83313>] xfs_da_do_buf+0x4d3/0x900 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a837d0>] xfs_da_read_buf+0x30/0x40 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a8e0cf>] xfs_dir2_leafn_lookup_int+0x28f/0x520 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a8e0cf>] xfs_dir2_leafn_lookup_int+0x28f/0x520 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a89215>] xfs_dir2_data_log_unused+0x55/0x70 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a837d0>] xfs_da_read_buf+0x30/0x40 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a8c782>] xfs_dir2_node_removename+0x312/0x500 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a8c782>] xfs_dir2_node_removename+0x312/0x500 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a87337>] xfs_dir_removename+0xf7/0x100 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8a9720d>] xfs_ilock_nowait+0xcd/0x100 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ab9783>] xfs_remove+0x393/0x4c0 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43ef>] xfs_vn_permission+0xf/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43e0>] xfs_vn_permission+0x0/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac4123>] xfs_vn_unlink+0x23/0x60 [xfs] Jul 17 07:33:53 pulsar kernel: [<c017a223>] mntput_no_expire+0x13/0x70 Jul 17 07:33:53 pulsar kernel: [<c016e0c1>] link_path_walk+0x71/0xf0 Jul 17 07:33:53 pulsar kernel: [<f8ab0638>] xfs_trans_unlocked_item+0x38/0x60 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ab63ff>] xfs_access+0x3f/0x50 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43ef>] xfs_vn_permission+0xf/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43e0>] xfs_vn_permission+0x0/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<c016bdca>] permission+0x8a/0xc0 Jul 17 07:33:53 pulsar kernel: [<c016c3e9>] may_delete+0x39/0x120 Jul 17 07:33:53 pulsar kernel: [<c016c957>] vfs_unlink+0x87/0xe0 Jul 17 07:33:53 pulsar kernel: [<c016e96c>] do_unlinkat+0xcc/0x150 Jul 17 07:33:53 pulsar kernel: [<c0102fbf>] syscall_call+0x7/0xb Jul 17 07:33:53 pulsar kernel: Filesystem "dm-6": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c. Caller 0xf8ab97d7 Jul 17 07:33:53 pulsar kernel: [<f8aaf91d>] xfs_trans_cancel+0xdd/0x100 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ab97d7>] xfs_remove+0x3e7/0x4c0 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ab97d7>] xfs_remove+0x3e7/0x4c0 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43ef>] xfs_vn_permission+0xf/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43e0>] xfs_vn_permission+0x0/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac4123>] xfs_vn_unlink+0x23/0x60 [xfs] Jul 17 07:33:53 pulsar kernel: [<c017a223>] mntput_no_expire+0x13/0x70 Jul 17 07:33:53 pulsar kernel: [<c016e0c1>] link_path_walk+0x71/0xf0 Jul 17 07:33:53 pulsar kernel: [<f8ab0638>] xfs_trans_unlocked_item+0x38/0x60 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ab63ff>] xfs_access+0x3f/0x50 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43ef>] xfs_vn_permission+0xf/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<f8ac43e0>] xfs_vn_permission+0x0/0x20 [xfs] Jul 17 07:33:53 pulsar kernel: [<c016bdca>] permission+0x8a/0xc0 Jul 17 07:33:53 pulsar kernel: [<c016c3e9>] may_delete+0x39/0x120 Jul 17 07:33:53 pulsar kernel: [<c016c957>] vfs_unlink+0x87/0xe0 Jul 17 07:33:53 pulsar kernel: [<c016e96c>] do_unlinkat+0xcc/0x150 Jul 17 07:33:53 pulsar kernel: [<c0102fbf>] syscall_call+0x7/0xb Jul 17 07:33:53 pulsar kernel: xfs_force_shutdown(dm-6,0x8) called from line 1139 of file fs/xfs/xfs_trans.c. Return address = 0xf8ac77bc Jul 17 07:33:53 pulsar kernel: Filesystem "dm-6": Corruption of in-memory data detected. Shutting down filesystem: dm-6 Jul 17 07:33:53 pulsar kernel: Please umount the filesystem, and rectify the problem(s) Jul 17 07:39:32 pulsar kernel: Reducing readahead size to 32K Jul 17 07:39:32 pulsar kernel: Reducing readahead size to 8K
That problem occured during a dist-upgrade, dm-6 is my /usr partition. Funny enough this happened a few months after finally replaced my ancient disk with a RAID1 array to make sure I do not lose data ;)
In any case it seems like the XFS driver in 2.6.18-rc1 is decently broken. After booting into 2.6.17 again, I could use /usr again but random files contain null bytes, firefox segfaults instead of starting up and a number of programs fail in mysterious ways. I tried to recover using xfs_repair but I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)
If more information from my logs is required, I can make it available (and any part of the partition if required).
Greetings
Torsten
Quoted 1 line · Torsten Landschoff
Hi friends,
Hi Torsten,
Quoted 6 lines
I upgraded to 2.6.18-rc1 on sunday, with the following results (taken from my
/var/log/kern.log), which ultimately led me to reinstall my system:Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538
I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
Quoted 3 lines
of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)
2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
thanks.
Signature · 1 line
Nathan
On Tuesday 18 July 2006 23:57, Nathan Scott wrote:
[snip]
Quoted 7 lines
Quoted 3 lines
of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
Just out of interest, I've got a few XFS volumes that were created 24 months ago on a machine that I upgraded to 2.6.17 about a month ago. I haven't seen any crashes so far.
Assuming I get the newest XFS repair tools on there, what's the disadvantage of repairing versus creating a new filesystem? What special circumstances are required to cause a crash?
Signature · 5 lines
Cheers, Alistair. Final year Computer Science undergraduate. 1F2 55 South Clerk Street, Edinburgh, UK.
Quoted 10 lines · Alistair John Strachan
On Tuesday 18 July 2006 23:57, Nathan Scott wrote:
[snip]Quoted 7 lines
Quoted 3 lines
of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
Quoted 7 lines
Just out of interest, I've got a few XFS volumes that were created 24 months ago on a machine that I upgraded to 2.6.17 about a month ago. I haven't seen any crashes so far.
Assuming I get the newest XFS repair tools on there, what's the disadvantage of repairing versus creating a new filesystem? What special circumstances are required to cause a crash?
There should be no disadvantage to repairing. I will update the FAQ shortly to describe all the details of the problem, recommendations on how to address it, which kernel version is affected, etc.
cheers.
Signature · 1 line
Nathan
Quoted 15 lines · Nathan Scott
Quoted 14 lines · Alistair John Strachan
On Tuesday 18 July 2006 23:57, Nathan Scott wrote:
[snip]Quoted 7 lines
Quoted 3 lines
of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
Just out of interest, I've got a few XFS volumes that were created 24 months ago on a machine that I upgraded to 2.6.17 about a month ago. I haven't seen any crashes so far.
Quoted 7 lines
Quoted 3 lines
Assuming I get the newest XFS repair tools on there, what's the disadvantage of repairing versus creating a new filesystem? What special circumstances are required to cause a crash?
There should be no disadvantage to repairing. I will update the FAQ shortly to describe all the details of the problem, recommendations on how to address it, which kernel version is affected, etc.
this FAQ, is it this: http://oss.sgi.com/projects/xfs/faq.html#dir2 ? (btw, it seems that while only in the TOC once, you have the same about 2.6.17 twice..)..
which version of xfsprogs should i use while doing the xfs_check ?
Quoted 3 lines
cheers.
Quoted 9 lines · Nathan Scott
Quoted 1 line · Torsten Landschoff
Hi friends,
Hi Torsten,
Quoted 4 lines
I upgraded to 2.6.18-rc1 on sunday, with the following results (taken from my
/var/log/kern.log), which ultimately led me to reinstall my system:
Quoted 6 lines
Quoted 2 lines
Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
This has me very worried.
i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before. does this mean my .17-rc3 may have corrupted my filesystem?
what action do you suggest i do now?
Quoted 11 lines
Quoted 4 lines
of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
thanks.
Quoted 17 lines · Kasper Sandberg
Quoted 9 lines · Nathan Scott
Quoted 1 line · Torsten Landschoff
Hi friends,
Hi Torsten,
Quoted 4 lines
I upgraded to 2.6.18-rc1 on sunday, with the following results (taken from my
/var/log/kern.log), which ultimately led me to reinstall my system:Quoted 6 lines
Quoted 2 lines
Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
Quoted 17 lines
This has me very worried.
i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before. does this mean my .17-rc3 may have corrupted my filesystem?
what action do you suggest i do now?
Quoted 10 lines
Quoted 4 lines
of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
thanks.
According to another thread Nathan just responded to, it sounds like we need to wait for a new version of the xfsprogs package, and then run xfs_repair on the affected filesystems. I wouldn't worry about it too much if you've not had any crashes. The damage can be repaired, just not right now.
I'm still waiting for a crash on a machine that has been under heavy load for 28 days, so it's obviously not _that_ easy to trigger.
Signature · 5 lines
Cheers, Alistair. Third year Computer Science undergraduate. 1F2 55 South Clerk Street, Edinburgh, UK.
Quoted 34 lines · Alistair John Strachan
Quoted 33 lines · Kasper Sandberg
Quoted 9 lines · Nathan Scott
Quoted 1 line · Torsten Landschoff
Hi friends,
Hi Torsten,
Quoted 4 lines
I upgraded to 2.6.18-rc1 on sunday, with the following results (taken from my
/var/log/kern.log), which ultimately led me to reinstall my system:Quoted 6 lines
Quoted 2 lines
Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
This has me very worried.
i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before. does this mean my .17-rc3 may have corrupted my filesystem?
what action do you suggest i do now?
Quoted 9 lines
Quoted 4 lines
of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
Quoted 6 lines
Quoted 1 line
Quoted 1 line
thanks.
According to another thread Nathan just responded to, it sounds like we need to wait for a new version of the xfsprogs package, and then run
xfs_repairon the affected filesystems. I wouldn't worry about it too much if you've not had any crashes. The damage can be repaired, just not right now.
without ANY loss? because even though it would be abit painful for me to do, i do have the option of smashing in a new drive, copy everything, and reinitialize my filesystem.
Quoted 3 lines
I'm still waiting for a crash on a machine that has been under heavy load for 28 days, so it's obviously not _that_ easy to trigger.
so basically if i upgrade to a safe kernel before i do get these errors, im good?
Quoted 1 line
Quoted 16 lines · Kasper Sandberg
Quoted 9 lines · Nathan Scott
Quoted 1 line · Torsten Landschoff
Hi friends,
Hi Torsten,
Quoted 4 lines
I upgraded to 2.6.18-rc1 on sunday, with the following results (taken from my
/var/log/kern.log), which ultimately led me to reinstall my system:Quoted 6 lines
Quoted 2 lines
Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
Quoted 6 lines
This has me very worried.
i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before. does this mean my .17-rc3 may have corrupted my filesystem?
what action do you suggest i do now?
The odds are decent that you're unaffected. You can check your filesystem using xfs_check or xfs_repair -n and these will give you a good indication as to whether further action is required.
cheers.
Signature · 1 line
Nathan
FAQ updated (was Re: XFS breakage...)
In reply to Kasper SandbergQuoted 9 lines · Kasper Sandberg
Quoted 8 lines · Nathan Scott
Quoted 3 lines · Torsten Landschoff
Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
Correction there - no -stable exists with this yet, I guess that'll be 2.6.17.7 once its out though.
Quoted 1 line
what action do you suggest i do now?
I've captured the state of this issue here, with options and ways to correct the problem:
http://oss.sgi.com/projects/xfs/faq.html#dir2
Hope this helps.
cheers.
Signature · 1 line
Nathan
Re: FAQ updated (was Re: XFS breakage...)
Am Donnerstag 20 Juli 2006 09:13 schrieb Nathan Scott:
Quoted 7 lines
Quoted 1 line
what action do you suggest i do now?
I've captured the state of this issue here, with options and ways to correct the problem:
http://oss.sgi.com/projects/xfs/faq.html#dir2Hope this helps.
Hello Nathan,
thanks a lot.
If you need anyone to test a new xfs_repair let me know. I have a partition which that kind of corruption at hand and can compile a new xfs_repair under Knoppix 5.
If you do not need any additional testers I will fix it the manual way.
Regards,
Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Re: FAQ updated (was Re: XFS breakage...)
In reply to Nathan ScottHi Nathan,
Am Donnerstag, 20. Juli 2006 09:13 schrieb Nathan Scott:
Quoted 4 lines
I've captured the state of this issue here, with options and ways to correct the problem:
http://oss.sgi.com/projects/xfs/faq.html#dir2
Thanks for the pointer. I think, it is valuable for all XFS users, but reading the FAQ with a decent webbrowser on linux (konqueror 3.5.3 and firefox 1.5.0.4 in my case) is very painful, due to the overlong lines, not to speak of printing such texts. Try yourself: load that page into konqueror, hit print, select preview and 'have fun', hmm, suffer..
Pete
Re: FAQ updated (was Re: XFS breakage...)
In reply to Nathan ScottQuoted 10 lines · Nathan Scott
Correction there - no -stable exists with this yet, I guess that'll be 2.6.17.7 once its out though.
Quoted 1 line
what action do you suggest i do now?
I've captured the state of this issue here, with options and ways to correct the problem:
http://oss.sgi.com/projects/xfs/faq.html#dir2Hope this helps.
It does, thanks :)
Does this problem exist in 2.16.6.x??
Quoted 1 line
From various comments like:
Unless 2.6.16.x is a dead-end could we please also have this patch put into there?
and
a result (I believe) of the corruption bug that was in 2.6.16/17.
and
I just want to confirm this bug as well and unfortunately it was my system disk too who had to take the hit. Im running 2.6.16
I assume it does.
But the FAQ says:
Q: What is the issue with directory corruption in Linux 2.6.17? In the Linux kernel 2.6.17 release a subtle bug...
which implies it's not...
HELP
So given this is from 2.6.16.9:
/*
* One less used entry in the free table.
*/
INT_MOD(free->hdr.nused, ARCH_CONVERT, -1);
xfs_dir2_free_log_header(tp, fbp);and it looks awfully similar to the patch which says:
fs/xfs/xfs_dir2_node.c +1 -1
@@ -970,7 +970,7 @@ xfs_dir2_leafn_remove(/* * One less used entry in the free table. */ - free->hdr.nused =cpu_to_be32 (-1 ); +be32_add (&free->hdr.nused,-1 );xfs_dir2_free_log_header (tp, fbp);
Should 2.6.16.x replace
INT_MOD(free->hdr.nused, ARCH_CONVERT, -1);
with
be32_add(&free->hdr.nused, -1);
I hope so because I assumed there simply wasn't a patch for 2.6.16 and applied this 'best guess' to my servers and rebooted/remounted successfully.
David
Re: FAQ updated (was Re: XFS breakage...)
Quoted 1 line · David Greaves
Does this problem exist in 2.16.6.x??
The change was merged after 2.6.16.x was branched, I was mistaken in how long I thought the bug has been about.
Quoted 2 lines
I hope so because I assumed there simply wasn't a patch for 2.6.16 and applied this 'best guess' to my servers and rebooted/remounted successfully.
Doing the correct change to 2.6.16.x won't hurt, but it's not necessary.
Re: FAQ updated (was Re: XFS breakage...)
Quoted 4 lines · Chris Wedgwood
Quoted 1 line · David Greaves
Does this problem exist in 2.16.6.x??
Quoted 8 lines
The change was merged after 2.6.16.x was branched, I was mistaken in how long I thought the bug has been about.
Quoted 2 lines
I hope so because I assumed there simply wasn't a patch for 2.6.16 and applied this 'best guess' to my servers and rebooted/remounted successfully.
Doing the correct change to 2.6.16.x won't hurt, but it's not necessary.
Yep. As Chris said, 2.6.17 is the only affected kernel. I've fixed up the whacky html formatting and my merge error (thanks to all for reporting those) so its a bit more readable now.
cheers.
Signature · 1 line
Nathan
Re: FAQ updated (was Re: XFS breakage...)
Nathan,
Does the bug only occur during a crash?
I have been running 2.6.17.x for awhile now (multiple XFS filesystems, all on UPS) - no issue?
Justin.
Quoted 28 lines · Nathan Scott
Quoted 4 lines · Chris Wedgwood
Quoted 1 line · David Greaves
Does this problem exist in 2.16.6.x??
Quoted 8 lines
The change was merged after 2.6.16.x was branched, I was mistaken in how long I thought the bug has been about.
Quoted 2 lines
I hope so because I assumed there simply wasn't a patch for 2.6.16 and applied this 'best guess' to my servers and rebooted/remounted successfully.
Doing the correct change to 2.6.16.x won't hurt, but it's not necessary.
Yep. As Chris said, 2.6.17 is the only affected kernel. I've fixed up the whacky html formatting and my merge error (thanks to all for reporting those) so its a bit more readable now.
cheers.
Signature · 6 lines
Nathan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: FAQ updated (was Re: XFS breakage...)
Quoted 3 lines · Justin Piszcz
Nathan,
Does the bug only occur during a crash?
No, its unrelated to crashing. Only when adding/removing from a directory that is in a specific node/btree format (many entries), and only under a specific set of conditions (like what directory entry names were used, which blocks they've hashed to and how they ended up being allocated and in what order each block gets removed from the directory).
Quoted 2 lines
I have been running 2.6.17.x for awhile now (multiple XFS filesystems, all on UPS) - no issue?
Could be an issue, could be none. xfs_check it to be sure.
cheers.
Signature · 1 line
Nathan
Re: FAQ updated (was Re: XFS breakage...)
Quoted 12 lines · Nathan Scott
Quoted 3 lines · Justin Piszcz
Nathan,
Does the bug only occur during a crash?
No, its unrelated to crashing. Only when adding/removing from a directory that is in a specific node/btree format (many entries), and only under a specific set of conditions (like what directory entry names were used, which blocks they've hashed to and how they ended up being allocated and in what order each block gets removed from the directory).
Quoted 11 lines
Quoted 2 lines
I have been running 2.6.17.x for awhile now (multiple XFS filesystems, all on UPS) - no issue?
Could be an issue, could be none.
xfs_checkit to be sure.cheers.
Signature · 1 line
Nathan
p34:~# xfs_check -v /dev/md3 xfs_check: out of memory p34:~#
D'oh...
1GB ram, 2GB swap trying to check a 2.6T fs, no dice.
As long as it mounted ok with the patched kernel, should one be ok?
Re: FAQ updated (was Re: XFS breakage...)
Quoted 5 lines · Justin Piszcz
p34:~# xfs_check -v /dev/md3 xfs_check: out of memory p34:~#D'oh...
xfs_repair -n is another option, it has a cheaper (memory wise, usually) checking algorithm.
Quoted 1 line
As long as it mounted ok with the patched kernel, should one be ok?
Not necessarily, no - mount will only read the root inode.
cheers.
Signature · 1 line
Nathan
Re: FAQ updated (was Re: XFS breakage...)
Nasty!
- agno = 37
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
free block 16777216 for directory inode 2684356622 bad nused free block 16777216 for directory inode 2147485710 bad nused
- traversal finished ...
- traversing all unattached subtrees ...
- traversals finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting. p34:~#
I applied the "one line fix" - I should be ok now?
Quoted 24 lines · Nathan Scott
Quoted 5 lines · Justin Piszcz
p34:~# xfs_check -v /dev/md3 xfs_check: out of memory p34:~#D'oh...
xfs_repair-n is another option, it has a cheaper (memory wise, usually) checking algorithm.Quoted 1 line
As long as it mounted ok with the patched kernel, should one be ok?
Not necessarily, no - mount will only read the root inode.
cheers.
Signature · 6 lines
Nathan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: FAQ updated (was Re: XFS breakage...)
Erm, the xfs_repair -n only prints out what it needs to fix, I read somewhere that xfs_repair may make things worse?
What is the 'correct' fix?
Quoted 47 lines · Justin Piszcz
Nasty!
- agno = 37
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
free block 16777216 for directory inode 2684356622 bad nused free block 16777216 for directory inode 2147485710 bad nused
- traversal finished ...
- traversing all unattached subtrees ...
- traversals finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting. p34:~#I applied the "one line fix" - I should be ok now?
Quoted 24 lines · Nathan Scott
Quoted 5 lines · Justin Piszcz
p34:~# xfs_check -v /dev/md3 xfs_check: out of memory p34:~#D'oh...
xfs_repair-n is another option, it has a cheaper (memory wise, usually) checking algorithm.Quoted 1 line
As long as it mounted ok with the patched kernel, should one be ok?
Not necessarily, no - mount will only read the root inode.
cheers.
Signature · 6 lines
Nathan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: FAQ updated (was Re: XFS breakage...)
In reply to Justin PiszczQuoted 7 lines · Justin Piszcz
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
free block 16777216 for directory inode 2684356622 bad nused free block 16777216 for directory inode 2147485710 bad nused
- traversal finished ...
...
I applied the "one line fix" - I should be ok now?
You have two corrupt directory inodes (caused by this bug, that is exactly the signature I'd expect - it was a nused field that was affected by the dodgey endian change). The two inodes need to be fixed - consult the FAQ for details.
Once fixed, and with a patched kernel, you're set.
cheers.
Signature · 1 line
Nathan
Re: FAQ updated (was Re: XFS breakage...)
Nathan,
Running xfs_repair multiple times (after following the FAQ for the write core.mode 0 fix), I get this:
- agno = 3
- agno = 4
- agno = 5
- agno = 6
entry ".." at block 0 offset 1352 in directory inode 3221227534 references free inode 2112
clearing inode number in entry at offset 1352...
no .. entry for directory 3221227534
- agno = 7
- agno = 8
- agno = 9
disconnected inode 2684386082, moving to lost+found
disconnected inode 2684386083, moving to lost+found
disconnected inode 2684386084, moving to lost+found
disconnected inode 2684386085, moving to lost+found
disconnected inode 2684386086, moving to lost+found
disconnected inode 2684386087, moving to lost+found
disconnected inode 2684386088, moving to lost+found
disconnected inode 2684386089, moving to lost+found
disconnected inode 2684386090, moving to lost+found
disconnected inode 2684386091, moving to lost+found
disconnected inode 2684386092, moving to lost+found
disconnected inode 2684386093, moving to lost+found
disconnected inode 2684386094, moving to lost+found
disconnected inode 2684386095, moving to lost+found
disconnected inode 2684386096, moving to lost+found
disconnected inode 2684386097, moving to lost+found
disconnected inode 2684386098, moving to lost+found
disconnected inode 2684386099, moving to lost+found
disconnected inode 2684386100, moving to lost+found
disconnected inode 2684386101, moving to lost+found
disconnected inode 2684386102, moving to lost+found
disconnected inode 2684386103, moving to lost+found
disconnected inode 2684386104, moving to lost+found
disconnected inode 2684386105, moving to lost+found
disconnected inode 2684386106, moving to lost+found
disconnected inode 2684386107, moving to lost+found
disconnected inode 2684386108, moving to lost+found
disconnected inode 2684386109, moving to lost+found
disconnected inode 2684386110, moving to lost+found
disconnected inode 2684386111, moving to lost+found
disconnected inode 2684386112, moving to lost+found
disconnected inode 2684386113, moving to lost+found
disconnected inode 2684386114, moving to lost+found
disconnected inode 2684386115, moving to lost+found
disconnected inode 2684386116, moving to lost+found
disconnected inode 2684386117, moving to lost+found
disconnected inode 2684386118, moving to lost+found
disconnected inode 2684386119, moving to lost+found
disconnected inode 2684386120, moving to lost+found
disconnected inode 2684386121, moving to lost+found
disconnected inode 2684386122, moving to lost+found
disconnected inode 2684386123, moving to lost+found
disconnected inode 2684386124, moving to lost+found
disconnected inode 2684386125, moving to lost+found
disconnected inode 2684386126, moving to lost+found
disconnected inode 2684386127, moving to lost+found
disconnected inode 2684386128, moving to lost+found
disconnected inode 2684386129, moving to lost+found
disconnected inode 2684386130, moving to lost+found
disconnected inode 2684386131, moving to lost+found
disconnected inode 2684386132, moving to lost+found
disconnected inode 2684386133, moving to lost+found
disconnected inode 2684386134, moving to lost+found
disconnected inode 2684386135, moving to lost+found
disconnected inode 2684386136, moving to lost+found
disconnected inode 2684386137, moving to lost+found
disconnected inode 2684386138, moving to lost+found
disconnected inode 2684386139, moving to lost+found
disconnected inode 2684386140, moving to lost+found
disconnected inode 2684386141, moving to lost+found
disconnected inode 2684386142, moving to lost+found
disconnected inode 2684386143, moving to lost+found
disconnected inode 2684386144, moving to lost+found
disconnected inode 2684386145, moving to lost+found
disconnected inode 2684386146, moving to lost+found
disconnected inode 2684386147, moving to lost+found
disconnected inode 2684386148, moving to lost+found
disconnected inode 2684386149, moving to lost+found
disconnected inode 2684386150, moving to lost+found
disconnected inode 2684386151, moving to lost+found
disconnected inode 2684386152, moving to lost+found
disconnected inode 2684386153, moving to lost+found
disconnected inode 2684386154, moving to lost+found
disconnected inode 2684386155, moving to lost+found
disconnected inode 2684386156, moving to lost+found
disconnected inode 2684386157, moving to lost+found
disconnected inode 2684386158, moving to lost+found
disconnected inode 2684386159, moving to lost+found
disconnected inode 2684386160, moving to lost+found
disconnected inode 2684386161, moving to lost+found
disconnected inode 2684386162, moving to lost+found
disconnected inode 2684386163, moving to lost+found
disconnected inode 2684386164, moving to lost+found
disconnected inode 2684386165, moving to lost+found
disconnected inode 2684653605, moving to lost+found
disconnected dir inode 3221227534, moving to lost+found
Phase 7 - verify and correct link counts...
resetting inode 3221227534 nlinks from 3 to 2
done
p34:~#
I can run this over and over, and the result is the same?
Quoted 27 lines · Nathan Scott
Quoted 8 lines · Justin Piszcz
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
free block 16777216 for directory inode 2684356622 bad nused free block 16777216 for directory inode 2147485710 bad nused- traversal finished ...
...
I applied the "one line fix" - I should be ok now?You have two corrupt directory inodes (caused by this bug, that is exactly the signature I'd expect - it was a nused field that was affected by the dodgey endian change). The two inodes need to be fixed - consult the FAQ for details.
Once fixed, and with a patched kernel, you're set.
cheers.
Signature · 6 lines
Nathan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: FAQ updated (was Re: XFS breakage...)
Quoted 1 line · Justin Piszcz
I can run this over and over, and the result is the same?
lost+found is recreated every time, rename it and you'll get less output
Re: FAQ updated (was Re: XFS breakage...)
Thanks, that was it, after removing the lost+found directory & re-running xfs_repair, I no longer have any errors, onthat device anyway.
Quoted 12 lines · Chris Wedgwood
Quoted 1 line · Justin Piszcz
I can run this over and over, and the result is the same?
lost+found is recreated every time, rename it and you'll get less output
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: FAQ updated (was Re: XFS breakage...)
In reply to Chris WedgwoodQuoted 6 lines · Chris Wedgwood
Quoted 1 line · Justin Piszcz
I can run this over and over, and the result is the same?
lost+found is recreated every time, rename it and you'll get less output
Yes this is the current xfs_repair behaviour (any previously unlinked inodes will be found as unlinked on each successive run, due to lost+found being recreated). This will likely be rethought soon (not far off), since it confuses everyone.
So, its all good - xfs_repair has fixed things and you're all set now.
cheers.
Signature · 1 line
Nathan
Re: FAQ updated (was Re: XFS breakage...)
In reply to Nathan ScottQuoted 14 lines · Nathan Scott
Quoted 9 lines · Kasper Sandberg
Quoted 8 lines · Nathan Scott
Quoted 3 lines · Torsten Landschoff
Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
Correction there - no -stable exists with this yet, I guess that'll be 2.6.17.7 once its out though.
Quoted 7 lines
Quoted 1 line
what action do you suggest i do now?
I've captured the state of this issue here, with options and ways to correct the problem:
http://oss.sgi.com/projects/xfs/faq.html#dir2Hope this helps.
I actually tried the xfs_db method to fix my / filesystem (as you had outlined in http://marc.theaimsgroup.com/?l=linux-kernel&m=115070320401919&w=2),
and while it's quite possible that I screwed it up, after a subsequent xfs_repair run (which completed successfully and moved lots of stuff to /lost+found, as I would expect), the XFS code had serious problems with various parts of my filesystem (like "ls /lost+found", which would cause lots of errors to be logged, although not a complete fs shutdown). After another run through xfs_repair resulted in a filesystem that would no longer even successfully boot.
Unfortunately it was a mostly-full 74GB big-/ partition on my primary machine (a laptop), so I don't have a dump of it for you and my report is probably pretty useless. :( But on the bright side, virtually all of the filesystem was otherwise intact and I was able to get all my data off before rebuilding my system.
Signature · 3 lines
Kevin 'radsaq' Radloff [email protected] http://thesaq.com/
Re: FAQ updated (was Re: XFS breakage...)
Quoted 33 lines · Kevin Radloff
Quoted 13 lines · Nathan Scott
Quoted 8 lines · Kasper Sandberg
Quoted 7 lines · Nathan Scott
Quoted 2 lines · Torsten Landschoff
Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216 Jul 17 07:33:53 pulsar kernel: dir: inode 54526538I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
Correction there - no -stable exists with this yet, I guess that'll be 2.6.17.7 once its out though.
Quoted 7 lines
Quoted 1 line
what action do you suggest i do now?
I've captured the state of this issue here, with options and ways to correct the problem:
http://oss.sgi.com/projects/xfs/faq.html#dir2Hope this helps.
I actually tried the
xfs_dbmethod to fix my / filesystem (as you had outlined in
http://marc.theaimsgroup.com/?l=linux-kernel&m=115070320401919&w=2), and while it's quite possible that I screwed it up, after a subsequentxfs_repairrun (which completed successfully and moved lots of stuff to /lost+found, as I would expect), the XFS code had serious problems with various parts of my filesystem (like "ls /lost+found", which would cause lots of errors to be logged, although not a complete fs shutdown). After another run throughxfs_repairresulted in a filesystem that would no longer even successfully boot.
Quoted 5 lines
Unfortunately it was a mostly-full 74GB big-/ partition on my primary machine (a laptop), so I don't have a dump of it for you and my report is probably pretty useless. :( But on the bright side, virtually all of the filesystem was otherwise intact and I was able to get all my data off before rebuilding my system.
I've been hit by this on my root filesystem today, and when I followed the instructions, I was able to retrieve my data. It turned out that the only corrupted inode was /, which unfortunately meant I had to repair the filesystem with a boot-cd. However, it was obvious which inodes corresponded to which directories, and I was able to repair it.
I'm not sure this advice is sound, but it seems to me that if you're running an affected 2.6.17 kernel (or ever have) on an XFS volume, it's not worth risking destruction if you haven't had any oopses. The filesystem will get worse, but hopefully in a non-fatal way, and the XFS guys will hopefully have an xfs_repair up that works, soon.
Right now I'd highly recommend copying as much as possible from the corrupted filesystem )after following the instructions) to a new filesystem (with an unaffected kernel, of course) and destroying the old one. I still have inconsistencies on the filesystem I "repaired", and it was a fairly painful process.
Signature · 5 lines
Cheers, Alistair. Final year Computer Science undergraduate. 1F2 55 South Clerk Street, Edinburgh, UK.
Re: FAQ updated (was Re: XFS breakage...)
In reply to Nathan ScottNathan Scott wrote:
: I've captured the state of this issue here, with options and ways : to correct the problem:
: http://oss.sgi.com/projects/xfs/faq.html#dir2
:
: Hope this helps.
I have been hit with this bug as well - I tried to clear the
two corrupted directory inodes with xfs_db (as the FAQ entry says), then ran xfs_repair (lots of files ended up in lost+found), but apparently the volume is still not OK - when I tried to use it (this volume is a public FTP archive), I got the following traces:
Jul 30 16:04:49 odysseus kernel: Filesystem "md5": XFS internal error xfs_da_do_buf(2) at line 2212 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff80324221
Jul 30 16:04:49 odysseus kernel:
Jul 30 16:04:49 odysseus kernel: Call Trace: <ffffffff803331ac>{xfs_corruption_error+228}
Jul 30 16:04:49 odysseus kernel: <ffffffff8035630e>{kmem_zone_alloc+86} <ffffffff803240f0>{xfs_da_do_buf+1359}
Jul 30 16:04:49 odysseus kernel: <ffffffff80324221>{xfs_da_read_buf+22} <ffffffff80323aba>{xfs_da_buf_make+31}
Jul 30 16:04:49 odysseus kernel: <ffffffff80324221>{xfs_da_read_buf+22} <ffffffff803263e7>{xfs_da_node_lookup_int+112}
Jul 30 16:04:49 odysseus kernel: <ffffffff803263e7>{xfs_da_node_lookup_int+112} <ffffffff8032c7b8>{xfs_dir2_node_lookup+70}
Jul 30 16:04:50 odysseus kernel: <ffffffff80327b35>{xfs_dir2_isleaf+25} <ffffffff803280d6>{xfs_dir2_lookup+256}
Jul 30 16:04:51 odysseus kernel: <ffffffff8034dd10>{xfs_dir_lookup_int+55} <ffffffff803511af>{xfs_lookup+79}
Jul 30 16:04:51 odysseus kernel: <ffffffff8035c95a>{xfs_vn_lo7b35>{xfs_dir2_isleaf+25} <ffffffff803280d6>{xfs_dir2_lookup+256}
Jul 30 16:04:52 odysseus kernel: <ffffffff8034dd10>{xfs_dir_lookup_int+55} <ffffffff803511af>{xfs_lookup+79}
Jul 30 16:04:53 odysseus kernel: <ffffffff8035c95a>{xfs_vn_lookup+48} <ffffffff80270b45>{do_lookup+196}
Jul 30 16:04:53 odysseus rpc.statd[3145]: Caught signal 15, un-registering and exiting.
Jul 30 16:04:53 odysseus kernel: <ffffffff802729c6>{__link_path_walk+2435} <ffffffff80272f40>{link_path_walk+89}
Jul 30 16:04:53 odysseus kernel: <ffffffff8049c0d2>{__sched_text_start+290} <ffffffff80273396>{do_path_lookup+614}
Jul 30 16:04:53 odysseus kernel: <ffffffff80271e47>{getname+347} <ffffffff80273bd6>{__user_walk_fd+55}
Jul 30 16:04:53 odysseus kernel: <ffffffff8026cba7>{vfs_lstat_fd+21} <ffffffff8049c0d2>{__sched_text_start+290}
Jul 30 16:04:53 odysseus kernel: <ffffffff8026cd92>{sys_newlstat+25} <ffffffff80265023>{vfs_write+283}
Jul 30 16:04:53 odysseus kernel: <ffffffff8026554c>{sys_write+69} <ffffffff80209826>{system_call+126}
Jul 30 16:04:53 odysseus kernel: 0x0: 00 00 00 00 00 00 00 00 00 00 00 00 00 0000 00
This is 2.6.17.7 dual x86_64 (Fedora Core 5). It has been unfortunately
running 2.6.17.1 for some time.
I will probably have to recreate the volume and restore its
contents from backups. Or is there any better solution?
Thanks,
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |Quoted 2 lines
I will never go to meetings again because I think face to face meetings <
are the biggest waste of time you can ever have. --Linus Torvalds <
Re: FAQ updated (was Re: XFS breakage...)
Quoted 38 lines · Jan Kasprzak
Nathan Scott wrote:
: I've captured the state of this issue here, with options and ways : to correct the problem:
: http://oss.sgi.com/projects/xfs/faq.html#dir2
:
: Hope this helps.I have been hit with this bug as well - I tried to clear the
two corrupted directory inodes withxfs_db(as the FAQ entry says), then ranxfs_repair(lots of files ended up in lost+found), but apparently the volume is still not OK - when I tried to use it (this volume is a public FTP archive), I got the following traces:Jul 30 16:04:49 odysseus kernel: Filesystem "md5": XFS internal error xfs_da_do_buf(2) at line 2212 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff80324221 Jul 30 16:04:49 odysseus kernel: Jul 30 16:04:49 odysseus kernel: Call Trace: <ffffffff803331ac>{xfs_corruption_error+228} Jul 30 16:04:49 odysseus kernel: <ffffffff8035630e>{kmem_zone_alloc+86} <ffffffff803240f0>{xfs_da_do_buf+1359} Jul 30 16:04:49 odysseus kernel: <ffffffff80324221>{xfs_da_read_buf+22} <ffffffff80323aba>{xfs_da_buf_make+31} Jul 30 16:04:49 odysseus kernel: <ffffffff80324221>{xfs_da_read_buf+22} <ffffffff803263e7>{xfs_da_node_lookup_int+112} Jul 30 16:04:49 odysseus kernel: <ffffffff803263e7>{xfs_da_node_lookup_int+112} <ffffffff8032c7b8>{xfs_dir2_node_lookup+70} Jul 30 16:04:50 odysseus kernel: <ffffffff80327b35>{xfs_dir2_isleaf+25} <ffffffff803280d6>{xfs_dir2_lookup+256} Jul 30 16:04:51 odysseus kernel: <ffffffff8034dd10>{xfs_dir_lookup_int+55} <ffffffff803511af>{xfs_lookup+79} Jul 30 16:04:51 odysseus kernel: <ffffffff8035c95a>{xfs_vn_lo7b35>{xfs_dir2_isleaf+25} <ffffffff803280d6>{xfs_dir2_lookup+256} Jul 30 16:04:52 odysseus kernel: <ffffffff8034dd10>{xfs_dir_lookup_int+55} <ffffffff803511af>{xfs_lookup+79} Jul 30 16:04:53 odysseus kernel: <ffffffff8035c95a>{xfs_vn_lookup+48} <ffffffff80270b45>{do_lookup+196} Jul 30 16:04:53 odysseus rpc.statd[3145]: Caught signal 15, un-registering and exiting. Jul 30 16:04:53 odysseus kernel: <ffffffff802729c6>{__link_path_walk+2435} <ffffffff80272f40>{link_path_walk+89} Jul 30 16:04:53 odysseus kernel: <ffffffff8049c0d2>{__sched_text_start+290} <ffffffff80273396>{do_path_lookup+614} Jul 30 16:04:53 odysseus kernel: <ffffffff80271e47>{getname+347} <ffffffff80273bd6>{__user_walk_fd+55} Jul 30 16:04:53 odysseus kernel: <ffffffff8026cba7>{vfs_lstat_fd+21} <ffffffff8049c0d2>{__sched_text_start+290} Jul 30 16:04:53 odysseus kernel: <ffffffff8026cd92>{sys_newlstat+25} <ffffffff80265023>{vfs_write+283} Jul 30 16:04:53 odysseus kernel: <ffffffff8026554c>{sys_write+69} <ffffffff80209826>{system_call+126} Jul 30 16:04:53 odysseus kernel: 0x0: 00 00 00 00 00 00 00 00 00 00 00 00 00 0000 00
This is 2.6.17.7 dual
x86_64(Fedora Core 5). It has been unfortunately
running 2.6.17.1 for some time.
Quoted 19 lines
I will probably have to recreate the volume and restore its
contents from backups. Or is there any better solution?Thanks,
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E | | http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |Quoted 2 lines
I will never go to meetings again because I think face to face meetings <
are the biggest waste of time you can ever have. --Linus Torvalds <-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
If you unmount, xfs_repair -n /dev/md5, what does it show currently?
Re: FAQ updated (was Re: XFS breakage...)
In reply to Jan KasprzakQuoted 7 lines · Jan Kasprzak
Nathan Scott wrote:
: I've captured the state of this issue here, with options and ways : to correct the problem:
: http://oss.sgi.com/projects/xfs/faq.html#dir2
:
: Hope this helps.
Quoted 5 lines
I have been hit with this bug as well - I tried to clear the
two corrupted directory inodes withxfs_db(as the FAQ entry says), then ranxfs_repair(lots of files ended up in lost+found), but apparently the volume is still not OK - when I tried to use it (this volume is a public FTP archive), I got the following traces:
There is now a fixed version of xfs_repair available - its in xfsprogs-2.8.10, source is on oss.sgi.com in the XFS ftp area. A number of people have reported success with Barry's earlier patch, noone's reported anything bad, so 2.8.10 is out now with the fix merged.
cheers.
Signature · 1 line
Nathan
Hi Nathan,
Quoted 3 lines · Nathan Scott
I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
That probably the cause of my problem. Thanks for the info!
BTW: I think there was nothing important on the broken filesystems, but I'd like to keep what's still there anyway just in case... How would you suggest should I copy that data? I fear, just mounting and using cp might break and shutdown the FS again, would xfsdump be more appropriate?
Thanks for XFS, I am using it for years in production servers!
Greetings
Torsten
Quoted 5 lines · Torsten Landschoff
Quoted 3 lines · Nathan Scott
I suspect you had some residual directory corruption from using the 2.6.17 XFS (which is known to have a lurking dir2 corruption issue, fixed in the latest -stable point release).
Quoted 7 lines
That probably the cause of my problem. Thanks for the info!
BTW: I think there was nothing important on the broken filesystems, but I'd like to keep what's still there anyway just in case... How would you suggest should I copy that data? I fear, just mounting and using cp might break and shutdown the FS again, would xfsdump be more appropriate?
Yeah, xfsdumps not a bad idea, the interfaces it uses may well be able to avoid the cases that trigger shutdown. Otherwise it is a case of identifying the problem directory inode (the inum is reported in the shutdown trace) and avoiding that path when cp'ing - you can match inum to path via xfs_ncheck.
Quoted 1 line
Thanks for XFS, I am using it for years in production servers!
Thanks for the kind words, they're much appreciated at times like these. :-]
cheers.
Signature · 1 line
Nathan
Hi folks,
Quoted 3 lines · Nathan Scott
2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
well, at least for me, corruption/errors *started* with 2.6.18-rc1:
http://oss.sgi.com/archives/xfs/2006-07/msg00151.html
I downgraded to 2.6.17.5 and the errors stopped. Now I've upgraded to 2.6.18-rc2 and see the same errors:
xfs_da_do_buf: bno 16777216
dir: inode 24472381
Filesystem "md0": XFS internal error xfs_da_do_buf(1) at line 1992 of file fs/xfs/xfs_da_btree.c. Caller 0xc0219230 Filesystem "md0": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c. Caller 0xc024d717
Please see the whole error/.config/logs here:
http://nerdbynature.de/bits/2.6.18-rc2/
Thanks,
Christian.
Signature · 3 lines
BOFH excuse #38: secretary plugged hairdryer into UPS
Quoted 5 lines · Christian Kujau
Quoted 3 lines · Nathan Scott
2.6.18-rc1 should be fine (contains the corruption fix). Did you mkfs and restore? Or at least get a full repair run? If you did, and you still see issues in .18-rc1, please let me know asap.
Quoted 7 lines
well, at least for me, corruption/errors *started* with 2.6.18-rc1:
...
I downgraded to 2.6.17.5 and the errors stopped. Now I've upgraded to 2.6.18-rc2 and see the same errors:
xfs_da_do_buf: bno 16777216
dir: inode 24472381
This is an ondisk corruption - downgrading the kernel will not resolve it. The problem must be triggered by a combination of operations on a directory; I'm certain that if you access inode 24472381 on your filesystem on 2.6.17, that it'll shutdown your filesystem too. See the FAQ entry for a description on how to translate inums to paths, and also the repair -n step to detect any corruption ondisk.
cheers.
Signature · 1 line
Nathan
Now that 2.6.17.7 is finally there, I am going to eyeball my XFS filesystems. Does xfsprogs 2.7.11 suffice?
Please update the XFS FAQ about this problem (and the solution, for others that did not read LKML), thanks!
Jan Engelhardt
Quoted 2 lines · Jan Engelhardt
Now that 2.6.17.7 is finally there, I am going to eyeball my XFS filesystems. Does xfsprogs 2.7.11 suffice?
Sure, except for the repair issue which Barry is working on (see the FAQ).
Quoted 2 lines
Please update the XFS FAQ about this problem (and the solution, for others that did not read LKML), thanks!
That was done last week. You shoulda read it before sending this, and saved me some time replying. ;)
cheers.
Signature · 1 line
Nathan
Hello again,
Quoted 3 lines · Nathan Scott
filesystem too. See the FAQ entry for a description on how to translate inums to paths, and also the repair -n step to detect any corruption ondisk.
I had two xfs filesystems and I first noticed that /data/Scratch was befallen from this bug. I did not care much about this (hence the name :)) and I wanted to postpone the xfs_db surgery.
Unfortunately I forgot that "/" was also an XFS and it crashed yesterday. remounting ro helped a bit (so no process attempted to write on it. however, cp'ing from the ro-mounted xfs sometimes hung, unkillable), I setup a mini-root somewhere else and followed the
instructions in the FAQ. It did not go too well, lots of stuff was moved to lost+found, but every subsequent xfs_repair run found more and more errors. I decided to mkfs the partition and make use of my backups. my other "scratch" partition is still XFS but mounted ro and I'll try the xfsprogs fixes Nathan published on this one.
Oh, and I dd'ed the corrupt xfs-filesystem to a file, so I can play around with this one as well.
If anyone is interested, here are the typescripts from the horrible xfs_repair runs: http://nerdbynature.de/bits/2.6.18-rc2/log/
cheers,
Christian.
Signature · 3 lines
BOFH excuse #21: POSIX compliance problem
Quoted 7 lines · Christian Kujau
I had two xfs filesystems and I first noticed that /data/Scratch was befallen from this bug. I did not care much about this (hence the name :)) and I wanted to postpone the
xfs_dbsurgery.
...
found more and more errors. I decided to mkfs the partition and make use of my backups. my other "scratch" partition is still XFS but mounted ro and I'll try the xfsprogs fixes Nathan published on this one.
Barry sent an xfs_repair patch to resolve this issue to the [email protected] list yesterday; please give that a go and let us know how it fares.
cheers.
Signature · 1 line
Nathan
* Nathan Scott <[email protected]>:
Quoted 2 lines
Barry sent an
xfs_repairpatch to resolve this issue to the [email protected] list yesterday; please give that a go and let us know how it fares.
Just to let you know, I did a cvs checkout of xfs-cmds as described on http://oss.sgi.com/projects/xfs/source.html
Then I saved the patch from
http://oss.sgi.com/archives/xfs/2006-07/msg00374.html using the "Original" link on hat page.
I build a xfs_Repair binary using that, transferred it onto an old KLAX boot cd I had and repaired the XFS root on my laptop.
I got 5000 files in lost and found, mostly the whole manpages from my system. Had to reinstall a few packages to restore lost binaries, but that's all.
When will that horrible bug be fixed in 2.6.x?
Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrums) [email protected] Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to [email protected]
Quoted 12 lines · Ralf Hildebrandt
* Nathan Scott <[email protected]>:
Quoted 2 lines
Barry sent an
xfs_repairpatch to resolve this issue to the [email protected] list yesterday; please give that a go and let us know how it fares.Just to let you know, I did a cvs checkout of xfs-cmds as described on http://oss.sgi.com/projects/xfs/source.html
Then I saved the patch from
http://oss.sgi.com/archives/xfs/2006-07/msg00374.html using the "Original" link on hat page.
Quoted 9 lines
I build a
xfs_Repairbinary using that, transferred it onto an old KLAX boot cd I had and repaired the XFS root on my laptop.I got 5000 files in lost and found, mostly the whole manpages from my system. Had to reinstall a few packages to restore lost binaries, but that's all.
When will that horrible bug be fixed in 2.6.x?
The bug is fixed in 2.6.17.7.
David
Quoted 5 lines · Torsten Landschoff
Hi friends,
I upgraded to 2.6.18-rc1 on sunday, with the following results (taken from my
/var/log/kern.log), which ultimately led me to reinstall my system:
[snip]
Quoted 12 lines
That problem occured during a dist-upgrade, dm-6 is my /usr partition. Funny enough this happened a few months after finally replaced my ancient disk with a RAID1 array to make sure I do not lose data ;)
In any case it seems like the XFS driver in 2.6.18-rc1 is decently broken. After booting into 2.6.17 again, I could use /usr again but random files contain null bytes, firefox segfaults instead of starting up and a number of programs fail in mysterious ways. I tried to recover using
xfs_repairbut I feel that my partition is thorougly borked. Of course no data was lost due to backups but still I'd like this bug to be fixed ;-)
Quoted 2 lines
If more information from my logs is required, I can make it available (and any part of the partition if required).
That looks like the death knell of my /, which succumbed on Friday as a result (I believe) of the corruption bug that was in 2.6.16/17. Ironically enough, I also saw the problem during an aptitude upgrade.
Also see this thread:
http://marc.theaimsgroup.com/?l=linux-kernel&m=115070320401919&w=2
Signature · 3 lines
Kevin 'radsaq' Radloff [email protected] http://thesaq.com/