Corrupted file system

DISCUSSION Diane Trout 4 participants 9 messages Sep 23, 2014 – Sep 25, 2014

Share & tools

Diane Trout

author

2014-09-23 22:03 UTC #0 lore raw

Hi,

I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a metadata dump but is there a point to making it available?

It does crash repeatedly at the same place

(I'm not subscribed to the list, so could you reply directly as well?)

disconnected inode 15276, moving to lost+found disconnected inode 15277, moving to lost+found disconnected inode 15278, moving to lost+found disconnected inode 15279, moving to lost+found disconnected inode 15280, moving to lost+found disconnected inode 15281, moving to lost+found disconnected inode 15282, moving to lost+found disconnected inode 15283, moving to lost+found disconnected inode 15284, moving to lost+found disconnected inode 15286, moving to lost+found disconnected inode 15287, moving to lost+found disconnected inode 15288, moving to lost+found disconnected inode 15289, moving to lost+found disconnected inode 15290, moving to lost+found disconnected inode 15291, moving to lost+found disconnected inode 15292, moving to lost+found disconnected inode 15293, moving to lost+found disconnected inode 15294, moving to lost+found disconnected inode 15295, moving to lost+found disconnected inode 15360, moving to lost+found
corrupt dinode 15360, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].
cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0)

fatal error -- 117 - couldn't iget disconnected inode

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Sean Caron

2014-09-23 22:12 UTC #1 lore raw

Hi Diane,

Probably best to reformat and restore from a backup at this point. I'm notorious for my views on xfs_repair but most folks here will agree that in the case where there has been a major underlying array failure, there is not much it can do to help. Better to just mount ro,noreplaylog and get what you can in these sorts of scenarios, IMO.

"Did you try the latest copy of xfs_repair"? Sometimes it will get further or not crash, but it likely will still maul whatever's left of your filesystem.

Best of luck,

Sean

Quoted 36 lines · Diane Trout

Hi,
I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a
metadata dump but is there a point to making it available?
It does crash repeatedly at the same place
(I'm not subscribed to the list, so could you reply directly as well?)
disconnected inode 15276, moving to lost+found disconnected inode 15277, moving to lost+found disconnected inode 15278, moving to lost+found disconnected inode 15279, moving to lost+found disconnected inode 15280, moving to lost+found disconnected inode 15281, moving to lost+found disconnected inode 15282, moving to lost+found disconnected inode 15283, moving to lost+found disconnected inode 15284, moving to lost+found disconnected inode 15286, moving to lost+found disconnected inode 15287, moving to lost+found disconnected inode 15288, moving to lost+found disconnected inode 15289, moving to lost+found disconnected inode 15290, moving to lost+found disconnected inode 15291, moving to lost+found disconnected inode 15292, moving to lost+found disconnected inode 15293, moving to lost+found disconnected inode 15294, moving to lost+found disconnected inode 15295, moving to lost+found disconnected inode 15360, moving to lost+found
corrupt dinode 15360, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].
cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0)

Quoted 8 lines

fatal error -- 117 - couldn't iget disconnected inode
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Diane Trout

author

2014-09-23 22:29 UTC #2 lore raw

Hello,

Yes, I had my doubts about that there's anything reasonable one could do after part of the file system was randomized. I'll trying updating to xfsprogs 3.2.1 and see if that does any better.

Thank you for the advice.

Diane

Quoted 12 lines · On Tuesday, September 23, 2014 18:12:22 Sean Caron wrote:

Hi Diane,
Probably best to reformat and restore from a backup at this point. I'm notorious for my views on xfs_repair but most folks here will agree that in the case where there has been a major underlying array failure, there is not much it can do to help. Better to just mount ro,noreplaylog and get what you can in these sorts of scenarios, IMO.
"Did you try the latest copy of xfs_repair"? Sometimes it will get further or not crash, but it likely will still maul whatever's left of your filesystem.

Quoted 49 lines

Best of luck,
Sean
Quoted 44 lines · Diane Trout
Hi,
I had a raid failure at work that ended up corrupting an xfs filesystem the
tail of the xfs_repair command looks like the below. I was able to generate a
metadata dump but is there a point to making it available?
It does crash repeatedly at the same place
(I'm not subscribed to the list, so could you reply directly as well?)
disconnected inode 15276, moving to lost+found disconnected inode 15277, moving to lost+found disconnected inode 15278, moving to lost+found disconnected inode 15279, moving to lost+found disconnected inode 15280, moving to lost+found disconnected inode 15281, moving to lost+found disconnected inode 15282, moving to lost+found disconnected inode 15283, moving to lost+found disconnected inode 15284, moving to lost+found disconnected inode 15286, moving to lost+found disconnected inode 15287, moving to lost+found disconnected inode 15288, moving to lost+found disconnected inode 15289, moving to lost+found disconnected inode 15290, moving to lost+found disconnected inode 15291, moving to lost+found disconnected inode 15292, moving to lost+found disconnected inode 15293, moving to lost+found disconnected inode 15294, moving to lost+found disconnected inode 15295, moving to lost+found disconnected inode 15360, moving to lost+found
corrupt dinode 15360, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].
cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0)
fatal error -- 117 - couldn't iget disconnected inode
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Eric Sandeen

2014-09-23 23:12 UTC #3 lore raw

Quoted 5 lines · Diane Trout

Hello,
Yes, I had my doubts about that there's anything reasonable one could do after part of the file system was randomized. I'll trying updating to xfsprogs 3.2.1 and see if that does any better.

Yes - you didn't mention which version you used, but I'd definitely try something current. The bug rings a bell, maybe fixed now.

To be safe, you can always make an xfs_metadump image (use -o to not obfuscate filenames for convenience), xfs_mdrestore that, and point repair at that fs image. Then you'll see what it *would* do before you *do* do. ;)

But yeah, depending on the damage done to the storage, you may be in trouble that repair can't help with. Depends on the failure, though.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Emmanuel Florac

2014-09-24 10:06 UTC #4 lore raw

In reply to Diane Trout

Le Tue, 23 Sep 2014 15:02:57 -0700
Diane Trout <[email protected]> écrivait:

Quoted 6 lines

I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a metadata dump but is there a point to making it available?
It does crash repeatedly at the same place

Did you try the very latest xfs_repair?

Signature · 11 lines

------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Diane Trout

author

2014-09-24 20:38 UTC #5 lore raw

Quoted 11 lines · On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:

Le Tue, 23 Sep 2014 15:02:57 -0700
Diane Trout <[email protected]> écrivait:
Quoted 6 lines
I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a metadata dump but is there a point to making it available?
It does crash repeatedly at the same place
Did you try the very latest xfs_repair?

I grabbed the one out of debian unstable (3.2.1), verified that it was your latest release and used that. I didn't try building the version in git.

It did get much further than version 3.1.7+b1 (debian stable)

Diane

Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in ino 2051.
bad hash table for directory inode 256 (no data entry): rebuilding rebuilding directory inode 256
7f70c79c8700: Badness in key lookup (length)
bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) Metadata corruption detected at block 0x4000070/0x1000 xfs_da_do_buf(2): XFS_CORRUPTION_ERROR

fatal error -- can't read block 8388608 for directory inode 259, error 117

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Eric Sandeen

2014-09-24 20:57 UTC #6 lore raw

If you want to make a metadata image, compress it, and send it my way I'd be glad to look at it - I've been doing some repair work lately.

-Eric

Quoted 33 lines · Diane Trout

Quoted 11 lines · On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:
Le Tue, 23 Sep 2014 15:02:57 -0700
Diane Trout <[email protected]> écrivait:
Quoted 6 lines
I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a metadata dump but is there a point to making it available?
It does crash repeatedly at the same place
Did you try the very latest xfs_repair?
I grabbed the one out of debian unstable (3.2.1), verified that it was your latest release and used that. I didn't try building the version in git.
It did get much further than version 3.1.7+b1 (debian stable)
Diane
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in ino 2051.
bad hash table for directory inode 256 (no data entry): rebuilding rebuilding directory inode 256
7f70c79c8700: Badness in key lookup (length)
bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) Metadata corruption detected at block 0x4000070/0x1000 xfs_da_do_buf(2): XFS_CORRUPTION_ERROR

Quoted 8 lines

fatal error -- can't read block 8388608 for directory inode 259, error 117
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Diane Trout

author

2014-09-25 00:25 UTC #7 lore raw

Huh.

I'd managed to make a metadump using the 3.1.7 version of the tools. Then I tried to run xfs_repair on the partition, and now xfs_metadump is segfaulting on the partition.

Would the older metadata dump be useful?

I'd need to rebuild xfsprogs before the stack trace would be useful... debian likes to strip the symbols and they didn't make a -dbg package.

Diane

Quoted 48 lines · On Wednesday, September 24, 2014 15:57:53 Eric Sandeen wrote:

If you want to make a metadata image, compress it, and send it my way I'd be glad to look at it - I've been doing some repair work lately.
-Eric
Quoted 42 lines · Diane Trout
Quoted 11 lines · On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:
Le Tue, 23 Sep 2014 15:02:57 -0700
Diane Trout <[email protected]> écrivait:
Quoted 6 lines
I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a metadata dump but is there a point to making it available?
It does crash repeatedly at the same place
Did you try the very latest xfs_repair?
I grabbed the one out of debian unstable (3.2.1), verified that it was your
latest release and used that. I didn't try building the version in git.
It did get much further than version 3.1.7+b1 (debian stable)
Diane
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in ino 2051.
bad hash table for directory inode 256 (no data entry): rebuilding rebuilding directory inode 256
7f70c79c8700: Badness in key lookup (length)
bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) Metadata corruption detected at block 0x4000070/0x1000 xfs_da_do_buf(2): XFS_CORRUPTION_ERROR
fatal error -- can't read block 8388608 for directory inode 259, error 117
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Eric Sandeen

2014-09-25 00:50 UTC #8 lore raw

Quoted 7 lines · Diane Trout

Huh.
I'd managed to make a metadump using the 3.1.7 version of the tools. Then I tried to run xfs_repair on the partition, and now xfs_metadump is segfaulting on the partition.
Would the older metadata dump be useful?

sure. Compress it & send me a private note with it attached (or with a location if it's too big to email)

-Eric

Quoted 55 lines

I'd need to rebuild xfsprogs before the stack trace would be useful... debian likes to strip the symbols and they didn't make a -dbg package.
Diane
Quoted 48 lines · On Wednesday, September 24, 2014 15:57:53 Eric Sandeen wrote:
If you want to make a metadata image, compress it, and send it my way I'd be glad to look at it - I've been doing some repair work lately.
-Eric
Quoted 42 lines · Diane Trout
Quoted 11 lines · On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:
Le Tue, 23 Sep 2014 15:02:57 -0700
Diane Trout <[email protected]> écrivait:
Quoted 6 lines
I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a metadata dump but is there a point to making it available?
It does crash repeatedly at the same place
Did you try the very latest xfs_repair?
I grabbed the one out of debian unstable (3.2.1), verified that it was your
latest release and used that. I didn't try building the version in git.
It did get much further than version 3.1.7+b1 (debian stable)
Diane
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in ino 2051.
bad hash table for directory inode 256 (no data entry): rebuilding rebuilding directory inode 256
7f70c79c8700: Badness in key lookup (length)
bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) Metadata corruption detected at block 0x4000070/0x1000 xfs_da_do_buf(2): XFS_CORRUPTION_ERROR
fatal error -- can't read block 8388608 for directory inode 259, error 117
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs