Skip to contentSkip to conversation

xfs_repair fails: mismatched UUID?

DISCUSSION Link Dupont 2 participants 6 messages Aug 17, 2017

lore
Share & tools
lore
#0 lore raw

Hello,

I have a JMicron USB RAID enclosure that is exhibiting read failures.

Quoted 11 lines
[59996.137762] sd 5:0:0:0: [sdd] tag#0 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[59996.137786] sd 5:0:0:0: [sdd] tag#0 Sense Key : Aborted Command [current] 
[59996.137792] sd 5:0:0:0: [sdd] tag#0 Add. Sense: No additional sense
information
[59996.137798] sd 5:0:0:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 a6 42
1d d8 00 00 00 08 00 00
[59996.137804] blk_update_request: I/O error, dev sdd, sector 2789350872
[59996.137816] Buffer I/O error on dev sdd, logical block 348668859, async

page read

I ran ddrescue to block copy the device to another external USB drive. This appears to have been successful (though not without errors). The new drive shows an identical UUID to the original.

Quoted 4 lines
# xfs_admin -u /dev/sdc1
UUID = 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
# xfs_admin -u /dev/sdd1
UUID = 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e

I'm still unable to mount the file-system on sdc1.

Quoted 2 lines
# mount /dev/sdc1 mnt
mount: /root/mnt: mount(2) system call failed: Structure needs cleaning.

I ran xfs_repair.

Quoted 6 lines
# xfs_repair /dev/sdc1
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
* ERROR: mismatched uuid in log
Quoted 9 lines
*            SB : 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
*            log: 00000001-0038-8e56-0000-000100388e2e
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

I tried xfs_repair with -L to destroy the log. The file-system does clean, but when it mounts, there's nothing present; no files or directories. Not even an lost+found.

I /can/ successfully mount the drive with the norecovery option, though directory and file names are mangled (unsurprisingly). As I poke around on the filesystem, I get certain block read errors:

Quoted 9 lines

37473-[ 4120.910844] XFS (sdc1): metadata I/O error: block 0x115a33400 ("xfs_trans_read_buf_map") error 117 numblks 32
37586:[ 4120.910850] XFS (sdc1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
37671-[ 4120.911231] XFS (sdc1): Metadata corruption detected at xfs_inode_buf_verify+0x73/0xf0 [xfs], xfs_inode block 0x115a33400 37796-[ 4120.911234] XFS (sdc1): Unmount and run xfs_repair 37850-[ 4120.911235] XFS (sdc1): First 64 bytes of corrupted metadata buffer:

Many files are corrupt (I get the "Structure needs cleaning" error when I attempt to cat the files). I found a past thread[1] that provided some useful tips. I looked at sb 0 and sb 1 and compared the two. They were identical except for the versionnum. I tried replicating the sb 1 versionnum value to sb 0, but the log still won't replay.

All of this has been performed on my ddrescue clone (not the original device). And with kernel 4.12.5 and xfsprogs 4.10.0.

I understand corrupt blocks and the data lost incurred from that. I can poke around and try to recover what I can when mounted in 'norecovery'. What I am perplexed by is the total loss of file-system info when xfs_repair -L is run; not even a lost+found?

Thanks!

Link

1: https://www.spinics.net/lists/xfs/msg38754.html

#1 lore raw
Quoted 25 lines · Link Dupont

Hello,

I have a JMicron USB RAID enclosure that is exhibiting read failures.

Quoted 11 lines
[59996.137762] sd 5:0:0:0: [sdd] tag#0 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[59996.137786] sd 5:0:0:0: [sdd] tag#0 Sense Key : Aborted Command [current] 
[59996.137792] sd 5:0:0:0: [sdd] tag#0 Add. Sense: No additional sense
information
[59996.137798] sd 5:0:0:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 a6 42
1d d8 00 00 00 08 00 00
[59996.137804] blk_update_request: I/O error, dev sdd, sector 2789350872
[59996.137816] Buffer I/O error on dev sdd, logical block 348668859, async

page read

I ran ddrescue to block copy the device to another external USB drive. This appears to have been successful (though not without errors). The new drive shows an identical UUID to the original.

Quoted 4 lines
# xfs_admin -u /dev/sdc1
UUID = 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
# xfs_admin -u /dev/sdd1
UUID = 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
Quoted 15 lines

I'm still unable to mount the file-system on sdc1.

Quoted 2 lines
# mount /dev/sdc1 mnt
mount: /root/mnt: mount(2) system call failed: Structure needs cleaning.

I ran xfs_repair.

Quoted 8 lines
# xfs_repair /dev/sdc1
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
* ERROR: mismatched uuid in log
*            SB : 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
*            log: 00000001-0038-8e56-0000-000100388e2e

Thjat looks suspiciously /not/ like a UUID.

Which makes me think that the "log" was not the log, and your enclosure scrambled data around somehow.

Forcing that mess through repair may have yielded more harm than good. So, good thing it was on the clone.

Did the enclosure contain multiple disks? Did they possibly get re-ordered or otherwise scrambled?

...

Quoted 1 line

All of this has been performed on my ddrescue clone (not the original device).

<cue applause!> :)

Quoted 6 lines

And with kernel 4.12.5 and xfsprogs 4.10.0.

I understand corrupt blocks and the data lost incurred from that. I can poke around and try to recover what I can when mounted in 'norecovery'. What I am perplexed by is the total loss of file-system info when xfs_repair -L is run; not even a lost+found?

Collecting and sharing the xfs_repair output might yield some insight as well, but first I'd look at what you're copying from; just how badly damaged is it? xfs_repair restores filesystem consistency but it is not a data recovery or storage repair tool. ;)

-Eric

#2 lore raw
Quoted 44 lines · Eric Sandeen
Quoted 25 lines · Link Dupont

Hello,

I have a JMicron USB RAID enclosure that is exhibiting read failures.

Quoted 11 lines
[59996.137762] sd 5:0:0:0: [sdd] tag#0 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[59996.137786] sd 5:0:0:0: [sdd] tag#0 Sense Key : Aborted Command [current] 
[59996.137792] sd 5:0:0:0: [sdd] tag#0 Add. Sense: No additional sense
information
[59996.137798] sd 5:0:0:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 a6 42
1d d8 00 00 00 08 00 00
[59996.137804] blk_update_request: I/O error, dev sdd, sector 2789350872
[59996.137816] Buffer I/O error on dev sdd, logical block 348668859, async

page read

I ran ddrescue to block copy the device to another external USB drive. This appears to have been successful (though not without errors). The new drive shows an identical UUID to the original.

Quoted 4 lines
# xfs_admin -u /dev/sdc1
UUID = 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
# xfs_admin -u /dev/sdd1
UUID = 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
Quoted 15 lines

I'm still unable to mount the file-system on sdc1.

Quoted 2 lines
# mount /dev/sdc1 mnt
mount: /root/mnt: mount(2) system call failed: Structure needs cleaning.

I ran xfs_repair.

Quoted 8 lines
# xfs_repair /dev/sdc1
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
* ERROR: mismatched uuid in log
*            SB : 7b2fc6f7-0f6b-40fb-b264-75d25e9d0d1e
*            log: 00000001-0038-8e56-0000-000100388e2e

Thjat looks suspiciously /not/ like a UUID.

Quoted 8 lines

Which makes me think that the "log" was not the log, and your enclosure scrambled data around somehow.

Forcing that mess through repair may have yielded more harm than good. So, good thing it was on the clone.

Did the enclosure contain multiple disks? Did they possibly get re-ordered or otherwise scrambled?

Oh right, you did say RAID enclosure.

My money is on the enclosure no longer presenting you with sectors in the original order, or in an otherwise severely damaged form, for whatever reason.

IOWS: I think this is a storage problem that xfs_repair can't solve.

-Eric

#3 lore raw
Quoted 5 lines · Eric Sandeen

Oh right, you did say RAID enclosure.

My money is on the enclosure no longer presenting you with sectors in the original order, or in an otherwise severely damaged form, for whatever reason.

The problem *could* be from the enclosure itself; I don't have a second one to test. I'm fairly sure the cause is a failing drive (I get a red LED from one of the two status LEDs on the front of the enclosure).

Quoted 2 lines

xfs_repair restores filesystem consistency but it is not a data recovery or storage repair tool. ;)

I get that. But since the files are there, it seemed like the file-system could be repaired. I'll dig up the output.

Quoted 4 lines
Quoted 1 line

All of this has been performed on my ddrescue clone (not the original

device).

<cue applause!> :)

I don't always directly edit properties of my filesystems, but when I do, I do it on a clone. :)

#4 lore raw
Quoted 11 lines · Link Dupont
Quoted 5 lines · Eric Sandeen

Oh right, you did say RAID enclosure.

My money is on the enclosure no longer presenting you with sectors in the original order, or in an otherwise severely damaged form, for whatever reason.

The problem *could* be from the enclosure itself; I don't have a second one to test. I'm fairly sure the cause is a failing drive (I get a red LED from one of the two status LEDs on the front of the enclosure).

Quoted 5 lines
Quoted 2 lines

xfs_repair restores filesystem consistency but it is not a data recovery or storage repair tool. ;)

I get that. But since the files are there, it seemed like the file-system could be repaired. I'll dig up the output.

My worry is that the "log uuid" was neither zeros nor the proper UUID; it was something else. So it wasn't bad, unreadable sectors which got filled in with zeros by ddrescue, it was apparently some random data coughed up by the enclosure for some unknown reason.

If it did that to the log, it may well have done that to many other parts of the filesystem as well.

-Eric

#5 lore raw
Quoted 29 lines · Eric Sandeen
Quoted 15 lines · Link Dupont
Quoted 6 lines · Eric Sandeen

Oh right, you did say RAID enclosure.

My money is on the enclosure no longer presenting you with sectors in the original order, or in an otherwise severely damaged form, for whatever reason.

The problem *could* be from the enclosure itself; I don't have a second one to test. I'm fairly sure the cause is a failing drive (I get a red LED from one of the two status LEDs on the front of the enclosure).

Quoted 5 lines
Quoted 2 lines

xfs_repair restores filesystem consistency but it is not a data recovery or storage repair tool. ;)

I get that. But since the files are there, it seemed like the file-system could be repaired. I'll dig up the output.

My worry is that the "log uuid" was neither zeros nor the proper UUID; it was something else. So it wasn't bad, unreadable sectors which got filled in with zeros by ddrescue, it was apparently some random data coughed up by the enclosure for some unknown reason.

Quoted 5 lines

If it did that to the log, it may well have done that to many other parts of the filesystem as well.

-Eric

This almost makes it seem more like the enclosure controller is at fault here. I'll chase down another one.