Skip to contentSkip to conversation

Need help to recover root filesystem after a power supply issue

DISCUSSION Andrey Zhunev 3 participants 7 messages Jul 10, 2019

lore
Share & tools
lore

Andrey Zhunev

author
#0 lore raw

Hello All,

I am struggling to recover my system after a PSU failure, and I was suggested to ask here for support.

One of the hard drives throws some read errors, and that happen to be my root drive...
My system is CentOS 7, and the root partition is a part of LVM.

[root@mgmt ~]# lvscan

  ACTIVE            '/dev/centos/root' [<98.83 GiB] inherit
  ACTIVE            '/dev/centos/home' [<638.31 GiB] inherit
  ACTIVE            '/dev/centos/swap' [<7.52 GiB] inherit

[root@mgmt ~]#

[root@tftp ~]# file -s /dev/centos/root /dev/centos/root: symbolic link to `../dm-3'
[root@tftp ~]# file -s /dev/centos/home /dev/centos/home: symbolic link to `../dm-4'
[root@tftp ~]# file -s /dev/dm-3
/dev/dm-3: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs) [root@tftp ~]# file -s /dev/dm-4
/dev/dm-4: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)

[root@tftp ~]# xfs_repair /dev/centos/root
Phase 1 - find and verify superblock...
superblock read failed, offset 53057945600, size 131072, ag 2, rval -1

fatal error -- Input/output error
[root@tftp ~]#

smartctl shows some pending sectors on /dev/sda, and no reallocated sectors (yet?).

Can someone please give me a hand to bring root partition back to life (ideally)? Or, at least, recover a couple of critical configuration files...

---
Best regards,
Andrey

#1 lore raw
Quoted 25 lines · Andrey Zhunev

Hello All,

I am struggling to recover my system after a PSU failure, and I was suggested to ask here for support.

One of the hard drives throws some read errors, and that happen to be my root drive...
My system is CentOS 7, and the root partition is a part of LVM.

[root@mgmt ~]# lvscan

  ACTIVE            '/dev/centos/root' [<98.83 GiB] inherit
  ACTIVE            '/dev/centos/home' [<638.31 GiB] inherit
  ACTIVE            '/dev/centos/swap' [<7.52 GiB] inherit

[root@mgmt ~]#

[root@tftp ~]# file -s /dev/centos/root /dev/centos/root: symbolic link to `../dm-3'
[root@tftp ~]# file -s /dev/centos/home /dev/centos/home: symbolic link to `../dm-4'
[root@tftp ~]# file -s /dev/dm-3
/dev/dm-3: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs) [root@tftp ~]# file -s /dev/dm-4
/dev/dm-4: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)

Quoted 5 lines

[root@tftp ~]# xfs_repair /dev/centos/root
Phase 1 - find and verify superblock...
superblock read failed, offset 53057945600, size 131072, ag 2, rval -1

fatal error -- Input/output error

look at dmesg, see what the kernel says about the read failure.

You might be able to use https://www.gnu.org/software/ddrescue/
to read as many sectors off the device into an image file as possible, and that image might be enough to work with for recovery. That would be my first approach:

1) use dd-rescue to create an image file of the device
2) make a copy of that image file
3) run xfs_repair -n on the copy to see what it would do
4) if that looks reasonable run xfs_repair on the copy
5) mount the copy and see what you get

But if your drive simply cannot be read at all, this is not a filesystem problem, it is a hardware problem. If this is critical data you may wish to hire a data recovery service.

-Eric

Quoted 15 lines

[root@tftp ~]#

smartctl shows some pending sectors on /dev/sda, and no reallocated sectors (yet?).

Can someone please give me a hand to bring root partition back to life (ideally)? Or, at least, recover a couple of critical configuration files...

---
Best regards,
Andrey

author
#2 lore raw
Quoted 45 lines · you
Quoted 25 lines · Andrey Zhunev

Hello All,

I am struggling to recover my system after a PSU failure, and I was suggested to ask here for support.

One of the hard drives throws some read errors, and that happen to be my root drive...
My system is CentOS 7, and the root partition is a part of LVM.

[root@mgmt ~]# lvscan

  ACTIVE            '/dev/centos/root' [<98.83 GiB] inherit
  ACTIVE            '/dev/centos/home' [<638.31 GiB] inherit
  ACTIVE            '/dev/centos/swap' [<7.52 GiB] inherit

[root@mgmt ~]#

[root@tftp ~]# file -s /dev/centos/root /dev/centos/root: symbolic link to `../dm-3'
[root@tftp ~]# file -s /dev/centos/home /dev/centos/home: symbolic link to `../dm-4'
[root@tftp ~]# file -s /dev/dm-3
/dev/dm-3: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs) [root@tftp ~]# file -s /dev/dm-4
/dev/dm-4: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)

Quoted 5 lines

[root@tftp ~]# xfs_repair /dev/centos/root
Phase 1 - find and verify superblock...
superblock read failed, offset 53057945600, size 131072, ag 2, rval -1

fatal error -- Input/output error

look at dmesg, see what the kernel says about the read failure.

You might be able to use https://www.gnu.org/software/ddrescue/
to read as many sectors off the device into an image file as possible, and that image might be enough to work with for recovery. That would be my first approach:

1) use dd-rescue to create an image file of the device
2) make a copy of that image file
3) run xfs_repair -n on the copy to see what it would do
4) if that looks reasonable run xfs_repair on the copy
5) mount the copy and see what you get

Quoted 5 lines

But if your drive simply cannot be read at all, this is not a filesystem problem, it is a hardware problem. If this is critical data you may wish to hire a data recovery service.

-Eric

Hi Eric,

Thanks for your message!
I already started to copy the failing drive with ddrescue. This is a large drive, so it takes some time to complete...

When I tried to run xfs_repair on the original (failing) drive, the xfs_repair was unable to read the superblock and then just quitted with an 'io error'.
Do you think it can behave differently on a copied image ?

I will definitely give it a try once the ddrescue finishes.

P.S. The data on this drive is not THAT critical to hire a professional data recovery service. Still, there are some files I would really like to restore (mostly settings and configuration files - nothing large, but important)... This will save me weeks to reconfigure and get the system back to its original state... Backups, always make backups... yeah, I know... :(

---
Best regards,
Andrey

#3 lore raw
Quoted 56 lines · Andrey Zhunev
Quoted 45 lines · you
Quoted 25 lines · Andrey Zhunev

Hello All,

I am struggling to recover my system after a PSU failure, and I was suggested to ask here for support.

One of the hard drives throws some read errors, and that happen to be my root drive...
My system is CentOS 7, and the root partition is a part of LVM.

[root@mgmt ~]# lvscan

  ACTIVE            '/dev/centos/root' [<98.83 GiB] inherit
  ACTIVE            '/dev/centos/home' [<638.31 GiB] inherit
  ACTIVE            '/dev/centos/swap' [<7.52 GiB] inherit

[root@mgmt ~]#

[root@tftp ~]# file -s /dev/centos/root /dev/centos/root: symbolic link to `../dm-3'
[root@tftp ~]# file -s /dev/centos/home /dev/centos/home: symbolic link to `../dm-4'
[root@tftp ~]# file -s /dev/dm-3
/dev/dm-3: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs) [root@tftp ~]# file -s /dev/dm-4
/dev/dm-4: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)

Quoted 5 lines

[root@tftp ~]# xfs_repair /dev/centos/root
Phase 1 - find and verify superblock...
superblock read failed, offset 53057945600, size 131072, ag 2, rval -1

fatal error -- Input/output error

look at dmesg, see what the kernel says about the read failure.

You might be able to use https://www.gnu.org/software/ddrescue/
to read as many sectors off the device into an image file as possible, and that image might be enough to work with for recovery. That would be my first approach:

1) use dd-rescue to create an image file of the device
2) make a copy of that image file
3) run xfs_repair -n on the copy to see what it would do
4) if that looks reasonable run xfs_repair on the copy
5) mount the copy and see what you get

Quoted 5 lines

But if your drive simply cannot be read at all, this is not a filesystem problem, it is a hardware problem. If this is critical data you may wish to hire a data recovery service.

-Eric

Hi Eric,

Quoted 8 lines

Thanks for your message!
I already started to copy the failing drive with ddrescue. This is a large drive, so it takes some time to complete...

When I tried to run xfs_repair on the original (failing) drive, the xfs_repair was unable to read the superblock and then just quitted with an 'io error'.
Do you think it can behave differently on a copied image ?

As I said, look at dmesg to see what failed on the original drive read attempt.

ddrescue will fill unreadable sectors with 0, and then of course that can be read from the image file.

-Eric

Quoted 15 lines

I will definitely give it a try once the ddrescue finishes.

P.S. The data on this drive is not THAT critical to hire a professional data recovery service. Still, there are some files I would really like to restore (mostly settings and configuration files - nothing large, but important)... This will save me weeks to reconfigure and get the system back to its original state... Backups, always make backups... yeah, I know... :(

---
Best regards,
Andrey

author
#4 lore raw
Quoted 66 lines · you
Quoted 56 lines · Andrey Zhunev
Quoted 45 lines · you
Quoted 25 lines · Andrey Zhunev

Hello All,

I am struggling to recover my system after a PSU failure, and I was suggested to ask here for support.

One of the hard drives throws some read errors, and that happen to be my root drive...
My system is CentOS 7, and the root partition is a part of LVM.

[root@mgmt ~]# lvscan

  ACTIVE            '/dev/centos/root' [<98.83 GiB] inherit
  ACTIVE            '/dev/centos/home' [<638.31 GiB] inherit
  ACTIVE            '/dev/centos/swap' [<7.52 GiB] inherit

[root@mgmt ~]#

[root@tftp ~]# file -s /dev/centos/root /dev/centos/root: symbolic link to `../dm-3'
[root@tftp ~]# file -s /dev/centos/home /dev/centos/home: symbolic link to `../dm-4'
[root@tftp ~]# file -s /dev/dm-3
/dev/dm-3: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs) [root@tftp ~]# file -s /dev/dm-4
/dev/dm-4: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)

Quoted 5 lines

[root@tftp ~]# xfs_repair /dev/centos/root
Phase 1 - find and verify superblock...
superblock read failed, offset 53057945600, size 131072, ag 2, rval -1

fatal error -- Input/output error

look at dmesg, see what the kernel says about the read failure.

You might be able to use https://www.gnu.org/software/ddrescue/
to read as many sectors off the device into an image file as possible, and that image might be enough to work with for recovery. That would be my first approach:

1) use dd-rescue to create an image file of the device
2) make a copy of that image file
3) run xfs_repair -n on the copy to see what it would do
4) if that looks reasonable run xfs_repair on the copy
5) mount the copy and see what you get

Quoted 5 lines

But if your drive simply cannot be read at all, this is not a filesystem problem, it is a hardware problem. If this is critical data you may wish to hire a data recovery service.

-Eric

Hi Eric,

Quoted 8 lines

Thanks for your message!
I already started to copy the failing drive with ddrescue. This is a large drive, so it takes some time to complete...

When I tried to run xfs_repair on the original (failing) drive, the xfs_repair was unable to read the superblock and then just quitted with an 'io error'.
Do you think it can behave differently on a copied image ?

Quoted 5 lines

As I said, look at dmesg to see what failed on the original drive read attempt.

ddrescue will fill unreadable sectors with 0, and then of course that can be read from the image file.

Ooops, I forgot to paste the error message from dmesg. Here it is:

Jul 10 11:48:05 mgmt kernel: ata1.00: exception Emask 0x0 SAct 0x180000 SErr 0x0 action 0x0
Jul 10 11:48:05 mgmt kernel: ata1.00: irq_stat 0x40000008
Jul 10 11:48:05 mgmt kernel: ata1.00: failed command: READ FPDMA QUEUED
Jul 10 11:48:05 mgmt kernel: ata1.00: cmd 60/00:98:28:ac:3e/01:00:03:00:00/40 tag 19 ncq 131072 in#012         res 41/40:00:08:ad:3e/00:00:03:00:00/40 Emask 0x409 (media error) <F>
Jul 10 11:48:05 mgmt kernel: ata1.00: status: { DRDY ERR }
Jul 10 11:48:05 mgmt kernel: ata1.00: error: { UNC }
Jul 10 11:48:05 mgmt kernel: ata1.00: configured for UDMA/133
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 Sense Key : Medium Error [current] [descriptor]
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 CDB: Read(16) 88 00 00 00 00 00 03 3e ac 28 00 00 01 00 00 00
Jul 10 11:48:05 mgmt kernel: blk_update_request: I/O error, dev sda, sector 54439176
Jul 10 11:48:05 mgmt kernel: ata1: EH complete

There are several of these.
At the moment ddrescue reports 22 read errors (with 35% of the data copied to a new storage). If I remember correctly, the LVM with my root partition is at the end of the drive. This means more errors will likely come... :(

The way I interpret the dmesg message, that's just a read error. I'm not sure, but maybe a complete wipe of the drive will even overwrite / clear these unreadable sectors.
Well, that's something to be checked after the copy process finishes.

---
Best regards,
Andrey

#5 lore raw
Quoted 1 line · Andrey Zhunev

Wednesday, July 10, 2019, 5:23:41 PM, you wrote:

...

Quoted 30 lines
Quoted 5 lines

As I said, look at dmesg to see what failed on the original drive read attempt.

ddrescue will fill unreadable sectors with 0, and then of course that can be read from the image file.

Ooops, I forgot to paste the error message from dmesg. Here it is:

Jul 10 11:48:05 mgmt kernel: ata1.00: exception Emask 0x0 SAct 0x180000 SErr 0x0 action 0x0
Jul 10 11:48:05 mgmt kernel: ata1.00: irq_stat 0x40000008
Jul 10 11:48:05 mgmt kernel: ata1.00: failed command: READ FPDMA QUEUED
Jul 10 11:48:05 mgmt kernel: ata1.00: cmd 60/00:98:28:ac:3e/01:00:03:00:00/40 tag 19 ncq 131072 in#012         res 41/40:00:08:ad:3e/00:00:03:00:00/40 Emask 0x409 (media error) <F>
Jul 10 11:48:05 mgmt kernel: ata1.00: status: { DRDY ERR }
Jul 10 11:48:05 mgmt kernel: ata1.00: error: { UNC }
Jul 10 11:48:05 mgmt kernel: ata1.00: configured for UDMA/133
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 Sense Key : Medium Error [current] [descriptor]
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 CDB: Read(16) 88 00 00 00 00 00 03 3e ac 28 00 00 01 00 00 00
Jul 10 11:48:05 mgmt kernel: blk_update_request: I/O error, dev sda, sector 54439176
Jul 10 11:48:05 mgmt kernel: ata1: EH complete

There are several of these.
At the moment ddrescue reports 22 read errors (with 35% of the data copied to a new storage). If I remember correctly, the LVM with my root partition is at the end of the drive. This means more errors will likely come... :(

Quoted 4 lines

The way I interpret the dmesg message, that's just a read error. I'm not sure, but maybe a complete wipe of the drive will even overwrite / clear these unreadable sectors.
Well, that's something to be checked after the copy process finishes.

Yep so hardware error, ddrescue will fill unreadable sectors with zeros, then you can see whether or not xfs_repair can cope with what is left.

overwriting the sectors may "fix" them but I would never trust that drive after this, personally. ;)

-Eric

#6 lore raw
In reply to Andrey Zhunev
Quoted 17 lines · Andrey Zhunev

Ooops, I forgot to paste the error message from dmesg. Here it is:

Jul 10 11:48:05 mgmt kernel: ata1.00: exception Emask 0x0 SAct 0x180000 SErr 0x0 action 0x0
Jul 10 11:48:05 mgmt kernel: ata1.00: irq_stat 0x40000008
Jul 10 11:48:05 mgmt kernel: ata1.00: failed command: READ FPDMA QUEUED
Jul 10 11:48:05 mgmt kernel: ata1.00: cmd 60/00:98:28:ac:3e/01:00:03:00:00/40 tag 19 ncq 131072 in#012         res 41/40:00:08:ad:3e/00:00:03:00:00/40 Emask 0x409 (media error) <F>
Jul 10 11:48:05 mgmt kernel: ata1.00: status: { DRDY ERR }
Jul 10 11:48:05 mgmt kernel: ata1.00: error: { UNC }
Jul 10 11:48:05 mgmt kernel: ata1.00: configured for UDMA/133
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 Sense Key : Medium Error [current] [descriptor]
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 10 11:48:05 mgmt kernel: sd 0:0:0:0: [sda] tag#19 CDB: Read(16) 88 00 00 00 00 00 03 3e ac 28 00 00 01 00 00 00
Jul 10 11:48:05 mgmt kernel: blk_update_request: I/O error, dev sda, sector 54439176
Jul 10 11:48:05 mgmt kernel: ata1: EH complete
Quoted 7 lines

There are several of these.
At the moment ddrescue reports 22 read errors (with 35% of the data copied to a new storage). If I remember correctly, the LVM with my root partition is at the end of the drive. This means more errors will likely come... :(

The way I interpret the dmesg message, that's just a read error.

"auto realocate failed" is important. Might indicate the realocation area is full :-?

Quoted 4 lines

I'm
not sure, but maybe a complete wipe of the drive will even overwrite / clear these unreadable sectors.
Well, that's something to be checked after the copy process finishes.

Run the SMART long test after you have made a copy, and watch specially for the Current_Pending_Sector, Offline_Uncorrectable, and Reallocated_Sector_Ct values. Then overwrite the entire disk with zeroes and repeat the test. If the bad sector number increases, dump the disk.

Signature · 4 lines
Cheers / Saludos,

		Carlos E. R.
		(from 15.0 x86_64 at Telcontar)