Skip to contentSkip to conversation

Raid recovery. Help wanted!

DISCUSSION Evgeny Koryanov 3 participants 4 messages Apr 21, 2013 MD RAID

lore
Share & tools
lore
#0 lore raw

Hello everybody!

Yesterday I met a problem with one of raid5 arrays build by mdadm on three (sd[bcd]) 1.5T devices.
I found array in degraded state with sdd fail. Drive becomes fail state after power jump.
Server supplied by UPS but this seems was not good enough - server was not rebooted but one drive as I said becomes fail state. I simply reattaches it and array started rebuilding but fails after couple of %'s passed with sdc becomes fail!!! I assemble array again with sd[bc] and tried to attach sdd again: picture repeated rebuild fails.
So I have sdb in sync state, sdc - failed and sdd spare. I checked SMARTs of drives to understand reason of such behavior and found it clean on all devices. Than I tried to dd if=/dev/sd[bcd] of=/dev/null and found that dd also fails with IO error. After dd bad blocks started appears in SMART :)
Finally I have:
sdb - sync
sdc - fail
sdd - spare
states and a number of bads on each hdd in random places...

Could any one suggest how can I assemble this array now in read-only mode to try to copy data?!
Theoretically data on sdd should not be rewritten and it still should be possible to try recover data (meaning that bads appears in quite different places)... May be you know utility which helps recover data or the way how to start array in read-only mode preventing becomes it to degraded state and force md device to try recover data using readable places from each devise???
Or any other ideas appreciated! Thanks, any way...

Best regards,
Evgeny.

#1 lore raw
Quoted 34 lines · Evgeny Koryanov

Hello everybody!

Yesterday I met a problem with one of raid5 arrays build by mdadm on three (sd[bcd]) 1.5T devices.
I found array in degraded state with sdd fail. Drive becomes fail state after power jump.
Server supplied by UPS but this seems was not good enough - server was not rebooted but one drive as I said becomes fail state.
I simply reattaches it and array started rebuilding but fails after couple of %'s passed with sdc becomes fail!!!
I assemble array again with sd[bc] and tried to attach sdd again: picture repeated rebuild fails.
So I have sdb in sync state, sdc - failed and sdd spare. I checked SMARTs of drives to understand reason of such behavior and
found it clean on all devices. Than I tried to dd if=/dev/sd[bcd] of=/dev/null and found that dd also fails with IO error.
After dd bad blocks started appears in SMART :)
Finally I have:
sdb - sync
sdc - fail
sdd - spare
states and a number of bads on each hdd in random places...

Could any one suggest how can I assemble this array now in read-only mode to try to copy data?!
Theoretically data on sdd should not be rewritten and it still should be possible to try recover data (meaning that bads appears in quite different places)...
May be you know utility which helps recover data or the way how to start array in read-only mode preventing becomes it to degraded state and force md device to try recover data using readable places from each devise???
Or any other ideas appreciated! Thanks, any way...

Quoted 6 lines

Best regards,
Evgeny.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi,

Could you post the smartctl -a output of all the drives? If 2 drives are failling you might want to derescue them somewhere and assemble the RAID from that.

Mathias

#2 lore raw
Quoted 41 lines · Mathias Burén
Quoted 34 lines · Evgeny Koryanov

Hello everybody!

Yesterday I met a problem with one of raid5 arrays build by mdadm on three (sd[bcd]) 1.5T devices.
I found array in degraded state with sdd fail. Drive becomes fail state after power jump.
Server supplied by UPS but this seems was not good enough - server was not rebooted but one drive as I said becomes fail state.
I simply reattaches it and array started rebuilding but fails after couple of %'s passed with sdc becomes fail!!!
I assemble array again with sd[bc] and tried to attach sdd again: picture repeated rebuild fails.
So I have sdb in sync state, sdc - failed and sdd spare. I checked SMARTs of drives to understand reason of such behavior and
found it clean on all devices. Than I tried to dd if=/dev/sd[bcd] of=/dev/null and found that dd also fails with IO error.
After dd bad blocks started appears in SMART :)
Finally I have:
sdb - sync
sdc - fail
sdd - spare
states and a number of bads on each hdd in random places...

Could any one suggest how can I assemble this array now in read-only mode to try to copy data?!
Theoretically data on sdd should not be rewritten and it still should be possible to try recover data (meaning that bads appears in quite different places)...
May be you know utility which helps recover data or the way how to start array in read-only mode preventing becomes it to degraded state and force md device to try recover data using readable places from each devise???
Or any other ideas appreciated! Thanks, any way...

Quoted 6 lines

Best regards,
Evgeny.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines

Hi,

Could you post the smartctl -a output of all the drives? If 2 drives are failling you might want to derescue them somewhere and assemble the RAID from that.

Mathias

Hi, Mathis.

Will post it tomorrow - it's down now and I'm not around.
But as per your suggestion: still not clear for me is it good idea - as soon as I will copy valid data to another drive
information about bad's places will be lost (for md device driver) and bad blocks physically will be replaced by zeros.
And what will happen after assembling and trying to read places where bad blocks was (where array marked sync and blocks are
not consistent - redundant part zeroed) - will md read properly? will md mark drive fail as soon as find async (zeroed) block ans start resync... Actually I did not found in mdadm read-only assembling mode which will prevent such behavior!

Best regards,
Evgeny.

* English - detected
* English
* Russian
* Norwegian

* English
* Russian
* Norwegian

<javascript:void(0);>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 52 lines · Evgeny Koryanov
Quoted 41 lines · Mathias Burén
Quoted 34 lines · Evgeny Koryanov

Hello everybody!

Yesterday I met a problem with one of raid5 arrays build by mdadm on three (sd[bcd]) 1.5T devices.
I found array in degraded state with sdd fail. Drive becomes fail state after power jump.
Server supplied by UPS but this seems was not good enough - server was not rebooted but one drive as I said becomes fail state.
I simply reattaches it and array started rebuilding but fails after couple of %'s passed with sdc becomes fail!!!
I assemble array again with sd[bc] and tried to attach sdd again: picture repeated rebuild fails.
So I have sdb in sync state, sdc - failed and sdd spare. I checked SMARTs of drives to understand reason of such behavior and
found it clean on all devices. Than I tried to dd if=/dev/sd[bcd] of=/dev/null and found that dd also fails with IO error.
After dd bad blocks started appears in SMART :)
Finally I have:
sdb - sync
sdc - fail
sdd - spare
states and a number of bads on each hdd in random places...

Could any one suggest how can I assemble this array now in read-only mode to try to copy data?!
Theoretically data on sdd should not be rewritten and it still should be possible to try recover data (meaning that bads appears in quite different places)...
May be you know utility which helps recover data or the way how to start array in read-only mode preventing becomes it to degraded state and force md device to try recover data using readable places from each devise???
Or any other ideas appreciated! Thanks, any way...

Quoted 6 lines

Best regards,
Evgeny.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines

Hi,

Could you post the smartctl -a output of all the drives? If 2 drives are failling you might want to derescue them somewhere and assemble the RAID from that.

Mathias

Hi, Mathis.

Quoted 9 lines

Will post it tomorrow - it's down now and I'm not around.
But as per your suggestion: still not clear for me is it good idea - as soon as I will copy valid data to another drive
information about bad's places will be lost (for md device driver) and bad blocks physically will be replaced by zeros.
And what will happen after assembling and trying to read places where bad blocks was (where array marked sync and blocks are
not consistent - redundant part zeroed) - will md read properly? will md mark drive fail as soon as find async (zeroed) block ans start resync... Actually I did not found in mdadm read-only assembling mode which will prevent such behavior!

Best regards,
Evgeny.

You want to use "ddrescue" which includes a log of bad blocks... (Be sure to use a logfile when you run ddrescue for each drive) It can often get all data off of failed drives because it will retry the bad blocks, and if not it will hopefully only have a very small area.

When it is done, you could conceivably have some program to do a recovery including the data in the log file to rebuild any other missing data... But I don't think such a program is written yet.

Essentially your first step is to recover as much of the actual data to reliable disks as possible, then you will know what your situation really is...

Sam--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html