Skip to contentSkip to conversation

unable to recover RAID 5 due to bad block

DISCUSSION Chris Purves 5 participants 11 messages Nov 10, 2011 – Nov 11, 2011 1 message not in the archive MD RAID

lore
Share & tools
lore
#0 lore raw

Hello,

I have a 5 disk RAID 5 where one of the disks died completely. I replaced the failed disk and added a new disk to the array. While it was rebuilding the new disk there was a read error on one of the "good" disks that caused the disk to be reset by the controller and the array was brought down.

I can bring the array back up in degraded mode by running mdadm --assemble --force /dev/md1; however, I have been unable to build the fifth disk. I would appreciate any help.

Signature · 1 line
Chris Purves
#1 lore raw

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk. Assemble the array using the new disk instead of the one with the read error. You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;) Alex.

Zitat von Chris Purves <[email protected]>:

Quoted 14 lines

Hello,

I have a 5 disk RAID 5 where one of the disks died completely. I replaced the failed disk and added a new disk to the array. While it was rebuilding the new disk there was a read error on one of the "good" disks that caused the disk to be reset by the controller and the array was brought down.

I can bring the array back up in degraded mode by running mdadm --assemble --force /dev/md1; however, I have been unable to build the fifth disk. I would appreciate any help.

Signature · 5 lines
Chris Purves
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
Quoted 8 lines · Alexander Kühn

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk.
Assemble the array using the new disk instead of the one with the read error. You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;) Alex.

You may be better cloning the original failed disk - that way it will have a chance of actually recovering the read error (rather than having to lose the blocks). It depends on why/how/when the original disk failed though.

Cheers,
Robin

Signature · 4 lines
     ___        
    ( ' }     |       Robin Hill        <[email protected]> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |
#3 lore raw
Quoted 10 lines · Robin Hill
Quoted 8 lines · Alexander Kühn

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk.
Assemble the array using the new disk instead of the one with the read error. You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;) Alex.

Quoted 7 lines

You may be better cloning the original failed disk - that way it will have a chance of actually recovering the read error (rather than having to lose the blocks). It depends on why/how/when the original disk failed though.

Cheers,
Robin

The original failed disk can't be recognized by the controller. I tried to read the disk on another machine with no luck. It appears to be completely dead.

Signature · 5 lines
Chris Purves
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw
In reply to Alexander Kühn
Quoted 5 lines · Alexander Kühn

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk. Assemble the array using the new disk instead of the one with the read error. You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;)

I have tried this already. After cloning the disk with errors, I replaced it with the clone and tried to re-start the array using

mdadm --assemble --force /dev/md1

mdadm assigned the new disk as a spare and said there were only three disks to start the array and so couldn't start it.

After I clone the disk with the error, how precisely should I re-start the array?

Signature · 5 lines
Chris Purves
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw

Try --re-add.

Zitat von Chris Purves <[email protected]>:

Quoted 12 lines · Alexander Kühn
Quoted 7 lines

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk.
Assemble the array using the new disk instead of the one with the read error.
You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;)

I have tried this already. After cloning the disk with errors, I replaced it with the clone and tried to re-start the array using

Quoted 15 lines

mdadm --assemble --force /dev/md1

mdadm assigned the new disk as a spare and said there were only three disks to start the array and so couldn't start it.

After I clone the disk with the error, how precisely should I re-start the array?

Signature · 5 lines
Chris Purves
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 2 lines · Alexander Kühn

Try --re-add.

I'll give it a try. It'll take several hours to clone the disk again (I had tried other things in the meantime) and then I'll report back.

Thanks.

Signature · 5 lines
Chris Purves
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#7 lore raw
In reply to Alexander Kühn
Quoted 1 line · On Thu, 10 Nov 2011 18:02:34 +0100 Alexander Kühn <[email protected]>…

Try --re-add.

Nope. That doesn't make any sense at all.

NeilBrown

Quoted 35 lines

Zitat von Chris Purves <[email protected]>:

Quoted 12 lines · Alexander Kühn
Quoted 7 lines

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk.
Assemble the array using the new disk instead of the one with the read error.
You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;)

I have tried this already. After cloning the disk with errors, I replaced it with the clone and tried to re-start the array using

Quoted 15 lines

mdadm --assemble --force /dev/md1

mdadm assigned the new disk as a spare and said there were only three disks to start the array and so couldn't start it.

After I clone the disk with the error, how precisely should I re-start the array?

Signature · 5 lines
Chris Purves
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw
In reply to Chris Purves
Quoted 7 lines · On Thu, 10 Nov 2011 12:50:46 -0400 Chris Purves <[email protected]> wrote:
Quoted 5 lines · Alexander Kühn

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk. Assemble the array using the new disk instead of the one with the read error. You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;)

Quoted 9 lines

I have tried this already. After cloning the disk with errors, I replaced it with the clone and tried to re-start the array using

mdadm --assemble --force /dev/md1

mdadm assigned the new disk as a spare and said there were only three disks to start the array and so couldn't start it.

After I clone the disk with the error, how precisely should I re-start the array?

This should have worked. So presumably some unstated requirement wasn't met.

Give details. Names of devices, sizes of devices, "mdadm --examine" of device. Are you using partitions or whole disk ....

The best assemble command would be:
mdadm --assemble /dev/md1 -vv ...list...of.devices..you...want.it.to.include.

NeilBrown

#9 lore raw

Re: unable to recover RAID 5 due to bad block [solved]

Quoted 21 lines · NeilBrown
Quoted 7 lines · On Thu, 10 Nov 2011 12:50:46 -0400 Chris Purves<[email protected]> wrote:
Quoted 5 lines · Alexander Kühn

ddrescue to the rescue!
Get a another new disk, then ddrescue the one with the read error to the new disk. Assemble the array using the new disk instead of the one with the read error. You will loose the blocks that can't be read of course.
And in the future do run raid check/scrubbing at regular intervals. ;)

Quoted 9 lines

I have tried this already. After cloning the disk with errors, I replaced it with the clone and tried to re-start the array using

mdadm --assemble --force /dev/md1

mdadm assigned the new disk as a spare and said there were only three disks to start the array and so couldn't start it.

After I clone the disk with the error, how precisely should I re-start the array?

This should have worked. So presumably some unstated requirement wasn't met.

Quoted 8 lines

Give details. Names of devices, sizes of devices, "mdadm --examine" of device. Are you using partitions or whole disk ....

The best assemble command would be:

   mdadm --assemble /dev/md1 -vv ...list...of.devices..you...want.it.to.include.

NeilBrown

After ddrescue finished, I unplugged the disk with the bad sector (all 4 KB worth), and ran mdadm --assemble --force.

This time the array started up with four disks (three old, one new). I'm not sure why it didn't work the previous time; I must have done something differently.

In any case, it's working now. Thanks to everyone for your assistance. It was a big help knowing that replacing the damaged disk with a clone should work. It saved me a lot of hassle restoring from backup. Now I just need to add a fifth disk and I'll be back to normal.

Signature · 5 lines
Chris Purves
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CALJXSJrdBX8xqkwamauRsz27LUQYg-gV-G4K+RrNkdhB5ki31w@mail.gmail.com

#11 lore raw
Quoted 1 line · Chris Purves

I can bring the array back up in degraded mode by running mdadm --assemble --force /dev/md1; however, I have been unable to build the fifth disk.  I would appreciate any help.

The best way to proceed would be with the array down, you can ddrescue the semi-good disk to a new good disk, then start the array and recover by adding a new disk to the array.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html