Skip to contentSkip to conversation

Recovery raid5 after sata cable failure.

DISCUSSION Francisco Zafra 4 participants 7 messages Jul 17, 2005 – Oct 12, 2005 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi all,

I have raid5 array working without problem for some months. A SATA cable
failed and de raid5 works fine keeping the superblock persistent, but now, I can't get the old device inserted into the array.
This is the array just now:

root@Torero-2:/mnt/raid5 # mdadm --detail /dev/md0 /dev/md0:

        Version : 00.90.01
  Creation Time : Tue May 31 19:37:37 2005
     Raid Level : raid5
     Array Size : 1367507456 (1304.16 GiB 1400.33 GB)
    Device Size : 195358208 (186.31 GiB 200.05 GB)

Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Sun Jul 17 13:06:17 2005
State : clean, degraded
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

UUID : c4ed8e45:2a036953:92bff479:7cf5bac9
Events : 0.162797

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1
6 8 97 6 active sync /dev/sdg1
7 0 0 - removed

And I try to re-add the old disk in this way:

root@Torero-2:/mnt/raid5 # mdadm /dev/md0 -a /dev/sdh1
mdadm: Cannot open /dev/sdh1: Device or resource busy

What is wrong? What I am doing bad? Sdh1 is absolutely unused, so I
don't understand the error "resource busy"

Thanks,

Paco Zafra.

#1 lore raw
Quoted 5 lines · On Sunday July 17, [email protected] wrote:

Hi all,

I have raid5 array working without problem for some months. A SATA cable
failed and de raid5 works fine keeping the superblock persistent, but now, I can't get the old device inserted into the array.

...

Quoted 8 lines

And I try to re-add the old disk in this way:

root@Torero-2:/mnt/raid5 # mdadm /dev/md0 -a /dev/sdh1
mdadm: Cannot open /dev/sdh1: Device or resource busy

What is wrong? What I am doing bad? Sdh1 is absolutely unused, so I
don't understand the error "resource busy"

Well, it definitely is busy...

Maybe it is still part of md0, but marked as 'faulty'.
If so (cat /proc/mdstat would tell you) you need to remove it first.
mdadm /dev/md0 -r /dev/sdh1
mdadm /dev/md0 -a /dev/sdh1

NeilBrown

#2 lore raw

I already tried that:

root@Torero-2:~ # cat /proc/mdstat
Personalities : [linear] [raid5]
md0 : active raid5 sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]

      1367507456 blocks level 5, 512k chunk, algorithm 2 [8/7] [UUUUUUU_]

unused devices: <none>
root@Torero-2:~ # mdadm /dev/md0 -r /dev/sdh1
mdadm: hot remove failed for /dev/sdh1: No such device or address root@Torero-2:~ # mdadm /dev/md0 -a /dev/sdh1
mdadm: Cannot open /dev/sdh1: Device or resource busy root@Torero-2:~ #

With no luck :(

Paco.

Quoted 35 lines

-----Mensaje original-----
De: Neil Brown [mailto:[email protected]] Enviado el: lunes, 18 de julio de 2005 0:11
Para: Francisco Zafra
CC: [email protected]
Asunto: Re: Recovery raid5 after sata cable failure.

Quoted 3 lines · On Sunday July 17, [email protected] wrote:

Hi all,

I have raid5 array working without problem for some

months. A SATA

Quoted 2 lines

cable failed and de raid5 works fine keeping the superblock persistent, but now, I can't get the old device inserted

into the array.
...

Quoted 7 lines

And I try to re-add the old disk in this way:

root@Torero-2:/mnt/raid5 # mdadm /dev/md0 -a /dev/sdh1
mdadm: Cannot open /dev/sdh1: Device or resource busy

What is wrong? What I am doing bad? Sdh1 is absolutely

unused, so I

Quoted 1 line

don't understand the error "resource busy"

Well, it definitely is busy...

Maybe it is still part of md0, but marked as 'faulty'.
If so (cat /proc/mdstat would tell you) you need to remove it first.
mdadm /dev/md0 -r /dev/sdh1
mdadm /dev/md0 -a /dev/sdh1

NeilBrown

#3 lore raw
Quoted 5 lines · On Monday July 18, [email protected] wrote:

I already tried that:

root@Torero-2:~ # cat /proc/mdstat
Personalities : [linear] [raid5]
md0 : active raid5 sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]

Quoted 7 lines
      1367507456 blocks level 5, 512k chunk, algorithm 2 [8/7] [UUUUUUU_]

unused devices: <none>
root@Torero-2:~ # mdadm /dev/md0 -r /dev/sdh1
mdadm: hot remove failed for /dev/sdh1: No such device or address root@Torero-2:~ # mdadm /dev/md0 -a /dev/sdh1
mdadm: Cannot open /dev/sdh1: Device or resource busy

Uhm, you might have a buggy version of mdadm. If you have 1.10.0, get an upgrade.

Otherwise either sdh1 or sdh must be:
open by some process with O_EXCL
open via a /dev/raw/* device
part of an md device (which it obviously isn't)
part of a dm device
mounted as a filesystem
an external-journal device for a jfs or ext3 or xfs filesystem in use as a swap device
open for writing under a security level of 1 (whatever that means..) an mtd device that is open

(those are all the places that I can find that take an exclusive lock
on a block device).

NeilBrown

#4 lore raw

Hi Neil,
Since some hours I am trying to solved it with the last version:
root@Torero-2:~ # mdadm --version
mdadm - v2.0-devel-2 - DEVELOPMENT VERSION NOT FOR REGULAR USE - 7 July 2005

With the same results :(

I really don't think it is locked I dd it in act of desperation and I have no problems:
root@Torero-2:~ # dd if=/dev/zero of=/dev/sdh bs=1k count=1000 1000+0 records in
1000+0 records out
1024000 bytes transferred in 0.417862 seconds (2450570 bytes/sec)

No locked or anything... I have really get out of ideas with this...

Thanks for all your help.

Paco.

Quoted 43 lines

-----Mensaje original-----
De: Neil Brown [mailto:[email protected]] Enviado el: lunes, 18 de julio de 2005 11:41
Para: Francisco Zafra
CC: [email protected]
Asunto: RE: Recovery raid5 after sata cable failure.

Quoted 5 lines · On Monday July 18, [email protected] wrote:

I already tried that:

root@Torero-2:~ # cat /proc/mdstat
Personalities : [linear] [raid5]
md0 : active raid5 sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2]

sdb1[1] sda1[0]

Quoted 8 lines
      1367507456 blocks level 5, 512k chunk, algorithm 2 [8/7] 

[UUUUUUU_]

unused devices: <none>
root@Torero-2:~ # mdadm /dev/md0 -r /dev/sdh1
mdadm: hot remove failed for /dev/sdh1: No such device or address root@Torero-2:~ # mdadm /dev/md0 -a /dev/sdh1
mdadm: Cannot open /dev/sdh1: Device or resource busy

Uhm, you might have a buggy version of mdadm. If you have 1.10.0, get an upgrade.

Otherwise either sdh1 or sdh must be:
open by some process with O_EXCL
open via a /dev/raw/* device
part of an md device (which it obviously isn't)
part of a dm device
mounted as a filesystem
an external-journal device for a jfs or ext3 or xfs filesystem in use as a swap device
open for writing under a security level of 1 (whatever that means..) an mtd device that is open

(those are all the places that I can find that take an exclusive lock on a block device).

NeilBrown

#5 lore raw
Quoted 4 lines · Francisco Zafra

Hi Neil, Since some hours I am trying to solved it with the last version:
root@Torero-2:~ # mdadm --version
mdadm - v2.0-devel-2 - DEVELOPMENT VERSION NOT FOR REGULAR USE - 7 July 2005

Quoted 9 lines

With the same results :(

I really don't think it is locked I dd it in act of desperation and I have no problems:
root@Torero-2:~ # dd if=/dev/zero of=/dev/sdh bs=1k count=1000 1000+0 records in
1000+0 records out
1024000 bytes transferred in 0.417862 seconds (2450570 bytes/sec)

Asking a silly question perhaps..

fuser /dev/sdh

Regards,
Brad

Signature · 3 lines
"Human beings, who are almost unique in having the ability
to learn from the experience of others, are also remarkable
for their apparent disinclination to do so." -- Douglas Adams
#6 lore raw

safe to test SATA array by pulling cables?

In reply to Francisco Zafra

I've read conflicting views on whether it's safe to pull either or both the SATA data cable or power cable from a disk in an array (when they are NOT in a hotswap cage) to test whether things works as expected during a real life disk failure.

Is there a consensus on this? Theoretically either should be handled by the controller (a 3ware 9500s) since this is a real life possibility, but will either event cause damage to the disk? I'd suspect that removal of the data cable would simulate a disk loss with less physical trauma to the disk in question.

Signature · 3 lines
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - [email protected] 
            <<plain text preferred>>