Desperate: mdadm fails to assemble raid saying devices are busy.

Hi,

I am on Arch Linux. I have a 4 device RAID 5 of which 1 device is no longer working. I am unable to assemble the raid.

The active devices are:/dev/sda2, /dev/sdb1, /dev/sdc2

The errors I get are like:

# mdadm -Avf /dev/md127
mdadm: looking for devices for /dev/md127
mdadm: no RAID superblock on /dev/dm-6
mdadm: no RAID superblock on /dev/dm-5
mdadm: no RAID superblock on /dev/dm-4
mdadm: no RAID superblock on /dev/dm-3
mdadm: no RAID superblock on /dev/dm-2
mdadm: no RAID superblock on /dev/dm-1
mdadm: no RAID superblock on /dev/dm-0
mdadm: /dev/sdc2 is busy - skipping
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb1 is busy - skipping
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 is busy - skipping
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda

I would like to know what is making these devices busy. I do not see any indication from lsof or from the device mapper that these devices are being used. I have even changed the initramfs and the fstab etc. to prevent this array from being started at boot time - still unable to assemble it.

This array contains my /home and I do not know how to recover it.

Help please ...

Thanks,
Sachin

cat /proc/mdstat

It is probably already partially assembled (and since no force not active--note there is a kernel optionmd_mod.start_dirty_degraded=1 that will enable if the array can at least work ) so not operational, typically you can stop it and it will no longer show up in mdstat then you can do what you are trying to do.

lsof will not show things that are being held open/busy in the kernel, lsof only shows userspace useage.

what does "cat /proc/mdstat" say? Also post mdadm and kernel version.

Mikael Abrahamsson    email: [email protected]

Hi Roger & Mikael,

Thanks a lot for your replies. I tried the following and still to no avail.

Here are the results of the various commands:

# cat /proc/mdstat
Personalities :
md0 : inactive sda2[0](S) sdb1[4](S) sdc2[1](S)
      480215144 blocks super 1.2

unused devices: <none>

# mdadm -V
mdadm - v3.2.6 - 25th October 2012

# uname -a
Linux triveni 3.10.6-2-ARCH #1 SMP PREEMPT Tue Aug 13 09:47:01 CEST 2013
x86_64 GNU/Linux

# mdadm -Esv
ARRAY /dev/md/0  level=raid5 metadata=1.2 num-devices=4
UUID=01b153ea:a93c7325:c13bf9cf:3195666f name=sysresccd:0
   devices=/dev/sdc2,/dev/sdb1,/dev/sda2

Based on Roger's mail, set the kernel option "md_mod.start_dirty_degraded=1":

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=/dev/mapper/triveni--lvm--320gp3-root rw
md_mod.start_dirty_degraded=1

Still, I get:

# mdadm -Avfs
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/dm-6
mdadm: no RAID superblock on /dev/dm-5
mdadm: no RAID superblock on /dev/dm-4
mdadm: no RAID superblock on /dev/dm-3
mdadm: no RAID superblock on /dev/dm-2
mdadm: no RAID superblock on /dev/dm-1
mdadm: no RAID superblock on /dev/dm-0
mdadm: /dev/sdc2 is busy - skipping
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb1 is busy - skipping
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 is busy - skipping
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda

Do "mdadm --stop /dev/md0" and then try

"mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb1 /dev/sdc2"

See if this removes the busy problem. Also post output from this operation as well as what comes up in dmesg when you do this.

Mikael Abrahamsson    email: [email protected]

Thanks, Mikael - that worked. The output of the various commands are given below.

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0

[22874.978883] md: md0 stopped.
[22874.978900] md: unbind<sda2>
[22875.018193] md: export_rdev(sda2)
[22875.018284] md: unbind<sdb1>
[22875.059733] md: export_rdev(sdb1)
[22875.059821] md: unbind<sdc2>
[22875.079735] md: export_rdev(sdc2)
[22875.652012] md: md0 stopped.

# mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb1 /dev/sdc2
mdadm: /dev/md0 assembled from 3 drives - not enough to start the array
while not clean - consider --force.

[22875.653625] md: bind<sdc2>
[22875.653943] md: bind<sdb1>
[22875.654302] md: bind<sda2>

# mdadm --assemble --force /dev/md0 /dev/sda2 /dev/sdb1 /dev/sdc2

This worked!!!

# mdadm -Dv /dev/md0

/dev/md0:

        Version : 1.2

Creation Time : Sat Mar 2 11:03:45 2013
Raid Level : raid5
Array Size : 468475392 (446.77 GiB 479.72 GB)
Used Dev Size : 156158464 (148.92 GiB 159.91 GB)
Raid Devices : 4
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Mon Sep 23 15:41:08 2013
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : sysresccd:0
UUID : 01b153ea:a93c7325:c13bf9cf:3195666f
Events : 118445

Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 34 1 active sync /dev/sdc2
2 0 0 2 removed
4 8 17 3 active sync /dev/sdb1

Now, I need to find a new hard drive and put it in Slot 2. What would be the best way to do that, apart from RTFM ?

Thanks for all your help.

Best,
Sachin

I recommend you to do RTFM anyhow, it'll help you in the long run. "--add" is what you're looking for for above operation.

Mikael Abrahamsson    email: [email protected]