Raid5 Reshape failed can't assemble array

Hi,

I wonder if anyone can help me, one of my drives was kicked during reshape (growing from 5 to 6 devices) and after a reboot my array refused to start. It's very similar to the case here http://marc.info/?t=125218236000001&r=1&w=2 . I have followed the steps presented in that thread but so far my array refuses to assemble.

I'm running Ubutunu 9.10 which had mdadm v2.7, that segfaulted when trying to assemble so I got v3.1.1 but still the assemble isn't working:

mdadm: looking for devices for /dev/md0
mdadm: /dev/block/252:0 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: cannot open device /dev/sdh1: Device or resource busy
mdadm: /dev/sdh1 has wrong uuid.
mdadm: /dev/sdh is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: /dev/sdg1 has wrong uuid.
mdadm: /dev/sdg is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: /dev/sdf is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: /dev/sde1 has wrong uuid.
mdadm: /dev/sde is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: /dev/sdd1 has wrong uuid.
mdadm: /dev/sdd is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sdc1 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sdc is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sdb5 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sdb2 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sdb1 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sdb is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sda5 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sda2 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sda1 is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1
mdadm: /dev/sda is not one of /dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1

Not sure why it says uuid is wrong as it is set correctly in the conf

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=5
   UUID=be282ff9:764d9beb:74ac4a35:dfcae213
   devices=/dev/sdf1,/dev/sde1,/dev/sdh1,/dev/sdg1,/dev/sdd1,dev/sdc1

# This file was auto-generated on Thu, 12 Nov 2009 01:12:06 +0800

# by mkconf $Id$

and here are details of my array

/dev/sdc1:

         Magic : a92b4efc
       Version : 0.91.00
          UUID : be282ff9:764d9beb:74ac4a35:dfcae213
 Creation Time : Sun Mar 15 05:22:48 2009
    Raid Level : raid5
 Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
    Array Size : 4883799680 (4657.55 GiB 5001.01 GB)

Raid Devices : 6
Total Devices : 6
Preferred Minor : 0

Reshape pos'n : 4156779840 (3964.21 GiB 4256.54 GB) Delta Devices : 1 (5->6)

Update Time : Mon Jan 18 00:00:51 2010
State : clean
Active Devices : 5
Working Devices : 6
Failed Devices : 1
Spare Devices : 1
Checksum : f0ffe91a - correct
Events : 764121

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 6 8 33 6 spare /dev/sdc1

  0     0       8       81        0      active sync   /dev/sdf1
  1     1       8       65        1      active sync   /dev/sde1
  2     2       8      113        2      active sync   /dev/sdh1
  3     3       8       97        3      active sync   /dev/sdg1
  4     4       8       49        4      active sync   /dev/sdd1
  5     5       0        0        5      faulty removed
  6     6       8       33        6      spare   /dev/sdc1
/dev/sdd1:
         Magic : a92b4efc
       Version : 0.91.00
          UUID : be282ff9:764d9beb:74ac4a35:dfcae213
 Creation Time : Sun Mar 15 05:22:48 2009
    Raid Level : raid5
 Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
    Array Size : 4883799680 (4657.55 GiB 5001.01 GB)

Raid Devices : 6
Total Devices : 6
Preferred Minor : 0

Reshape pos'n : 4157438400 (3964.84 GiB 4257.22 GB) Delta Devices : 1 (5->6)

Update Time : Mon Jan 18 00:01:10 2010
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 1
Spare Devices : 0
Checksum : f1085969 - correct
Events : 764126

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 4 8 49 4 active sync /dev/sdd1

  0     0       8       81        0      active sync   /dev/sdf1
  1     1       8       65        1      active sync   /dev/sde1
  2     2       8      113        2      active sync   /dev/sdh1
  3     3       8       97        3      active sync   /dev/sdg1
  4     4       8       49        4      active sync   /dev/sdd1
  5     5       0        0        5      faulty removed
/dev/sde1:
         Magic : a92b4efc
       Version : 0.91.00
          UUID : be282ff9:764d9beb:74ac4a35:dfcae213
 Creation Time : Sun Mar 15 05:22:48 2009
    Raid Level : raid5
 Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
    Array Size : 4883799680 (4657.55 GiB 5001.01 GB)

Raid Devices : 6
Total Devices : 6
Preferred Minor : 0

Reshape pos'n : 4157438400 (3964.84 GiB 4257.22 GB) Delta Devices : 1 (5->6)

Update Time : Mon Jan 18 00:01:10 2010
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 1
Spare Devices : 0
Checksum : f1085973 - correct
Events : 764126

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 1 8 65 1 active sync /dev/sde1

  0     0       8       81        0      active sync   /dev/sdf1
  1     1       8       65        1      active sync   /dev/sde1
  2     2       8      113        2      active sync   /dev/sdh1
  3     3       8       97        3      active sync   /dev/sdg1
  4     4       8       49        4      active sync   /dev/sdd1
  5     5       0        0        5      faulty removed

Any help or ideas appreciated.

Thanks

Craig

I figured out my problem, just had to stop the array first. Strangely though assemble reported not enough drives to start array but when I checked /dev/md0 it shows it as degraded and recovering. Anyway looks like my file system is fine, that is a relief.

dev/md0:

        Version : 0.91

Creation Time : Sun Mar 15 05:22:48 2009
Raid Level : raid5
Array Size : 3907039744 (3726.04 GiB 4000.81 GB)
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Raid Devices : 6
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Sat Jan 23 19:33:23 2010
State : clean, degraded, recovering
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Reshape Status : 86% complete
Delta Devices : 1, (5->6)

UUID : be282ff9:764d9beb:74ac4a35:dfcae213
Events : 0.764194

Number Major Minor RaidDevice State
0 8 81 0 active sync /dev/sdf1
1 8 65 1 active sync /dev/sde1
2 8 113 2 active sync /dev/sdh1
3 8 97 3 active sync /dev/sdg1
4 8 49 4 active sync /dev/sdd1
5 0 0 5 removed