raid10 issues after reorder of boot drives.

I have a strange issue on my raid10 8x400GB array. I cannot assemble my array anymore and it's gotten to a point that I don't know what to do to recover my data. I was hoping I could get some advice on it. Below is some info which I hope will help troubleshoot this issue.

ARRAY /dev/md/1 metadata=1.0 num-devices=0 UUID=828ed03d:0c28afda:4a636e88:7b29ec9f name=Darkside:1
spares=7
devices=/dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdc3,/dev/sda3,/dev/sdb3 ARRAY /dev/md/1 level=raid10 metadata=1.0 num-devices=8 UUID=828ed03d:0c28afda:4a636e88:7b29ec9f name=Darkside:1
devices=8:131

mdadm - v3.2.3 - 23rd December 2011

${i};done

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f013

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sda2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sda3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e460f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdb2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdb3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7d10530f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdc2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdc3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x81a213ab

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdd2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdd3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sde: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f00e

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sde2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sde3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdf: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f00c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdf2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdf3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdg: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x327d8d82

   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdg2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdg3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdi: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e460f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdi1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdi2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdi3 73779200 976773119 451496960 fd Linux raid
autodetect

3.3.3-gentoo

Thanks in advanced.

adding more verbose info gives me:

mdadm: looking for devices for /dev/md1
mdadm: /dev/dm-8 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdi1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdi is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sr0 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-7 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-6 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-5 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-4 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-3 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/dm-0 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/md/0 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdh3 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdh2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdh1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdh is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: cannot open device /dev/sdg3: Device or resource busy
mdadm: /dev/sdg2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdg1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdg is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: /dev/sdf2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdf1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdf is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: cannot open device /dev/sde3: Device or resource busy
mdadm: /dev/sde2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sde1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sde is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: cannot open device /dev/sdd3: Device or resource busy
mdadm: /dev/sdd2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdd1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdd is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: cannot open device /dev/sdb3: Device or resource busy
mdadm: /dev/sdb2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdb1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdb is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: cannot open device /dev/sda3: Device or resource busy
mdadm: /dev/sda2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sda1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sda is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: cannot open device /dev/sdc3: Device or resource busy
mdadm: /dev/sdc2 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdc1 is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
mdadm: /dev/sdc is not one of /dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3

You seem to have an explicit list of devices in /etc/mdadm.conf This is not a good idea for 'sd' devices as they can change their names, which can mean they aren't on the list any more. You should remove that once you get this all sorted out.

NeilBrown

@Neil sorry but I didn't get to reply to all on my last 2 emails, so here is goes again so it's archived.

/dev/sdh3:

          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
           Name : Darkside:1  (local to host Darkside)

Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : raid10
Raid Devices : 8

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)

     Array Size : 3611971584 (1722.32 GiB 1849.33 GB)
  Used Dev Size : 902992896 (430.58 GiB 462.33 GB)
   Super Offset : 902993904 sectors
          State : clean
    Device UUID : 00565578:e2eaaba3:f1eae17c:f474ee8d

    Update Time : Wed Apr 25 17:22:58 2012
       Checksum : 1e7c3692 - correct
         Events : 82942

         Layout : far=2
     Chunk Size : 256K

Device Role : Active device 0
Array State : AAAAAAAA ('A' == active, '.' == missing)

The only drive that didn't get affected is far=3. Any suggestions? I have the drives on separate controllers and when I created the array I set up the order as /dev/sda3 /dev/sde3 /dev/sdb3 /dev/sdf3 and so on. so I would assume the same order would be used, also note that I ran luksFormat on /dev/md1 then ran pvcreate /dev/md1 and so on. Will I have issues with luksOpen after recreating the array? I removed the /dev/sdh1 drive so now the output is like:

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f013

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sda2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sda3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e460f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdb2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdb3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7d10530f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdc2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdc3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x81a213ab

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdd2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdd3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sde: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f00e

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sde2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sde3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdf: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f00c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdf2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdf3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdg: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x327d8d82

   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdg2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdg3 73779200 976773119 451496960 fd Linux raid
autodetect

Disk /dev/sdh: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e460f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1            2048     2099199     1048576   82  Linux swap / Solaris
/dev/sdh2         2099200    73779199    35840000   fd  Linux raid

autodetect
/dev/sdh3 73779200 976773119 451496960 fd Linux raid
autodetect

and

cat /proc/mdstat
Personalities : [raid10]
md127 : inactive sdb3[8](S) sdf3[13](S) sda3[11](S) sdc3[9](S) sdd3[12](S) sdg3[10](S) sde3[15](S)

      3160477768 blocks super 1.0

md0 : active raid10 sdd2[12] sdh2[0] sdb2[8] sdf2[13] sdg2[10] sda2[11] sde2[14] sdc2[9]

      143358976 blocks super 1.0 256K chunks 2 near-copies [8/8] [UUUUUUUU]

unused devices: <none>

and the output from you for loop.

/dev/sda3 and /dev/sdc3 seem to match
/dev/sda3 and /dev/sde3 seem to match
/dev/sda3 and /dev/sdg3 seem to match
/dev/sdc3 and /dev/sda3 seem to match
/dev/sdc3 and /dev/sde3 seem to match
/dev/sdc3 and /dev/sdg3 seem to match
/dev/sde3 and /dev/sda3 seem to match
/dev/sde3 and /dev/sdc3 seem to match
/dev/sde3 and /dev/sdg3 seem to match
/dev/sdg3 and /dev/sda3 seem to match
/dev/sdg3 and /dev/sdc3 seem to match
/dev/sdg3 and /dev/sde3 seem to match

Thanks in advanced Neil.
likewhoa

As the array is "far=2" you will need to create with --layout=f2

That means you cannot easily detect pairs by comparing the first few kilobytes.

However if you are confident that you know the order of the drives (because of they arrangement of controllers) then maybe you can just create the array. Make sure you set --chunk=256

Yes, you will need to 'luksOpen' after recreating the array. That will quite possibly fail if you have the order wrong.
Then you would need to "pvscan" or whatever one does to find LVM components. If both those succeed, then try "fsck -n".
If they fail, try a different ordering.

NeilBrown

I am not sure how to proceed now with the output that shows possible pairs as it won't allow me to setup all 8 devices on the array but only
4. Should I run the array creation with -x4 and set the available spare devicesor or just create the array as I can remember which was one pair from each controller. i.e /dev/sda3 /dev/sde3 ...?

ok I was able to recreate the array with correct order which I took from my /dev/md0's --details output and was able to decrypt the luks mapping but XFS didn't open and xfs_repair is currently doing the matrix. I will keep this posted with updates.

Thanks again Neil.
WRT 3.3.3 should I just go back to 3.3.2 which seemed to run fine and wait until there is a release of 3.3.3 that has fix?

I hope the order really is correct.... I wouldn't expect xfs to find problems if it was...

3.3.4 has the fix and was just released.
3.3.1, 3.3.2 and 3.3.3 all have the bug. It only triggers on shutdown and even then only occasionally.
So I recommend 3.3.4.

NeilBrown

The reason I believe it was correct was that 'cryptsetup luksOpen /dev/md1 md1' worked. I really do hope that it was correct too because after opening the luks mapping I assume there is no going back.

Opening the luks mapping could just mean that the first few blocks are correct. So the first disk is right but others might not be.

There is going backup unless something has been written to the array. Once that happens anything could be corrupted. So if the xfs check you are doing is read-only you could still have room to move.

With a far=2 array, each first half of each device is mirrored on the second half. So you can probably recover the ordering by finding which pairs match.

The "Used Dev Size" is 902992896 sectors. Half of that is 451496448 or 231166181376 bytes.

So to check if two devices are adjacent in the mapping you can try:

cmp --ignore-initial=0:231166181376 --bytes=231166181376 first-dev second-dev

You could possibly use a smaller --bytes= number, at least on the first attempt.
You a similar 'for' loop to before an use this command and it might tell you which pairs of devices are consecutive. From that you should be able to get the full order.

NeilBrown

I don't see why xfs_repair would write data unless it actually finds the superblock but I am not sure so I will take my chances since it's still searching for the secondary superblock now.

After running the for loop all night which produced this output.

/dev/sda3 /dev/sdb3 differ: byte 1, line 1
/dev/sda3 /dev/sdc3 differ: byte 262145, line 2
/dev/sda3 /dev/sdd3 differ: byte 1, line 1
/dev/sda3 /dev/sde3 differ: byte 1, line 1
/dev/sda3 and /dev/sdf3 seem to match
/dev/sda3 /dev/sdg3 differ: byte 262145, line 2
/dev/sda3 /dev/sdh3 differ: byte 1, line 1
/dev/sdb3 /dev/sda3 differ: byte 1, line 1
/dev/sdb3 /dev/sdc3 differ: byte 1, line 1
/dev/sdb3 /dev/sdd3 differ: byte 1, line 1
/dev/sdb3 and /dev/sde3 seem to match
/dev/sdb3 /dev/sdf3 differ: byte 1, line 1
/dev/sdb3 /dev/sdg3 differ: byte 1, line 1
/dev/sdb3 /dev/sdh3 differ: byte 1, line 1
/dev/sdc3 /dev/sda3 differ: byte 262145, line 2
/dev/sdc3 /dev/sdb3 differ: byte 1, line 1
/dev/sdc3 /dev/sdd3 differ: byte 1, line 1
/dev/sdc3 /dev/sde3 differ: byte 1, line 1
/dev/sdc3 /dev/sdf3 differ: byte 262145, line 2
/dev/sdc3 and /dev/sdg3 seem to match
/dev/sdc3 /dev/sdh3 differ: byte 1, line 1
/dev/sdd3 /dev/sda3 differ: byte 1, line 1
/dev/sdd3 /dev/sdb3 differ: byte 1, line 1
/dev/sdd3 /dev/sdc3 differ: byte 1, line 1
/dev/sdd3 /dev/sde3 differ: byte 1, line 1
/dev/sdd3 /dev/sdf3 differ: byte 1, line 1
/dev/sdd3 /dev/sdg3 differ: byte 1, line 1
/dev/sdd3 and /dev/sdh3 seem to match
/dev/sde3 /dev/sda3 differ: byte 262145, line 2
/dev/sde3 /dev/sdb3 differ: byte 1, line 1
/dev/sde3 and /dev/sdc3 seem to match
/dev/sde3 /dev/sdd3 differ: byte 1, line 1
/dev/sde3 /dev/sdf3 differ: byte 262145, line 2
/dev/sde3 /dev/sdg3 differ: byte 262145, line 2
/dev/sde3 /dev/sdh3 differ: byte 1, line 1
/dev/sdf3 /dev/sda3 differ: byte 1, line 1
/dev/sdf3 /dev/sdb3 differ: byte 1, line 1
/dev/sdf3 /dev/sdc3 differ: byte 1, line 1
/dev/sdf3 and /dev/sdd3 seem to match
/dev/sdf3 /dev/sde3 differ: byte 1, line 1
/dev/sdf3 /dev/sdg3 differ: byte 1, line 1
/dev/sdf3 /dev/sdh3 differ: byte 1, line 1
/dev/sdg3 and /dev/sda3 seem to match
/dev/sdg3 /dev/sdb3 differ: byte 1, line 1
/dev/sdg3 /dev/sdc3 differ: byte 262145, line 2
/dev/sdg3 /dev/sdd3 differ: byte 1, line 1
/dev/sdg3 /dev/sde3 differ: byte 1, line 1
/dev/sdg3 /dev/sdf3 differ: byte 262145, line 2
/dev/sdg3 /dev/sdh3 differ: byte 1, line 1
/dev/sdh3 /dev/sda3 differ: byte 1, line 1
/dev/sdh3 and /dev/sdb3 seem to match
/dev/sdh3 /dev/sdc3 differ: byte 1, line 1
/dev/sdh3 /dev/sdd3 differ: byte 1, line 1
/dev/sdh3 /dev/sde3 differ: byte 1, line 1
/dev/sdh3 /dev/sdf3 differ: byte 1, line 1
/dev/sdh3 /dev/sdg3 differ: byte 1, line 1

I manage recover my luks+xfs with this --create command \o/

--layout=f2 --assume-clean /dev/sdh3 /dev/sdb3 /dev/sde3 /dev/sdc3 /dev/sdg3 /dev/sda3 /dev/sdf3 /dev/sdd3

Thank you Neil for your assistance you rock! With regards to my /etc/mdadm.conf the explicit references to the sd drives was generated with 'mdadm -Esv', my question is do you suggest I not even populate /etc/mdadm.conf with such output because of change in drives and just use 'mdadm -s -A /dev/mdX'? Also after running into this nasty bug I get the feeling that I should really keep a copy of all my future 'mdadm --create ...' commands handy just for such situations, do you agree?

Thanks again and have a GREAT weekend.
Fernando Vasquez a.k.a likewhoa

cool!! I love it when something like that produces a nice clean usable result.

The output of '-Ds' and '-Es' was only ever meant to be a starting point for mdadm.conf, not the final content..

I think it is good to populate mdadm.conf, though in simple cases it isn't essential.
I would just list the UUID for each array:
ARRAY /dev/md0 uuid=.......
and leave it at that.

Keeping a copy of the output of "mdadm -E" of each device could occasionally be useful. You would need to update this copy any time any config change happened to the array such as a device failure or a spare being added.

Glad it all worked out!

NeilBrown

I used the output of 'mdadm -Esv' then removed everything but UUID=... but after doing 'mdadm -A /dev/md1' I get "mdadm: failed to add 8:131 to /dev/md1: Invalid argument
mdadm: /dev/md1 assembled from 0 drives - not enough to start the array." mdadm.conf has "ARRAY /dev/md/1 UUID=158918a9:52356b95:f4926a78:a55d4509" so not sure what could be the problem since it should just work.

I'm afraid you've been bitten by a rather nasty bug which is present in 3.3 and got back ported to some -stable kernel. The fix has been submitted and should be appearing in -stable kernels soon (maybe already).

The effect of this bug is to remove key data from the metadata. It makes the devices look like a spare for an array of unknown level with no devices and no chunk size.

It seems that you have one device which wasn't corrupted: 8:131 which should be /dev/sdi3. The others are messed up.

What you will need to do is re-create the array. You can look at /dev/sdi3 to see what the chunksize and layout is:
mdadm --examine /dev/sdi3

and then run a command like:

mdadm --stop /dev/md1
mdadm --create /dev/md1 --metadata=1.0 -l10 -n8 --chunk=XXX --layout=YY \

  --assume-clean \
    /dev/sdg3 /dev/sda3 /dev/sdi3 /dev/sdb3 /dev/sdf3 /dev/sde3 /dev/sdc3 /dev/sdh3

(probably have some device names wrong)

Note that the order of the devices is *very* important, and that is information that has been lost. However some of it can be recovered. If this is an 'n2' or 'near=2' array, which is the default, then adjacent devices will be identical. You can find identical pairs with:

for i in a b c d e f g h i
do for j in a b c d e f g h i
do if [ $i = $j ] ; then continue fi

     if cmp -s -n 32768 /dev/sd${i}3 /dev/sd${j}3
     then echo /dev/sd${i}3 and /dev/sd${j}3 seem to match
     fi

done
done

That should help you find pairs. Then you have at most 16 possible orderings of those 4 pairs to test.

Note that the "--create" command doesn't touch the data. It just writes new metadata and assembles the array. Then you can run "fsck -n" to check if the array looks good.

Good luck, and please ask if anything here isn't clear.

NeilBrown

note this strange problem which doesn't seem related.

Personalities : [raid10]
md127 : inactive sdb3[8](S) sdf3[13](S) sda3[11](S) sdc3[9](S) sdd3[12](S) sdg3[10](S) sde3[15](S)

      3160477768 blocks super 1.0

md0 : active raid10 sdd2[12] sdh2[0] sdb2[8] sdf2[13] sdg2[10] sda2[11] sde2[14] sdc2[9]

      143358976 blocks super 1.0 256K chunks 2 near-copies [8/8] [UUUUUUUU]

unused devices: <none>

mdadm: error opening /dev/md127: No such file or directory

Do you happen to know if this bug was present in ubuntu 11.10, kernel versions 3.0.0-15-server or 3.0.0-16-server ?

I saw failures to assemble RAID6 arrays after unclean shutdowns on both of these, just in test environments. If it happens again I'll send mdadm --examine output. I just tried to replicate it and failed :-(

Regards,

Brian.

I don't keep track of what ubuntu (or anyone but suse) put in their kernels, sorry.

To replicate:

- create an array.
- stop the array
- assemble the array with at least one missing device e.g.:

    mdadm -A /dev/md0 /dev/sda1 /dev/sdb1

if it was a 3-device array
- check in /proc/mdstat that it is listed as "inactivate"
- reboot
- now "mdadm -E" the devices. If the raid level is -unknown- then the bug has hit.

NeilBrown

Thank you for this info.

After reboot the array came up as active again, and mdadm -E showed "Raid Level : raid6" for all drives, so that's not the problem I had before.

Regards,

Brian.

I wondered about Ubuntu and checked the kernel package changelog:
http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_3.0.0-19.33/changelog

Cross-checking with the commit nominated by Jan Ceuleers to this list on 30/04/12 (c744a65c1e2d59acc54333ce8) and the Ubuntu changelog (see grep below) - I don't believe Ubuntu 11.10 includes the bug.

Cheers,

lb

$ grep ' md' changelog
  * md/bitmap: ensure to load bitmap when creating via sysfs.
  * md/raid1,raid10: avoid deadlock during resync/recovery.
  * net: Fix driver name for mdio-gpio.c
  * tcp: md5: using remote adress for md5 lookup in rst packet
  * md/raid5: fix bug that could result in reads from a failed device.
  * md/raid5: abort any pending parity operations when array fails.
  * md: Avoid waking up a thread after it has been freed.
  * md/raid5: fix bug that could result in reads from a failed device.
  * tcp: properly handle md5sig_pool references
  * md/linear: avoid corrupting structure while waiting for rcu_free to
  * md: Fix handling for devices from 2TB to 4TB in 0.90 metadata.
  * winbond: include linux/delay.h for mdelay et al
  * md: fix 'degraded' calculation when starting a reshape.
  * md: fix small irregularity with start_ro module parameter
  * SCSI: st: fix mdata->page_order handling
  * md: Fix unfortunate interaction with evms
  * md/bitmap: protect against bitmap removal while being updated.
  * md: Fix "strchr" [drivers/md/dm-log-userspace.ko] undefined!