Need help with recovery

Hi guys.

I've been having trouble with a four-disk raid5 array, which I now believe were caused by a flaky eSATA cable. The cable has been replaced, and the controller now seems to reliably see all four disks.

In fact I have five disks, and so a partition on the fifth disk acts as a spare. Over the past several days I've been connecting and disconnecting drives in ways that I didn't realise were unreliable (due to the faulty cable).

This has resulted in the md metadata being out of sync on the different partitions. Please could someone provide advice as to how to recover this array (which contains an ext3 filesystem)?

Many thanks!

Jan

/dev/sda2:

          Magic : a92b4efc
        Version : 00.90.00
           UUID : 714e46f1:479268a7:895e209c:936fa570
  Creation Time : Fri Jun 22 20:56:51 2007
     Raid Level : raid5
  Used Dev Size : 311588608 (297.15 GiB 319.07 GB)
     Array Size : 934765824 (891.46 GiB 957.20 GB)

Raid Devices : 4
Total Devices : 4
Preferred Minor : 2

Update Time : Wed Jul 14 13:19:56 2010
State : active
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Checksum : 242755b4 - correct
Events : 24565

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 1 8 2 1 active sync /dev/sda2

   0     0       8       66        0      active sync   /dev/sde2
   1     1       8        2        1      active sync   /dev/sda2
   2     2       8       18        2      active sync   /dev/sdb2
   3     3       0        0        3      faulty removed
   4     4       8       34        4      spare   /dev/sdc2
/dev/sdb2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 714e46f1:479268a7:895e209c:936fa570
  Creation Time : Fri Jun 22 20:56:51 2007
     Raid Level : raid5
  Used Dev Size : 311588608 (297.15 GiB 319.07 GB)
     Array Size : 934765824 (891.46 GiB 957.20 GB)

Raid Devices : 4
Total Devices : 5
Preferred Minor : 2

Update Time : Tue Jul 13 03:47:51 2010
State : active
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Checksum : 2425708a - correct
Events : 21017

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 0 8 2 0 active sync /dev/sda2

   0     0       8        2        0      active sync   /dev/sda2
   1     1       8       18        1      active sync   /dev/sdb2
   2     2       8       34        2      active sync   /dev/sdc2
   3     3       8       50        3      active sync   /dev/sdd2
   4     4       8       82        4      spare   /dev/sdf2
/dev/sdc2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 714e46f1:479268a7:895e209c:936fa570
  Creation Time : Fri Jun 22 20:56:51 2007
     Raid Level : raid5
  Used Dev Size : 311588608 (297.15 GiB 319.07 GB)
     Array Size : 934765824 (891.46 GiB 957.20 GB)

Raid Devices : 4
Total Devices : 4
Preferred Minor : 2

Update Time : Wed Jul 14 13:20:13 2010
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 2427b5fa - correct
Events : 24574

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 4 8 34 4 spare /dev/sdc2

   0     0       8       66        0      active sync   /dev/sde2
   1     1       0        0        1      faulty removed
   2     2       8       18        2      active sync   /dev/sdb2
   3     3       0        0        3      faulty removed
   4     4       8       34        4      spare   /dev/sdc2
/dev/sdd2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 714e46f1:479268a7:895e209c:936fa570
  Creation Time : Fri Jun 22 20:56:51 2007
     Raid Level : raid5
  Used Dev Size : 311588608 (297.15 GiB 319.07 GB)
     Array Size : 934765824 (891.46 GiB 957.20 GB)

Raid Devices : 4
Total Devices : 4
Preferred Minor : 2

Update Time : Wed Jul 14 13:20:13 2010
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 2427b5ec - correct
Events : 24574

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 2 8 18 2 active sync /dev/sdb2

   0     0       8       66        0      active sync   /dev/sde2
   1     1       0        0        1      faulty removed
   2     2       8       18        2      active sync   /dev/sdb2
   3     3       0        0        3      faulty removed
   4     4       8       34        4      spare   /dev/sdc2
/dev/sde2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 714e46f1:479268a7:895e209c:936fa570
  Creation Time : Fri Jun 22 20:56:51 2007
     Raid Level : raid5
  Used Dev Size : 311588608 (297.15 GiB 319.07 GB)
     Array Size : 934765824 (891.46 GiB 957.20 GB)

Raid Devices : 4
Total Devices : 4
Preferred Minor : 2

Update Time : Wed Jul 14 13:20:13 2010
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 2427b618 - correct
Events : 24574

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 0 8 66 0 active sync /dev/sde2

   0     0       8       66        0      active sync   /dev/sde2
   1     1       0        0        1      faulty removed
   2     2       8       18        2      active sync   /dev/sdb2
   3     3       0        0        3      faulty removed
   4     4       8       34        4      spare   /dev/sdc2

mdadm -A /dev/md2 --force /dev/sd[ade]2

sdc2 this it is a spare, so you don't want it just now.
sdb2 and sde2 both think that are 'device 0', but sde2 is clearly newer.

So run the above mdadm comment (assuming none of the devices have been renamed again since you created the --examine output), then
fsck -n /dev/md2

and make sure it is happy.

If it is you can added sdb2 and sde2 as spares.

NeilBrown

Neil,

Many thanks for your feedback. I will try this when I get home.

So if I boil this down to the principles, your advice is to assemble the array in degraded mode using the components that have the most recent update timestamps. Degraded, because this prevents resynchronisation.

Thanks, Jan

It is more a case of "degraded because that uses fewer devices, and so fewer old devices which could introduce errors." though avoiding resync is a good thing too.

The key bit is "--force" which tell mdadm to assemble it anyway even if something looks out of data. mdadm will only include just enough devices to make the array work.
You don't really need to be selective about the devices you choose. If you give "mdadm -Af" all of the devices, it will just pick the ones that are best. I just listed them for you to make it more clear what was happening.

NeilBrown

Neil,

Your advice has worked a treat. Thank you very much; my server is back up again.

Thanks, Jan