Raid 5 rebuild with only 2 spare devices

Hi!

Sorry for my bad English. I'm from Austria and this is also my first "mailinglist-post".

I have a problem with my RAID5. The raid has only 1 active devices out of 3. The other 2 devices are detected as spare. This is what happens when I try to assemble the raid(I'm using loop devices because I'm working with backup files):

root@backup-server:/media# mdadm --assemble --force /dev/md2 /dev/loop0 /dev/loop1 /dev/loop2
mdadm: /dev/md2 assembled from 1 drive and 2 spares - not enough to start the array.

root@backup-server:/media# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : inactive loop1[0](S) loop2[4](S) loop0[3](S)
      4390443648 blocks

unused devices: <none>

root@backup-server:/media# mdadm -R /dev/md2
mdadm: failed to run array /dev/md2: Input/output error

root@backup-server:/media# mdadm -D /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Thu Nov 19 21:09:37 2009
     Raid Level : raid5
  Used Dev Size : 1463481216 (1395.68 GiB 1498.60 GB)
   Raid Devices : 3
  Total Devices : 1
Preferred Minor : 2
    Persistence : Superblock is persistent

Update Time : Sun Nov 14 14:12:44 2010
State : active, FAILED, Not Started
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : 9665c475:31f17aa2:83a3570a:c5b3b84e
Events : 0.3352467

Number Major Minor RaidDevice State
0 7 1 0 active sync /dev/loop1
1 0 0 1 removed
2 0 0 2 removed

root@backup-server:/media# mdadm /dev/md2 -a /dev/loop0
mdadm: re-added /dev/loop0
root@backup-server:/media# mdadm /dev/md2 -a /dev/loop2
mdadm: re-added /dev/loop2
root@backup-server:/media# mdadm -D /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Thu Nov 19 21:09:37 2009
     Raid Level : raid5
  Used Dev Size : 1463481216 (1395.68 GiB 1498.60 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 2
    Persistence : Superblock is persistent

Update Time : Sun Nov 14 14:12:44 2010
State : active, FAILED, Not Started
Active Devices : 1
Working Devices : 3
Failed Devices : 0
Spare Devices : 2

Layout : left-symmetric
Chunk Size : 64K

UUID : 9665c475:31f17aa2:83a3570a:c5b3b84e
Events : 0.3352467

Number Major Minor RaidDevice State
0 7 1 0 active sync /dev/loop1
1 0 0 1 removed
2 0 0 2 removed

3 7 0 - spare /dev/loop0
4 7 2 - spare /dev/loop2

I also tried to recreate the raid:

root@backup-server:/media# mdadm -Cv /dev/md2 -n3 -l5 /dev/loop0 /dev/loop1 /dev/loop2
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: /dev/loop0 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Thu Nov 19 21:09:37 2009
mdadm: layout defaults to left-symmetric
mdadm: /dev/loop1 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Thu Nov 19 21:09:37 2009
mdadm: layout defaults to left-symmetric
mdadm: /dev/loop2 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Thu Nov 19 21:09:37 2009
mdadm: size set to 1463479808K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.

root@backup-server:/media# mdadm -D /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Fri Feb  4 17:05:18 2011
     Raid Level : raid5
     Array Size : 2926959616 (2791.37 GiB 2997.21 GB)
  Used Dev Size : 1463479808 (1395.68 GiB 1498.60 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

Update Time : Fri Feb 4 17:05:18 2011
State : clean, degraded
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 512K

Name : backup-server:2 (local to host backup-server) UUID : c37336d0:9811f9d1:294aa588:a85a5096
Events : 0

Number Major Minor RaidDevice State
0 7 0 0 active sync /dev/loop0
1 7 1 1 active sync /dev/loop1
2 0 0 2 removed

3 7 2 - spare /dev/loop2

root@backup-server:/media# mdadm /dev/md2 -r /dev/loop2
mdadm: hot removed /dev/loop2 from /dev/md2

root@backup-server:/media# mdadm /dev/md2 -a /dev/loop2
mdadm: re-added /dev/loop2
root@backup-server:/media# mdadm -D /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Fri Feb  4 17:05:18 2011
     Raid Level : raid5
     Array Size : 2926959616 (2791.37 GiB 2997.21 GB)
  Used Dev Size : 1463479808 (1395.68 GiB 1498.60 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

Update Time : Fri Feb 4 17:15:25 2011
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 512K

Rebuild Status : 0% complete

Name : backup-server:2 (local to host backup-server) UUID : c37336d0:9811f9d1:294aa588:a85a5096
Events : 6

Number Major Minor RaidDevice State
0 7 0 0 active sync /dev/loop0
1 7 1 1 active sync /dev/loop1
3 7 2 2 spare rebuilding /dev/loop2

root@backup-server:/media# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid5 loop2[3] loop0[0] loop1[1]
      2926959616 blocks super 1.2 level 5, 512k chunk, algorithm 2 
[3/2] [UU_]
      [=>...................]  recovery =  5.0% (74496424/1463479808) 
finish=188.7min speed=122624K/sec

unused devices: <none>

When I to that I can't find the LVM that should be inside the raid. So I reloaded the Backup so I'm back at the beginning. I know that my data is more or less intact because I can find a few files with testdisks photorec(after I rebuild the raid with the command above).

Best regards,

Thomas

Hi Thomas,

Welcome! (Your English looks fine to me--and I've had 40+ years of practice.)

Working from backups is a very good plan!

Hmmm. Not enough info here, and further steps destroy it. Good thing you started over.

Please show "mdadm -E /dev/loop[0-2]" on fresh loop copies *before* trying any "create" or "add" operations.

Yeah, mdadm was trying to tell you not to do that. "--assume-clean" is really important when trying to recreate an array with existing data.

[trim /]

If the problem is just the event counts, "mdadm --assemble --force" is probably what you want, followed by "mdadm --readonly". If pvscan shows your LVM subsystem at that point, try an fsck to see how much trouble you are in.

Phil

Hi,
thanks for the quick answer.

2011/2/10 Phil Turmel <[email protected]>:

Well, not having a backup has brought me into this problem so I learned form my mistake.

Unfortunately I'm not that familiar the -E option so I don't really understand what all that means. But I think it's interesting that there are sometimes 4 devices although the raid only has 3 in reality.

root@backup-server:/media# mdadm -E /dev/loop[0-2]
/dev/loop0:
         Magic : a92b4efc
       Version : 0.90.00
          UUID : 9665c475:31f17aa2:83a3570a:c5b3b84e
 Creation Time : Thu Nov 19 21:09:37 2009
    Raid Level : raid5
 Used Dev Size : 1463481216 (1395.68 GiB 1498.60 GB)
    Array Size : 2926962432 (2791.37 GiB 2997.21 GB)
  Raid Devices : 3
 Total Devices : 2
Preferred Minor : 2

   Update Time : Sun Nov 14 02:08:16 2010
         State : active
 Active Devices : 1
Working Devices : 2
 Failed Devices : 2
 Spare Devices : 1
      Checksum : aa2d9609 - correct
        Events : 3352465

        Layout : left-symmetric
    Chunk Size : 64K

     Number   Major   Minor   RaidDevice State
this     3       8       35        3      spare

  0     0       8       19        0      active sync 1     1       0        0        1      faulty removed 2     2       0        0        2      faulty removed 3     3       8       35        3      spare /dev/loop1:
         Magic : a92b4efc
       Version : 0.90.00
          UUID : 9665c475:31f17aa2:83a3570a:c5b3b84e Creation Time : Thu Nov 19 21:09:37 2009
    Raid Level : raid5
 Used Dev Size : 1463481216 (1395.68 GiB 1498.60 GB) Array Size : 2926962432 (2791.37 GiB 2997.21 GB) Raid Devices : 3
 Total Devices : 1
Preferred Minor : 2

   Update Time : Sun Nov 14 14:12:44 2010
         State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 2
 Spare Devices : 0
      Checksum : aa2e3f94 - correct
        Events : 3352467

        Layout : left-symmetric
    Chunk Size : 64K

     Number   Major   Minor   RaidDevice State this     0       8       19        0      active sync

  0     0       8       19        0      active sync 1     1       0        0        1      faulty removed 2     2       0        0        2      faulty removed /dev/loop2:
         Magic : a92b4efc
       Version : 0.90.00
          UUID : 9665c475:31f17aa2:83a3570a:c5b3b84e Creation Time : Thu Nov 19 21:09:37 2009
    Raid Level : raid5
 Used Dev Size : 1463481216 (1395.68 GiB 1498.60 GB) Array Size : 2926962432 (2791.37 GiB 2997.21 GB) Raid Devices : 3
 Total Devices : 1
Preferred Minor : 2

   Update Time : Sun Nov 14 01:41:40 2010
         State : active
 Active Devices : 1
Working Devices : 1
 Failed Devices : 2
 Spare Devices : 0
      Checksum : aa2d8f80 - correct
        Events : 3352459

        Layout : left-symmetric
    Chunk Size : 64K

     Number   Major   Minor   RaidDevice State
this     4       8        3       -1      spare

  0     0       8       19        0      active sync 1     1       0        0        1      faulty removed 2     2       0        0        2      faulty removed

I will try that at some point but not right now because the restore from my backups takes about 15 hours.

Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 10/02/2011 18:03, Thomas Heilberg wrote:
[...]

[...]

[...]

[...]

Those loop devices are now trashed since you didn't re-create the array with exactly the parameters with which it was initially created. Your settings make me think the array was created with an older version of mdadm; the defaults for metadata version and chunk size changed a little while ago. Anyway, if you're trying again, you should specify -e 0.90 -c 64. While you're at it, add --assume-clean to avoid any rebuild, which in your case may in fact destroy good data (though the array's parity would end up consistent). Or if as you noted in your other reply you're going to have to wait 15 hours before trying anything, maybe wait until The Boss[1] makes a more intelligent suggestion than I can; he usually posts at times that appear to be overnight to me but are presumably sensible times of day for him.

Cheers,

John.

[1] Neil Brown, who lives in Sydney where it's revoltingly early in the morning.

2011/2/10 John Robinson <[email protected]>:

It worked! Although I am not quite sure why.
This is what I did:

root@backup-server:/media# mdadm -Cv /dev/md2 -e 0.90 -c 64
--assume-clean -n3 -l5 /dev/loop[012]
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: /dev/loop0 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Thu Nov 19 21:09:37 2009
mdadm: layout defaults to left-symmetric
mdadm: /dev/loop1 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Thu Nov 19 21:09:37 2009
mdadm: layout defaults to left-symmetric
mdadm: /dev/loop2 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Thu Nov 19 21:09:37 2009
mdadm: size set to 1463481216K
Continue creating array? y
mdadm: array /dev/md2 started.

root@backup-server:/media# mdadm -D /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Sat Feb 12 18:25:55 2011
     Raid Level : raid5
     Array Size : 2926962432 (2791.37 GiB 2997.21 GB)
  Used Dev Size : 1463481216 (1395.68 GiB 1498.60 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 2
    Persistence : Superblock is persistent

Update Time : Sat Feb 12 18:25:55 2011
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : b5a7fcfb:b98b8cb8:41761e78:ef14cd93 (local to host
backup-server)
Events : 0.1

Number Major Minor RaidDevice State
0 7 0 0 active sync /dev/loop0
1 7 1 1 active sync /dev/loop1
2 7 2 2 active sync /dev/loop2

root@backup-server:/media# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md2 : active (auto-read-only) raid5 loop2[2] loop1[1] loop0[0]
      2926962432 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>

After that pvscan found my LVM:

root@backup-server:/media# pvscan
  PV /dev/md2    VG server   lvm2 [2,73 TiB / 86,37 GiB free]
root@backup-server:/media# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "server" using metadata type lvm2
root@backup-server:/media# lvs
  LV        VG     Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  daten     server -wi---   2,58t
  gentoo    server -wi---  20,00g
  home      server -wi---  20,00g
  root      server -wi---  25,00g

root@backup-server:/media# vgchange -ay
  4 logical volume(s) in volume group "server" now active

Then of course I checked all filesystems with e2fsck and it appears that all my data is ok. I'm so happy thank you both for the help. :)

But there is one thing I don't understand. I did that recreating act 2 times. The first time it didn't work and I had to restore the raid-partitions from my backup(this time onto a btrfs partition in the hope I could use its snapshot feature) and then it worked. Is it possible that the order of the devices is important for the recreate process?
I mean mdadm -C... /dev/loop1 /dev/loop0 /dev/loop2 instead of the normal order? Because I did that by mistake(or to be more precise I "mounted" the second image into loop0)

This would be important to me because, then I could just directly recreate my raid and wouldn't need to copy 2,2TB over LAN.

Anyway thank you again for the help.

Thomas

Wonderful!

[trim /]

Yes, the order of devices matters for "create". With the original images, "mdadm -E" on each should report which "slot" or raid device number they expect to be. That's used in normal assembly to keep everything in order (the kernel doesn't guarantee consistent block device names or load order).

So, if you are recreating an array (with --assume-clean), you need to specify them in the same order that "mdadm -E" sees.

Phil