Creating RAID5 with four devices and end up with 5 (one removed and one spare). Why?

DISCUSSION Tor Arne Vestbø 3 participants 6 messages Mar 3, 2008 – Mar 4, 2008 MD RAID

Share & tools

Tor Arne Vestbø

author

2008-03-03 08:55 UTC #0 lore raw

Hi!

I'm trying to build a Linux RAID5 with four (4) 750GB disks, but not matter what I do I end up with mdadm listing five (5) devices and telling me that one of them is a spare, and another one is failed/removed. I've been googling and reading HOWTOs for a week now, but can't figure it out. Here's what I do:

monstre:~/buildroot # mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd[cdef]1

mdadm: /dev/sdc1 appears to contain an ext2fs file system

    size=-2097251968K  mtime=Wed Feb 27 00:26:57 2008
mdadm: /dev/sdc1 appears to be part of a raid array:
    level=raid5 devices=4 ctime=Tue Feb 26 20:47:24 2008
mdadm: /dev/sdd1 appears to be part of a raid array:
    level=raid5 devices=4 ctime=Tue Feb 26 20:47:24 2008
mdadm: /dev/sde1 appears to be part of a raid array:
    level=raid5 devices=4 ctime=Tue Feb 26 20:47:24 2008
mdadm: /dev/sdf1 appears to contain an ext2fs file system
    size=-1560381056K  mtime=Wed Feb 27 00:26:57 2008
mdadm: /dev/sdf1 appears to be part of a raid array:
    level=raid5 devices=4 ctime=Tue Feb 26 20:47:24 2008

Continue creating array? y
mdadm: array /dev/md0 started.

monstre:~/buildroot # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active(auto-read-only) raid5 sdf1[4](S) sde1[2] sdd1[1] sdc1[0]

      2197715712 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

unused devices: <none>

monstre:~/buildroot # mdadm --detail /dev/md0 /dev/md0:

        Version : 00.90.03

Creation Time : Tue Feb 26 20:57:46 2008
Raid Level : raid5
Array Size : 2197715712 (2095.91 GiB 2250.46 GB)
Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Tue Feb 26 20:57:46 2008
State : clean, degraded
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

UUID : 8b498455:0ab45ac1:ebf26e3f:56a9fc2f
Events : 0.1

Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 65 2 active sync /dev/sde1
3 0 0 3 removed

4 8 81 - spare /dev/sdf1

monstre: # mdadm --examine /dev/sdd1 /dev/sdd1:

          Magic : a92b4efc
        Version : 00.90.00
           UUID : a0186556:4ffb5a2a:822f8875:94ae7d2c
  Creation Time : Sun Mar  2 22:52:53 2008
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 2197715712 (2095.91 GiB 2250.46 GB)

Raid Devices : 4
Total Devices : 4
Preferred Minor : 0

Update Time : Sun Mar 2 22:59:54 2008
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Checksum : 6b5e8442 - correct
Events : 0.22

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 1 8 49 1 active sync /dev/sdd1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       0        0        3      faulty removed
   4     4       8       81        4      spare   /dev/sdf1

-------------------

So what i don't get is:

1. Why is mdadm --examine listing "3 3 0 0 3
faulty removed" and telling me I have a failed device?
2. Why is one of the actual disks (sdf) used as a spare, even though I didn't ask for it?

Thanks for any tips or insights which may put me on the right track :)

Tor Arne

Robin Hill

2008-03-03 09:19 UTC #1 lore raw

Quoted 57 lines · Tor Arne Vestbø

Hi!
I'm trying to build a Linux RAID5 with four (4) 750GB disks, but not matter what I do I end up with mdadm listing five (5) devices and telling me that one of them is a spare, and another one is failed/removed. I've been googling and reading HOWTOs for a week now, but can't figure it out. Here's what I do:
monstre:~/buildroot # mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd[cdef]1
mdadm: array /dev/md0 started.
monstre:~/buildroot # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active(auto-read-only) raid5 sdf1[4](S) sde1[2] sdd1[1] sdc1[0]
      2197715712 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
unused devices: <none>
monstre: # mdadm --examine /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : a0186556:4ffb5a2a:822f8875:94ae7d2c
  Creation Time : Sun Mar  2 22:52:53 2008
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 2197715712 (2095.91 GiB 2250.46 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Sun Mar 2 22:59:54 2008
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Checksum : 6b5e8442 - correct
Events : 0.22
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 49 1 active sync /dev/sdd1
   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       0        0        3      faulty removed
   4     4       8       81        4      spare   /dev/sdf1
-------------------

Quoted 9 lines

So what i don't get is:
1. Why is mdadm --examine listing "3 3 0 0 3
faulty removed" and telling me I have a failed device?
2. Why is one of the actual disks (sdf) used as a spare, even though I didn't ask for it?
Thanks for any tips or insights which may put me on the right track :)

This is perfectly normal (and explained in the manual page) - the RAID5 array is created in an initially degraded state, then rebuilt. This means the array can be available for use immediately, with the rebuild taking place in the background. You'll need to run 'mdadm -w /dev/md0' to force the array into read-write mode (it's currently started in auto-read-only mode) and the resync will then begin.

HTH,
Robin

Signature · 4 lines

     ___        
    ( ' }     |       Robin Hill        <[email protected]> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

Tor Arne Vestbø

author

2008-03-03 09:29 UTC #2 lore raw

Thanks for your reply Robin!

Quoted 5 lines · Robin Hill

Quoted 5 lines
So what i don't get is:
1. Why is mdadm --examine listing "3 3 0 0 3 faulty removed" and telling me I have a failed device?
2. Why is one of the actual disks (sdf) used as a spare, even though I didn't ask for it?

Quoted 8 lines

Quoted 2 lines
Thanks for any tips or insights which may put me on the right track :)
This is perfectly normal (and explained in the manual page) - the RAID5 array is created in an initially degraded state, then rebuilt. This means the array can be available for use immediately, with the rebuild taking place in the background. You'll need to run 'mdadm -w /dev/md0' to force the array into read-write mode (it's currently started in auto-read-only mode) and the resync will then begin.

I had a suspicion this was the case for question number two -- why one of the disks initially are marked as spares. Good to have that cleared up!

But does that explain why mdadm believes I have another disk, a fifth disk, that has been removed? The ID numbers of the real disks are 0,1,2,4 but I would expect 0,1,2,3:

Number Major Minor RaidDevice State
this 1 8 49 1 active sync /dev/sdd1

    0     0       8       33        0      active sync   /dev/sdc1
    1     1       8       49        1      active sync   /dev/sdd1
    2     2       8       65        2      active sync   /dev/sde1
    3     3       0        0        3      faulty removed <-- What's up?
    4     4       8       81        4      spare   /dev/sdf1

Thanks!

Tor Arne

Mario 'BitKoenig' Holbe

2008-03-03 09:55 UTC #3 lore raw

Quoted 3 lines · Tor Arne Vestbø

But does that explain why mdadm believes I have another disk, a fifth disk, that has been removed? The ID numbers of the real disks are 0,1,2,4 but I would expect 0,1,2,3:

Once the rebuild has finished, you will have exactly the latter. md stores raid-disks in different slots than not-yet-raid-disks.

regards
Mario

Quoted 2 lines

As Luke Leighton said once on samba-ntdom, "now, what was that about
rebooting? that was so long ago, i had to look it up with man -k."

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Tor Arne Vestbø

author

2008-03-03 10:01 UTC #4 lore raw

Quoted 5 lines · Mario 'BitKoenig' Holbe

Quoted 1 line · Tor Arne Vestbø
But does that explain why mdadm believes I have another disk, a fifth disk, that has been removed? The ID numbers of the real disks are 0,1,2,4 but I would expect 0,1,2,3:
Once the rebuild has finished, you will have exactly the latter. md stores raid-disks in different slots than not-yet-raid-disks.

I see :) Thank you both!

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Tor Arne Vestbø

author

2008-03-04 09:47 UTC #5 lore raw

In reply to Mario 'BitKoenig' Holbe

Quoted 5 lines · Mario 'BitKoenig' Holbe

Quoted 1 line · Tor Arne Vestbø
But does that explain why mdadm believes I have another disk, a fifth disk, that has been removed? The ID numbers of the real disks are 0,1,2,4 but I would expect 0,1,2,3:
Once the rebuild has finished, you will have exactly the latter. md stores raid-disks in different slots than not-yet-raid-disks.

Of course, you were right :) I waited for the array to sync, and what do you know! My 0,1,2,3,4 one failure, one spare array turned into a 0,1,2,3 clean everything fine array :P

Thanks!

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html