Skip to contentSkip to conversation

Raid 10 question/problem [ot]

DISCUSSION Marc Perkel 4 participants 15 messages Jan 27, 2007 – Jan 29, 2007

lore
Share & tools
lore
#0 lore raw

I'm a little stumped trying to set up raid 10. I set it up and it worked but after a reboot it forgets my raid setup.

Created 2 raid 1 arrays in md0 and md1 and that works and survives a reboot.

However - I created a raid 0 on /dev/md2 made up of /dev/md0 and /dev/md1 and it worked but it forgets it after I reboot. The device /dev/md2 fails to survive a reboot.

Created the /etc/mdadm.conf file but that doesn't seem to have made a difference.

What am I missing? Thanks in advance.

____________________________________________________________________________________
Don't pick lemons.
See all the new 2007 cars at Yahoo! Autos.
http://autos.yahoo.com/new_cars.html

#1 lore raw
Quoted 4 lines

I'm a little stumped trying to set up raid 10. I set it up and it worked but after a reboot it forgets my raid setup.

Now, let's hear the name of the distribution you use.

BTW, is md1 also disappearing?

	-`J'
#2 lore raw
Quoted 12 lines · --- Jan Engelhardt
Quoted 2 lines

I'm a little stumped trying to set up raid 10. I

set

Quoted 1 line

it up and it worked but after a reboot it forgets

my

Quoted 1 line

raid setup.

Now, let's hear the name of the distribution you
use.

BTW, is md1 also disappearing?

Sorry about that. I'm using Fedora Core 6. /dev/md0 and /dev/md1, buth of which are raid 1 arrays survive the reboot. But when I make a raid 0 out of those two raid arrays that's what is vanishing.

Thanks for your help.

____________________________________________________________________________________
Finding fabulous fares is fun.
Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains.
http://farechase.yahoo.com/promo-generic-14795097

#3 lore raw
Quoted 11 lines · Marc Perkel
Quoted 10 lines · --- Jan Engelhardt
Quoted 2 lines

I'm a little stumped trying to set up raid 10. I

set

Quoted 1 line

it up and it worked but after a reboot it forgets

my

Quoted 1 line

raid setup.

Now, let's hear the name of the distribution you
use.

Quoted 6 lines
Quoted 1 line

BTW, is md1 also disappearing?

Sorry about that. I'm using Fedora Core 6. /dev/md0 and /dev/md1, buth of which are raid 1 arrays survive the reboot. But when I make a raid 0 out of those two raid arrays that's what is vanishing.

That's interesting. I am using Aurora Corona, and all but md0 vanishes. (Reason for that is that udev does not create the nodes md1-md31 on boot, so mdadm cannot assemble the arrays.)

	-`J'
#4 lore raw
Quoted 20 lines · --- Jan Engelhardt
Quoted 5 lines · Marc Perkel
Quoted 4 lines · --- Jan Engelhardt
Quoted 2 lines

I'm a little stumped trying to set up raid 10. I

set

Quoted 1 line

it up and it worked but after a reboot it

forgets

Quoted 10 lines
Quoted 7 lines

my

Quoted 1 line

raid setup.

Now, let's hear the name of the distribution you
use.

BTW, is md1 also disappearing?

Sorry about that. I'm using Fedora Core 6. /dev/md0 and /dev/md1, buth of which are raid 1 arrays

survive

Quoted 1 line

the reboot. But when I make a raid 0 out of those

Quoted 9 lines

two

Quoted 1 line

raid arrays that's what is vanishing.

That's interesting. I am using Aurora Corona, and
all but md0 vanishes.
(Reason for that is that udev does not create the
nodes md1-md31 on
boot, so mdadm cannot assemble the arrays.)

What do you have to do to get UDEV to create /dev/md2? Is there a config file for that?

____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

#5 lore raw
Quoted 11 lines · Marc Perkel
Quoted 8 lines
Quoted 4 lines

I'm using Fedora Core 6. /dev/md0 and /dev/md1, buth of which are raid 1 arrays survive the reboot. But when I make a raid 0 out of those two raid arrays that's what is vanishing.

That's interesting. I am using Aurora Corona [FC6+RHide], and all but md0 vanishes. (Reason for that is that udev does not create the nodes md1-md31 on boot, so mdadm cannot assemble the arrays.)

What do you have to do to get UDEV to create /dev/md2? Is there a config file for that?

That's the big question. On openSUSE 10.2, all the md devices get created automatically. I suppose that happens as part of udev processing all the queued kernel events at bootup.

On default FC6 install (i.e. without any raid), only /dev/md0 is present (like in Aurora). That alone, and that you got md1 there, and I don't, is strange.

I think I found it. udev does not do md at all, for some reason. This line in /etc/rc.d/rc.sysinit is quite "offending":

[ -x /sbin/nash ] && echo "raidautorun /dev/md0" | nash --quiet

Starting a init=/bin/bash prompt and doing "/usr/sbin/udevmonitor &" there reveals:

bash-3.1# echo raidautorun /dev/md0 | /sbin/nash --quiet UEVENT[1169934663.372139] add@/block/md0 bash-3.1# echo raidautorun /dev/md1 | /sbin/nash --quiet UEVENT[1169934667.601027] add@/block/md1

No sign of md1. (Wtf here!) I can see why it's broken, but to nash every md device sounds like the worst solution around. I'd say the Fedora boot process is severely broken wrt. md. Well, what's your rc.sysinit looking like, since you seem to be having a md1 floating around?

	-`J'
#7 lore raw
In reply to Jan Engelhardt
Quoted 1 line · Jan Engelhardt

On Jan 27 2007 10:31, Marc Perkel wrote:

[]

Quoted 8 lines
Quoted 4 lines

Sorry about that. I'm using Fedora Core 6. /dev/md0 and /dev/md1, buth of which are raid 1 arrays survive the reboot. But when I make a raid 0 out of those two raid arrays that's what is vanishing.

That's interesting. I am using Aurora Corona, and all but md0 vanishes. (Reason for that is that udev does not create the nodes md1-md31 on boot, so mdadm cannot assemble the arrays.)

This is nonsense.

Mdadm creates those nodes automa[tg]ically - man mdadm, search for --auto. Udev has exactly nothing to do with mdX nodes.

In order for an md array to be started up on boot, it has to be specified in /etc/mdadm.conf. With proper DEVICE line in there. That's all.

If you're using raid0 on top of two raid1s (as opposed to using raid10 directly - which is more efficient and flexible), the DEVICE line in mdadm.conf should be either `partitions' (usually preferred way), or - in case direct device list is specified - should contain both real disks AND the raid1 arrays.

But in any case, this has exactly nothing to do with kernel. It's 100% userspace issues, I'd say distribution-specific issues.

/mjt

#8 lore raw
Quoted 10 lines · Michael Tokarev
Quoted 4 lines · Jan Engelhardt

That's interesting. I am using Aurora Corona, and all but md0 vanishes. (Reason for that is that udev does not create the nodes md1-md31 on boot, so mdadm cannot assemble the arrays.)

This is nonsense.

Mdadm creates those nodes automa[tg]ically - man mdadm, search for --auto. Udev has exactly nothing to do with mdX nodes.

Note that mdadm -As _is_ run on FC6 boot.

Quoted 2 lines

In order for an md array to be started up on boot, it has to be specified in /etc/mdadm.conf. With proper DEVICE line in there. That's all.

That's how it is, and it does not work.

openSUSE 10.2:
no mdadm.conf _at all_, /etc/init.d/boot.d/boot.md is chkconfig'ed _out_, _no_ md kernel module is loaded, and I still have all the /dev/md nodes.

FC6 standard install:
no mdadm.conf, otherwise regular boot. /dev/md0 exists. Uhuh.

FC6 with two raids:

# fdisk -l
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         123      987966   fd  Linux raid autodetect
/dev/sdb2             124         246      987997+  fd  Linux raid autodetect
/dev/sdb3             247         369      987997+  fd  Linux raid autodetect
/dev/sdb4             370        1044     5421937+  fd  Linux raid autodetect

Disk /dev/sdc: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1         123      987966   fd  Linux raid autodetect
/dev/sdc2             124         246      987997+  fd  Linux raid autodetect
/dev/sdc3             247         369      987997+  fd  Linux raid autodetect
/dev/sdc4             370        1044     5421937+  fd  Linux raid autodetect
# mdadm -C /dev/md0 -e 1.0 -l 1 -n 2 /dev/sdb1 /dev/sdc1
mdadm: array /dev/md0 started.
# mdadm -C /dev/md1 -e 1.0 -l 1 -n 2 /dev/sdb2 /dev/sdc2
mdadm: error opening /dev/md1: No such file or directory

Showstopper.

# mknod /dev/md1 b 9 1
# mdadm -C /dev/md1 -e 1.0 -l 1 -n 2 /dev/sdb2 /dev/sdc2
mdadm: array /dev/md1 started.
# cat /etc/mdadm.conf
cat: /etc/mdadm.conf: No such file or directory
# echo "DEVICE /dev/sd[a-z][0-9]" >/etc/mdadm.conf
# mdadm --detail --scan >>/etc/mdadm.conf
# cat /etc/mdadm.conf
DEVICE /dev/sd[a-z][0-9]
ARRAY /dev/md0 level=raid1 num-devices=2 name=0 UUID=5ded6a11:3b9072f6:ae46efc7:d1628ea7
ARRAY /dev/md1 level=raid1 num-devices=2 name=1 UUID=2fda5608:d63d8287:761a7a09:68fe743f
# reboot

...
Starting udev: [ OK ]
Loading default keymap (us): [ OK ]
Setting hostname fc6.site: [ OK ]
mdadm: /dev/md0 has been started with 2 drives.
mdadm: error opening /dev/md1: No such file or directory No devices found
Setting up Logical Volume Management: No volume groups found [ OK ]
...

Now with "DEVICE partitions" in mdadm.conf:

mdadm: /dev/md0 has been started with 2 drives.
mdadm: error opening /dev/md1: No such file or directory

You see, I have all the reason to be confused.

Quoted 2 lines

But in any case, this has exactly nothing to do with kernel. It's 100% userspace issues, I'd say distribution-specific issues.

At least I can agree.

	-`J'
#9 lore raw
Quoted 1 line · Jan Engelhardt

On Jan 28 2007 12:05, Michael Tokarev wrote:

[]

Quoted 3 lines
Quoted 1 line

Mdadm creates those nodes automa[tg]ically - man mdadm, search for --auto.

Note that mdadm -As _is_ run on FC6 boot.

See above -- man mdadm, search for --auto. -A = --assemble, -s = --scan.

Quoted 4 lines
Quoted 2 lines

In order for an md array to be started up on boot, it has to be specified in /etc/mdadm.conf. With proper DEVICE line in there. That's all.

That's how it is, and it does not work.

Sure. Because of this missing --auto flag.

Quoted 3 lines

openSUSE 10.2:
no mdadm.conf _at all_, /etc/init.d/boot.d/boot.md is chkconfig'ed _out_, _no_ md kernel module is loaded, and I still have all the /dev/md nodes.

And no udev. Or, alternatively, *all* md devices are created by some other script or somesuch. There's No Magic (tm).

[]

Quoted 4 lines
# mdadm -C /dev/md1 -e 1.0 -l 1 -n 2 /dev/sdb2 /dev/sdc2
mdadm: error opening /dev/md1: No such file or directory

Showstopper.

Nonsense. See above again, man mdadm, search for --auto.

[]

Quoted 1 line

You see, I have all the reason to be confused.

Yeah, this is quite... confusing.

It's all due to the way how mdadm iteracts with the kernel and how udev works - all together. The thing is - in order to assemble an array, proper device node has to be in place. But udev wont create it until array is assembled. Chicken&eggs problem.

Exactly due to this, the node(s) can be created with mdadm (with --auto option, whicih can be specified in mdadm.conf too), AND/OR with some startup script before invoking mdadm, AND/OR when the system isn't broken by udevd (with ol'good static /dev).

Quoted 2 lines
Quoted 2 lines

But in any case, this has exactly nothing to do with kernel. It's 100% userspace issues, I'd say distribution-specific issues.

..And each distribution uses its own kludge/workaround/solution for this stuff - which you demonstrated... ;)

/mjt

#10 lore raw
Quoted 5 lines · Michael Tokarev
Quoted 3 lines
Quoted 1 line

Mdadm creates those nodes automa[tg]ically - man mdadm, search for --auto.

Note that mdadm -As _is_ run on FC6 boot.

See above -- man mdadm, search for --auto. -A = --assemble, -s = --scan.

Oops, thank you. So....

  mdadm -A -s --auto=yes

did the right thing, and probably is what was intended. And now for tonight's 1000$ question: why does fedora don't do that, making me edit the mdadm -A -s line in /etc/rc.d/rc.sysvinit?

	-`J'
#11 lore raw
In reply to Marc Perkel
Quoted 12 lines · Marc Perkel

I'm a little stumped trying to set up raid 10. I set it up and it worked but after a reboot it forgets my raid setup.

Created 2 raid 1 arrays in md0 and md1 and that works and survives a reboot.

However - I created a raid 0 on /dev/md2 made up of /dev/md0 and /dev/md1 and it worked but it forgets it after I reboot. The device /dev/md2 fails to survive a reboot.

Quoted 4 lines

Created the /etc/mdadm.conf file but that doesn't seem to have made a difference.

What am I missing? Thanks in advance.

RAID-10 is not the same as RAID 0+1.

There's a linux-raid mailing list, the archives may assist in understanding this. Either use RAID-10 or add md2 to the mdadm.conf to get it started at boot. I suggest using RAID-10.

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot
#12 lore raw

Bill Davidsen wrote:
[]

Quoted 1 line

RAID-10 is not the same as RAID 0+1.

It is. Yes, there's separate module for raid10, but what it - basically - does is the same as raid0 module over two raid1 modules will do. It's just a bit more efficient (less levels, more room for optimisations), easy to use (you'll have single array instead of at least 3), and a bit more flexible; at the same way it's less widely tested...

But the end result is basically the same for both ways.

/mjt

#13 lore raw
Quoted 9 lines · Michael Tokarev

Bill Davidsen wrote:
[]

Quoted 1 line

RAID-10 is not the same as RAID 0+1.

It is. Yes, there's separate module for raid10, but what it - basically - does is the same as raid0 module over two raid1 modules will do. It's just a bit more efficient (less levels, more room for optimisations), easy to use (you'll have single array instead of at least 3), and a bit more flexible; at the same way it's less widely tested...

And most importantly, raid10 allows you to spread the array data over an odd number of devices while still having [at least] 2 copies of each block.

Hm, I really wished resizing was implemented for raid0 and raid10 too... ;)

	-`J'
#14 lore raw
In reply to Michael Tokarev
Quoted 4 lines · Michael Tokarev

Bill Davidsen wrote:
[]

Quoted 1 line

RAID-10 is not the same as RAID 0+1.

Quoted 8 lines

It is. Yes, there's separate module for raid10, but what it - basically - does is the same as raid0 module over two raid1 modules will do. It's just a bit more efficient (less levels, more room for optimisations), easy to use (you'll have single array instead of at least 3), and a bit more flexible; at the same way it's less widely tested...

But the end result is basically the same for both ways.

For values of "same" which exclude consideration of the disk layout, throughput, overhead, system administration, and use of spares. Those are different. But both methods do write multiple copies of ones and zeros to storage media.

Neil brown, 08/23/2005:
- A raid10 can consist of an odd number of drives (if you have a cabinet with, say, 8 slots, you can have 1 hot spare, and 7 drives in a raid10. You cannot do that with LVM (or raid0) over raid1).
- raid10 has a layout ('far') which theoretically can provide sequential read throughput that scales by number of drives, rather than number of raid1 pairs. I say 'theoretically' because I think there are still issues with the read-balancing code that make this hard to get in practice (though increasing the read-ahead seems to help).

After about 40 configurations tested, I can say that write performance is better as well, for any given stripe cache size up to 4x stripe size. I was looking at something else, but the numbers happen to be available.

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot