Skip to contentSkip to conversation

starting an array on Linux, advise needed

DISCUSSION Arkadiusz Miskiewicz 5 participants 6 messages Jan 16, 2004 – Jan 17, 2004 MD RAID

lore
Share & tools
lore
#0 lore raw

I've asked Neil about that previously but he didn't respond, so I'm asking here....

---------- Przekazana wiadomość ----------

Subject: starting an array on Linux, advise needed
Date: pon 15. grudnia 2003 13:35
From: Arkadiusz Miskiewicz <[email protected]>
To: [email protected]

Hi,

I need smallest possible version of tool for starting md array on Linux.
Right now I'm doing:

raid_start /dev/md0 /dev/hda1 /dev/hdb1 /dev/hdc1

and code does
fd_md = open(argv[1], O_RDWR, 0);

for (i = 2; i < argc; i++) {
if (stat(argv[i], &st)) {
bb_perror_msg("error stating raid builing device

 `%s'", argv[i]);
                       continue;
        }

        if (ioctl(fd_md, START_ARRAY, (unsigned long) st.st_rdev) == -1) {
                       bb_perror_msg("error starting raid device `%s:%s'",
argv[1], argv[i]);
                       continue;
         }
         return 0;
}

return 1;

so it starts using first working device and tries to assemble array from
other devices found in superblock.

Unfortunately there are some hard disk controllers which on 2.4 provide /dev/hdX devices while on 2.6 these are visible as /dev/sdX so even raid_start /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 won't work because it will try to use /dev/sda1 and hdb1+hdc1 (from superblock).

How simplest possible version should look like - is this below ok?

fd_md = open(argv[1], O_RDWR, 0);
ioctl(fd_md, SET_ARRAY_INFO, NULL);

for (i = 2; i < argc; i++) {
       stat(argv[i], &disk);
       ioctl(fd_md, ADD_NEW_DISK, &disk);
}

ioctl(fd_md, RUN_ARRAY, NULL);

and what if some of these ioctls fails - would be safe to do ioctl(fd_md, STOP_ARRAY, NULL)?

Thanks for help.
--
Arkadiusz Miśkiewicz CS at FoE, Wroclaw University of Technology
arekm.pld-linux.org AM2-6BONE, 1024/3DB19BBD, arekm(at)ircnet, PLD/Linux

-------------------------------------------------------

The other problem is that after switching from 2.4.21 to 2.6.1 I got:
FS: Mounted root (romfs filesystem) readonly.
+ insmod /lib/modumd: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27 les/2.6.1-1.csetraid5: measuring checksumming speed
20040116_0629smp 8regs : 916.000 MB/sec
/kernel/drivers/ 8regs_prefetch: 732.000 MB/sec
md/md.ko

Using    32regs    :   408.000 MB/sec
/lib/modules/2.6   32regs_prefetch:   400.000 MB/sec
.1-1.cset2004011   pIII_sse  :   996.000 MB/sec
6_0629smp/kernel   pII_mmx   :  1252.000 MB/sec
/drivers/md/md.k   p5_mmx    :  1308.000 MB/sec

o
+ insmod /libraid5: using function: pIII_sse (996.000 MB/sec)
/modules/2.6.1-1md: raid5 personality registered as nr 4
.cset20040116_06SCSI subsystem initialized
29smp/kernel/drivers/md/xor.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/md/xor.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/md/raid5.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/md/raid5.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/scsi_mod.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/scsi_mod.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/sd_mod.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/sd_mod.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/jbd/jbd.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/jbd/jbd.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/mbcache.kmd: could not lock unknown-block(8,34). o
Using /lib/momd: could not import unknown-block(8,34)!
dules/2.6.1-1.csmd: autostart unknown-block(0,2082) failed!
et20040116_0629smd: could not lock unknown-block(8,18).
mp/kernel/fs/mbcmd: could not import unknown-block(8,18)!
ache.ko
+ insmomd: autostart unknown-block(0,2066) failed!
d /lib/modules/2md: could not lock unknown-block(8,2).
.6.1-1.cset20040md: could not import unknown-block(8,2)!
116_0629smp/kernmd: autostart unknown-block(0,2050) failed!
el/fs/ext3/ext3.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/ext3/ext3.ko
+ raid_start /dev/md0 /dev/sdc2 /dev/sdb2 /dev/sda2
raid_start: error starting raid device `/dev/md0:/dev/sdc2': Invalid argument
raid_start: error starting raid device `/dev/md0:/dev/sdb2': Invalid argument
raid_start: error starting raid device `/dev/md0:/dev/sda2': Invalid argument
romfs: unable to read superblock
EXT3-fs: unable to read superblock

Kernel panic: VFS: Unable to mount root fs on md0

(kernel messages are unfortunately mixed with messages from userspace apps)

Signature · 6 lines
Arkadiusz Miśkiewicz    CS at FoE, Wroclaw University of Technology
arekm.pld-linux.org AM2-6BONE, 1024/3DB19BBD, arekm(at)ircnet, PLD/Linux
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
In reply to Arkadiusz Miskiewicz

I found this old message, it may help.
======================================
http://www.corpit.ru/mjt/mdctl.c - a very small utility similar to raidtools suite. I wrote this program to use it inside an initrd to bring raid arrays up at boot time - where the size of executable and simplicity is at premium. Differences between this implementation and raidtools are:

- size. When compiled with dietlibc, executable is about

    7kb on i386.

- it does not have mkraid utility and other fun stuff. Supported are
- raidstart
- raidstop
- raidhot{add,remove,generateerror)
- raidsetfaulty
i.e., only "simple" commands - just ioctls in fact.

- it uses NO config file like all the raidtools suite. I.e., it is more like mdadm in this respect. All parameters are accepted in the command line. This makes the utility handy inside read-only boot-time environment (you may e.g. parse md=... kernel parameter and convert it into mdctl commandline)

- `mdctl start' tries to bring an array using ALL components of the array given in the command line, until successeful (raidstart tries only first component, and if that is missing but other component(s) are available, operation will fail).

Usage:

mdctl start /dev/mdN /dev/hda1 /dev/hdb1 /dev/hdc1
equivalent to md=N,/dev/hda1,/dev/hdb1,/dev/hdc1 boot parameter, but you can't specify raid level (it will be read from the superblock), and kernel may choose to bring up another mdN (as indicated by super-minor)

mdctl {stop,readonly,ro,readwrite,rw} /dev/mdN

mdctl {add,remove,setfaulty,generaterror} /dev/mdN /dev/hda1

This tiny code released under GPL licence... in a hope it will be useful to someone else too, not only to me ;)

Enjoy.

/mjt

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Arkadiusz Miskiewicz
Sent: Friday, January 16, 2004 6:47 PM
To: [email protected]
Subject: Fwd: starting an array on Linux, advise needed

I've asked Neil about that previously but he didn't respond, so I'm asking here....

---------- Przekazana wiadomość ----------

Subject: starting an array on Linux, advise needed
Date: pon 15. grudnia 2003 13:35
From: Arkadiusz Miskiewicz <[email protected]>
To: [email protected]

Hi,

I need smallest possible version of tool for starting md array on Linux.
Right now I'm doing:

raid_start /dev/md0 /dev/hda1 /dev/hdb1 /dev/hdc1

and code does
fd_md = open(argv[1], O_RDWR, 0);

for (i = 2; i < argc; i++) {
if (stat(argv[i], &st)) {
bb_perror_msg("error stating raid builing device

 `%s'", argv[i]);
                       continue;
        }

        if (ioctl(fd_md, START_ARRAY, (unsigned long) st.st_rdev) == -1) {
                       bb_perror_msg("error starting raid device `%s:%s'",
argv[1], argv[i]);
                       continue;
         }
         return 0;
}

return 1;

so it starts using first working device and tries to assemble array from
other devices found in superblock.

Unfortunately there are some hard disk controllers which on 2.4 provide /dev/hdX devices while on 2.6 these are visible as /dev/sdX so even raid_start /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 won't work because it will try to use /dev/sda1 and hdb1+hdc1 (from superblock).

How simplest possible version should look like - is this below ok?

fd_md = open(argv[1], O_RDWR, 0);
ioctl(fd_md, SET_ARRAY_INFO, NULL);

for (i = 2; i < argc; i++) {
       stat(argv[i], &disk);
       ioctl(fd_md, ADD_NEW_DISK, &disk);
}

ioctl(fd_md, RUN_ARRAY, NULL);

and what if some of these ioctls fails - would be safe to do ioctl(fd_md, STOP_ARRAY, NULL)?

Thanks for help.
--
Arkadiusz Miśkiewicz CS at FoE, Wroclaw University of Technology
arekm.pld-linux.org AM2-6BONE, 1024/3DB19BBD, arekm(at)ircnet, PLD/Linux

-------------------------------------------------------

The other problem is that after switching from 2.4.21 to 2.6.1 I got:
FS: Mounted root (romfs filesystem) readonly.
+ insmod /lib/modumd: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27 les/2.6.1-1.csetraid5: measuring checksumming speed
20040116_0629smp 8regs : 916.000 MB/sec
/kernel/drivers/ 8regs_prefetch: 732.000 MB/sec
md/md.ko

Using    32regs    :   408.000 MB/sec
/lib/modules/2.6   32regs_prefetch:   400.000 MB/sec
.1-1.cset2004011   pIII_sse  :   996.000 MB/sec
6_0629smp/kernel   pII_mmx   :  1252.000 MB/sec
/drivers/md/md.k   p5_mmx    :  1308.000 MB/sec

o
+ insmod /libraid5: using function: pIII_sse (996.000 MB/sec) /modules/2.6.1-1md: raid5 personality registered as nr 4 .cset20040116_06SCSI subsystem initialized 29smp/kernel/drivers/md/xor.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/md/xor.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/md/raid5.ko Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/md/raid5.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/scsi_mod.ko Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/scsi_mod.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/sd_mod.ko Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/drivers/scsi/sd_mod.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/jbd/jbd.ko Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/jbd/jbd.ko
+ insmod /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/mbcache.kmd:
could not lock unknown-block(8,34).
o
Using /lib/momd: could not import unknown-block(8,34)! dules/2.6.1-1.csmd: autostart unknown-block(0,2082) failed!
et20040116_0629smd: could not lock unknown-block(8,18). mp/kernel/fs/mbcmd: could not import unknown-block(8,18)! ache.ko
+ insmomd: autostart unknown-block(0,2066) failed!
d /lib/modules/2md: could not lock unknown-block(8,2). .6.1-1.cset20040md: could not import unknown-block(8,2)! 116_0629smp/kernmd: autostart unknown-block(0,2050) failed! el/fs/ext3/ext3.ko
Using /lib/modules/2.6.1-1.cset20040116_0629smp/kernel/fs/ext3/ext3.ko
+ raid_start /dev/md0 /dev/sdc2 /dev/sdb2 /dev/sda2
raid_start: error starting raid device `/dev/md0:/dev/sdc2': Invalid argument
raid_start: error starting raid device `/dev/md0:/dev/sdb2': Invalid argument
raid_start: error starting raid device `/dev/md0:/dev/sda2': Invalid argument
romfs: unable to read superblock
EXT3-fs: unable to read superblock

Kernel panic: VFS: Unable to mount root fs on md0

(kernel messages are unfortunately mixed with messages from userspace apps)

Signature · 12 lines
Arkadiusz Miśkiewicz    CS at FoE, Wroclaw University of Technology
arekm.pld-linux.org AM2-6BONE, 1024/3DB19BBD, arekm(at)ircnet, PLD/Linux
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw
Quoted 3 lines · Guy

I found this old message, it may help.
======================================
http://www.corpit.ru/mjt/mdctl.c - a very small utility similar to raidtools

Note it will not work correctly in the situation described. Or at least it may not work. For the same reason as original code does not work: "renamed" devices in 2.4 vs 2.6.

With raidstart (and this utility follows it logic), it is the kernel who is reading the raid superblock, opening devices and assembles the array. Most important part is the superblock:
kernel will pick up any devices found there (by major/minor number). When booting 2.4 and 2.6, or when your devices are using dynamically-assigned numbers, that numbers may be different on next reboot, so kernel will not be able to find all required devices.

While mdadm tries to examine all superblocks and find all devices in between the ones listed on command line, in the superblock or ones that are really present on your system at any given time (--scan; depends on your mdadm.conf also).

So when you expect your device names/numbers to be instable between reboots, it's best to use mdadm together with --scan and proper mdadm.conf or with all possible devices listed in command line. Mdassemble by Luca Berra will be useful here too.

/mjt

P.S. Please trim original message somehow when replying.

#4 lore raw
Quoted 8 lines
Quoted 3 lines · Guy

I found this old message, it may help.
======================================
http://www.corpit.ru/mjt/mdctl.c - a very small utility similar to raidtools

Note it will not work correctly in the situation described. Or at least it may not work. For the same reason as original code does not work: "renamed" devices in 2.4 vs 2.6.

<snip>

I've been following this thread for a while and have now become a bit confused. What are the implications for an autostart array marked 'FD' in 2.6 ?? Will it start correctly? hmmm.... I think I missed something in the discussion..... How does one specify the raidtab under this circumstance if the device names are not stable.

Michael
[email protected]

#5 lore raw

raidtab?
Your sad devotion to that ancient software has not helped you conjure up the superblocks.

Read:
man mdadm

I have this line in my mdadm.conf:
DEVICE /dev/sd[abcdefghijklmnopqrstuvwxyz][12]
It handles the first 2 partitions of the first 26 SCSI disks! I only have 17 disks, most only have 1 partition, some have 2.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Michael
Sent: Friday, January 16, 2004 7:48 PM
To: [email protected]
Subject: Re: starting an array on Linux, advise needed

Quoted 4 lines
Quoted 3 lines · Guy

I found this old message, it may help.
======================================
http://www.corpit.ru/mjt/mdctl.c - a very small utility similar to

raidtools

Quoted 4 lines

Note it will not work correctly in the situation described. Or at least it may not work. For the same reason as original code does not work: "renamed" devices in 2.4 vs 2.6.

<snip>

I've been following this thread for a while and have now become a bit confused. What are the implications for an autostart array marked 'FD' in 2.6 ?? Will it start correctly? hmmm.... I think I missed something in the discussion..... How does one specify the raidtab under this circumstance if the device names are not stable.

Michael
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html