2.6.15: mdrun, udev -- who creates nodes?

DISCUSSION Moshe Yudkowsky 10 participants 17 messages Jan 29, 2006 – Feb 10, 2006 MD RAID

lore

Share & tools

lore

Moshe Yudkowsky

author

2006-01-29 11:40 UTC #0 lore raw

Here's a snippet from /sbin/mdrun:

Quoted 3 lines

mdadm -A -a $NUMBER -f arr MDS $MD && setarr MDS $MD "started" # just to be sure
ln /dev/md/$NUMBER /dev/md$NUMBER 2>/dev/null

The first line is the problem I'm having, namely, that the arrays are being assembled with the wrong numbers. I'm about to try a replacement strategy that ignores /sbin/mdrun entirely.

The last line of the snippet shows /sbin/mdrun attempting to create a device that udev creates.

This won't crash anything, to the best of my knowledge; but which package is supposed to have "control" of creating /dev/md[0-9]*? udev has a rule (under compat-full) that creates this same symlink.

Signature · 14 lines

  Moshe Yudkowsky
  work: http://www.Disaggregate.com
  book: http://www.PebbleAndAvalanche.com


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Marco d'Itri

2006-01-29 12:35 UTC #1 lore raw

Quoted 3 lines · Moshe Yudkowsky

This won't crash anything, to the best of my knowledge; but which package is supposed to have "control" of creating /dev/md[0-9]*? udev has a rule (under compat-full) that creates this same symlink.

Indeed I do not understand why mdrun does this.

Anyway, the problem is that while udev is supposed to create the devices it cannot do it until the RAID array has been assembled, and the array cannot be assembled unless the device node is available.
This needs to be fixed in the kernel, but apparently nobody cares enough.

Signature · 13 lines

ciao,
Marco


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Moshe Yudkowsky

2006-01-29 14:49 UTC #2 lore raw

In reply to Moshe Yudkowsky

Quoted 7 lines · Marco d'Itri

Quoted 3 lines · Moshe Yudkowsky
This won't crash anything, to the best of my knowledge; but which package is supposed to have "control" of creating /dev/md[0-9]*? udev has a rule (under compat-full) that creates this same symlink.

Quoted 6 lines

Indeed I do not understand why mdrun does this.
Anyway, the problem is that while udev is supposed to create the devices it cannot do it until the RAID array has been assembled, and the array cannot be assembled unless the device node is available.
This needs to be fixed in the kernel, but apparently nobody cares enough.

There's an option on mdadm, --auto, that creates the device nodes if needed.

That's in theory. In practice, I find that "mdadm --assemble --scan --config=/etc/mdadm/mdadm.conf --auto" doesn't work. All three drives listed in the config file fail to start. I'm wondering just what the problem might be (config file requires UUID, not just ARRAY?); but it's clearly an issue for mdadm. And the inability to startup 2.6.15 using RAID drives that don't start at /dev/md/0 is quite disquieting.

Signature · 16 lines

Moshe Yudkowsky * [email protected] * www.pobox.com/~moshe
"I leave my soul to G-d, my service to my Prince, my good wishes to my 
friends,
and my love and charity to you all."  - Montrose, on the gallows, 21 May 
1650


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

linas

2006-01-30 20:42 UTC #3 lore raw

In reply to Moshe Yudkowsky

On Sun, Jan 29, 2006 at 08:49:56AM -0600, Moshe Yudkowsky was heard to remark:

Quoted 2 lines

That's in theory. In practice, I find that "mdadm --assemble --scan --config=/etc/mdadm/mdadm.conf --auto" doesn't work. All three drives

I'm surprised by this. Disk drives which have partitiions marked as being of type "Linux md" will have an md superblock on them. This superblock identifies what md device the disk partition belongs to. Manually specifying /etc/mdadm/mdadm.conf is dangerous, because it is likely to introduce erroneous information about the actual structure of the system.

The correct mode of operation should be this:
-- udev creates /dev/hda1, hda2, etc.
-- kernel scans /dev/hda1, looking for md superblock
-- kernel assembles devices according to info found in the superblocks -- udev creates /dev/md0, etc.

Disclaimer: this is a knee-jerk email; I have not actually thought about the issues.

--linas

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Marco d'Itri

2006-01-30 20:47 UTC #4 lore raw

In reply to Moshe Yudkowsky

Quoted 3 lines · linas

-- kernel scans /dev/hda1, looking for md superblock
-- kernel assembles devices according to info found in the superblocks -- udev creates /dev/md0, etc.

The problem is that some users and distributions build the drivers as modules and/or disable in-kernel auto-assembly.

Signature · 2 lines

ciao,
Marco

Jason Lunz

2006-01-31 16:40 UTC #5 lore raw

In reply to Moshe Yudkowsky

Quoted 5 lines · [email protected] said:

Quoted 3 lines
-- kernel scans /dev/hda1, looking for md superblock
-- kernel assembles devices according to info found in the superblocks -- udev creates /dev/md0, etc.
The problem is that some users and distributions build the drivers as modules and/or disable in-kernel auto-assembly.

Not only that, the raid developers themselves consider autoassembly deprecated.

http://article.gmane.org/gmane.linux.kernel/373620

With some platforms (partition types, really) it's never been supported in the first place.

Jason

linas

2006-01-31 19:26 UTC #6 lore raw

On Tue, Jan 31, 2006 at 04:40:46PM +0000, Jason Lunz was heard to remark:

Quoted 11 lines

Quoted 5 lines · [email protected] said:
Quoted 3 lines
-- kernel scans /dev/hda1, looking for md superblock
-- kernel assembles devices according to info found in the superblocks -- udev creates /dev/md0, etc.
The problem is that some users and distributions build the drivers as modules and/or disable in-kernel auto-assembly.
Not only that, the raid developers themselves consider autoassembly deprecated.
http://article.gmane.org/gmane.linux.kernel/373620

Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that this is one of the finest features of linux raid, so why remove it?

Speaking as a real-life sysadmin, with actual servers and actual failed disks, disk cables and disk controllers, this is a life-saving feature. Persistant naming of devices in Linux has long been a problem, and in this case, it seemed to work.

<story>
I once had an ide controller fail on an x86 board. I bought a new controller at the local store, recabled the disks, and booted. I was alarmed to find that the system was trying to mount /home as /usr, and /usr as /lib, etc. Turned out that /dev/hdc had gotten renamed as /dev/hde, etc. and had to go through a long, painful, rocket-science (yes, I *do* have a PhD) boot-floppy rescue to restore the system to working order.

I shudder to think what would have happened if RAID reconstruction had started based on faulty device names. Worse, as part of my rescue ops, I had to make multle copies of /etc/fstab, which resided on different disks (my root volume was raided), as well as the boot floppy, and each contained inconsistent info (needed to bootstrap my way back). Along the way, I made multiple errors in editing the /etc/fstab since I could not keep them straight; twiddling BIOS settings added to the confusion. If this had been /etc/raid.conf instead, with reconstruction triggered off of it, this could have been an absolute disaster.
</story>

Based on the above, real-life experience, my gut reaction is raid assembly based on config files is a bad idea. I don't understand how innocent, "minor" errors made by the sysadmin won't result in catastrophic data loss.

--linas

Molle Bestefich

2006-01-31 20:19 UTC #7 lore raw

Quoted 7 lines · [email protected] wrote:

Quoted 4 lines
Not only that, the raid developers themselves
consider autoassembly deprecated.
http://article.gmane.org/gmane.linux.kernel/373620
Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that this is one of the finest features of linux raid, so why remove it?

I *think* that "the raid developers" may be, for once, choosing words not-so-wisely when talking about "deprecating autoassembly".

Last time I heard that I choked as well, only to find out later that Neil's notion of what auto-assembly is differed substantially from my own.

Isn't there a faq/wiki somewhere where the official opinion on "autoassembly deprecation" and exactly what that means can go?

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Jason Lunz

2006-01-31 20:52 UTC #8 lore raw

Quoted 2 lines · Molle Bestefich

I *think* that "the raid developers" may be, for once, choosing words not-so-wisely when talking about "deprecating autoassembly".

You're right, I should be careful not to imply anything Neil didn't actually say. What he said was:

Quoted 1 line

From: Neil Brown <neilb <at> suse.de>

...

Quoted 4 lines

And if other partition styles wanted to add support for raid auto detect, tell them "no". It is perfectly possible and even preferable
to live without autodetect. We should support legacy usage (those
above) but should discourage any new usage.

which is a far cry from officially deprecating anything. My point was that the outlined solution for having udev create /dev/md* nodes could not work on all platforms because it required raid autodetect, and it looks unlikely that raid autodetect will be universally supported.

Jason

linas

2006-01-31 21:13 UTC #9 lore raw

I was reacting to the following:

Quoted 3 lines

Quoted 3 lines
From: Neil Brown <neilb <at> suse.de>
It is perfectly possible and even preferable
to live without autodetect.

I didn't see how it could be "preferable", and illustrated this with the sysadmin story.

--linas

Luca Berra

2006-01-31 21:44 UTC #15 lore raw

In reply to linas

Quoted 8 lines · linas

On Tue, Jan 31, 2006 at 04:40:46PM +0000, Jason Lunz was heard to remark:
Quoted 7 lines
Quoted 5 lines · [email protected] said:
Quoted 3 lines
-- kernel scans /dev/hda1, looking for md superblock
-- kernel assembles devices according to info found in the superblocks -- udev creates /dev/md0, etc.
The problem is that some users and distributions build the drivers as modules and/or disable in-kernel auto-assembly.

Quoted 7 lines

Quoted 4 lines
Not only that, the raid developers themselves consider autoassembly deprecated.
http://article.gmane.org/gmane.linux.kernel/373620
Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that this is one of the finest features of linux raid, so why remove it?

please, no, not again:
this particular horse has been beaten to death many times before.

the in kernel auto assembly should be removed for good (read the linux-raid list archive to understand why)

it should be replaced by auto assembly in user space (mdadm), which does not suffer from the problems that in-kernel has. neither it does suffer from a poor configuration file like the (unmaintained) raidtools had. and is much more cleaner and maintainable.

Signature · 17 lines

Luca Berra -- [email protected]
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Molle Bestefich

2006-01-31 21:58 UTC #10 lore raw

In reply to linas

Quoted 7 lines · linas

I was reacting to the following:
Quoted 3 lines
Quoted 3 lines
From: Neil Brown <neilb <at> suse.de>
It is perfectly possible and even preferable
to live without autodetect.
I didn't see how it could be "preferable", and illustrated this with the sysadmin story.

Certainly it isn't, and now I remember the point.

Quoting Luca Berra:

Quoted 3 lines

the in kernel auto assembly should be removed for good it should be replaced by auto assembly in user space (mdadm), which does not suffer from the problems that in-kernel has.

If people could start saying
"we need to get rid of in-kernel autodetection"
instead of
"we need to get rid of autodetection",
I think there would be much less confusion.

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Andy Smith

2006-02-01 09:27 UTC #11 lore raw

Quoted 10 lines · Molle Bestefich

Quoting Luca Berra:
Quoted 3 lines
the in kernel auto assembly should be removed for good it should be replaced by auto assembly in user space (mdadm), which does not suffer from the problems that in-kernel has.
If people could start saying
"we need to get rid of in-kernel autodetection"
instead of
"we need to get rid of autodetection",
I think there would be much less confusion.

And force us all to use initrd/initramfs :(

Signature · 2 lines

http://strugglers.net/wiki/Xen_hosting -- A Xen VPS hosting hobby
Encrypted mail welcome - keyid 0x604DE5DB

Luca Berra

2006-02-01 19:44 UTC #12 lore raw

Quoted 6 lines · Andy Smith

Quoted 5 lines · Molle Bestefich
Quoting Luca Berra:
Quoted 3 lines
the in kernel auto assembly should be removed for good it should be replaced by auto assembly in user space (mdadm), which does not suffer from the problems that in-kernel has.

Quoted 8 lines

Quoted 5 lines
If people could start saying
"we need to get rid of in-kernel autodetection"
instead of
"we need to get rid of autodetection",
I think there would be much less confusion.
And force us all to use initrd/initramfs :(

i believe that if you looked in this ml archives you'll know you can use the kernel command line to build an array, if you don't want to use initrd/initramfs.

Signature · 6 lines

Luca Berra -- [email protected]
       Communication Media & Services S.r.l.
/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

Andy Smith

2006-02-01 22:14 UTC #13 lore raw

Quoted 8 lines · Luca Berra

Quoted 7 lines · Andy Smith
Quoted 5 lines · Molle Bestefich
If people could start saying
"we need to get rid of in-kernel autodetection"
instead of
"we need to get rid of autodetection",
I think there would be much less confusion.

Quoted 5 lines

Quoted 2 lines
And force us all to use initrd/initramfs :(
i believe that if you looked in this ml archives you'll know you can use the kernel command line to build an array, if you don't want to use initrd/initramfs.

I am aware and I do. However that is in-kernel. If it needs userland as has been stated here then it requires initrd, no?

Signature · 2 lines

http://strugglers.net/wiki/Xen_hosting -- A Xen VPS hosting hobby
Encrypted mail welcome - keyid 0x604DE5DB

Neil Brown

2006-02-02 01:27 UTC #14 lore raw

In reply to Andy Smith

Quoted 6 lines · On Wednesday February 1, [email protected] wrote:

Quoted 5 lines · Molle Bestefich
Quoting Luca Berra:
Quoted 3 lines
the in kernel auto assembly should be removed for good it should be replaced by auto assembly in user space (mdadm), which does not suffer from the problems that in-kernel has.

Quoted 7 lines

Quoted 5 lines
If people could start saying
"we need to get rid of in-kernel autodetection"
instead of
"we need to get rid of autodetection",
I think there would be much less confusion.
And force us all to use initrd/initramfs :(

Don't think of it as 'forcing' you. Think of it as encouraging you to be an early adopter...
I suspect it won't be many years (maybe 1) before you have to have an initramfs to boot. There will be a standard one distributed with the kernel, but it will (hopefully) be fairly easy to add stuff.

So get with the program. Initramfs is *easy*. Just DO it.

(mdadm-2.2 comes with a script to make an initramfs which "works for me").

NeilBrown

Bill Davidsen

2006-02-10 22:46 UTC #16 lore raw

In reply to linas

Quoted 51 lines · linas

On Tue, Jan 31, 2006 at 04:40:46PM +0000, Jason Lunz was heard to remark:
Quoted 8 lines
Quoted 5 lines · [email protected] said:
Quoted 5 lines
-- kernel scans /dev/hda1, looking for md superblock
-- kernel assembles devices according to info found in the superblocks -- udev creates /dev/md0, etc.
Quoted 10 lines
Quoted 4 lines
The problem is that some users and distributions build the drivers as modules and/or disable in-kernel auto-assembly.
Not only that, the raid developers themselves consider autoassembly deprecated.
http://article.gmane.org/gmane.linux.kernel/373620
Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that this is one of the finest features of linux raid, so why remove it?
Speaking as a real-life sysadmin, with actual servers and actual failed disks, disk cables and disk controllers, this is a life-saving feature. Persistant naming of devices in Linux has long been a problem, and in this case, it seemed to work.
<story>
I once had an ide controller fail on an x86 board. I bought a new controller at the local store, recabled the disks, and booted. I was alarmed to find that the system was trying to mount /home as /usr, and /usr as /lib, etc. Turned out that /dev/hdc had gotten renamed as /dev/hde, etc. and had to go through a long, painful, rocket-science (yes, I *do* have a PhD) boot-floppy rescue to restore the system to working order.
I shudder to think what would have happened if RAID reconstruction had started based on faulty device names. Worse, as part of my rescue ops, I had to make multle copies of /etc/fstab, which resided on different disks (my root volume was raided), as well as the boot floppy, and each contained inconsistent info (needed to bootstrap my way back). Along the way, I made multiple errors in editing the /etc/fstab since I could not keep them straight; twiddling BIOS settings added to the confusion. If this had been /etc/raid.conf instead, with reconstruction triggered off of it, this could have been an absolute disaster.
</story>

Quoted 5 lines

Based on the above, real-life experience, my gut reaction is raid assembly based on config files is a bad idea. I don't understand how innocent, "minor" errors made by the sysadmin won't result in catastrophic data loss.

I fear you don't understand how the auto detect and assemble works. Or more to the point what it does, since how it works is somewhat more complex. If you use partitions and UUID, you can just plug in the drives any old place and they will be found and recognised in spite of that. As long as you have a boot drive where the BIOS will use it, mdadm with find your stuff and put it together correctly. Neil does more magic than harry Potter!

I know someone who gave this a real life test, although I'd not say who.

Signature · 15 lines

bill davidsen <[email protected]>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel