2.6.12 initrd module loading seems parallel on bootup
The kernel is panic'ed trying to mount even before the sym53c8xx driver has completed SCSI bus detection, the MD driver does not find any volumes (since neither the cpqarray.ko or sym53c8xx.ko have registered any disks on the system yet).
I have also witnessed (when booting 2.6.12.1) what appeared to be ext3.ko load failure, I think due to jbd.ko not having chance to register its exported symbols before ext3.ko needs to find them.
The broken sequence on 2.6.12 (excuse any errors, this has been typed in not pasted) it would be handy to allow Shift-Page Up to work from a panic, "unable to mount root" or "unable to exec init" if that were possible.
You can see the out of place "input: AT Translated Set 2 keyboard on isa0060/serio0" and "Compaq SMART2 Driver (v 2.6.0)" the elapsed time to print all these messages is < 1.5 seconds.
RedHat nash version 4.1.18 stable
Mounting /proc filesystem
Mounting sysfs
Creating /dev
Starting udev
input: AT Translated Set 2 keyboard on isa0060/serio0 Loading scsi_mod.ko module
Loading sd_mod.ko module
SCSI subsystem initialized
Loading scsi_transport_spi.ko module
Loading sym53c8xx.ko module
Loading cpqarray.ko module
Loading raid1.ko module
Loading jbd.ko module
Loading ext3.ko module
ACPI: PCI interrupt 0000:05:04.0[A] -> GSI 22 (level, low) -> IRQ 169
sym0: <896> rev 0x5 at pci 0000:05:04.0 irq 177
Compaq SMART2 Driver (v 2.6.0)
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
sym0: SCSI BUS has been reset.
scsi0 : sym-2.2.0
md: raid1 personality registered as nr 3
md: Autodetecting RAID arrays.
md: ... autorun DONE.
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
Creating root device
Mounting root filesystem
mount: error 19 mounting ext3
mount: error 2 mounting none
Switching to new root
switchroot: mount failed: 22
umount /initrd /dev failed: 2
kernel panic - not syncing: Attempted to kill init! <0>Rebooting in 240 seconds...
The following is what a healthy 2.6.11.12 bootup on the same system looks like:
Freeing unused kernel memory: 232k freed
SCSI subsystem initialized
Compaq SMART2 Driver (v 2.6.0)
cpqarray: Device 0xae10 has been found at bus 6 dev 0 func 0
ACPI: PCI interrupt 0000:06:00.0[A] -> GSI 30 (level, low) -> IRQ 169
cpqarray: Finding drives on ida0 (Smart Array 3200) cpqarray ida/c0d0: blksz=512 nr_blks=35561280
cpqarray ida/c0d1: blksz=512 nr_blks=71130720
cpqarray ida/c0d2: blksz=512 nr_blks=17764320
cpqarray ida/c0d3: blksz=512 nr_blks=35838720
ida/c0d0: p1 p2
ida/c0d1: p1
ida/c0d2: p1 p2
ida/c0d3: p1
ACPI: PCI interrupt 0000:05:04.0[A] -> GSI 22 (level, low) -> IRQ 177
sym0: <896> rev 0x5 at pci 0000:05:04.0 irq 177
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
sym0: SCSI BUS has been reset.
scsi0 : sym-2.1.18n
Vendor: COMPAQ Model: BD30088279 Rev: HPB0
Type: Direct-Access ANSI SCSI revision: 03
sym0:0:0: tagged command queuing enabled, command queue depth 16. target0:0:0: Beginning Domain Validation
sym0:0: wide asynchronous.
sym0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25.0 ns, offset 31) target0:0:0: Ending Domain Validation
SCSI device sda: 585937500 512-byte hdwr sectors (300000 MB) SCSI device sda: drive cache: write through
SCSI device sda: 585937500 512-byte hdwr sectors (300000 MB) SCSI device sda: drive cache: write through
sda: sda1 sda2 sda3 sda4
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Vendor: COMPAQ Model: BD30088279 Rev: HPB0
Type: Direct-Access ANSI SCSI revision: 03
sym0:4:0: tagged command queuing enabled, command queue depth 16. target0:0:4: Beginning Domain Validation
sym0:4: wide asynchronous.
sym0:4: FAST-40 WIDE SCSI 80.0 MB/s ST (25.0 ns, offset 31) target0:0:4: Ending Domain Validation
SCSI device sdb: 585937500 512-byte hdwr sectors (300000 MB) SCSI device sdb: drive cache: write through
SCSI device sdb: 585937500 512-byte hdwr sectors (300000 MB) SCSI device sdb: drive cache: write through
SCSI device sdb: drive cache: write through
sdb: sdb1 sdb2 sdb3 sdb4
Attached scsi disk sdb at scsi0, channel 0, id 4, lun 0
ACPI: PCI interrupt 0000:05:04.1[B] -> GSI 21 (level, low) -> IRQ 185 sym1: <896> rev 0x5 at pci 0000:05:04.1 irq 185 sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking sym1: SCSI BUS has been reset. scsi1 : sym-2.1.18n md: raid1 personality registered as nr 3 md: Autodetecting RAID arrays. md: invalid raid superblock magic on sda4 md: sda4 has invalid sb, not importing! md: invalid raid superblock magic on sdb4 md: sdb4 has invalid sb, not importing! md: autorun ... md: considering sdb3 ... md: adding sdb3 ... md: sdb2 has different UUID to sdb3 md: sdb1 has different UUID to sdb3 md: adding sda3 ... md: sda2 has different UUID to sdb3 md: sda1 has different UUID to sdb3 md: created md2 md: bind<sda3> md: bind<sdb3> md: running: <sdb3><sda3> raid1: raid set md2 active with 2 out of 2 mirrors md: considering sdb2 ... md: adding sdb2 ... md: sdb1 has different UUID to sdb2 md: adding sda2 ... md: sda1 has different UUID to sdb2 md: created md0 md: bind<sda2> md: bind<sdb2> md: running: <sdb2><sda2> raid1: raid set md0 active with 2 out of 2 mirrors md: considering sdb1 ... md: adding sdb1 ... md: adding sda1 ... md: created md1 md: bind<sda1> md: bind<sdb1> md: running: <sdb1><sda1> raid1: raid set md1 active with 2 out of 2 mirrors md: ... autorun DONE. md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE. md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Hope this helps
Signature · 1 line
Darryl L. Miles
Well, although I cannot comment on your exact situation, I guess I see the same problem here. When checking out 2.6.12 for the first time, my little early-userspace environment had some serious trouble configuring the scsi|md|lvm setup which worked flawlessly before. The plan was to load required modules for disks, then md, bring up the arrays and activate the LVs. But it takes several seconds (!) until device nodes appear after modprobe has terminated correctly. For me, the quick ugly fix was to introduce a delay loop in the bash scripts which waits for important device nodes before proceeding, similar to that:
function wait_for_node{} (
echo -en "Waiting for ${1} to appear..."
while [ ! -e ${1} ]
do
echo -en "."
sleep 1s
done
echo " done.")
This one can be called with any node important for the next initialization step. In my case, I wait for disk partitions to appear before RAID initialization:
wait_for_node /dev/sda1
You could wait for your root partition to come up, instead of running into the vfs panic while trying to mount it. But that won't solve the original problem, I fear...
Yours,
Chris
Quoted 3 lines · Darryl L. Miles
[ modules getting loaded out-of-order and in parallel from initrd ]
On June 7, Martin Wilck reported:
Quoted 7 lines
It turned out to be a problem with Red Hat's nash that didn't check the returned pid in it's
wait4()call and thus ended up insmod'ing mutliple modules simultaneously, leading to "Unkown symbol" errors. Yuck, it took me a day figure that out.That bug is fixed in redhat's "mkinitrd" package 4.2.0.3-1 and later, but that package is currently only in Fedora's "Development" tree.
I'd like to know what changed in the kernel to make nash's behaviour change. Martin, did you work that out?
Quoted 2 lines · Andrew Morton
I'd like to know what changed in the kernel to make nash's behaviour change. Martin, did you work that out?
See http://lkml.org/lkml/2005/1/17/132
Signature · 1 line
pozsy
Quoted 15 lines · Andrew Morton
Quoted 3 lines · Darryl L. Miles
[ modules getting loaded out-of-order and in parallel from initrd ]
On June 7, Martin Wilck reported:
Quoted 5 lines
It turned out to be a problem with Red Hat's nash that didn't check the returned pid in it's
wait4()call and thus ended up insmod'ing mutliple modules simultaneously, leading to "Unkown symbol" errors. Yuck, it took me a day figure that out.That bug is fixed in redhat's "mkinitrd" package 4.2.0.3-1 and later, but that package is currently only in Fedora's "Development" tree.
Found the thread: http://www.ussg.iu.edu/hypermail/linux/kernel/0506.0/1556.html
This works for me:
wget http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/SRPMS/udev-058-1.src.rpm
rpmbuild --rebuild udev-058-1.src.rpm
rpm -Uvh /usr/src/redhat/RPMS/i386/udev-058-1.i386.rpm
wget http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/SRPMS/mkinitrd-4.2.17-1.src.rpm
rpmbuild --rebuild mkinitrd-4.2.17-1.src.rpm
rpm -Uvh /usr/src/redhat/RPMS/i386/mkinitrd-4.2.17-1.i386.rpm
Thanks
Signature · 1 line
Darryl L. Miles
Quoted 2 lines · Andrew Morton
I'd like to know what changed in the kernel to make nash's behaviour change. Martin, did you work that out?
Quoting from Prarit Bhargava's email (http://lkml.org/lkml/2005/6/14/149)
-
"The issue is that David Howells posted a patch that changed the behaviour of kallsyms/insmod/rmmod sometime ago. The patch *is correct* in what it does, however, the patch requires that /sbin/sh must be aware of pid returns by wait() - http://lkml.org/lkml/2005/1/17/132"
IIRC this issue was same as the one faced by Martin.
Parag
Quoted 1 line · Pozsár Balázs
The patch was against 2.6.11-rc1, has it not been adapted into mainline? As far as I can see, most of the hunks were already applied, and some parts of the code have evolved further since. BTW, I never had this problem with any 2.6.11 until .12 so far. Thanks for your pointer!
Quoted 1 line · Darryl L. Miles
Andrew Morton wrote:
[snip]
Quoted 5 lines
Found the thread: http://www.ussg.iu.edu/hypermail/linux/kernel/0506.0/1556.html
This works for me:
[snip]
Thanks, but still no go, as I am using Gentoo and a totally bloated self-grown glibc initramfs environment. Whenever I boot 2.6.11.12, there is no problem whatsoever about modprobe returning before device nodes are created. Any ideas? I changed udev versions back and forth between 058 and 056, did not make any difference either...
I can send any information of interest later tonight, but have to run for now. Thanks in advance!
Quoted 12 lines · Christian Trefzer
Quoted 1 line · Darryl L. Miles
Andrew Morton wrote:
[snip]
Quoted 6 lines
Found the thread:
http://www.ussg.iu.edu/hypermail/linux/kernel/0506.0/1556.htmlThis works for me:
Quoted 7 lines
[snip]
Thanks, but still no go, as I am using Gentoo and a totally bloated self-grown glibc initramfs environment. Whenever I boot 2.6.11.12, there is no problem whatsoever about modprobe returning before device nodes are created. Any ideas? I changed udev versions back and forth between 058 and 056, did not make any difference either...
Well, I'm running Gentoo Linux x86_64 with a 2.6.12 kernel. I didn't experience any problems. I use initrd because most filesystems are located on LVM partitions.
I build my kernel with the following command:
genkernel --no-mrproper --oldconfig --lvm2 --udev all
Work like a charm.
Regards,
Toon.
Are we sure we are not talking about two different problems here.
I'm using RedHat and mkinitrd, in the initrd image there is already a skeleton /dev tree that contains my real-root device. udev also exists in the initrd image. I don't think any /dev device magic was necessary for me too mount root.
It is not clear Chris which tools you are using on initrd, standard Gentoo methods or a home brew setup ? What shell is calling modprobe ? Can you confirm at what point you are seeing modprobe exit ?
* Immediatly (before device driver has detected hardware and reported / registered its findings). This is the symptom I was seeing, but the cause was incorrect shell handling. * After detection but before device node creation.
* After device node creation.
FYI - This looks like the snippet from nash.c
@@ -403,7 +450,7 @@
int otherCommand(char * bin, char * cmd, char * end, int doFork) {
char ** args;
char ** nextArg;
- int pid;
+ int pid, wpid;
int status;
char fullPath[255];
static const char * sysPath = PATH;@@ -479,10 +526,20 @@
close(stdoutFd);
- wait4(-1, &status, 0, NULL);
- if (!WIFEXITED(status) || WEXITSTATUS(status)) {
- printf("ERROR: %s exited abnormally!\n", args[0]);
- return 1;
+ for (;;) {
+ wpid = wait4(-1, &status, 0, NULL);
+ if (wpid == -1) {
+ printf("ERROR: Failed to wait for process %d\n", wpid);
+ }
+
+ if (wpid != pid)
+ continue;
+
+ if (!WIFEXITED(status) || WEXITSTATUS(status)) {
+ printf("ERROR: %s exited abnormally with value %d ! (pid %d)\n", args[0], WEXITSTATUS(status), pid);
+ return 1;
+ }
+ break;
}
}Signature · 1 line
Darryl L. Miles
Hi Darry,
first of all thanks for your questions, and sorry for answering late, I've been away for some time.
Quoted 6 lines · Darryl L. Miles
Are we sure we are not talking about two different problems here.
I'm using RedHat and mkinitrd, in the initrd image there is already a skeleton /dev tree that contains my real-root device. udev also exists in the initrd image. I don't think any /dev device magic was necessary for me too mount root.
It is not clear Chris which tools you are using on initrd, standard Gentoo methods or a home brew setup ? What shell is calling modprobe ? Can you confirm at what point you are seeing modprobe exit ?
I am using a self-made initramfs setup using the "standard" binaries from the live system, usable for instance as rescue system with tools identical to anything I have installed, including bash as shell for the init script, udev, module-init-tools, lvm2, mdadm, and so on and so forth. There are only the most basic device nodes in the supplied /dev tree, like null, console, etc. Unfortunately, I don't really "see" modprobe exit, it just takes less time until the "modprobe $mod && e_done" prints its "done." yet related device nodes are still missing. Waiting for any required nodes fixes the problem so far, but maybe bash-3.00 is having a similar issue as nash. I'll try using sash or busybox ASAP to check if the problem is somewhere else.
The funny thing is that with kernels up to 2.6.11.12, I don't see this problem. The patch mentioned further up this thread was against 2.6.11-rc1 and should have been applied long before I even touched 2.6.11, so I just don't understand why I have problems now.
Quoted 1 line
* Immediatly (before device driver has detected hardware and reported / registered its findings). This is the symptom I was seeing, but the cause was incorrect shell handling. * After detection but before device node creation.
Well, so far I cannot distinguish both of these cases, but it definitely is _not_ the third one:
Quoted 3 lines
* After device node creation.
Otherwise I would not run into problems like mdadm failing trying to assemble arrays while disk nodes are still missing in action.
Quoted 3 lines
FYI - This looks like the snippet from
nash.c[snip]
Thanks for the patch, I'll look into similarities in the bash code if the shell really is the cause of my trouble.
The exact patch for kernel/module.c that was marked for 2.6.11-rcX hit general release in patch-2.6.12.
What $PID is bash running as ? Martin's comments on this are seem most relevant.
Can you build your own (2 pages of code) init process, that does something along the lines of
* gracefully handles SIGCHLD
* forks
* executes bash
* waits for bash to exit much like the patch does
So bash is not running a pid 1. While nash is expected to run as init, bash is not, so fixing bash might also break it (its a complex beast).
Signature · 1 line
Darryl L. Miles
Quoted 2 lines · Darryl L. Miles
The exact patch for
kernel/module.cthat was marked for 2.6.11-rcX hit general release in patch-2.6.12.
That explains a lot. Thanks for clarification : )
Quoted 1 line
What $PID is bash running as ? Martin's comments on this are seem most relevant.
The initramfs contains /init which is a shell script, while /bin/sh is bash. So I figure bash is running with PID 1, but I'll verify that by adding echo $$ to the init script.
Quoted 8 lines
Can you build your own (2 pages of code) init process, that does something along the lines of
* gracefully handles SIGCHLD
* forks
* executes bash
* waits for bash to exit much like the patch doesSo bash is not running a pid 1. While nash is expected to run as init, bash is not, so fixing bash might also break it (its a complex beast).
I'm not (yet) too much into C programming, so I'm afraid I won't be able to do that. As this problem arises only due to my own early-userspace setup, I'd rather try using different shells for init-as-shell-script, and maybe do some not-so-clean hacks to get the bigger picture.
As this is not a kernel issue, I'd rather take this one off LKML and write a success report once appropriate.
Thanks for your help to everyone involved : )