system fails to boot
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517
As a matter of fact, there are 2 bugs.
1)root=/dev/sda1, system boot randomly fails. Mostly, boot for 5 times and fails once. nash has a bug. Some of its functions misuse return value 0. Sometimes, 0 means timeout and no uevent available. Sometimes, 0 means nash gets an uevent, but the uevent isn't block-related (for exmaple, usb). If by coincidence, kernel tells nash that uevents are available, but kernel also set timeout, nash might stops collecting other uevents in queue if current uevent isn't block-related. I work out a patch for nash to fix it.
http://bugzilla.kernel.org/attachment.cgi?id=18858
2) root=LABEL=/, system always can't boot. initrd init reports
switchroot fails. Here is an executation branch of nash when booting:
(1) nash read /sys/block/sda/dev; Assume major is 8 (on my desktop)
(2) nash query /proc/devices with the major number; It found line "8 sd";
(3) nash use 'sd' to search its own probe table to find device (DISK) type for the device and add it to its own list;
(4) Later on, it probes all devices in its list to get filesystem labels;
scsi register "8 sd" always.
When major is 259, nash fails to find the device(DISK) type. I enables CONFIG_DEBUG_BLOCK_EXT_DEVT=y when compiling kernel, so 259 is picked up for device /dev/sda1, which causes nash to fail to find device (DISK) type.
To fixing issue 2), I create a patch for nash and another patch for kernel.
http://bugzilla.kernel.org/attachment.cgi?id=18859
http://bugzilla.kernel.org/attachment.cgi?id=18837
Below is the patch for kernel 2.6.28-rc4. It registers blkext, a new block device in proc/devices.
With 2 patches on nash and 1 patch on kernel, I boot my machines for dozens of times without failure.
Signed-off-by Zhang Yanmin <[email protected]>
Would you like to accept the kernel patch into your testing tree? Pls. do CC to me when replying as I couldn't subscribe LKML emails now.
---
block/genhd.c +1 -0
@@ -1028,6 +1028,7 @@ static int __init proc_genhd_init(void) {proc_create ("diskstats" ,0 ,NULL , &proc_diskstats_operations);proc_create ("partitions" ,0 ,NULL , &proc_partitions_operations); +register_blkdev (BLOCK_EXT_MAJOR ,"blkext" );return 0 ; }module_init (proc_genhd_init);
Tejun Heo
Acked-byQuoted 1 line · Yanmin
Signed-off-by Zhang Yanmin <[email protected]>
Signature · 1 line
tejun
Quoted 4 lines · Tejun Heo
Quoted 1 line · Yanmin
???Signed-off-by Zhang Yanmin <[email protected]>
Acked-by Tejun Heo <[email protected]>
Good debugging, thanks a lot Zhang. I've applied it, going upstream soon.
Signature · 1 line
Jens Axboe
Quoted 52 lines · Yanmin
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517As a matter of fact, there are 2 bugs.
1)root=/dev/sda1, system boot randomly fails. Mostly, boot for 5 times and fails once. nash has a bug. Some of its functions misuse return value 0. Sometimes, 0 means timeout and no uevent available. Sometimes, 0 means nash gets an uevent, but the uevent isn't block-related (for exmaple, usb). If by coincidence, kernel tells nash that uevents are available, but kernel also set timeout, nash might stops collecting other uevents in queue if current uevent isn't block-related. I work out a patch for nash to fix it.
http://bugzilla.kernel.org/attachment.cgi?id=188582) root=LABEL=/, system always can't boot. initrd init reports
switchroot fails. Here is an executation branch of nash when booting:
(1) nash read/sys/block/sda/dev; Assume major is 8 (on my desktop)
(2) nash query/proc/deviceswith the major number; It found line "8 sd";
(3) nash use 'sd' to search its own probe table to find device (DISK) type for the device and add it to its own list;
(4) Later on, it probes all devices in its list to get filesystem labels;
scsi register "8 sd" always.
When major is 259, nash fails to find the device(DISK) type. I enablesCONFIG_DEBUG_BLOCK_EXT_DEVT=ywhen compiling kernel, so 259 is picked up for device/dev/sda1, which causes nash to fail to find device (DISK) type.
To fixing issue 2), I create a patch for nash and another patch for kernel.
http://bugzilla.kernel.org/attachment.cgi?id=18859
http://bugzilla.kernel.org/attachment.cgi?id=18837Below is the patch for kernel 2.6.28-rc4. It registers blkext, a new block device in proc/devices.
With 2 patches on nash and 1 patch on kernel, I boot my machines for dozens of times without failure.
Signed-off-by Zhang Yanmin <[email protected]>
Would you like to accept the kernel patch into your testing tree? Pls. do CC to me when replying as I couldn't subscribe LKML emails now.---
block/genhd.c static int __init proc_genhd_init(void) · line 1034
Quoted 7 lines
block/genhd.c +1
-0@@ -1028,6 +1028,7 @@ static int __init proc_genhd_init(void) {proc_create ("diskstats" ,0 ,NULL , &proc_diskstats_operations);proc_create ("partitions" ,0 ,NULL , &proc_partitions_operations); +register_blkdev (BLOCK_EXT_MAJOR ,"blkext" );return 0 ; }module_init (proc_genhd_init);
It's procfs-specific init, what's up?
Quoted 5 lines · Alexey Dobriyan
Quoted 5 lines
Would you like to accept the kernel patch into your testing tree? Pls. do CC to me when replying as I couldn't subscribe LKML emails now.
---
Quoted 12 lines
block/genhd.c static int __init proc_genhd_init(void) · line 1034
Quoted 10 lines
block/genhd.c +1
-0@@ -1028,6 +1028,7 @@ static int __init proc_genhd_init(void) {proc_create ("diskstats" ,0 ,NULL , &proc_diskstats_operations);proc_create ("partitions" ,0 ,NULL , &proc_partitions_operations); +register_blkdev (BLOCK_EXT_MAJOR ,"blkext" );return 0 ; }module_init (proc_genhd_init);It's procfs-specific init, what's up?
Ah... right, better to move it to genhd_device_init(). Thanks.
Signature · 1 line
tejun
Quoted 19 lines · Tejun Heo
Quoted 5 lines · Alexey Dobriyan
Quoted 5 lines
Would you like to accept the kernel patch into your testing tree? Pls. do CC to me when replying as I couldn't subscribe LKML emails now.
---
Quoted 12 lines
block/genhd.c static int __init proc_genhd_init(void) · line 1034
Quoted 10 lines
block/genhd.c +1
-0@@ -1028,6 +1028,7 @@ static int __init proc_genhd_init(void) {proc_create ("diskstats" ,0 ,NULL , &proc_diskstats_operations);proc_create ("partitions" ,0 ,NULL , &proc_partitions_operations); +register_blkdev (BLOCK_EXT_MAJOR ,"blkext" );return 0 ; }module_init (proc_genhd_init);It's procfs-specific init, what's up?
Quoted 1 line
Ah... right, better to move it to
genhd_device_init(). Thanks.
Thanks. I thought nash reads /proc/devices so just added there. Below patch moves it to genhd_device_init. I tested it on my Nehalem machine.
---
block/genhd.c +2 -0
@@ -768,6 +768,8 @@ static int __init genhd_device_init(void bdev_map =kobj_map_init (base_probe, &block_class_lock);blk_dev_init (); +register_blkdev (BLOCK_EXT_MAJOR ,"blkext" ); +#ifndef CONFIG_SYSFS_DEPRECATED /* create top-level block dir */ block_depr =kobject_create_and_add ("block" ,NULL );
Quoted 38 lines · Alexey Dobriyan
Quoted 37 lines · Yanmin
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517As a matter of fact, there are 2 bugs.
1)root=/dev/sda1, system boot randomly fails. Mostly, boot for 5 times and fails once. nash has a bug. Some of its functions misuse return value 0. Sometimes, 0 means timeout and no uevent available. Sometimes, 0 means nash gets an uevent, but the uevent isn't block-related (for exmaple, usb). If by coincidence, kernel tells nash that uevents are available, but kernel also set timeout, nash might stops collecting other uevents in queue if current uevent isn't block-related. I work out a patch for nash to fix it.
http://bugzilla.kernel.org/attachment.cgi?id=188582) root=LABEL=/, system always can't boot. initrd init reports
switchroot fails. Here is an executation branch of nash when booting:
(1) nash read/sys/block/sda/dev; Assume major is 8 (on my desktop)
(2) nash query/proc/deviceswith the major number; It found line "8 sd";
(3) nash use 'sd' to search its own probe table to find device (DISK) type for the device and add it to its own list;
(4) Later on, it probes all devices in its list to get filesystem labels;
scsi register "8 sd" always.
When major is 259, nash fails to find the device(DISK) type. I enablesCONFIG_DEBUG_BLOCK_EXT_DEVT=ywhen compiling kernel, so 259 is picked up for device/dev/sda1, which causes nash to fail to find device (DISK) type.
To fixing issue 2), I create a patch for nash and another patch for kernel.
http://bugzilla.kernel.org/attachment.cgi?id=18859
http://bugzilla.kernel.org/attachment.cgi?id=18837
Quoted 3 lines
Quoted 2 lines
Below is the patch for kernel 2.6.28-rc4. It registers blkext, a new block device in proc/devices.
It's procfs-specific init, what's up?
nash (FC9 uses nash to explain the init script in initrd) reads /proc/devices to check the type of root device. When CONFIG_DEBUG_BLOCK_EXT_DEVT=y, the root device MAJOR is 259. Current kernel doesn't register block device for 259 in /proc/devices.
It's hard to explain in a short statement. Would you like to read it from http://bugzilla.kernel.org/show_bug.cgi?id=11899?
Quoted 15 lines · Yanmin
Quoted 14 lines · Alexey Dobriyan
Quoted 13 lines · Yanmin
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517As a matter of fact, there are 2 bugs.
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
2) root=LABEL=/, system always can't boot. initrd init reports
switchroot fails. Here is an executation branch of nash when booting:
(1) nash read/sys/block/sda/dev; Assume major is 8 (on my desktop)
(2) nash query/proc/deviceswith the major number; It found line "8 sd";
(3) nash use 'sd' to search its own probe table to find device (DISK) type for the device and add it to its own list;Quoted 8 lines
Quoted 8 lines
(4) Later on, it probes all devices in its list to get filesystem labels;
scsi register "8 sd" always.
When major is 259, nash fails to find the device(DISK) type. I enablesCONFIG_DEBUG_BLOCK_EXT_DEVT=ywhen compiling kernel, so 259 is picked up for device/dev/sda1, which causes nash to fail to find device (DISK) type.
To fixing issue 2), I create a patch for nash and another patch for kernel.
http://bugzilla.kernel.org/attachment.cgi?id=18859
http://bugzilla.kernel.org/attachment.cgi?id=18837
As for issue 2) with root=LABEL=/, I double-checked nash codes. That's really beyond what I imagined. I'm not an expert of nash. kernel might allocate MINOR number from MAX_EXT_DEVT (259) for any type of disk (cciss/ataraid/sd/ide/floppy/md ...), while nash assumes a MAJOR number is used by one of them exclusively. In the other hand, nash probes scsi/ide/usb serially as long as the type is DEV_TYPE_DISK. I won't say nash codes are not perfect, but nash is growing.
Peter Jones,
You maintain nash. What's your opinion?
-yanmin
Quoted 11 lines · Yanmin
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517
Hi Yanmin,
It seems I still have this problem with -rc6, both root=LABEL and root=/dev/sda8. I have replaced my nash/mkinitrd with patched ones.
Maybe a different issue just with the same appearance? Can you boot -rc6 smothly on your T61?
--
Thanks,
Jike
Quoted 14 lines · Jike Song
Quoted 11 lines · Yanmin
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517
Quoted 4 lines
Hi Yanmin,
It seems I still have this problem with -rc6, both root=LABEL and root=/dev/sda8. I have replaced my nash/mkinitrd with patched ones.
Did you recreate the initrd after updating nash/mkinitrd tools? What's the boot log?
Quoted 2 lines
Maybe a different issue just with the same appearance?
Maybe, but mostly little possibility.
Quoted 2 lines
Can you boot
-rc6 smothly on your T61?
Yes, I can boot 2.6.28-rc6 on my T61 and Nehalem machine. Without the patch, 2.6.28-rc6 boot randomly fails on T61.
Quoted 16 lines · Yanmin
Quoted 14 lines · Jike Song
Quoted 11 lines · Yanmin
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517
Quoted 5 lines
Quoted 4 lines
Hi Yanmin,
It seems I still have this problem with -rc6, both root=LABEL and root=/dev/sda8. I have replaced my nash/mkinitrd with patched ones.
Did you recreate the initrd after updating nash/mkinitrd tools?
Yes, I run "make install" at first, and found the failure of boot; Then I recreated initrd manually with mkinitrd/nash (patched by you), and booted -rc6 again - failed again.
Quoted 1 line
What's the boot log?
Sorry I don't have the laptop with me now, will post it thisevening(Ah, only possible if I can get netconsole work). Anyway,
your patches did work for me with -rc4, so I think I'll bisect it.
Signature · 2 lines
Thanks, Jike
Quoted 20 lines · Jike Song
Quoted 18 lines · Yanmin
Quoted 16 lines · Jike Song
Quoted 11 lines · Yanmin
Jens,
We run into system boot failure with kernel 2.6.28-rc. We found it on a couple of machines, including T61 notebook, nehalem machine, and another HPC NX6325 notebook. All the machines use FedoraCore 8 or FedoraCore 9. With kernel prior to 2.6.28-rc, system boot doesn't fail.
I debug it and locate the root cause. Pls. see
http://bugzilla.kernel.org/show_bug.cgi?id=11899
https://bugzilla.redhat.com/show_bug.cgi?id=471517Hi Yanmin,
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
It seems I still have this problem with -rc6, both root=LABEL and root=/dev/sda8. I have replaced my nash/mkinitrd with patched ones.
Did you recreate the initrd after updating nash/mkinitrd tools?
Yes, I run "make install" at first, and found the failure of boot; Then I recreated initrd manually with mkinitrd/nash (patched by you), and booted -rc6 again - failed again.
Would you like to write down the detailed steps?
Quoted 5 lines
Quoted 2 lines
What's the boot log?
Sorry I don't have the laptop with me now, will post it this
evening(Ah, only possible if I can get netconsole work). Anyway,
your patches did work for me with -rc4, so I think I'll bisect it.
That's good if the boot always fails. But when root=/dev/sdaXXX, system boot fails randomly.