Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow
----- Original Message ----
Quoted 116 lines
From: Martin Knoblauch <[email protected]>
To: Andrew Morton <[email protected]>
Cc: Mike Galbraith <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Kay Sievers <[email protected]>; [email protected]; Jesse Barnes <[email protected]>; Matthew Wilcox <[email protected]>
Sent: Wednesday, May 27, 2009 11:14:06 AM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow----- Original Message ----
Quoted 3 lines
From: Andrew Morton
To: Martin Knoblauch
Cc: Mike Galbraith ; [email protected]; [email protected];[email protected]; [email protected]; Kay Sievers ; [email protected]; Jesse Barnes
; Matthew WilcoxQuoted 29 lines
Sent: Wednesday, May 27, 2009 8:31:02 AM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowQuoted 6 lines · On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch wrote:
----- Original Message ----
Quoted 3 lines
From: Mike Galbraith
To: Martin Knoblauch
Cc: Andrew Morton ; [email protected];[email protected]; [email protected]; [email protected]
Quoted 10 lines
Quoted 10 lines
Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slowQuoted 1 line · Martin Knoblauch
just to bring this back to my problem :-)
Good idea :-)
Quoted 1 line
Last week I reported that the "new" sysfs entry in
/proc/mountsalreadycomes
Quoted 4 lines
Quoted 4 lines
out of initrd. Does this ring a bell?
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Nope, no bells.
The only thing I can suggest is that you try a bisection.
-Mike
OK, so I finally managed to bisect the issue down to the following commit.
Not much that I can say about it. Someone else suggested that it might all be
a
Quoted 25 lines
question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.
Quoted 10 lines
|commit 1120f8b8169fb2cb51219d326892d963e762edb6 |Author: Stephen Hemminger |Date: Thu Dec 18 09:17:16 2008 -0800 | | PCI: handle long delays in VPD access | | Accessing the VPD area can take a long time. The existing | VPD access code fails consistently on my hardware. There are comments |Quoted 9 lines
| Change the access routines to: | * use a mutex rather than spinning with IRQ's disabled and lock held | * have a much longer timeout | * call cond_resched while spinning | | Signed-off-by: Stephen Hemminger | Reviewed-by: Matthew Wilcox | Signed-off-by: Jesse BarnesQuoted 4 lines
So afacit what's happening is that the above change caused one of your PCI devices to take a very long time to initialise, yes? Was it the CCISS driver?
the whole thing is not understood. I mentioned CCISS only because it is the most visible difference difference between my two test platforms.
Quoted 6 lines
If you add "printk.time=y" to the kernel boot command line then you'll get timestamped boot messages which will make it easier to determine where the time was consumed. Adding `initcall_debug' to the boot line will help us delve further into the delay, assuming that the offending driver is build into vmlinux (which it might not be).
added both options. "dmesg" output from both is appended. The initcall timings vary in both directions. For CCISS, they are actually faster for the "bad" case.
Quoted 3 lines
Either way, it would be useful to know which driver the above change broke.
agreed.
Quoted 4 lines
Once we know that, the questions is: doe sthe driver still work? If so, then presumably the hardware if behaving unexpectedly, or in a way which we're failing to cope with.
if it is CCISS, I can definitely say that it does work OK. As far as I can see, the whole system works OK, besides the duplicate sysfs line coming out of initrd.
Quoted 5 lines
Or perhaps that patch was simply buggy.
btw, I don't agree that this report should be closed for "fuzziness"! AFACIT the regression clearly and reproducibly occurs on one of your machines, yes? That ain't fuzzy!
Quoted 7 lines
frankly, I will stop caring about the DL380s before 2.6.31 gets released. My production kernels are not affected and the hotplug scripts can easily be fixed for testting. So, my interest is more curiosity. And the day-job does not really justify spending much more time on it. Aren't day-jobs annoying ...
Cheers
Martin
FWIW, I compiled the CCISS driver into the kernel. This makes the second "/sys" line in /proc/mounts go away, dmesg attached. But does it prove anything? The initialization of the CCISS hardware now happens about 2 seconds earlier in the bootup sequence. Does this hint to a problem with CCISS, or just confirms that the whole issue is really timing dependent? Anyway, I add Mike to CC.
Cheers
Martin
Quoted 2 lines · On Wed, 27 May 2009 04:25:57 -0700 (PDT) Martin Knoblauch <[email protected]>…
FWIW, I compiled the CCISS driver into the kernel. This makes the second "/sys" line in
/proc/mountsgo away, dmesg attached. But does it prove anything? The initialization of the CCISS hardware now happens about 2 seconds earlier in the bootup sequence. Does this hint to a problem with CCISS, or just confirms that the whole issue is really timing dependent? Anyway, I add Mike to CC.
It seems that the PCI change caused timing changes which triggered a udev/sysfs/whatever problem, which manifests as the duplicated /proc/mounts entry to turn up.
What we don't know (afaik) is why the kernel permitted two entries in /proc/mounts. That might be a bug.
It could be that if dual /proc/mounts problem gets fixed, everything works OK - by intent or by accident, the userspace startup scripts may then work acceptably.
I think Al asked you a few questions around the behaviour of mount(8) and the mount syscall, so we could delve further into why /proc/mounts is getting mucked up. Did you end up running those tests?
Quoted 13 lines · Andrew Morton
Quoted 2 lines · On Wed, 27 May 2009 04:25:57 -0700 (PDT) Martin Knoblauch <[email protected]>…
FWIW, I compiled the CCISS driver into the kernel. This makes the second "/sys" line in
/proc/mountsgo away, dmesg attached. But does it prove anything? The initialization of the CCISS hardware now happens about 2 seconds earlier in the bootup sequence. Does this hint to a problem with CCISS, or just confirms that the whole issue is really timing dependent? Anyway, I add Mike to CC.It seems that the PCI change caused timing changes which triggered a udev/sysfs/whatever problem, which manifests as the duplicated
/proc/mountsentry to turn up.What we don't know (afaik) is why the kernel permitted two entries in
/proc/mounts. That might be a bug.
Quoted 7 lines
It could be that if dual
/proc/mountsproblem gets fixed, everything works OK - by intent or by accident, the userspace startup scripts may then work acceptably.I think Al asked you a few questions around the behaviour of mount(8) and the mount syscall, so we could delve further into why
/proc/mountsis getting mucked up. Did you end up running those tests?
I expect the duplicate comes from a left-over mount in initramfs which isn't a duplicate in the sense of a bug in vfs or mount or anything. I guess, it is just still mounted in the initial kernel rootfs, below the root from the disk. It could be that a umount from initramfs did go wrong because of a changed timing.
Thanks,
Kay
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
----- Original Message ----
Quoted 7 lines
From: Kay Sievers <[email protected]>
To: Andrew Morton <[email protected]>
Cc: Martin Knoblauch <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Wednesday, May 27, 2009 10:56:45 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowOn Wed, May 27, 2009 at 22:31, Andrew Morton wrote:
Quoted 25 lines
Quoted 4 lines
Quoted 1 line · On Wed, 27 May 2009 04:25:57 -0700 (PDT) Martin Knoblauch wrote:
FWIW, I compiled the CCISS driver into the kernel. This makes the second
"/sys" line in
/proc/mountsgo away, dmesg attached. But does it prove anything? The initialization of the CCISS hardware now happens about 2 seconds earlier in the bootup sequence. Does this hint to a problem with CCISS, or just confirms that the whole issue is really timing dependent? Anyway, I add Mike to CC.Quoted 16 lines
Quoted 1 line
It seems that the PCI change caused timing changes which triggered a udev/sysfs/whatever problem, which manifests as the duplicated
/proc/mountsentry to turn up.What we don't know (afaik) is why the kernel permitted two entries in
/proc/mounts. That might be a bug.It could be that if dual
/proc/mountsproblem gets fixed, everything works OK - by intent or by accident, the userspace startup scripts may then work acceptably.I think Al asked you a few questions around the behaviour of mount(8) and the mount syscall, so we could delve further into why
/proc/mountsis getting mucked up. Did you end up running those tests?
I do not recall any questions from Al. If he asked, I am pretty sure I answered :-)
Quoted 6 lines
I expect the duplicate comes from a left-over mount in initramfs which isn't a duplicate in the sense of a bug in vfs or mount or anything. I guess, it is just still mounted in the initial kernel rootfs, below the root from the disk. It could be that a umount from initramfs did go wrong because of a changed timing.
This is what I suspect as well. I know for sure that the first sysfs-line in /proc/mounts
| none /sys sysfs rw 0 0
is already there (2.6.29-rc1 and up) when entering startup-skripts. It is supposed to be unmounted before, but something seems to prevent it. I have idea how to capture debug output from the initrd/init script :-(
Cheers
Martin
Quoted 10 lines · On Thu, 28 May 2009 02:14:46 -0700 (PDT) Martin Knoblauch <[email protected]>…
Quoted 6 lines
I expect the duplicate comes from a left-over mount in initramfs which isn't a duplicate in the sense of a bug in vfs or mount or anything. I guess, it is just still mounted in the initial kernel rootfs, below the root from the disk. It could be that a umount from initramfs did go wrong because of a changed timing.
This is what I suspect as well. I know for sure that the first
sysfs-line in/proc/mounts
Quoted 6 lines
| none /sys sysfs rw 0 0
is already there (2.6.29-rc1 and up) when entering startup-skripts. It is supposed to be unmounted before, but something seems to prevent it. I have idea how to capture debug output from the initrd/init script :-(
What's the latest here Martin? It sounded like this was a userspace issue, with something reading the VPD over and over? Or was it just a longer timeout that caused a specific driver to slow everything down?
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
----- Original Message ----
Quoted 20 lines
From: Jesse Barnes <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Kay Sievers <[email protected]>; Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Tuesday, June 16, 2009 9:25:47 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowQuoted 12 lines · On Thu, 28 May 2009 02:14:46 -0700 (PDT) Martin Knoblauch wrote:
Quoted 6 lines
I expect the duplicate comes from a left-over mount in initramfs which isn't a duplicate in the sense of a bug in vfs or mount or anything. I guess, it is just still mounted in the initial kernel rootfs, below the root from the disk. It could be that a umount from initramfs did go wrong because of a changed timing.
This is what I suspect as well. I know for sure that the first
sysfs-line in/proc/mounts| none /sys sysfs rw 0 0
Quoted 9 lines
Quoted 4 lines
is already there (2.6.29-rc1 and up) when entering startup-skripts. It is supposed to be unmounted before, but something seems to prevent it. I have idea how to capture debug output from the initrd/init script :-(
What's the latest here Martin? It sounded like this was a userspace issue, with something reading the VPD over and over? Or was it just a longer timeout that caused a specific driver to slow everything down?
Not sure about the VPD thing. Anyway, no real news. Still happens in 2.6.30. But it only happens on a certain HW platform (HP/DL380G4). The folks at HP try to reproduce in their environment.
Cheers
Martin
Quoted 30 lines · Martin Knoblauch
----- Original Message ----
Quoted 19 lines
From: Jesse Barnes <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Kay Sievers <[email protected]>; Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Tuesday, June 16, 2009 9:25:47 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowQuoted 11 lines · On Thu, 28 May 2009 02:14:46 -0700 (PDT) Martin Knoblauch wrote:
Quoted 6 lines
I expect the duplicate comes from a left-over mount in initramfs which isn't a duplicate in the sense of a bug in vfs or mount or anything. I guess, it is just still mounted in the initial kernel rootfs, below the root from the disk. It could be that a umount from initramfs did go wrong because of a changed timing.
This is what I suspect as well. I know for sure that the first
sysfs-line in/proc/mounts| none /sys sysfs rw 0 0
Quoted 8 lines
Quoted 4 lines
is already there (2.6.29-rc1 and up) when entering startup-skripts. It is supposed to be unmounted before, but something seems to prevent it. I have idea how to capture debug output from the initrd/init script :-(
What's the latest here Martin? It sounded like this was a userspace issue, with something reading the VPD over and over? Or was it just a longer timeout that caused a specific driver to slow everything down?
Quoted 4 lines
Not sure about the VPD thing. Anyway, no real news. Still happens in 2.6.30. But it only happens on a certain HW platform (HP/DL380G4). The folks at HP try to reproduce in their environment.
Cheers
Martin
I reproduced this and verified Martin's analysis. Conclusions:
- >>> | none /sys sysfs rw 0 0
is because the initrd "umount /sys" fails with EBUSY
|commit 1120f8b8169fb2cb51219d326892d963e762edb6 |Author: Stephen Hemminger <[email protected]> |Date: Thu Dec 18 09:17:16 2008 -0800 | | PCI: handle long delays in VPD access
does not have a bug. The longer timeout makes the problem visible.
/sys is busy because udev is trying to read the vpd and the cciss pci device always fails the vpd with ETIMEOUT. If all timeouts are before or after the umount, no firmware load problem.
IMO there is either a vpd read bug on this platform or it is unsupported and ETIMEOUT is the wrong error.
... now I punt this to the HP platform/driver people.
jim
Quoted 24 lines · jim owens
Quoted 8 lines · Martin Knoblauch
----- Original Message ----
Quoted 5 lines
From: Jesse Barnes <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Kay Sievers <[email protected]>; Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Tuesday, June 16, 2009 9:25:47 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowNot sure about the VPD thing. Anyway, no real news. Still happens in 2.6.30. But it only happens on a certain HW platform (HP/DL380G4). The folks at HP try to reproduce in their environment.
I reproduced this and verified Martin's analysis. Conclusions:
- >>> | none /sys sysfs rw 0 0
is because the initrd "umount /sys" fails with EBUSY
|commit 1120f8b8169fb2cb51219d326892d963e762edb6 |Author: Stephen Hemminger <[email protected]> |Date: Thu Dec 18 09:17:16 2008 -0800 | | PCI: handle long delays in VPD accessdoes not have a bug. The longer timeout makes the problem visible.
Quoted 8 lines
/sys is busy because udev is trying to read the vpd and the cciss pci device always fails the vpd with ETIMEOUT. If all timeouts are before or after the umount, no firmware load problem.
IMO there is either a vpd read bug on this platform or it is unsupported and ETIMEOUT is the wrong error.
... now I punt this to the HP platform/driver people.
OK, let's add the PCI list in on this too ...
There are a bunch of PCI devices with buggy VPD. I've got one (it's a prototype card, I think, so I shan't name the vendor). lspci reports:
Capabilities: [84] Vital Product Data
Unknown small resource type 00
[last line repeated about 32768 times]
and it takes for-freaking-ever. I posted a patch to pciutils on May 13th to fix this:
---
I have several cards which report more-or-less garbage in their VPD. It can take an extraordinarily long time to read all their VPD and none of it is of interest. Instead, if we find an unknown resource type, just stop trying to read any more.
ls-vpd.c +1 -1
@@ -204,7 +204,7 @@ cap_vpd(struct device *d)default :printf ("\t\tUnknown %s resource type %02x\n" , (tag &0x80 ) ?"large" :"small" , tag & ~0x80 ); -break ; +return ; } res_addr += res_len;
-----
but I don't know what udev is doing. The udev source doesn't seem to read PCI vpd itself:
udev-0.141$ find -type f |xargs grep -il vpd./extras/volume_id/lib/adaptec_raid.c
./extras/scsi_id/scsi_id.8./extras/scsi_id/scsi.h./extras/scsi_id/scsi_id.config./extras/scsi_id/scsi_serial.c
so there must be some script that it's invoking which is doing that. Anyone familiar with udev?
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 10 lines · Matthew Wilcox
but I don't know what udev is doing. The udev source doesn't seem to read PCI vpd itself:
udev-0.141$ find -type f |xargs grep -il vpd
./extras/volume_id/lib/adaptec_raid.c
./extras/scsi_id/scsi_id.8./extras/scsi_id/scsi.h./extras/scsi_id/scsi_id.config./extras/scsi_id/scsi_serial.c
Quoted 2 lines
so there must be some script that it's invoking which is doing that. Anyone familiar with udev?
scsi_id is usually also called for cciss devices:
KERNEL=="cciss*", ..., IMPORT{program}="scsi_id ...
Kay
Quoted 12 lines · Kay Sievers
Quoted 10 lines · Matthew Wilcox
but I don't know what udev is doing. ??The udev source doesn't seem to read PCI vpd itself:
udev-0.141$ find -type f |xargs grep -il vpd
./extras/volume_id/lib/adaptec_raid.c
./extras/scsi_id/scsi_id.8./extras/scsi_id/scsi.h./extras/scsi_id/scsi_id.config./extras/scsi_id/scsi_serial.c
Quoted 5 lines
Quoted 2 lines
so there must be some script that it's invoking which is doing that. Anyone familiar with udev?
scsi_idis usually also called for cciss devices:
KERNEL=="cciss*", ..., IMPORT{program}="scsi_id ...
yes, but it's getting SCSI VPD (by asking for mode pages from the SCSI device). This problem is with PCI VPD which is totally different.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 13 lines · Matthew Wilcox
Quoted 12 lines · Kay Sievers
Quoted 10 lines · Matthew Wilcox
but I don't know what udev is doing. ??The udev source doesn't seem to read PCI vpd itself:
udev-0.141$ find -type f |xargs grep -il vpd
./extras/volume_id/lib/adaptec_raid.c
./extras/scsi_id/scsi_id.8./extras/scsi_id/scsi.h./extras/scsi_id/scsi_id.config./extras/scsi_id/scsi_serial.c
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
so there must be some script that it's invoking which is doing that. Anyone familiar with udev?
scsi_idis usually also called for cciss devices:
KERNEL=="cciss*", ..., IMPORT{program}="scsi_id ...yes, but it's getting SCSI VPD (by asking for mode pages from the SCSI device). This problem is with PCI VPD which is totally different.
Ah, I see. There is no tool around udev, I know of, which does this.
Maybe someone is still using the broken-by-design libsysfs, which opens _every_ file it can find in /sys, even when not asked for anything specific.
Thanks,
Kay
Quoted 5 lines · Kay Sievers
Ah, I see. There is no tool around udev, I know of, which does this.
Maybe someone is still using the broken-by-design libsysfs, which opens _every_ file it can find in /sys, even when not asked for anything specific.
I did consider that option, but dismissed it ... I didn't think anyone was using something that old. Martin, could you take a look at what scripts you're running?
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 9 lines · Matthew Wilcox
Quoted 5 lines · Kay Sievers
Ah, I see. There is no tool around udev, I know of, which does this.
Maybe someone is still using the broken-by-design libsysfs, which opens _every_ file it can find in /sys, even when not asked for anything specific.
I did consider that option, but dismissed it ... I didn't think anyone was using something that old.
My test is a fresh RHEL 4.3 install with 2.6.30-rc8. Whatever scripts are running are stock redhat and I'm still in initrd so what would I be looking for?
The initrd timeouts all seem to be triggered by the insmod and only 1 more vpd read happens in init 3 at the end after HAL starts.
This is the BUG() I set to ID (maybe incorrectly)
that the vpd reads were comming from udev:
[ 5.260664] ------------[ cut here ]------------ [ 5.260667] kernel BUG at /root/linux-2.6.30-rc8/drivers/pci/access.c:207! [ 5.260671] invalid opcode: 0000 [#1] SMP [ 5.260675] last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.2/0000:04:03.0/msi_bus [ 5.260678] CPU 1 [ 5.260681] Modules linked in: cciss(+) sd_mod scsi_mod [ 5.260689] Pid: 476, comm: udev Not tainted 2.6.30-rc8 #33 ProLiant DL380 G4 [ 5.260692] RIP: 0010:[<ffffffff803b282f>] [<ffffffff803b282f>] pci_vpd_pci22_wait+0xef/0x100 [ 5.260703] RSP: 0018:ffff8801199e7dd8 EFLAGS: 00010246 [ 5.260706] RAX: 0000000000000000 RBX: ffff88011ad60440 RCX: 0000000000001001 [ 5.260709] RDX: 0000000000007c7b RSI: 0000000000000001 RDI: ffffffff8072c160 [ 5.260712] RBP: ffff8801199e7e08 R08: 0000000000000001 R09: 0000000000000001 [ 5.260715] R10: 0000000000000000 R11: 0000000000000006 R12: ffff88011ac8d800 [ 5.260718] R13: 00000000fffee028 R14: ffff88011ac8d800 R15: ffff88011992e000 [ 5.260722] FS: 0000000000000000(0000) GS:ffff88002edd9000(0000) knlGS:0000000000000000 [ 5.260725] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 5.260728] CR2: 00000000022562b8 CR3: 00000001198ef000 CR4: 00000000000006e0 [ 5.260731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5.260734] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 5.260738] Process udev (pid: 476, threadinfo ffff8801199e6000, task ffff88011b78c3c0) [ 5.260740] Stack: [ 5.260742] ffff8801199e7e08 0000ffff803b268d 0000000000001000 0000000000000000 [ 5.260746] 0000000000000000 ffff88011ad60440 ffff8801199e7e68 ffffffff803b292c [ 5.260752] ffffffff803455f6 ffff88011ad60458 0000000000001000 0000000000001000 [ 5.260757] Call Trace: [ 5.260760] [<ffffffff803b292c>] pci_vpd_pci22_read+0xec/0x170 [ 5.260765] [<ffffffff803455f6>] ? read+0x96/0x1b0 [ 5.260771] [<ffffffff803b22a9>] pci_read_vpd+0x29/0x40 [ 5.260775] [<ffffffff803bab06>] read_vpd_attr+0x46/0x50 [ 5.260780] [<ffffffff803456fb>] read+0x19b/0x1b0 [ 5.260784] [<ffffffff80363566>] ? security_file_permission+0x16/0x20 [ 5.260790] [<ffffffff802e3f5d>] vfs_read+0xbd/0x190 [ 5.260797] [<ffffffff802e4365>] sys_read+0x55/0x90 [ 5.260801] [<ffffffff8020b71b>] system_call_fastpath+0x16/0x1b [ 5.260807] Code: c9 c3 8b 35 74 33 fe 00 31 c0 4c 89 e2 48 c7 c7 ed 89 6a 80 e8 b3 3d e9 ff 8b 05 5d 33 fe 00 ff c0 89 05 55 33 fe 00 ff c8 75 04 <0f> 0b eb fe b8 92 ff ff ff e9 35 ff ff ff 66 66 90 55 48 89 e5 [ 5.260845] RIP [<ffffffff803b282f>] pci_vpd_pci22_wait+0xef/0x100 [ 5.260849] RSP <ffff8801199e7dd8> [ 5.260854] ---[ end trace 37f1683404a28980 ]---
here is a single user boot with a timeout printk() and a hack in mkinitrd to sleep before and after the insmod cciss:
[ 3.858278] NET: Registered protocol family 17 [ 3.864293] Freeing unused kernel memory: 2416k freed
Red Hat nash version 4.2.1.6 starting
Mounted /proc filesystem
Mounting sysfs
Creating /dev
Starting udev
[ 3.876470] udev used greatest stack depth: 5632 bytes left Loading scsi_mod.ko module [ 3.977087] SCSI subsystem initialized Loading sd_mod.ko module [ 3.987672] Driver 'sd' needs updating - please use bus_type methods [ 4.280095] input: AT Translated Set 2 keyboard as /class/input/input0 [ 4.626432] input: PS/2 Generic Mouse as /class/input/input1 JIM sleep 1 Loading cciss.ko module [ 4.999782] HP CISS Driver (v 3.6.20) [ 5.003771] cciss 0000:04:03.0: PCI INT A -> GSI 51 (level, low) -> IRQ 51 [ 5.052021] IRQ 51/cciss0: IRQF_DISABLED is not guaranteed on shared IRQs [ 5.059001] cciss0: <0x46> at PCI 0000:04:03.0 IRQ 51 using DAC [ 5.065800] blocks= 71122560 block_size= 512 [ 5.070602] heads=255, sectors=32, cylinders=8716 [ 5.070603] [ 5.077428] blocks= 71122560 block_size= 512 [ 5.082151] heads=255, sectors=32, cylinders=8716 [ 5.082153] [ 5.089413] cciss/c0d0: p1 [ 5.131820] blocks= 71122560 block_size= 512 [ 5.139627] heads=255, sectors=32, cylinders=8716 [ 5.139629] [ 5.146117] blocks= 71122560 block_size= 512 [ 5.150887] heads=255, sectors=32, cylinders=8716 [ 5.150889] [ 5.157577] cciss/c0d1: p1 [ 5.160261] TIMEOUT 1 ffff88011ad54800 [ 5.164164] p2 p3 [ 5.208880] blocks= 142253280 block_size= 512 [ 5.215732] heads=255, sectors=32, cylinders=17433 [ 5.215734] [ 5.223191] blocks= 142253280 block_size= 512 [ 5.227965] [ 5.227966] TIMEOUT 2 ffff88011ad54800 [ 5.232942] heads=255, sectors=32, cylinders=17433 [ 5.232944] [ 5.239516] cciss/c0d2: p1 p2 p3 [ 5.292263] [ 5.292265] TIMEOUT 3 ffff88011ad54800 [ 5.356262] [ 5.356263] TIMEOUT 4 ffff88011ad54800 [ 5.420266] [ 5.420267] TIMEOUT 5 ffff88011ad54800 [ 5.484855] [ 5.484857] TIMEOUT 6 ffff88011ad54800 [ 5.491980] blocks= 71122560 block_size= 512 [ 5.496991] heads=255, sectors=32, cylinders=8716 [ 5.496993] [ 5.503484] blocks= 71122560 block_size= 512 [ 5.508761] heads=255, sectors=32, cylinders=8716 [ 5.508763] [ 5.515729] cciss/c0d3: p1 [ 5.548263] [ 5.548264] TIMEOUT 7 ffff88011ad54800 [ 5.610138] work_for_cpu used greatest stack depth: 4592 bytes left [ 5.612264] [ 5.612266] TIMEOUT 8 ffff88011ad54800 [ 5.674223] [ 5.674224] TIMEOUT 9 ffff88011ad54800 [ 5.736261] [ 5.736263] TIMEOUT 10 ffff88011ad54800 [ 5.800261] [ 5.800263] TIMEOUT 11 ffff88011ad54800 [ 5.864259] [ 5.864260] TIMEOUT 12 ffff88011ad54800 [ 5.928259] [ 5.928260] TIMEOUT 13 ffff88011ad54800 [ 5.992011] [ 5.992012] TIMEOUT 14 ffff88011ad54800 [ 6.056011] [ 6.056012] TIMEOUT 15 ffff88011ad54800 [ 6.120011] [ 6.120013] TIMEOUT 16 ffff88011ad54800 [ 6.184011] [ 6.184012] TIMEOUT 17 ffff88011ad54800 [ 6.248011] [ 6.248012] TIMEOUT 18 ffff88011ad54800 [ 6.312011] [ 6.312012] TIMEOUT 19 ffff88011ad54800 [ 6.376011] [ 6.376013] TIMEOUT 20 ffff88011ad54800
JIM sleep 1
Loading jbd.ko module
Loading ext3.ko module
Creating root device
Mounting root filesystem
[ 6.760540] kjournald starting. Commit interval 5 seconds [ 6.765559] EXT3-fs: mounted filesystem with writeback data mode. Switching to new root [ 6.946395] SELinux: Disabled at runtime. [ 6.950840] type=1404 audit(1245517131.950:2): selinux=0 auid=4294967295 ses=4294967295 INIT: version 2.85 booting [ 7.235810] uname used greatest stack depth: 4296 bytes left [ 7.304350] awk used greatest stack depth: 3688 bytes left Welcome to Red Hat Enterprise Linux AS Press 'I' to enter interactive startup.
Starting udev: [ OK ]
Initializing hardware... storage network audio done[ OK ] Configuring kernel parameters: [ OK ]
Setting clock (utc): Sat Jun 20 12:59:02 EDT 2009 [ OK ] Setting hostname dl380.lnx.usa.hp.com: [ OK ] Checking root filesystem
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/cciss/c0d1p3 /1: clean, 348377/4177920 files, 2477319/8351791 blocks [ OK ]
Remounting root filesystem in read-write mode: [ OK ] No devices found
no block devices found
Setting up Logical Volume Management: [ OK ] Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /boot] fsck.ext3 -a /dev/cciss/c0d1p1 /boot12: clean, 46/26104 files, 26042/104388 blocks [ OK ]
Mounting local filesystems: [ OK ] Enabling local filesystem quotas: [ OK ] Enabling swap space: [ OK ]
sh-3.00#
----- Original Message ----
Quoted 14 lines
From: jim owens <[email protected]>
To: Matthew Wilcox <[email protected]>
Cc: Kay Sievers <[email protected]>; Martin Knoblauch <[email protected]>; Jesse Barnes <[email protected]>; Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Saturday, June 20, 2009 11:17:06 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowQuoted 7 lines · Matthew Wilcox
Quoted 5 lines · Kay Sievers
Ah, I see. There is no tool around udev, I know of, which does this.
Maybe someone is still using the broken-by-design libsysfs, which opens _every_ file it can find in /sys, even when not asked for anything specific.
Quoted 7 lines
Quoted 2 lines
I did consider that option, but dismissed it ... I didn't think anyone was using something that old.
My test is a fresh RHEL 4.3 install with 2.6.30-rc8. Whatever scripts are running are stock redhat and I'm still in initrd so what would I be looking for?
same here. Anything in initrd is stock RedHat stuff, except the kernel modules
Quoted 55 lines
The initrd timeouts all seem to be triggered by the insmod and only 1 more vpd read happens in init 3 at the end after HAL starts.
This is the
BUG()I set to ID (maybe incorrectly)
that the vpd reads were comming from udev:[ 5.260664] ------------[ cut here ]------------ [ 5.260667] kernel BUG at /root/linux-2.6.30-rc8/drivers/pci/access.c:207! [ 5.260671] invalid opcode: 0000 [#1] SMP [ 5.260675] last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.2/0000:04:03.0/msi_bus [ 5.260678] CPU 1 [ 5.260681] Modules linked in: cciss(+) sd_mod scsi_mod [ 5.260689] Pid: 476, comm: udev Not tainted 2.6.30-rc8 #33 ProLiant DL380 G4 [ 5.260692] RIP: 0010:[] [] pci_vpd_pci22_wait+0xef/0x100 [ 5.260703] RSP: 0018:ffff8801199e7dd8 EFLAGS: 00010246 [ 5.260706] RAX: 0000000000000000 RBX: ffff88011ad60440 RCX: 0000000000001001 [ 5.260709] RDX: 0000000000007c7b RSI: 0000000000000001 RDI: ffffffff8072c160 [ 5.260712] RBP: ffff8801199e7e08 R08: 0000000000000001 R09: 0000000000000001 [ 5.260715] R10: 0000000000000000 R11: 0000000000000006 R12: ffff88011ac8d800 [ 5.260718] R13: 00000000fffee028 R14: ffff88011ac8d800 R15: ffff88011992e000 [ 5.260722] FS: 0000000000000000(0000) GS:ffff88002edd9000(0000) knlGS:0000000000000000 [ 5.260725] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 5.260728] CR2: 00000000022562b8 CR3: 00000001198ef000 CR4: 00000000000006e0 [ 5.260731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5.260734] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 5.260738] Process udev (pid: 476, threadinfo ffff8801199e6000, task ffff88011b78c3c0) [ 5.260740] Stack: [ 5.260742] ffff8801199e7e08 0000ffff803b268d 0000000000001000 0000000000000000 [ 5.260746] 0000000000000000 ffff88011ad60440 ffff8801199e7e68 ffffffff803b292c [ 5.260752] ffffffff803455f6 ffff88011ad60458 0000000000001000 0000000000001000 [ 5.260757] Call Trace: [ 5.260760] [] pci_vpd_pci22_read+0xec/0x170 [ 5.260765] [] ? read+0x96/0x1b0 [ 5.260771] [] pci_read_vpd+0x29/0x40 [ 5.260775] [] read_vpd_attr+0x46/0x50 [ 5.260780] [] read+0x19b/0x1b0 [ 5.260784] [] ? security_file_permission+0x16/0x20 [ 5.260790] [] vfs_read+0xbd/0x190 [ 5.260797] [] sys_read+0x55/0x90 [ 5.260801] [] system_call_fastpath+0x16/0x1b [ 5.260807] Code: c9 c3 8b 35 74 33 fe 00 31 c0 4c 89 e2 48 c7 c7 ed 89 6a 80 e8 b3 3d e9 ff 8b 05 5d 33 fe 00 ff c0 89 05 55 33 fe 00 ff c8 75 04 <0f> 0b eb fe b8 92 ff ff ff e9 35 ff ff ff 66 66 90 55 48 89 e5 [ 5.260845] RIP [] pci_vpd_pci22_wait+0xef/0x100 [ 5.260849] RSP [ 5.260854] ---[ end trace 37f1683404a28980 ]---
Quoted 3 lines
here is a single user boot with a timeout
printk()and a hack in mkinitrd to sleep before and after the insmod cciss:
How do you get to see the debug-output from initrd? I never managed that.
Quoted 127 lines
[ 3.858278] NET: Registered protocol family 17 [ 3.864293] Freeing unused kernel memory: 2416k freedRed Hat nash version 4.2.1.6 starting
Mounted /proc filesystem
Mounting sysfs
Creating /dev
Starting udev[ 3.876470] udev used greatest stack depth: 5632 bytes left Loading scsi_mod.ko module [ 3.977087] SCSI subsystem initialized Loading sd_mod.ko module [ 3.987672] Driver 'sd' needs updating - please use bus_type methods [ 4.280095] input: AT Translated Set 2 keyboard as /class/input/input0 [ 4.626432] input: PS/2 Generic Mouse as /class/input/input1 JIM sleep 1 Loading cciss.ko module [ 4.999782] HP CISS Driver (v 3.6.20) [ 5.003771] cciss 0000:04:03.0: PCI INT A -> GSI 51 (level, low) -> IRQ 51 [ 5.052021] IRQ 51/cciss0: IRQF_DISABLED is not guaranteed on shared IRQs [ 5.059001] cciss0: <0x46> at PCI 0000:04:03.0 IRQ 51 using DAC [ 5.065800] blocks= 71122560 block_size= 512 [ 5.070602] heads=255, sectors=32, cylinders=8716 [ 5.070603] [ 5.077428] blocks= 71122560 block_size= 512 [ 5.082151] heads=255, sectors=32, cylinders=8716 [ 5.082153] [ 5.089413] cciss/c0d0: p1 [ 5.131820] blocks= 71122560 block_size= 512 [ 5.139627] heads=255, sectors=32, cylinders=8716 [ 5.139629] [ 5.146117] blocks= 71122560 block_size= 512 [ 5.150887] heads=255, sectors=32, cylinders=8716 [ 5.150889] [ 5.157577] cciss/c0d1: p1 [ 5.160261] TIMEOUT 1 ffff88011ad54800 [ 5.164164] p2 p3 [ 5.208880] blocks= 142253280 block_size= 512 [ 5.215732] heads=255, sectors=32, cylinders=17433 [ 5.215734] [ 5.223191] blocks= 142253280 block_size= 512 [ 5.227965] [ 5.227966] TIMEOUT 2 ffff88011ad54800 [ 5.232942] heads=255, sectors=32, cylinders=17433 [ 5.232944] [ 5.239516] cciss/c0d2: p1 p2 p3 [ 5.292263] [ 5.292265] TIMEOUT 3 ffff88011ad54800 [ 5.356262] [ 5.356263] TIMEOUT 4 ffff88011ad54800 [ 5.420266] [ 5.420267] TIMEOUT 5 ffff88011ad54800 [ 5.484855] [ 5.484857] TIMEOUT 6 ffff88011ad54800 [ 5.491980] blocks= 71122560 block_size= 512 [ 5.496991] heads=255, sectors=32, cylinders=8716 [ 5.496993] [ 5.503484] blocks= 71122560 block_size= 512 [ 5.508761] heads=255, sectors=32, cylinders=8716 [ 5.508763] [ 5.515729] cciss/c0d3: p1 [ 5.548263] [ 5.548264] TIMEOUT 7 ffff88011ad54800 [ 5.610138] work_for_cpu used greatest stack depth: 4592 bytes left [ 5.612264] [ 5.612266] TIMEOUT 8 ffff88011ad54800 [ 5.674223] [ 5.674224] TIMEOUT 9 ffff88011ad54800 [ 5.736261] [ 5.736263] TIMEOUT 10 ffff88011ad54800 [ 5.800261] [ 5.800263] TIMEOUT 11 ffff88011ad54800 [ 5.864259] [ 5.864260] TIMEOUT 12 ffff88011ad54800 [ 5.928259] [ 5.928260] TIMEOUT 13 ffff88011ad54800 [ 5.992011] [ 5.992012] TIMEOUT 14 ffff88011ad54800 [ 6.056011] [ 6.056012] TIMEOUT 15 ffff88011ad54800 [ 6.120011] [ 6.120013] TIMEOUT 16 ffff88011ad54800 [ 6.184011] [ 6.184012] TIMEOUT 17 ffff88011ad54800 [ 6.248011] [ 6.248012] TIMEOUT 18 ffff88011ad54800 [ 6.312011] [ 6.312012] TIMEOUT 19 ffff88011ad54800 [ 6.376011] [ 6.376013] TIMEOUT 20 ffff88011ad54800JIM sleep 1
Loading jbd.ko module
Loading ext3.ko module
Creating root device
Mounting root filesystem[ 6.760540] kjournald starting. Commit interval 5 seconds [ 6.765559] EXT3-fs: mounted filesystem with writeback data mode. Switching to new root [ 6.946395] SELinux: Disabled at runtime. [ 6.950840] type=1404 audit(1245517131.950:2): selinux=0 auid=4294967295 ses=4294967295 INIT: version 2.85 booting [ 7.235810] uname used greatest stack depth: 4296 bytes left [ 7.304350] awk used greatest stack depth: 3688 bytes left Welcome to Red Hat Enterprise Linux AS Press 'I' to enter interactive startup.Starting udev: [ OK ]
Initializing hardware... storage network audio done[ OK ] Configuring kernel parameters: [ OK ]
Setting clock (utc): Sat Jun 20 12:59:02 EDT 2009 [ OK ] Setting hostname dl380.lnx.usa.hp.com: [ OK ] Checking root filesystem
[/sbin/fsck.ext3(1) -- /] fsck.ext3 -a/dev/cciss/c0d1p3/1: clean, 348377/4177920 files, 2477319/8351791 blocks [ OK ]
Remounting root filesystem in read-write mode: [ OK ] No devices found
no block devices found
Setting up Logical Volume Management: [ OK ] Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3(1) -- /boot] fsck.ext3 -a/dev/cciss/c0d1p1/boot12: clean, 46/26104 files, 26042/104388 blocks [ OK ]Mounting local filesystems: [ OK ] Enabling local filesystem quotas: [ OK ] Enabling swap space: [ OK ]sh-3.00#
Quoted 2 lines · Martin Knoblauch
How do you get to see the debug-output from initrd? I never managed that.
I did all my boot testing from ILO with VSP for a serial console.
jim
----- Original Message ----
Quoted 40 lines
From: jim owens <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Jesse Barnes <[email protected]>; Kay Sievers <[email protected]>; Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Saturday, June 20, 2009 6:37:34 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowQuoted 5 lines · Martin Knoblauch
----- Original Message ----
Quoted 3 lines
From: Jesse Barnes
To: Martin Knoblauch
Cc: Kay Sievers ; Andrew Morton; [email protected]; [email protected];
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]Quoted 25 lines
Quoted 24 lines
Sent: Tuesday, June 16, 2009 9:25:47 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slowQuoted 9 lines · On Thu, 28 May 2009 02:14:46 -0700 (PDT) Martin Knoblauch wrote:
Quoted 6 lines
I expect the duplicate comes from a left-over mount in initramfs which isn't a duplicate in the sense of a bug in vfs or mount or anything. I guess, it is just still mounted in the initial kernel rootfs, below the root from the disk. It could be that a umount from initramfs did go wrong because of a changed timing.
This is what I suspect as well. I know for sure that the first
sysfs-line in/proc/mountsQuoted 6 lines
| none /sys sysfs rw 0 0
is already there (2.6.29-rc1 and up) when entering startup-skripts. It is supposed to be unmounted before, but something seems to prevent it. I have idea how to capture debug output from the initrd/init script :-(
What's the latest here Martin? It sounded like this was a userspace issue, with something reading the VPD over and over? Or was it just a longer timeout that caused a specific driver to slow everything down?
Quoted 9 lines
Quoted 1 line
Not sure about the VPD thing. Anyway, no real news. Still happens in 2.6.30.
But it only happens on a certain HW platform (HP/DL380G4). The folks at HP try to reproduce in their environment.
Quoted 3 lines
Cheers
MartinI reproduced this and verified Martin's analysis. Conclusions:
Cool, so I am not seeing gremlins :-)
Cheers
Martin