Skip to contentSkip to conversation

Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

DISCUSSION Martin Knoblauch 9 participants 39 messages Apr 24, 2009 – May 28, 2009

lore
Share & tools
lore
#0 lore raw

----- Original Message ----

Quoted 111 lines

From: Martin Knoblauch <[email protected]>
To: Rafael J. Wysocki <[email protected]>
Cc: [email protected]; [email protected]
Sent: Wednesday, April 22, 2009 6:55:27 PM
Subject: Re: Booting 2.6.30-rc2-git7 very slow

----- Original Message ----

Quoted 6 lines

From: Martin Knoblauch
To: Rafael J. Wysocki
Cc: [email protected]; [email protected]
Sent: Wednesday, April 22, 2009 1:32:25 PM
Subject: Re: Booting 2.6.30-rc2-git7 very slow

Quoted 22 lines

----- Original Message ----

Quoted 20 lines

From: Martin Knoblauch
To: Rafael J. Wysocki
Cc: [email protected]
Sent: Wednesday, April 22, 2009 11:56:27 AM
Subject: Re: Booting 2.6.30-rc2-git7 very slow

Quoted 12 lines

From: Rafael J. Wysocki

To: Martin Knoblauch
Cc: [email protected]
Sent: Tuesday, April 21, 2009 9:12:20 PM
Subject: Re: Booting 2.6.30-rc2-git7 very slow

Quoted 4 lines · Martin Knoblauch

Hi, [please CC me on replies, as I am not subscribed]

booting 2.6.30-rc2-git7 on a HP/DL380G3 (x86_64, RHEL4.3, 64 bit

userspace)

Quoted 1 line

is much slower that it used to be. It seems I run into timeouts when

[trying

Quoted 46 lines
Quoted 5 lines

to]

Quoted 3 lines

load intel and tg3 microcodes:

Quoted 2 lines
[   14.478892] platform microcode: firmware: requesting 

intel-ucode/0f-04-0a

Quoted 1 line
Quoted 1 line
Quoted 1 line
[   74.476741] platform microcode: firmware: requesting 

intel-ucode/0f-04-0a

Quoted 1 line
Quoted 1 line
Quoted 1 line
[  134.476638] platform microcode: firmware: requesting 

intel-ucode/0f-04-0a

Quoted 1 line
Quoted 1 line
Quoted 1 line
[  194.476637] platform microcode: firmware: requesting 

intel-ucode/0f-04-0a

Quoted 25 lines
Quoted 16 lines
Quoted 1 line
[  254.476493] Microcode Update Driver: v2.00 , 

Peter Oruba

Quoted 14 lines
[  254.718489] Microcode Update Driver: v2.00 removed.

So, we see 60 seconds for eaoch of the CPUs

[  255.273426] tg3 0000:03:01.0: PME# disabled
[  257.833769] tg3: eth0: Link is up at 1000 Mbps, full duplex.
[  257.833775] tg3: eth0: Flow control is off for TX and off for RX.
[  269.643973] tg3 0000:03:01.1: firmware: requesting tigon/tg3_tso.bin
[  329.640456] eth1: Failed to load firmware "tigon/tg3_tso.bin"
[  329.640463] eth1: TSO capability disabled.
[  329.640487] tg3 0000:03:01.1: PME# disabled
[  333.081753] tg3: eth1: Link is up at 1000 Mbps, full duplex.
[  333.081759] tg3: eth1: Flow control is off for TX and off for RX.
Quoted 6 lines
Quoted 3 lines

We see 60 seconds for eth1, complaining about a failed firmware load.
/lib/firmware/tigon/tg3_tso.bin does exist and is from the current build.

Do I assume correctly that 2.6.29 did not have this problem?

Just checked. 2.6.29 has exactely the same problem. 2.6.28.2 was OK. This is

Quoted 8 lines

from the 2.6.29 boot:

[   14.308340] platform microcode: firmware: requesting intel-ucode/0f-04-0a
[   74.304612] platform microcode: firmware: requesting intel-ucode/0f-04-0a
[  134.304651] platform microcode: firmware: requesting intel-ucode/0f-04-0a
[  194.304638] platform microcode: firmware: requesting intel-ucode/0f-04-0a
[  254.304597] Microcode Update Driver: v2.00 ,  Peter Oruba
[  254.546200] Microcode Update Driver: v2.00 removed.
Quoted 19 lines

In case of the platform microcode, the delays happen when doing:

/sbin/modprobe microcode

from the init script. I have a "microcode.dat" File in both /etc/ and /etc/firmware.

Quoted 11 lines
[  255.088405] tg3 0000:03:01.0: PME# disabled
[  257.669617] tg3: eth0: Link is up at 1000 Mbps, full duplex.
[  257.669622] tg3: eth0: Flow control is off for TX and off for RX.
[  269.456132] tg3 0000:03:01.1: firmware: requesting tigon/tg3_tso.bin
[  329.456495] eth1: Failed to load firmware "tigon/tg3_tso.bin"
[  329.456500] eth1: TSO capability disabled.
[  329.456524] tg3 0000:03:01.1: PME# disabled
[  332.921832] tg3: eth1: Link is up at 1000 Mbps, full duplex.
[  332.921837] tg3: eth1: Flow control is off for TX and off for RX.

OK, I was able to solve the tg3 failure by adding "tigon/tg3_tso.bin" to CONFIG_EXTRA_FIRMWARE. I tried the same with the CPU microcode (after copying /etc/firmware/microcode.dat to $BUILDDIR/firmware/), but no success.

Searching the archives also found some mentioning that I might need special udev-rules to make firmware loading work again transparently. But no explicit solution.

Quoted 6 lines

I find this new (since 2.6.29) behaviour:

- very unexpected
- not well documented
- the EXTRA_FIRMWARE solution very unscalable

OK, I just found the reason for both intel-ucode and tg3 failures. Apparently between 2.6.28 and 2.6.29 the mount of sysfs has subtely changed from:

/sys /sys sysfs rw 0 0

to:

none /sys sysfs rw,relatime 0 0

The "none" breaks the RHEL-4 provided hotplug script "firmware.agent" when it tries to parse the mount point for "/sys". As a result, the firmware loading is never properly finished and the driver(s) just timeout on the value in /sys/class/firmware/timeout. Bingo. Simple fix in user-pace possible - cool down Martin :-)

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

Cheers
Martin

#1 lore raw
Quoted 9 lines · On Fri, 24 Apr 2009 05:45:19 -0700 (PDT) Martin Knoblauch <[email protected]>…

OK, I just found the reason for both intel-ucode and tg3 failures. Apparently between 2.6.28 and 2.6.29 the mount of sysfs has subtely changed from:

/sys /sys sysfs rw 0 0

to:

none /sys sysfs rw,relatime 0 0

I assume that you're referring to the contents of /proc/mounts?

Quoted 3 lines

The "none" breaks the RHEL-4 provided hotplug script "firmware.agent" when it tries to parse the mount point for "/sys". As a result, the firmware loading is never properly finished and the driver(s) just timeout on the value in /sys/class/firmware/timeout. Bingo. Simple fix in user-pace possible - cool down Martin :-)

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

#2 lore raw
Quoted 14 lines · Andrew Morton
Quoted 9 lines · On Fri, 24 Apr 2009 05:45:19 -0700 (PDT) Martin Knoblauch <[email protected]>…

OK, I just found the reason for both intel-ucode and tg3 failures. Apparently between 2.6.28 and 2.6.29 the mount of sysfs has subtely changed from:

/sys /sys sysfs rw 0 0

to:

none /sys sysfs rw,relatime 0 0

I assume that you're referring to the contents of /proc/mounts?

Quoted 7 lines
Quoted 3 lines

The "none" breaks the RHEL-4 provided hotplug script "firmware.agent" when it tries to parse the mount point for "/sys". As a result, the firmware loading is never properly finished and the driver(s) just timeout on the value in /sys/class/firmware/timeout. Bingo. Simple fix in user-pace possible - cool down Martin :-)

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

<paste>

Quoted 3 lines

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

(I wonder how the heck that is accomplished)

#3 lore raw
Quoted 21 lines · On Wed, 29 Apr 2009 05:51:36 +0200 Mike Galbraith <[email protected]> wrote:
Quoted 20 lines · Andrew Morton
Quoted 9 lines · On Fri, 24 Apr 2009 05:45:19 -0700 (PDT) Martin Knoblauch <[email protected]>…

OK, I just found the reason for both intel-ucode and tg3 failures. Apparently between 2.6.28 and 2.6.29 the mount of sysfs has subtely changed from:

/sys /sys sysfs rw 0 0

to:

none /sys sysfs rw,relatime 0 0

I assume that you're referring to the contents of /proc/mounts?

Quoted 3 lines

The "none" breaks the RHEL-4 provided hotplug script "firmware.agent" when it tries to parse the mount point for "/sys". As a result, the firmware loading is never properly finished and the driver(s) just timeout on the value in /sys/class/firmware/timeout. Bingo. Simple fix in user-pace possible - cool down Martin :-)

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

Quoted 9 lines
Quoted 1 line

I wonder how we did that.

<paste>

Quoted 3 lines

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

___(I wonder how the heck that is accomplished)

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Displaying relatime seems a bit pointless too.

#4 lore raw

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

----- Original Message ----

Quoted 46 lines

From: Andrew Morton <[email protected]>
To: Mike Galbraith <[email protected]>
Cc: Martin Knoblauch <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>; Al Viro <[email protected]>
Sent: Wednesday, April 29, 2009 10:17:55 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 2 lines · On Wed, 29 Apr 2009 05:51:36 +0200 Mike Galbraith wrote:
Quoted 1 line · Andrew Morton

On Fri, 24 Apr 2009 05:45:19 -0700 (PDT) Martin Knoblauch

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines

OK, I just found the reason for both intel-ucode and tg3 failures.

Apparently between 2.6.28 and 2.6.29 the mount of sysfs has subtely changed from:

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

/sys /sys sysfs rw 0 0

to:

none /sys sysfs rw,relatime 0 0

I assume that you're referring to the contents of /proc/mounts?

Quoted 1 line

The "none" breaks the RHEL-4 provided hotplug script "firmware.agent"

when it tries to parse the mount point for "/sys". As a result, the firmware loading is never properly finished and the driver(s) just timeout on the value in /sys/class/firmware/timeout. Bingo. Simple fix in user-pace possible - cool down Martin :-)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Questions remains: was this intentional? It breaks existing userspace and

should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

Quoted 10 lines
Quoted 9 lines

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

Quoted 8 lines
Quoted 2 lines

___(I wonder how the heck that is accomplished)

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Displaying relatime seems a bit pointless too.

Actually, relatime is added in 2.6.30. In 2.6.29 I only see the duplicate lines.

Cheers
Martin

#5 lore raw
In reply to Andrew Morton

----- Original Message ----

Quoted 46 lines

From: Andrew Morton <[email protected]>
To: Mike Galbraith <[email protected]>
Cc: Martin Knoblauch <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>; Al Viro <[email protected]>
Sent: Wednesday, April 29, 2009 10:17:55 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 2 lines · On Wed, 29 Apr 2009 05:51:36 +0200 Mike Galbraith wrote:
Quoted 1 line · Andrew Morton

On Fri, 24 Apr 2009 05:45:19 -0700 (PDT) Martin Knoblauch

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines

OK, I just found the reason for both intel-ucode and tg3 failures.

Apparently between 2.6.28 and 2.6.29 the mount of sysfs has subtely changed from:

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

/sys /sys sysfs rw 0 0

to:

none /sys sysfs rw,relatime 0 0

I assume that you're referring to the contents of /proc/mounts?

Quoted 1 line

The "none" breaks the RHEL-4 provided hotplug script "firmware.agent"

when it tries to parse the mount point for "/sys". As a result, the firmware loading is never properly finished and the driver(s) just timeout on the value in /sys/class/firmware/timeout. Bingo. Simple fix in user-pace possible - cool down Martin :-)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Questions remains: was this intentional? It breaks existing userspace and

should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

Quoted 10 lines
Quoted 9 lines

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

Quoted 8 lines
Quoted 2 lines

___(I wonder how the heck that is accomplished)

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Displaying relatime seems a bit pointless too.

Hmm. I actually believe the "none" line comes out of /etc/fstab, but was never before displayed in /proc/mount.

This is from 2.6.19:

[root@lpsdm60 ~]# grep sysfs /etc/fstab
none /sys sysfs defaults 0 0
[root@lpsdm60 ~]# mount | grep sysfs
none on /sys type sysfs (rw)
[root@lpsdm60 ~]# grep sysfs /proc/mounts
/sys /sys sysfs rw 0 0

And this is from 2.6.30:

[root@lpsdm52 linux-2.6.30-rc3-git2]# grep sysfs /etc/fstab
none /sys sysfs defaults 0 0
[root@lpsdm52 linux-2.6.30-rc3-git2]# mount | grep sysfs none on /sys type sysfs (rw)
[root@lpsdm52 linux-2.6.30-rc3-git2]# grep sysfs /proc/mounts none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

Any changes to mount-handling in 2.6.29?

Cheers
Martin

#6 lore raw
In reply to Andrew Morton
Quoted 11 lines · Andrew Morton
Quoted 11 lines
Quoted 5 lines
Quoted 1 line

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

<paste>

Quoted 3 lines

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

Quoted 5 lines
Quoted 1 line

___(I wonder how the heck that is accomplished)

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

#7 lore raw
Quoted 15 lines · Al Viro
Quoted 13 lines · Andrew Morton
Quoted 12 lines
Quoted 5 lines
Quoted 1 line

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

<paste>

Quoted 3 lines

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

___(I wonder how the heck that is accomplished)

Quoted 7 lines
Quoted 3 lines

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Yeah, but how does one go about doing that?

Using mount -f, I can convince mount to succeed, but I still have only one entry in /proc/mounts, despite what my mount binary imagines.

marge:..sys/vm # grep sysfs /proc/mounts
sysfs /sys sysfs rw,relatime 0 0

marge:..sys/vm # mount|grep sysfs
sysfs on /sys type sysfs (rw)
sys on /sys type sysfs (rw)
/sys on /sys type sysfs (rw)

-Mike

#8 lore raw
Quoted 7 lines · Mike Galbraith
Quoted 2 lines

/etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Yeah, but how does one go about doing that?

Using mount -f, I can convince mount to succeed, but I still have only one entry in /proc/mounts, despite what my mount binary imagines.

Huh?
-f Causes everything to be done except for the actual system call;
if it's not obvious, this ``fakes'' mounting the file system. This option is useful in conjunction with the -v flag to deter- mine what the mount command is trying to do. It can also be used to add entries for devices that were mounted earlier with the -n

What are you talking about?

The interesting part is why mount(2) doesn't fail with -EBUSY on that overmounting. Is there anything else mounted on /sys? That, or any interesting patches applied to the tree (fs/sysfs/mount.c, fs/namespace.c)

#9 lore raw

----- Original Message ----

Quoted 22 lines

From: Al Viro <[email protected]>
To: Mike Galbraith <[email protected]>
Cc: Andrew Morton <[email protected]>; Martin Knoblauch <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>
Sent: Wednesday, April 29, 2009 4:34:03 PM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 7 lines · Mike Galbraith
Quoted 2 lines

/etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Yeah, but how does one go about doing that?

Using mount -f, I can convince mount to succeed, but I still have only one entry in /proc/mounts, despite what my mount binary imagines.

Huh?
-f Causes everything to be done except for the actual system call;
if it's not obvious, this ``fakes'' mounting the file system. This option is useful in conjunction with the -v flag to deter- mine what the mount command is trying to do. It can also be used to add entries for devices that were mounted earlier with the -n

Quoted 5 lines

What are you talking about?

The interesting part is why mount(2) doesn't fail with -EBUSY on that overmounting. Is there anything else mounted on /sys? That, or any interesting patches applied to the tree (fs/sysfs/mount.c, fs/namespace.c)

In my 2.6.30-rc3-git2 there is definitely nothing interesting, just my NFS readahead patch. But that touches nothing near sysfs. And my 2.6.29 test-boot was from a plain build.

Cheers
Martin

#10 lore raw
In reply to Al Viro
Quoted 9 lines · Al Viro
Quoted 7 lines · Mike Galbraith
Quoted 2 lines

/etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Yeah, but how does one go about doing that?

Using mount -f, I can convince mount to succeed, but I still have only one entry in /proc/mounts, despite what my mount binary imagines.

Quoted 8 lines

Huh?
-f Causes everything to be done except for the actual system call;
if it's not obvious, this ``fakes'' mounting the file system. This option is useful in conjunction with the -v flag to deter- mine what the mount command is trying to do. It can also be used to add entries for devices that were mounted earlier with the -n

What are you talking about?

Me? My binary lost touch with reality, but I couldn't induce it to produce two /proc/mount entries. I thought that's what I said.

-Mike

#11 lore raw
In reply to Mike Galbraith
Quoted 29 lines · On Wed, 29 Apr 2009 16:18:45 +0200 Mike Galbraith <[email protected]> wrote:
Quoted 15 lines · Al Viro
Quoted 13 lines · Andrew Morton
Quoted 12 lines
Quoted 5 lines
Quoted 1 line

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

<paste>

Quoted 3 lines

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

___(I wonder how the heck that is accomplished)

Quoted 7 lines
Quoted 3 lines

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Yeah, but how does one go about doing that?

Using mount -f, I can convince mount to succeed, but I still have only one entry in /proc/mounts, despite what my mount binary imagines.

Quoted 8 lines

marge:..sys/vm # grep sysfs /proc/mounts
sysfs /sys sysfs rw,relatime 0 0

marge:..sys/vm # mount|grep sysfs
sysfs on /sys type sysfs (rw)
sys on /sys type sysfs (rw)
/sys on /sys type sysfs (rw)

So /proc/mounts is OK and /etc/mtab is wrong?

Obvious next step is to strace `mount -f', see what's happening around sys_mount(), please.

#12 lore raw
Quoted 41 lines · Andrew Morton
Quoted 29 lines · On Wed, 29 Apr 2009 16:18:45 +0200 Mike Galbraith <[email protected]> wrote:
Quoted 15 lines · Al Viro
Quoted 13 lines · Andrew Morton
Quoted 12 lines
Quoted 5 lines
Quoted 1 line

Questions remains: was this intentional? It breaks existing userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

<paste>

Quoted 3 lines

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

___(I wonder how the heck that is accomplished)

Quoted 7 lines
Quoted 3 lines

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Yeah, but how does one go about doing that?

Using mount -f, I can convince mount to succeed, but I still have only one entry in /proc/mounts, despite what my mount binary imagines.

Quoted 8 lines

marge:..sys/vm # grep sysfs /proc/mounts
sysfs /sys sysfs rw,relatime 0 0

marge:..sys/vm # mount|grep sysfs
sysfs on /sys type sysfs (rw)
sys on /sys type sysfs (rw)
/sys on /sys type sysfs (rw)

Quoted 4 lines

So /proc/mounts is OK and /etc/mtab is wrong?

Obvious next step is to strace `mount -f', see what's happening around sys_mount(), please.

Well, there is no syscall with -f.

I was trying various mount options to see if I could find a way to create bogons that could confuse scripts. I could create bogons in /etc/mtab with -f, or bogons in /proc/mounts by using --move. I could re-create the exact reported data with a combination of mount -n and mount --move. I could not get a double /proc/mounts entry without --move, and that seems unlikely to appear in boot scripts. So I still wonder how the heck it was accomplished.

I also now wonder why you can --move mounts on top of one another, but beck with it, ignorance conserves braincells I may some day need :)

-Mike

#13 lore raw

----- Original Message ----

Quoted 66 lines

From: Mike Galbraith <[email protected]>
To: Andrew Morton <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Wednesday, May 6, 2009 6:45:40 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Wed, 29 Apr 2009 16:18:45 +0200 Mike Galbraith wrote:
Quoted 3 lines · Al Viro
Quoted 1 line · Andrew Morton
Quoted 1 line
Quoted 1 line
Quoted 1 line

Questions remains: was this intentional? It breaks existing

userspace and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

Quoted 34 lines
Quoted 25 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 9 lines

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

___(I wonder how the heck that is accomplished)

Quoted 7 lines
Quoted 3 lines

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Yeah, but how does one go about doing that?

Using mount -f, I can convince mount to succeed, but I still have only one entry in /proc/mounts, despite what my mount binary imagines.

Quoted 8 lines

marge:..sys/vm # grep sysfs /proc/mounts
sysfs /sys sysfs rw,relatime 0 0

marge:..sys/vm # mount|grep sysfs
sysfs on /sys type sysfs (rw)
sys on /sys type sysfs (rw)
/sys on /sys type sysfs (rw)

Quoted 4 lines

So /proc/mounts is OK and /etc/mtab is wrong?

Obvious next step is to strace `mount -f', see what's happening around sys_mount(), please.

Well, there is no syscall with -f.

I was trying various mount options to see if I could find a way to create bogons that could confuse scripts. I could create bogons in /etc/mtab with -f, or bogons in /proc/mounts by using --move. I could re-create the exact reported data with a combination of mount -n and mount --move. I could not get a double /proc/mounts entry without --move, and that seems unlikely to appear in boot scripts. So I still wonder how the heck it was accomplished.

Quoted 3 lines

I also now wonder why you can --move mounts on top of one another, but beck with it, ignorance conserves braincells I may some day need :)

just to bring this back to my problem :-) Last week I reported that the "new" sysfs entry in /proc/mounts already comes out of initrd. Does this ring a bell?

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Cheers
Martin

#15 lore raw

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

----- Original Message ----

Quoted 12 lines

From: Mike Galbraith <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

just to bring this back to my problem :-)

Good idea :-)

Quoted 9 lines
Quoted 1 line

Last week I reported that the "new" sysfs entry in /proc/mounts already comes

out of initrd. Does this ring a bell?

Quoted 2 lines

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Nope, no bells.

The only thing I can suggest is that you try a bisection.

I was afraid that someone would bring up the bi-word :-) I am not using git so far, and my playground is behind a customers firewall.

In a first attempt I tried to be "cheap" and look for the first rcX, but unfortunatelly the new behaviour already starts with 29-rc1 :-(

Cheers
Martin

#16 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

In reply to Mike Galbraith

----- Original Message ----

Quoted 15 lines

From: Mike Galbraith <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

just to bring this back to my problem :-)

Good idea :-)

Quoted 1 line

Last week I reported that the "new" sysfs entry in /proc/mounts already comes

out of initrd. Does this ring a bell?

Quoted 1 line
Quoted 7 lines
Quoted 1 line

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Nope, no bells.

The only thing I can suggest is that you try a bisection.

-Mike

OK, so I finally managed to bisect the issue down to the following commit. Not much that I can say about it. Someone else suggested that it might all be a question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.

|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger <[email protected]>
|Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
|
|    Change the access routines to:
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger <[email protected]>
|    Reviewed-by: Matthew Wilcox <[email protected]>
|    Signed-off-by: Jesse Barnes <[email protected]>

Cheers
Martin

#17 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 37 lines · On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch <[email protected]>…

----- Original Message ----

Quoted 15 lines

From: Mike Galbraith <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

just to bring this back to my problem :-)

Good idea :-)

Quoted 1 line

Last week I reported that the "new" sysfs entry in /proc/mounts already comes

out of initrd. Does this ring a bell?

Quoted 1 line
Quoted 7 lines
Quoted 1 line

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Nope, no bells.

The only thing I can suggest is that you try a bisection.

-Mike

OK, so I finally managed to bisect the issue down to the following commit. Not much that I can say about it. Someone else suggested that it might all be a question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.

|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger <[email protected]>
|Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
|
Quoted 9 lines
|    Change the access routines to:
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger <[email protected]>
|    Reviewed-by: Matthew Wilcox <[email protected]>
|    Signed-off-by: Jesse Barnes <[email protected]>

<hello, any maintainers out there?>

So afacit what's happening is that the above change caused one of your PCI devices to take a very long time to initialise, yes? Was it the CCISS driver?

If you add "printk.time=y" to the kernel boot command line then you'll get timestamped boot messages which will make it easier to determine where the time was consumed. Adding `initcall_debug' to the boot line will help us delve further into the delay, assuming that the offending driver is build into vmlinux (which it might not be).

Either way, it would be useful to know which driver the above change broke.

Once we know that, the questions is: doe sthe driver still work? If so, then presumably the hardware if behaving unexpectedly, or in a way which we're failing to cope with.

Or perhaps that patch was simply buggy.

btw, I don't agree that this report should be closed for "fuzziness"! AFACIT the regression clearly and reproducibly occurs on one of your machines, yes? That ain't fuzzy!

#18 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

----- Original Message ----

Quoted 64 lines

From: Andrew Morton <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Mike Galbraith <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Kay Sievers <[email protected]>; [email protected]; Jesse Barnes <[email protected]>; Matthew Wilcox <[email protected]>
Sent: Wednesday, May 27, 2009 8:31:02 AM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 6 lines · On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch wrote:

----- Original Message ----

Quoted 3 lines

From: Mike Galbraith
To: Martin Knoblauch
Cc: Andrew Morton ; [email protected];

[email protected]; [email protected]; [email protected]

Quoted 10 lines
Quoted 10 lines

Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

just to bring this back to my problem :-)

Good idea :-)

Quoted 1 line

Last week I reported that the "new" sysfs entry in /proc/mounts already

comes

Quoted 11 lines
Quoted 9 lines

out of initrd. Does this ring a bell?

Quoted 2 lines

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Nope, no bells.

The only thing I can suggest is that you try a bisection.

-Mike

OK, so I finally managed to bisect the issue down to the following commit.

Not much that I can say about it. Someone else suggested that it might all be a question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.

Quoted 10 lines
|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger 
|Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
|
Quoted 9 lines
|    Change the access routines to:
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger 
|    Reviewed-by: Matthew Wilcox 
|    Signed-off-by: Jesse Barnes 
Quoted 4 lines

So afacit what's happening is that the above change caused one of your PCI devices to take a very long time to initialise, yes? Was it the CCISS driver?

the whole thing is not understood. I mentioned CCISS only because it is the most visible difference difference between my two test platforms.

Quoted 6 lines

If you add "printk.time=y" to the kernel boot command line then you'll get timestamped boot messages which will make it easier to determine where the time was consumed. Adding `initcall_debug' to the boot line will help us delve further into the delay, assuming that the offending driver is build into vmlinux (which it might not be).

added both options. "dmesg" output from both is appended. The initcall timings vary in both directions. For CCISS, they are actually faster for the "bad" case.

Quoted 3 lines

Either way, it would be useful to know which driver the above change broke.

agreed.

Quoted 4 lines

Once we know that, the questions is: doe sthe driver still work? If so, then presumably the hardware if behaving unexpectedly, or in a way which we're failing to cope with.

if it is CCISS, I can definitely say that it does work OK. As far as I can see, the whole system works OK, besides the duplicate sysfs line coming out of initrd.

Quoted 5 lines

Or perhaps that patch was simply buggy.

btw, I don't agree that this report should be closed for "fuzziness"! AFACIT the regression clearly and reproducibly occurs on one of your machines, yes? That ain't fuzzy!

frankly, I will stop caring about the DL380s before 2.6.31 gets released. My production kernels are not affected and the hotplug scripts can easily be fixed for testting. So, my interest is more curiosity. And the day-job does not really justify spending much more time on it. Aren't day-jobs annoying ...

Cheers
Martin

#19 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

In reply to Andrew Morton
Quoted 49 lines · Andrew Morton
Quoted 37 lines · On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch <[email protected]>…

----- Original Message ----

Quoted 15 lines

From: Mike Galbraith <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Andrew Morton <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

just to bring this back to my problem :-)

Good idea :-)

Quoted 1 line

Last week I reported that the "new" sysfs entry in /proc/mounts already comes

out of initrd. Does this ring a bell?

Quoted 1 line
Quoted 7 lines
Quoted 1 line

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Nope, no bells.

The only thing I can suggest is that you try a bisection.

-Mike

OK, so I finally managed to bisect the issue down to the following commit. Not much that I can say about it. Someone else suggested that it might all be a question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.

|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger <[email protected]>
|Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
|
Quoted 9 lines
|    Change the access routines to:
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger <[email protected]>
|    Reviewed-by: Matthew Wilcox <[email protected]>
|    Signed-off-by: Jesse Barnes <[email protected]>
Quoted 1 line

<hello, any maintainers out there?>

This is the first I've seen of this report ...

Quoted 9 lines

So afacit what's happening is that the above change caused one of your PCI devices to take a very long time to initialise, yes? Was it the CCISS driver?

If you add "printk.time=y" to the kernel boot command line then you'll get timestamped boot messages which will make it easier to determine where the time was consumed. Adding `initcall_debug' to the boot line will help us delve further into the delay, assuming that the offending driver is build into vmlinux (which it might not be).

The two message logs posted show NTP starting up within a second of each other. What was the problem again?

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#20 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

----- Original Message ----

Quoted 71 lines

From: Matthew Wilcox <[email protected]>
To: Andrew Morton <[email protected]>
Cc: Martin Knoblauch <[email protected]>; Mike Galbraith <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Kay Sievers <[email protected]>; [email protected]; Jesse Barnes <[email protected]>
Sent: Wednesday, May 27, 2009 1:21:53 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Andrew Morton

On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch

Quoted 7 lines · wrote:
Quoted 7 lines

----- Original Message ----

Quoted 3 lines

From: Mike Galbraith
To: Martin Knoblauch
Cc: Andrew Morton ; [email protected];

[email protected]; [email protected]; [email protected]

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

just to bring this back to my problem :-)

Good idea :-)

Quoted 1 line

Last week I reported that the "new" sysfs entry in /proc/mounts already

comes

Quoted 11 lines
Quoted 11 lines
Quoted 9 lines

out of initrd. Does this ring a bell?

Quoted 2 lines

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Nope, no bells.

The only thing I can suggest is that you try a bisection.

-Mike

OK, so I finally managed to bisect the issue down to the following commit.

Not much that I can say about it. Someone else suggested that it might all be a question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.

Quoted 11 lines
Quoted 11 lines
|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger 
|Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
|
|    Change the access routines to:
Quoted 10 lines
Quoted 8 lines
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger 
|    Reviewed-by: Matthew Wilcox 
|    Signed-off-by: Jesse Barnes 

This is the first I've seen of this report ...

Quoted 4 lines

So afacit what's happening is that the above change caused one of your PCI devices to take a very long time to initialise, yes? Was it the CCISS driver?

Quoted 9 lines
Quoted 5 lines

If you add "printk.time=y" to the kernel boot command line then you'll get timestamped boot messages which will make it easier to determine where the time was consumed. Adding `initcall_debug' to the boot line will help us delve further into the delay, assuming that the offending driver is build into vmlinux (which it might not be).

The two message logs posted show NTP starting up within a second of each other. What was the problem again?

Just to recap. Starting with 2.6.29-rc1, /proc/mounts has two "sysfs" lines:

|none /sys sysfs rw,relatime 0 0
| /sys /sys sysfs rw,relatime 0 0

The first of them already comes out of initrd, which somehow means that the explicit umount in the initrd/init script seems to fail. This is observable on a HP/DL380G4 with a "SmartArray 6" controller (cciss driver).

As a result of the two lines, the hotplug/firmware script in my RHEL4.3 userspace failed to determine the "/sys" mountpoint, which in turn resulted in a 60 second timeout for each firmware load attempt, adding 6 minutes to the boot sequence (4 CPUs, 2 tg3s). I have since then just fixed the hotplug script to do "the right thing". Therefore you are not seeing a big difference in the two posted dmesg logs. The underlying issue remains.

In a next step I managed to bisect the appearance of the second "sysfs" line down to the following commit:

|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger <[email protected]>
|Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
|
|    Change the access routines to:
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger <[email protected]>
|    Reviewed-by: Matthew Wilcox <[email protected]>
|    Signed-off-by: Jesse Barnes <[email protected]>

The commit itself may just show a problem/bug elsewhere. It definitely changes locking and timing around PCI.

Next, I tried to run an identically configured kernel on a different system (IBM/x3650 with aacraid). The problem was not observable. This now either points to CCISS, or leaves timing.

Finally, today I built the ccsii driver into the kernel. It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty? It is now loaded about 2 seconds earlier in the boot sequence, which is a big change in timing I guess.

Enlighten me :-)

Cheers
Martin

#21 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 94 lines · Martin Knoblauch

----- Original Message ----

Quoted 54 lines

From: Matthew Wilcox <[email protected]>
To: Andrew Morton <[email protected]>
Cc: Martin Knoblauch <[email protected]>; Mike Galbraith <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Kay Sievers <[email protected]>; [email protected]; Jesse Barnes <[email protected]>
Sent: Wednesday, May 27, 2009 1:21:53 PM
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Andrew Morton

On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch

Quoted 3 lines · wrote:
Quoted 3 lines

----- Original Message ----

Quoted 1 line

From: Mike Galbraith To: Martin Knoblauch Cc: Andrew Morton ; [email protected];

[email protected]; [email protected]; [email protected]

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Sent: Wednesday, May 6, 2009 10:37:45 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

just to bring this back to my problem :-)

Good idea :-)

Quoted 1 line

Last week I reported that the "new" sysfs entry in /proc/mounts already

comes

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

out of initrd. Does this ring a bell?

Quoted 1 line

http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html

Nope, no bells.

The only thing I can suggest is that you try a bisection.

-Mike

OK, so I finally managed to bisect the issue down to the following commit.

Not much that I can say about it. Someone else suggested that it might all be a question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.

Quoted 7 lines
Quoted 7 lines
|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger |Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
Quoted 9 lines
Quoted 8 lines
|
|    Change the access routines to:
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger |    Reviewed-by: Matthew Wilcox |    Signed-off-by: Jesse Barnes 

This is the first I've seen of this report ...

Quoted 4 lines

So afacit what's happening is that the above change caused one of your PCI devices to take a very long time to initialise, yes? Was it the CCISS driver?

Quoted 8 lines
Quoted 5 lines

If you add "printk.time=y" to the kernel boot command line then you'll get timestamped boot messages which will make it easier to determine where the time was consumed. Adding `initcall_debug' to the boot line will help us delve further into the delay, assuming that the offending driver is build into vmlinux (which it might not be).

The two message logs posted show NTP starting up within a second of each other. What was the problem again?

Just to recap. Starting with 2.6.29-rc1, /proc/mounts has two "sysfs" lines:

|none /sys sysfs rw,relatime 0 0
| /sys /sys sysfs rw,relatime 0 0

The first of them already comes out of initrd, which somehow means that the explicit umount in the initrd/init script seems to fail. This is observable on a HP/DL380G4 with a "SmartArray 6" controller (cciss driver).

As a result of the two lines, the hotplug/firmware script in my RHEL4.3 userspace failed to determine the "/sys" mountpoint, which in turn resulted in a 60 second timeout for each firmware load attempt, adding 6 minutes to the boot sequence (4 CPUs, 2 tg3s). I have since then just fixed the hotplug script to do "the right thing". Therefore you are not seeing a big difference in the two posted dmesg logs. The underlying issue remains.

In a next step I managed to bisect the appearance of the second "sysfs" line down to the following commit:

|commit 1120f8b8169fb2cb51219d326892d963e762edb6
|Author: Stephen Hemminger <[email protected]>
|Date:   Thu Dec 18 09:17:16 2008 -0800
|
|    PCI: handle long delays in VPD access
|
|    Accessing the VPD area can take a long time.  The existing
|    VPD access code fails consistently on my hardware. There are comments
|
|    Change the access routines to:
|      * use a mutex rather than spinning with IRQ's disabled and lock held
|      * have a much longer timeout
|      * call cond_resched while spinning
|
|    Signed-off-by: Stephen Hemminger <[email protected]>
|    Reviewed-by: Matthew Wilcox <[email protected]>
|    Signed-off-by: Jesse Barnes <[email protected]>
Quoted 7 lines

The commit itself may just show a problem/bug elsewhere. It definitely changes locking and timing around PCI.

Next, I tried to run an identically configured kernel on a different system (IBM/x3650 with aacraid). The problem was not observable. This now either points to CCISS, or leaves timing.

Finally, today I built the ccsii driver into the kernel. It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty? It is now loaded about 2 seconds earlier in the boot sequence, which is a big change in timing I guess.

Enlighten me :-)

No idea what is going on, but since I saw your May 20 message, I have been trying to wake up someone whose day job it should be to care about dl380s and smartarrays. :)

If they don't react soon, I'll make this my own day job to try to reproduce it. We will need hardware config details and firmware revs to start.

So I would not close the bug just yet.

jim

#22 lore raw

RE: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 7 lines
Quoted 2 lines

Finally, today I built the ccsii driver into the kernel.

It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty? It is now loaded about 2 seconds earlier in the boot sequence, which is a big change in timing I guess.

Quoted 1 line
Quoted 5 lines
Quoted 1 line

Enlighten me :-)

No idea what is going on, but since I saw your May 20 message, I have been trying to wake up someone whose day job it should be to care about dl380s and smartarrays. :)

What? I know Martin has pinged me in the past, but I do not think it was about this issue. If there's multiple sysfs entries for cciss I can't explain that offhand. We do little to nothing for sysfs in the driver. Had something similiar happen recently where "/" changed to "!". Had do to with our nested directory structure, /dev/cciss/name vs /dev/name. But we did not make that change, either.

-- mikem

Quoted 10 lines

If they don't react soon, I'll make this my own day job to try to reproduce it. We will need hardware config details and firmware revs to start.

So I would not close the bug just yet.

jim

#23 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 7 lines · Mike (OS Dev)
Quoted 4 lines
Quoted 1 line

Finally, today I built the ccsii driver into the kernel.

It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty? It is now loaded about 2 seconds earlier in the boot sequence, which is a big change in timing I guess.

Quoted 1 line

Enlighten me :-)

No idea what is going on, but since I saw your May 20 message, I have been trying to wake up someone whose day job it should be to care about dl380s and smartarrays. :)

What? I know Martin has pinged me in the past, but I do not think it was about this issue. If there's multiple sysfs entries for cciss I can't explain that offhand. We do little to nothing for sysfs in the driver. Had something similiar happen recently where "/" changed to "!". Had do to with our nested directory structure, /dev/cciss/name vs /dev/name. But we did not make that change, either.

It was not you that I was trying to get to look at this... but now that I have your attention :)

I did not think it pointed at cciss or the sysfs entry as being the problem. I was wondering more about platform probe timing, the bios, and how each card firmware reacts given what I thought the patch did. I could be totally wrong about that as I work at the filesystem level.

We have not seen the problem yet but we have different proliants/smartarrays.

If you can try it on matching hardware, great, if not then someone will eventually find a system to test it.

jim

#24 lore raw

RE: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 43 lines

-----Original Message-----
From: Owens, James
Sent: Wednesday, May 27, 2009 3:13 PM
To: Miller, Mike (OS Dev)
Cc: Martin Knoblauch; Matthew Wilcox; Andrew Morton; Mike Galbraith; [email protected]; [email protected]; [email protected]; [email protected]; Kay Sievers; [email protected]; Jesse Barnes
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 4 lines · Mike (OS Dev)
Quoted 4 lines
Quoted 1 line

Finally, today I built the ccsii driver into the kernel.

It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty? It is now loaded about 2 seconds earlier in the boot sequence, which

is a big

Quoted 3 lines
Quoted 3 lines

change in timing I guess.

Quoted 1 line

Enlighten me :-)

No idea what is going on, but since I saw your May 20

message, I have

Quoted 4 lines
Quoted 2 lines

been trying to wake up someone whose day job it should be to care about dl380s and smartarrays. :)

What? I know Martin has pinged me in the past, but I do not

think it was about this issue. If there's multiple sysfs entries for cciss I can't explain that offhand. We do little to nothing for sysfs in the driver.

Quoted 1 line

Had something similiar happen recently where "/" changed to

"!". Had do to with our nested directory structure, /dev/cciss/name vs /dev/name. But we did not make that change, either.

It was not you that I was trying to get to look at this... but now that I have your attention :)

I did not think it pointed at cciss or the sysfs entry as being the problem. I was wondering more about platform probe timing, the bios, and how each card firmware reacts given what I thought the patch did. I could be totally wrong about that as I work at the filesystem level.

Quoted 6 lines

We have not seen the problem yet but we have different proliants/smartarrays.

If you can try it on matching hardware, great, if not then someone will eventually find a system to test it.

Sorry, I don't have any hw that old. We're very limited on space these days.

-- mikem

#25 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

In reply to Miller, Mike (OS Dev)

----- Original Message ----

Quoted 19 lines

From: "Miller, Mike (OS Dev)" <[email protected]>
To: "Owens, James" <[email protected]>; Martin Knoblauch <[email protected]>
Cc: Matthew Wilcox <[email protected]>; Andrew Morton <[email protected]>; Mike Galbraith <[email protected]>; "[email protected]" <[email protected]>; "[email protected]" <[email protected]>; "[email protected]" <[email protected]>; "[email protected]" <[email protected]>; Kay Sievers <[email protected]>; "[email protected]" <[email protected]>; Jesse Barnes <[email protected]>
Sent: Wednesday, May 27, 2009 8:18:46 PM
Subject: RE: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 7 lines
Quoted 2 lines

Finally, today I built the ccsii driver into the kernel.

It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty? It is now loaded about 2 seconds earlier in the boot sequence, which is a big change in timing I guess.

Quoted 1 line
Quoted 5 lines
Quoted 1 line

Enlighten me :-)

No idea what is going on, but since I saw your May 20 message, I have been trying to wake up someone whose day job it should be to care about dl380s and smartarrays. :)

Quoted 7 lines

What? I know Martin has pinged me in the past, but I do not think it was about this issue. If there's multiple sysfs entries for cciss I can't explain that offhand. We do little to nothing for sysfs in the driver.
Had something similiar happen recently where "/" changed to "!". Had do to with our nested directory structure, /dev/cciss/name vs /dev/name. But we did not make that change, either.

Hi Mike, Jim

you are right. I never talked to you about this issue. Actually, I did not suspect CCISS or the DL380 in general to be involved before last week when I retested on the x3650 and the problem went away. Due to day-job and priorities I did not follow up.

Quoted 7 lines

-- mikem

Quoted 5 lines

If they don't react soon, I'll make this my own day job to try to reproduce it. We will need hardware config details and firmware revs to start.

Dl380/G4
2x3.4 GHz CPUs
8 GB Memeory
4x72 GB as RAID5 on SA6i controller

BIOS: P51 (07(19/2007)
ILO FW: 1.91
SA6i FW: 2.84

lspci output (-vvv and -xxx) appendend to prevent line wrapping.

Cheers
Martin

#26 lore raw

RE: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 55 lines

-----Original Message-----
From: Martin Knoblauch [mailto:[email protected]]
Sent: Thursday, May 28, 2009 4:00 AM
To: Miller, Mike (OS Dev); Owens, James
Cc: Matthew Wilcox; Andrew Morton; Mike Galbraith; [email protected]; [email protected]; [email protected]; [email protected]; Kay Sievers; [email protected]; Jesse Barnes
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

----- Original Message ----

Quoted 6 lines

From: "Miller, Mike (OS Dev)" <[email protected]>
To: "Owens, James" <[email protected]>; Martin Knoblauch <[email protected]>
Cc: Matthew Wilcox <[email protected]>; Andrew Morton <[email protected]>; Mike Galbraith <[email protected]>; "[email protected]" <[email protected]>; "[email protected]"

Quoted 13 lines

<[email protected]>; "[email protected]" <[email protected]>; "[email protected]" <[email protected]>; Kay Sievers <[email protected]>; "[email protected]" <[email protected]>; Jesse Barnes <[email protected]>
Sent: Wednesday, May 27, 2009 8:18:46 PM
Subject: RE: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 4 lines
Quoted 2 lines

Finally, today I built the ccsii driver into the kernel.

It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty?

It is now

Quoted 1 line
Quoted 1 line

loaded about 2 seconds earlier in the boot sequence,

which is a big

Quoted 9 lines
Quoted 7 lines

change in timing I guess.

Quoted 2 lines

Enlighten me :-)

No idea what is going on, but since I saw your May 20 message, I have been trying to wake up someone whose day job it should be to care about dl380s and smartarrays. :)

What? I know Martin has pinged me in the past, but I do not

think it

Quoted 2 lines

was about this issue. If there's multiple sysfs entries for cciss I can't explain that offhand. We do little to nothing for

sysfs in the driver.

Quoted 1 line

Had something similiar happen recently where "/" changed to

"!". Had

Quoted 3 lines

do to with our nested directory structure, /dev/cciss/name vs /dev/name. But we did not make that change, either.

Quoted 7 lines

Hi Mike, Jim

you are right. I never talked to you about this issue.
Actually, I did not suspect CCISS or the DL380 in general to be involved before last week when I retested on the x3650 and the problem went away. Due to day-job and priorities I did not follow up.

Martin,
Same exact OS install? Are you sure all the userspace stuff is the same?

-- mikem

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines

If they don't react soon, I'll make this my own day job to try to reproduce it. We will need hardware config details and firmware revs to start.

Dl380/G4
2x3.4 GHz CPUs
8 GB Memeory
4x72 GB as RAID5 on SA6i controller

BIOS: P51 (07(19/2007)
ILO FW: 1.91
SA6i FW: 2.84

lspci output (-vvv and -xxx) appendend to prevent line wrapping.

Cheers
Martin

#27 lore raw

Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

----- Original Message ----

Quoted 63 lines

From: "Miller, Mike (OS Dev)" <[email protected]>
To: Martin Knoblauch <[email protected]>; "Owens, James" <[email protected]>
Cc: Matthew Wilcox <[email protected]>; Andrew Morton <[email protected]>; Mike Galbraith <[email protected]>; "[email protected]" <[email protected]>; "[email protected]" <[email protected]>; "[email protected]" <[email protected]>; "[email protected]" <[email protected]>; Kay Sievers <[email protected]>; "[email protected]" <[email protected]>; Jesse Barnes <[email protected]>
Sent: Thursday, May 28, 2009 9:01:15 PM
Subject: RE: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 55 lines

-----Original Message-----
From: Martin Knoblauch [mailto:[email protected]]
Sent: Thursday, May 28, 2009 4:00 AM
To: Miller, Mike (OS Dev); Owens, James
Cc: Matthew Wilcox; Andrew Morton; Mike Galbraith; [email protected]; [email protected]; [email protected]; [email protected]; Kay Sievers; [email protected]; Jesse Barnes
Subject: Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

----- Original Message ----

Quoted 11 lines

From: "Miller, Mike (OS Dev)"
To: "Owens, James" ; Martin Knoblauch

Cc: Matthew Wilcox ; Andrew Morton
; Mike Galbraith ;
"[email protected]" ; "[email protected]"
; "[email protected]"
; "[email protected]"
; Kay Sievers ;
"[email protected]" ; Jesse Barnes

Quoted 8 lines

Sent: Wednesday, May 27, 2009 8:18:46 PM
Subject: RE: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Quoted 4 lines
Quoted 2 lines

Finally, today I built the ccsii driver into the kernel.

It was previously modularized and loaded from initrd. The second "sysfs" line went away. But does this make cciss guilty?

It is now

Quoted 1 line
Quoted 1 line

loaded about 2 seconds earlier in the boot sequence,

which is a big

Quoted 9 lines
Quoted 7 lines

change in timing I guess.

Quoted 2 lines

Enlighten me :-)

No idea what is going on, but since I saw your May 20 message, I have been trying to wake up someone whose day job it should be to care about dl380s and smartarrays. :)

What? I know Martin has pinged me in the past, but I do not

think it

Quoted 2 lines

was about this issue. If there's multiple sysfs entries for cciss I can't explain that offhand. We do little to nothing for

sysfs in the driver.

Quoted 1 line

Had something similiar happen recently where "/" changed to

"!". Had

Quoted 3 lines

do to with our nested directory structure, /dev/cciss/name vs /dev/name. But we did not make that change, either.

Hi Mike, Jim

Quoted 9 lines
Quoted 5 lines

you are right. I never talked to you about this issue.
Actually, I did not suspect CCISS or the DL380 in general to be involved before last week when I retested on the x3650 and the problem went away. Due to day-job and priorities I did not follow up.

Martin,
Same exact OS install? Are you sure all the userspace stuff is the same?

yup. Exactely same user-space. Same kernel config. Only difference is aacraid vs. cciss module in initrd

Quoted 23 lines

-- mikem

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines

If they don't react soon, I'll make this my own day job to try to reproduce it. We will need hardware config details and firmware revs to start.

Dl380/G4
2x3.4 GHz CPUs
8 GB Memeory
4x72 GB as RAID5 on SA6i controller

BIOS: P51 (07(19/2007)
ILO FW: 1.91
SA6i FW: 2.84

lspci output (-vvv and -xxx) appendend to prevent line wrapping.

Cheers
Martin

#28 lore raw
In reply to Al Viro

----- Original Message ----

Quoted 24 lines

From: Al Viro <[email protected]>
To: Andrew Morton <[email protected]>
Cc: Mike Galbraith <[email protected]>; Martin Knoblauch <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>
Sent: Wednesday, April 29, 2009 2:08:28 PM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Andrew Morton
Quoted 1 line
Quoted 1 line
Quoted 1 line

Questions remains: was this intentional? It breaks existing userspace

and should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

Quoted 12 lines
Quoted 11 lines
Quoted 9 lines

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

[hotplug]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

___(I wonder how the heck that is accomplished)

Quoted 7 lines
Quoted 3 lines

Beats me. I'm not seeing likely changes in fs/proc/base.c or around show_mountinfo(). Maybe sysfs broke in an ingenious way. (hopefully cc's viro).

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

One definitely comes from /etc/fstab, but I am not aware of any other script mounting sysfs in my userspace.

Cheers
Martin

#29 lore raw
Quoted 1 line · Martin Knoblauch

One definitely comes from /etc/fstab, but I am not aware of any other script

mounting sysfs in my userspace.

You said it was a RedHat box?

Look in /etc/rc.sysinit:

        mount -n -t sysfs /sys /sys >/dev/null 2>&1

(Near line 28 for RHEL 4, line 23 for RHEL5, and line 21 for Fedora Rawhide)

Probably your culprit.

#30 lore raw
Quoted 7 lines · On Wed, Apr 29, 2009 at 01:35:59PM -0400, [email protected] wrote:
Quoted 1 line · Martin Knoblauch

One definitely comes from /etc/fstab, but I am not aware of any other script

mounting sysfs in my userspace.

You said it was a RedHat box?

Quoted 7 lines

Look in /etc/rc.sysinit:

        mount -n -t sysfs /sys /sys >/dev/null 2>&1

(Near line 28 for RHEL 4, line 23 for RHEL5, and line 21 for Fedora Rawhide)

Probably your culprit.

Again, the interesting question is WTF had mount(2) not failed with -EBUSY

#31 lore raw
Quoted 1 line · Al Viro

Again, the interesting question is WTF had mount(2) not failed with -EBUSY

A change of '/' between the two mounts maybe? If one is done from initrd and the other after / is mounted?

OG.

#32 lore raw
In reply to [email protected]

----- Original Message ----

Quoted 13 lines

From: "[email protected]" <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Al Viro <[email protected]>; Andrew Morton <[email protected]>; Mike Galbraith <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>
Sent: Wednesday, April 29, 2009 7:35:59 PM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 1 line · Martin Knoblauch

One definitely comes from /etc/fstab, but I am not aware of any other script

mounting sysfs in my userspace.

You said it was a RedHat box?

Quoted 7 lines

Look in /etc/rc.sysinit:

        mount -n -t sysfs /sys /sys >/dev/null 2>&1

(Near line 28 for RHEL 4, line 23 for RHEL5, and line 21 for Fedora Rawhide)

Probably your culprit.

OK, you got me. But still something changed with 2.6.29. Before, there was only one line in /proc/mounts

Cheers
Martin

#33 lore raw
In reply to Martin Knoblauch
Quoted 5 lines · Martin Knoblauch
Quoted 3 lines

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

One definitely comes from /etc/fstab, but I am not aware of any other script mounting sysfs in my userspace.

Check in /etc/init.d; e.g. on debian it's mountkernfs. More interesting question is what else is mounted on /sys; how about the entire /proc/mounts contents?

#34 lore raw

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

----- Original Message ----

Quoted 12 lines

From: Al Viro <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Andrew Morton <[email protected]>; Mike Galbraith <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>
Sent: Wednesday, April 29, 2009 7:41:54 PM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 4 lines · Martin Knoblauch
Quoted 3 lines

Er... Somebody mounting sysfs twice? From some init script and from /etc/fstab, perhaps? That definitely looks like two mount(2) had to have been done to cause that...

Quoted 6 lines
Quoted 1 line

One definitely comes from /etc/fstab, but I am not aware of any other script

mounting sysfs in my userspace.

Check in /etc/init.d; e.g. on debian it's mountkernfs. More interesting question is what else is mounted on /sys; how about the entire /proc/mounts contents?

2.6.30-rc3-git2:

rootfs / rootfs rw 0 0
/proc /proc proc rw,relatime 0 0
none /sys sysfs rw,relatime 0 0
none /dev tmpfs rw,relatime,mode=755 0 0
/dev/root / ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev tmpfs rw,relatime,mode=755 0 0
/proc /proc proc rw,relatime 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0
none /dev/pts devpts rw,relatime,gid=5,mode=620 0 0 /dev/cciss/c0d0p1 /boot ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev/shm tmpfs rw,relatime 0 0
/dev/cciss/c0d0p4 /scratch ext2 rw,noatime,errors=continue 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0

2.6.19.2

rootfs / rootfs rw 0 0
/proc /proc proc rw 0 0
none /dev tmpfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
none /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
/sys /sys sysfs rw 0 0
none /dev/pts devpts rw 0 0
/dev/cciss/c0d0p1 /boot ext3 rw,data=ordered 0 0 none /dev/shm tmpfs rw 0 0 /dev/VolGroup00/LogVol02 /scratch ext2 rw,noatime 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

Cheers
Martin

#35 lore raw
Quoted 17 lines · Martin Knoblauch

2.6.30-rc3-git2:

rootfs / rootfs rw 0 0
/proc /proc proc rw,relatime 0 0
none /sys sysfs rw,relatime 0 0
none /dev tmpfs rw,relatime,mode=755 0 0
/dev/root / ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev tmpfs rw,relatime,mode=755 0 0
/proc /proc proc rw,relatime 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0
none /dev/pts devpts rw,relatime,gid=5,mode=620 0 0 /dev/cciss/c0d0p1 /boot ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev/shm tmpfs rw,relatime 0 0
/dev/cciss/c0d0p4 /scratch ext2 rw,noatime,errors=continue 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0

Cute... I wonder how that second mount(2) managed to succeed. Could you check what explicit mounting of sysfs on that point *again* does?

#36 lore raw

----- Original Message ----

Quoted 25 lines

From: Al Viro <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Andrew Morton <[email protected]>; Mike Galbraith <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>
Sent: Wednesday, April 29, 2009 8:10:41 PM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 17 lines · Martin Knoblauch

2.6.30-rc3-git2:

rootfs / rootfs rw 0 0
/proc /proc proc rw,relatime 0 0
none /sys sysfs rw,relatime 0 0
none /dev tmpfs rw,relatime,mode=755 0 0
/dev/root / ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev tmpfs rw,relatime,mode=755 0 0
/proc /proc proc rw,relatime 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0
none /dev/pts devpts rw,relatime,gid=5,mode=620 0 0 /dev/cciss/c0d0p1 /boot ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev/shm tmpfs rw,relatime 0 0
/dev/cciss/c0d0p4 /scratch ext2 rw,noatime,errors=continue 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0

Quoted 2 lines

Cute... I wonder how that second mount(2) managed to succeed. Could you check what explicit mounting of sysfs on that point *again* does?

Hi Al,

by now I know a bit more. Starting with 2.6.29, the

none /sys sysfs rw 0 0

line is already in /proc/mounts when entering userspace. So I guess it comes out of my initrd. Again, what changed?

Cheers
Martin

#37 lore raw
Quoted 30 lines · On Thu, 30 Apr 2009 02:12:00 -0700 (PDT) Martin Knoblauch <[email protected]>…

----- Original Message ----

Quoted 25 lines

From: Al Viro <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Andrew Morton <[email protected]>; Mike Galbraith <[email protected]>; Rafael J. Wysocki <[email protected]>; [email protected]; tigran aivazian <[email protected]>
Sent: Wednesday, April 29, 2009 8:10:41 PM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 17 lines · Martin Knoblauch

2.6.30-rc3-git2:

rootfs / rootfs rw 0 0
/proc /proc proc rw,relatime 0 0
none /sys sysfs rw,relatime 0 0
none /dev tmpfs rw,relatime,mode=755 0 0
/dev/root / ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev tmpfs rw,relatime,mode=755 0 0
/proc /proc proc rw,relatime 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0
none /dev/pts devpts rw,relatime,gid=5,mode=620 0 0 /dev/cciss/c0d0p1 /boot ext3 rw,relatime,errors=continue,data=writeback 0 0 none /dev/shm tmpfs rw,relatime 0 0
/dev/cciss/c0d0p4 /scratch ext2 rw,noatime,errors=continue 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0

Quoted 2 lines

Cute... I wonder how that second mount(2) managed to succeed. Could you check what explicit mounting of sysfs on that point *again* does?

Quoted 8 lines

Hi Al,

by now I know a bit more. Starting with 2.6.29, the

none /sys sysfs rw 0 0

line is already in /proc/mounts when entering userspace. So I guess it comes out of my initrd. Again, what changed?

afacit this didn't get understood, let alone fixed?

I see that you had a PCI-related issue which changed boot timing a lot, but that isn't the bug - it simply exposed the bug by changing timing?

#38 lore raw
In reply to Andrew Morton

----- Original Message ----

Quoted 28 lines

From: Andrew Morton <[email protected]>
To: Martin Knoblauch <[email protected]>
Cc: Rafael J. Wysocki <[email protected]>; [email protected]; [email protected]; tigran aivazian <[email protected]>
Sent: Wednesday, April 29, 2009 3:28:37 AM
Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow

Quoted 3 lines · On Fri, 24 Apr 2009 05:45:19 -0700 (PDT) Martin Knoblauch wrote:

OK, I just found the reason for both intel-ucode and tg3 failures. Apparently

between 2.6.28 and 2.6.29 the mount of sysfs has subtely changed from:

Quoted 6 lines

/sys /sys sysfs rw 0 0

to:

none /sys sysfs rw,relatime 0 0

I assume that you're referring to the contents of /proc/mounts?

Quoted 1 line

The "none" breaks the RHEL-4 provided hotplug script "firmware.agent" when it

tries to parse the mount point for "/sys". As a result, the firmware loading is never properly finished and the driver(s) just timeout on the value in /sys/class/firmware/timeout. Bingo. Simple fix in user-pace possible - cool down Martin :-)

Quoted 1 line
Quoted 8 lines
Quoted 1 line

Questions remains: was this intentional? It breaks existing userspace and

should therefore be considered a regression - right? On the other hand, it will never be a problem for RHEL-4/5 kernels, unless the change in 2.6.29 gets backported. Any ideas?

afaik that was unintentional and was probably a mistake.

I wonder how we did that.

Actually, what breaks the RHEL-4.3 script is not the "none", but the duplicate lines in /proc/mounts that I reported earlier in the "regression" thread.

[root@lpsdm52]# grep sysfs /proc/mounts
none /sys sysfs rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

One of them likely comes from the respective line in /etc/fstab, but where does the second one come from?

Cheers
Martin