introduce crashboot kernel command line parameter

o cciss driver initialization fails and hits BUG() if underlying device was active during the driver initialization. Device might be active if previous kernel crashed and this kernel is booting after that using kdump.

...

o If crash_boot parameter is set, then ignore the completed command messages sent by device which have not been issued in the context of this kernel.

Signed-off-by Vivek Goyal <[email protected]>

---

 linux-2.6.17-1M-vivek/drivers/block/cciss.c |    7 +++++++
 1 files changed, 7 insertions(+)

diff -puN drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix drivers/block/cciss.c

drivers/block/cciss.c renamed from drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix +0 -0

@@ -1976,6 +1976,13 @@ static int add_sendcmd_reject(__u8 cmd, 
 			ctlr, complete);

Quoted 28 lines · On Fri, 23 Jun 2006 17:04:24 -0400 Vivek Goyal <[email protected]> wrote:

o cciss driver initialization fails and hits BUG() if underlying device was active during the driver initialization. Device might be active if previous kernel crashed and this kernel is booting after that using kdump.

...

o If crash_boot parameter is set, then ignore the completed command messages sent by device which have not been issued in the context of this kernel.

Signed-off-by Vivek Goyal <[email protected]>

---

 linux-2.6.17-1M-vivek/drivers/block/cciss.c |    7 +++++++
 1 files changed, 7 insertions(+)

diff -puN drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix drivers/block/cciss.c

drivers/block/cciss.c renamed from drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix +3 -0

@@ -1976,6 +1976,13 @@ static int add_sendcmd_reject(__u8 cmd, 
 			ctlr, complete);
 		/* not much we can do. */
 #ifdef CONFIG_CISS_SCSI_TAPE
+		/* We might get notification of completion of commands
+		 * which we never issued in this kernel if this boot is
+		 * taking place after previous kernel's crash. Simply

drivers/block/cciss.c static int add_sendcmd_reject(__u8 cmd, · line 1986

Quoted 5 lines

drivers/block/cciss.c renamed from drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix +4 -0

+		 * ignore the commands in this case.
+		 */
+		if (crash_boot)
+			return 0;
 		return 1;

Looks like this is working around a driver problem rather than fixing it properly ;)

Quoted 31 lines · Andrew Morton

Quoted 28 lines · On Fri, 23 Jun 2006 17:04:24 -0400 Vivek Goyal <[email protected]> wrote:

o cciss driver initialization fails and hits BUG() if underlying device was active during the driver initialization. Device might be active if previous kernel crashed and this kernel is booting after that using kdump.

...

o If crash_boot parameter is set, then ignore the completed command messages sent by device which have not been issued in the context of this kernel.

Signed-off-by Vivek Goyal <[email protected]>

---

 linux-2.6.17-1M-vivek/drivers/block/cciss.c |    7 +++++++
 1 files changed, 7 insertions(+)

diff -puN drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix drivers/block/cciss.c

drivers/block/cciss.c renamed from drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix +3 -0

@@ -1976,6 +1976,13 @@ static int add_sendcmd_reject(__u8 cmd, 
 			ctlr, complete);
 		/* not much we can do. */
 #ifdef CONFIG_CISS_SCSI_TAPE
+		/* We might get notification of completion of commands
+		 * which we never issued in this kernel if this boot is
+		 * taking place after previous kernel's crash. Simply

Quoted 8 lines

drivers/block/cciss.c static int add_sendcmd_reject(__u8 cmd, · line 1986

Quoted 5 lines

drivers/block/cciss.c renamed from drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix +4 -0

+		 * ignore the commands in this case.
+		 */
+		if (crash_boot)
+			return 0;
 		return 1;

Looks like this is working around a driver problem rather than fixing it properly ;)

That's true. Its more of a working around the problem. I think in all such cases we should soft reset the device so that device drops the messages issued from the context of previous kernel and starts afresh.

But looks like not all the devices provide software reset facility (Or I can't find it out from the source code or limited documentation available). Mike, can I soft reset this device?

I am facing similar problem in megaraid driver as well where detailed technical documentation is not available and I can't find a way to soft reset the device.

Or is there a generic way to handle these situations? Fixing them driver by driver is a long painful process.

Quoted 20 lines

Quoted 11 lines

Quoted 11 lines

diff -puN drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix drivers/block/cciss.c

drivers/block/cciss.c renamed from drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix +3 -0

@@ -1976,6 +1976,13 @@ static int add_sendcmd_reject(__u8 cmd, 
 			ctlr, complete);
 		/* not much we can do. */
 #ifdef CONFIG_CISS_SCSI_TAPE
+		/* We might get notification of completion of commands
+		 * which we never issued in this kernel if this boot is
+		 * taking place after previous kernel's crash. Simply

Quoted 8 lines

drivers/block/cciss.c static int add_sendcmd_reject(__u8 cmd, · line 1986

Quoted 5 lines

drivers/block/cciss.c renamed from drivers/block/cciss.c~cciss-initialization-issue-over-kdump-fix +4 -0

+		 * ignore the commands in this case.
+		 */
+		if (crash_boot)
+			return 0;
 		return 1;

Looks like this is working around a driver problem rather than fixing it properly ;)

Quoted 3 lines

That's true. Its more of a working around the problem. I think in all such cases we should soft reset the device so that device drops the messages issued from the context of previous kernel and starts afresh.

Sounds good.

Quoted 4 lines

But looks like not all the devices provide software reset facility (Or I can't find it out from the source code or limited documentation available). Mike, can I soft reset this device?

Quoted 3 lines

I am facing similar problem in megaraid driver as well where detailed technical documentation is not available and I can't find a way to soft reset the device.

Megaraid has a maintainer who has documents and hardware engineers.

Quoted 2 lines

Or is there a generic way to handle these situations? Fixing them driver by driver is a long painful process.

Some generic way of whacking a PCI device via the standard PCI registers? Not that I know of.

Quoted 3 lines · Andrew Morton

Quoted 2 lines

Or is there a generic way to handle these situations? Fixing them driver by driver is a long painful process.

Quoted 2 lines

Some generic way of whacking a PCI device via the standard PCI registers? Not that I know of.

Somebody hinted that think of PCI bus reset. But I think PCI bus reset will require firware/BIOS to export a hook to software to so initiate PCI bus reset and I don't think many platforms do that. Infact I am not even aware of one platform who does that.

Quoted 5 lines · Vivek Goyal

Quoted 3 lines · Andrew Morton

Quoted 2 lines

Or is there a generic way to handle these situations? Fixing them driver by driver is a long painful process.

Quoted 7 lines

Quoted 2 lines

Some generic way of whacking a PCI device via the standard PCI registers? Not that I know of.

Somebody hinted that think of PCI bus reset. But I think PCI bus reset will require firware/BIOS to export a hook to software to so initiate PCI bus reset and I don't think many platforms do that. Infact I am not even aware of one platform who does that.

Not all pci busses support it but there is a standard pci bus reset bit in pci bridges.

I don't know if it would help but it might make sense to have a config option that can be used to mark drivers that are known to have problems, in these scenarios.

CONFIG_BRITTLE_INIT perhaps?

It would at least make it easier for people to see which drivers they don't want to use, and give people some incentive to fix things.

Quoted 18 lines · Eric W. Biederman

Quoted 5 lines · Vivek Goyal

Quoted 3 lines · Andrew Morton

Quoted 2 lines

Or is there a generic way to handle these situations? Fixing them driver by driver is a long painful process.

Quoted 7 lines

Quoted 2 lines

Some generic way of whacking a PCI device via the standard PCI registers? Not that I know of.

Somebody hinted that think of PCI bus reset. But I think PCI bus reset will require firware/BIOS to export a hook to software to so initiate PCI bus reset and I don't think many platforms do that. Infact I am not even aware of one platform who does that.

Not all pci busses support it but there is a standard pci bus reset bit in pci bridges.

Quoted 9 lines

I don't know if it would help but it might make sense to have a config option that can be used to mark drivers that are known to have problems, in these scenarios.

CONFIG_BRITTLE_INIT perhaps?

It would at least make it easier for people to see which drivers they don't want to use, and give people some incentive to fix things.

Vivek,

I think having something as Eric suggested instead of crashboot= is better. We can hve this config option set for kernel like dump capture kernel. (CONFIG_CRASH_DUMP=y). This should save some bytes on already longish kdump kernel boot paramenters.

On Mon, Jun 26, 2006 at 07:41:00AM +0530, Maneesh Soni wrote:

Maneesh, Keeping this code under a config option becomes a problem when we will have a relocatable kernel. At some point of time we got to have relocatable kernel so that people don't have to build two kernels. In fact this is becoming a pain area for distros. That's the reason I thought of making it a command line parameter.

I remember few months back, Eric had mentioned that he has got patches for relocatable kernel ready for review for i386 and x86_64. Eric, do you have any plans to post the patches for review?

Quoted 7 lines · Vivek Goyal

On Mon, Jun 26, 2006 at 07:41:00AM +0530, Maneesh Soni wrote:

Maneesh, Keeping this code under a config option becomes a problem when we will have a relocatable kernel. At some point of time we got to have relocatable kernel so that people don't have to build two kernels. In fact this is becoming a pain area for distros. That's the reason I thought of making it a command line parameter.

Ok. Even if we do this with a command line, we need to have a clean concept. If the concept is ignore devices with a brittle init routine that is comprehensible and potentially useful for other reasons than crash dumps.

If the concept is crashdump it is a poorly defined concept and all of Andrews objections apply.

Quoted 3 lines

I remember few months back, Eric had mentioned that he has got patches for relocatable kernel ready for review for i386 and x86_64. Eric, do you have any plans to post the patches for review?

I have some code that I keep intending to get to. It has probably bit rotted since I wrote it, but it shouldn't be too bad to clean up. Unfortunately the whole crashdump thing is fairly low on my priority list.

Quoted 4 lines · Eric W. Biederman

Quoted 2 lines · Vivek Goyal

On Mon, Jun 26, 2006 at 07:41:00AM +0530, Maneesh Soni wrote:

Quoted 8 lines

Quoted 5 lines

Maneesh, Keeping this code under a config option becomes a problem when we will have a relocatable kernel. At some point of time we got to have relocatable kernel so that people don't have to build two kernels. In fact this is becoming a pain area for distros. That's the reason I thought of making it a command line parameter.

Ok. Even if we do this with a command line, we need to have a clean concept. If the concept is ignore devices with a brittle init routine that is

comprehensible

Quoted 2 lines

and potentially useful for other reasons than crash dumps.

Looks like there are two problems to be solved.

- Framework/capability to mark and isolate the drivers, either at compile time or run time, which are not hardened enough to initialize properly when the underlying device is in operational or in unknown state.

- Actually hardening a driver to be able to initialize in a potentially unreliable environment.

Solving first problem will help more in terms of people knowing in advance that certain drivers are known to have problems in specific environemnt and a user has got the option of skipping the execution/compilation of those drivers. (This is something close to what CONFIG_EXPERIMENTAL does)

Second problem deals more with actually hardening the driver and not skipping its compilation/execution.

I think people would like to change a driver's behaviour at run time. For example if they are booting in a unreliable environment they would like to reset the device otherwise they would skip that as BIOS has already done that for them.

But looks like not all devices have got the capablity to be reset from software. In those cases probably one need to put some hooks, relax driver's consistency checks etc in special boot environment.

Here I am trying to solve the second problem so that a driver comes to know that it is initializing in a special boot environment and it can modify its behavior at run time.

Quoted 3 lines

If the concept is crashdump it is a poorly defined concept and all of Andrews objections apply.

I think this parameter is generic enough and not limited to crashdumps. If a user decides to implement a different scheme than kdump in kexec on panic and boot a customized kernel, he can very well use this parameter to make sure that next kernel is able to at least boot and not panic in between.

Solving first problem will help in doing a plain kexec. We can simply mark the drivers known to have problems and slowly people can fix those drivers. Fixing the driver in this case is different because most likely driver authors will provide a shutdown routine in the driver so that device can be shutdown and then one can boot into the second kernel. Till then a user can happily skip the drivers known to have problems.

In summary, we got two problems to solve. Currently I am focussed on solving second problem which enables boot a kernel in an unreliable environment and do some minimal specific operation and then boot back to regular kernela. So I think just introducing a command line parameter which drivers can use to determine that they are initializing in an special environement, solves it and is generic enough.

Options like COFIG_BRITTLE_INIT or sikkping execution of brittle driver based on a command line option seems to be the solution for the first problem.

Please correct me if I am wrong. I know little about drivers.

Quoted 4 lines

Quoted 3 lines

I remember few months back, Eric had mentioned that he has got patches for relocatable kernel ready for review for i386 and x86_64. Eric, do you have any plans to post the patches for review?

Quoted 4 lines

I have some code that I keep intending to get to. It has probably bit rotted since I wrote it, but it shouldn't be too bad to clean up. Unfortunately the whole crashdump thing is fairly low on my priority list.

I am willing to work on it. Building from scratch always takes more time. If you are willing, I will more than happy to build on top of your patches.

-----Original Message-----
From: Eric W. Biederman [mailto:[email protected]]
Sent: Monday, June 26, 2006 11:01 AM
To: [email protected]
Cc: Maneesh Soni; Andrew Morton; [email protected]; [email protected]; Miller, Mike (OS Dev); [email protected]; [email protected]
Subject: Re: [Fastboot] [RFC] [PATCH 2/2] kdump: cciss driver initialization issue fix

Quoted 6 lines · Vivek Goyal

Quoted 5 lines · Eric W. Biederman

Quoted 3 lines · Vivek Goyal

On Mon, Jun 26, 2006 at 07:41:00AM +0530, Maneesh Soni wrote:

Maneesh, Keeping this code under a config option becomes

a problem

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

when we will have a relocatable kernel. At some point of time we got to have relocatable kernel so that people don't have

to build

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

two kernels. In fact this is becoming a pain area for distros. That's the reason I thought of making it a command line

parameter.

Quoted 2 lines

Quoted 2 lines

Ok. Even if we do this with a command line, we need to

have a clean concept.

Quoted 1 line

Quoted 1 line

If the concept is ignore devices with a brittle init

routine that is

Quoted 7 lines

comprehensible

Quoted 2 lines

and potentially useful for other reasons than crash dumps.

Looks like there are two problems to be solved.

- Framework/capability to mark and isolate the drivers,

either at compile

Quoted 1 line

time or run time, which are not hardened enough to

initialize properly

Quoted 3 lines

when the underlying device is in operational or in unknown state.

- Actually hardening a driver to be able to initialize in a

potentially

Quoted 10 lines

unreliable environment.

Solving first problem will help more in terms of people knowing in advance that certain drivers are known to have problems in specific environemnt and a user has got the option of skipping the execution/compilation of those drivers. (This is something close to what CONFIG_EXPERIMENTAL does)

Second problem deals more with actually hardening the

driver and not

Quoted 3 lines

skipping its compilation/execution.

I think people would like to change a driver's behaviour at

run time.

Quoted 1 line

For example if they are booting in a unreliable environment

they would

Quoted 2 lines

like to reset the device otherwise they would skip that as BIOS has already done that for them.

In the general case the device reset does not hurt. Yes there is the case of the slow scsi probe. But a lot of that appears to be a poor implementation of the scsi probe. So I can see a kernel command line option to play fast and loose but we should be safe and thorough by default.

The more code paths you introduce the harder code is to maintain and test. The earlier discussion suggested you cannot harden some drivers. We can take action against drivers like that simply and easily.

Hacks in the driver initialization are a completely different story.

Quoted 1 line

But looks like not all devices have got the capablity to be

reset from

Quoted 2 lines

software. In those cases probably one need to put some hooks, relax driver's consistency checks etc in special boot environment.

Forget the concept of a special boot environment. A buggy BIOS or rebooting after being in windows can potentially have the same effect as a kdump, environment.

Quoted 1 line

Here I am trying to solve the second problem so that a

driver comes to

Quoted 1 line

know that it is initializing in a special boot environment

and it can

Quoted 1 line

modify its behavior at run time.

As Andrew said that encourages hacks.
For the specific megaraid example it would be simple enough to always ignore the condition and just print a warning.

There is no such thing as a special boot environment there are only quality of implementation differences. And in a kexec on a panic scenario the quality of implementation is terrible.

Quoted 1 line

Quoted 1 line

If the concept is crashdump it is a poorly defined concept

and all of

Quoted 4 lines

Quoted 2 lines

Andrews objections apply.

I think this parameter is generic enough and not limited to

crashdumps.

Quoted 1 line

If a user decides to implement a different scheme than

kdump in kexec

Quoted 2 lines

on panic and boot a customized kernel, he can very well use this parameter to make sure that next kernel is able to at least

boot and

Quoted 1 line

not panic in between.

The name crashboot is certainly not generic enough to make it clear what it means or to make it sound interesting outside of a crashdump scenario.

Quoted 1 line

Solving first problem will help in doing a plain kexec. We

can simply

Quoted 4 lines

mark the drivers known to have problems and slowly people can fix those drivers. Fixing the driver in this case is different because most likely driver authors will provide a shutdown routine in the driver so that device can be shutdown and then one can boot

into the second kernel.

Quoted 1 line

Till then a user can happily skip the drivers known to have

problems.

Quoted 2 lines

In summary, we got two problems to solve. Currently I am

focussed on

Quoted 2 lines

solving second problem which enables boot a kernel in an unreliable environment and do some minimal specific operation and then

boot back

Quoted 3 lines

to regular kernela. So I think just introducing a command line parameter which drivers can use to determine that they are initializing in an special environement, solves it and is

generic enough.

Quoted 3 lines

Options like COFIG_BRITTLE_INIT or sikkping execution of brittle driver based on a command line option seems to be the

solution for the

Quoted 1 line

first problem.

Among other things it is social engineering to solve the first problem.

Quoted 6 lines

Please correct me if I am wrong. I know little about drivers.

Quoted 3 lines

Quoted 3 lines

I remember few months back, Eric had mentioned that he has got patches for relocatable kernel ready for review for i386 and x86_64. Eric, do you have any plans to post the patches

for review?

Quoted 3 lines

Quoted 3 lines

I have some code that I keep intending to get to. It has probably bit rotted since I wrote it, but it shouldn't be too bad

to clean up.

Quoted 1 line

Quoted 1 line

Unfortunately the whole crashdump thing is fairly low on

my priority list.

Quoted 3 lines

Quoted 1 line

I am willing to work on it. Building from scratch always

takes more time.

Quoted 2 lines

If you are willing, I will more than happy to build on top of your patches.

I will see what I can dig up.

Eric

All,
Sorry to come in late and top post. I've been out of the office and I'm trying to get to the gist of this issue.
Exactly what is the problem? I'm not familiar with kdump so I don't have a clue about what's going on.

There are a couple of reset features supported by _some_ cciss controllers. I'd have to go back to the open spec to see whats in the public domain. We're trying to get the open spec updated and more complete but we're waiting on the lawyers. :(

All,
Sorry to come in late and top post. I've been out of the office and I'm trying to get to the gist of this issue.
Exactly what is the problem? I'm not familiar with kdump so I don't have a clue about what's going on.
There are a couple of reset features supported by _some_ cciss controllers. I'd have to go back to the open spec to see whats in the public domain. We're trying to get the open spec updated and more complete but we're waiting on the lawyers. :(

-----Original Message-----
From: Eric W. Biederman [mailto:[email protected]]
Sent: Monday, June 26, 2006 11:38 AM
To: Miller, Mike (OS Dev)
Cc: [email protected]; Maneesh Soni; Andrew Morton; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Fastboot] [RFC] [PATCH 2/2] kdump: cciss driver initialization issue fix

Quoted 7 lines · Mike (OS Dev)

All,
Sorry to come in late and top post. I've been out of the office and I'm trying to get to the gist of this issue.
Exactly what is the problem? I'm not familiar with kdump so I don't have a clue about what's going on.
There are a couple of reset features supported by _some_ cciss controllers. I'd have to go back to the open spec to see

whats in the

Quoted 2 lines

public domain. We're trying to get the open spec updated and more complete but we're waiting on the lawyers. :(

kdump or taking crash dumps using the kexec on panic mechanism could be called a drivers worst nightmare. In the latest distros this is becoming the way crash dump style information is captured.

Because the initial kernel is broken we do a jump into another kernel that is sufficient to record a crash dump. That second kernel initializes the hardware from whatever random state the first kernel left the drivers in. That first kernel is not permitted to do any device shutdown activities.

The problem is that a command the running instance of the driver did not initiate completes. At least if I read Vivek patch 2/2 correctly.

So we have three options.
- reset the card during initialization.
- handle the case of a command we did not initiate completing.
- mark the driver/card as impossibly hopeless for use in a crash dump scenario.

Eric

Quoted 15 lines

kdump or taking crash dumps using the kexec on panic mechanism could be called a drivers worst nightmare. In the latest distros this is becoming the way crash dump style information is captured.

Because the initial kernel is broken we do a jump into another kernel that is sufficient to record a crash dump. That second kernel initializes the hardware from whatever random state the first kernel left the drivers in. That first kernel is not permitted to do any device shutdown activities.

The problem is that a command the running instance of the driver did not initiate completes. At least if I read Vivek patch 2/2 correctly.

Quoted 8 lines

So we have three options.
- reset the card during initialization.
- handle the case of a command we did not initiate completing.
- mark the driver/card as impossibly hopeless for use in a crash dump scenario.

Eric

Thanks Eric, that helps me understand. Section 8.2.2 of the open cciss spec supports a reset message. Target 0x00 is the controller. We could add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal circumstances. And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it would not impact 99.9% of users.

Quoted 8 lines

Thanks Eric, that helps me understand. Section 8.2.2 of the open cciss spec supports a reset message. Target 0x00 is the controller. We could add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal circumstances. And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it would not impact 99.9% of users.

That's the precise reason of introducing the "crashboot" command line parameter. Driver authors can check against this condition and reset the device and 99.9% of the users are not impacted.

Quoted 24 lines

kdump or taking crash dumps using the kexec on panic mechanism could be called a drivers worst nightmare. In the latest distros this is becoming the way crash dump style information is captured.

Because the initial kernel is broken we do a jump into another kernel that is sufficient to record a crash dump. That second kernel initializes the hardware from whatever random state the first kernel left the drivers in. That first kernel is not permitted to do any device shutdown activities.

The problem is that a command the running instance of the driver did not initiate completes. At least if I read Vivek patch 2/2 correctly.

So we have three options.
- reset the card during initialization.
- handle the case of a command we did not initiate completing.
- mark the driver/card as impossibly hopeless for use in a crash dump scenario.

Quoted 1 line

Eric

Thanks Eric, that helps me understand. Section 8.2.2 of the open cciss spec supports a reset message. Target 0x00 is the controller. We could add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal circumstances. And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump.

Thanks Eric, that helps me understand. Section 8.2.2 of the open cciss spec supports a reset message. Target 0x00 is the controller. We could add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal circumstances.

And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it would not impact 99.9% of users.

Thoughts, comments, flames?

Quoted 4 lines · Mike (OS Dev)

Thanks Eric, that helps me understand. Section 8.2.2 of the open cciss spec supports a reset message. Target 0x00 is the controller. We could add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal circumstances.

Where does the init time penalty come from? How large is the init penalty? I suspect it is from waiting for the scsi disks to spin up. But I am just guessing in the dark.

Quoted 4 lines

And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it would not impact 99.9% of users.

I am wondering if it is possible to look at the controller and see if it is in a bad state, (i.e. in some state besides just coming out of reset) and if so issue a reset. If this really is a long operation that would be the ideal way to handle it.

If the amount of time is really user noticeable and testing for it is impossible then it is probably time to talk kernel command line options. >
Although it might simply be appropriate to handle commands completing you didn't start. I am not at all familiar with that particular piece of hardware so I can't make a good guess on what needs to happen there.

Quoted 1 line

Thoughts, comments, flames?

Good question.

It is a bit of a pain but not too hard to setup a test environment so you can reproduce this if you are interested. Vivek should be the authority there.

-----Original Message-----
From: Eric W. Biederman [mailto:[email protected]]
Sent: Monday, June 26, 2006 12:52 PM
To: Miller, Mike (OS Dev)
Cc: [email protected]; Maneesh Soni; Andrew Morton; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Fastboot] [RFC] [PATCH 2/2] kdump: cciss driver initialization issue fix

Quoted 1 line · Mike (OS Dev)

Thanks Eric, that helps me understand. Section 8.2.2 of the

open cciss

Quoted 1 line

spec supports a reset message. Target 0x00 is the

controller. We could

Quoted 2 lines

add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal

circumstances.

Where does the init time penalty come from? How large is the init penalty? I suspect it is from waiting for the scsi disks to spin up.
But I am just guessing in the dark.

Quoted 4 lines

And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it

would not

Quoted 1 line

impact 99.9% of users.

I am wondering if it is possible to look at the controller and see if it is in a bad state, (i.e. in some state besides just coming out of reset) and if so issue a reset. If this really is a long operation that would be the ideal way to handle it.

If the amount of time is really user noticeable and testing for it is impossible then it is probably time to talk kernel command line options.

Although it might simply be appropriate to handle commands completing you didn't start. I am not at all familiar with that particular piece of hardware so I can't make a good guess on what needs to happen there.

Quoted 18 lines

-----Original Message-----
From: Eric W. Biederman [mailto:[email protected]]
Sent: Monday, June 26, 2006 12:52 PM
To: Miller, Mike (OS Dev)
Cc: [email protected]; Maneesh Soni; Andrew Morton; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Fastboot] [RFC] [PATCH 2/2] kdump: cciss driver initialization issue fix

Quoted 1 line · Mike (OS Dev)

Thanks Eric, that helps me understand. Section 8.2.2 of the

open cciss

Quoted 1 line

spec supports a reset message. Target 0x00 is the

controller. We could

Quoted 2 lines

add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal

Quoted 6 lines

circumstances.

Where does the init time penalty come from? How large is the init penalty? I suspect it is from waiting for the scsi disks to spin up.
But I am just guessing in the dark.

The penalty is in the firmware and self-test operations.

Quoted 6 lines

Quoted 3 lines

And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it

would not

Quoted 1 line

impact 99.9% of users.

Quoted 4 lines

I am wondering if it is possible to look at the controller and see if it is in a bad state, (i.e. in some state besides just coming out of reset) and if so issue a reset. If this really is a long operation that would be the ideal way to handle it.

It's not really in a bad state at this time, is it? Maybe some commands hanging around.

Quoted 3 lines

If the amount of time is really user noticeable and testing for it is impossible then it is probably time to talk kernel command line options.

I was informed of the crashboot command line parameter. I can implement that as a test.

Quoted 4 lines

Although it might simply be appropriate to handle commands completing you didn't start. I am not at all familiar with that particular piece of hardware so I can't make a good guess on what needs to happen there.

Not sure about doing this.

Quoted 7 lines

Quoted 5 lines

Where does the init time penalty come from? How large is the init penalty? I suspect it is from waiting for the scsi disks to spin up.
But I am just guessing in the dark.

The penalty is in the firmware and self-test operations.

Ok. Reasonable. Roughly long does that take? 1 millisecond? 1 second? 1 minute? 1 hour?

Quoted 6 lines

Quoted 6 lines

Quoted 3 lines

And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it

would not

Quoted 1 line

impact 99.9% of users.

Quoted 7 lines

Quoted 4 lines

I am wondering if it is possible to look at the controller and see if it is in a bad state, (i.e. in some state besides just coming out of reset) and if so issue a reset. If this really is a long operation that would be the ideal way to handle it.

It's not really in a bad state at this time, is it? Maybe some commands hanging around.

Not bad as in broken. But bad as in unexpected. If it is just a matter of outstanding commands we might even be able to just ask the adapter to cancel all of the at initialization time.

Quoted 6 lines

Quoted 3 lines

If the amount of time is really user noticeable and testing for it is impossible then it is probably time to talk kernel command line options.

I was informed of the crashboot command line parameter. I can implement that as a test.

Sounds like a start.

Quoted 6 lines

Quoted 4 lines

Although it might simply be appropriate to handle commands completing you didn't start. I am not at all familiar with that particular piece of hardware so I can't make a good guess on what needs to happen there.

Not sure about doing this.

Well I would certainly print a warning.

-----Original Message-----
From: Eric W. Biederman [mailto:[email protected]]
Sent: Monday, June 26, 2006 2:21 PM
To: Miller, Mike (OS Dev)
Cc: [email protected]; Maneesh Soni; Andrew Morton; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Fastboot] [RFC] [PATCH 2/2] kdump: cciss driver initialization issue fix

Quoted 7 lines

Quoted 7 lines

Quoted 1 line · Mike (OS Dev)

Thanks Eric, that helps me understand. Section 8.2.2 of the

open cciss

Quoted 1 line

spec supports a reset message. Target 0x00 is the

controller. We could

Quoted 1 line

add this to the init routine to ensure the board is made

sane again

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

but this would drastically increase init time under normal

circumstances.

Where does the init time penalty come from? How large is the init penalty? I suspect it is from waiting for the scsi disks

to spin up.

Quoted 3 lines

Quoted 1 line

But I am just guessing in the dark.

The penalty is in the firmware and self-test operations.

Ok. Reasonable. Roughly long does that take? 1 millisecond? 1 second? 1 minute? 1 hour?

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we

could test

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

against and perform the reset when that condition is met it

would not

Quoted 1 line

impact 99.9% of users.

I am wondering if it is possible to look at the controller

and see if

Quoted 1 line

Quoted 1 line

it is in a bad state, (i.e. in some state besides just

coming out of

Quoted 1 line

Quoted 1 line

reset) and if so issue a reset. If this really is a long

operation

Quoted 4 lines

Quoted 1 line

that would be the ideal way to handle it.

It's not really in a bad state at this time, is it? Maybe some commands hanging around.

Not bad as in broken. But bad as in unexpected. If it is just a matter of outstanding commands we might even be able to just ask the adapter to cancel all of the at initialization time.

Quoted 3 lines

I was informed of the crashboot command line parameter. I can implement that as a test.

Sounds like a start.

Quoted 1 line

Quoted 1 line

Although it might simply be appropriate to handle commands

completing

Quoted 5 lines

Quoted 3 lines

you didn't start. I am not at all familiar with that particular piece of hardware so I can't make a good guess on what needs to happen there.

Not sure about doing this.

Well I would certainly print a warning.

Eric

Quoted 9 lines

Subject: Re: [Fastboot] [RFC] [PATCH 2/2] kdump: cciss driver initialization issue fix

Quoted 1 line · Mike (OS Dev)

Thanks Eric, that helps me understand. Section 8.2.2 of the

open cciss

Quoted 1 line

spec supports a reset message. Target 0x00 is the

controller. We could

Quoted 8 lines

Quoted 2 lines

add this to the init routine to ensure the board is made sane again but this would drastically increase init time under normal

circumstances.

Where does the init time penalty come from? How large is the init penalty? I suspect it is from waiting for the scsi disks to spin up.
But I am just guessing in the dark.

The penalty is in the firmware and self-test operations.

Quoted 7 lines

Quoted 4 lines

And I suspect this is a hard reset, also. Not sure if that would negatively impact kdump. If there were some condition we could test against and perform the reset when that condition is met it

would not

Quoted 1 line

impact 99.9% of users.

Quoted 4 lines

I am wondering if it is possible to look at the controller and see if it is in a bad state, (i.e. in some state besides just coming out of reset) and if so issue a reset. If this really is a long operation that would be the ideal way to handle it.

It's not really in a bad state at this time, is it? Maybe some commands hanging around.

Quoted 5 lines

If the amount of time is really user noticeable and testing for it is impossible then it is probably time to talk kernel command line options.

I was informed of the crashboot command line parameter. I can implement that as a test.

Quoted 19 lines

Looks like there are two problems to be solved.

- Framework/capability to mark and isolate the drivers, either at compile time or run time, which are not hardened enough to initialize properly when the underlying device is in operational or in unknown state.

- Actually hardening a driver to be able to initialize in a potentially unreliable environment.

Solving first problem will help more in terms of people knowing in advance that certain drivers are known to have problems in specific environemnt and a user has got the option of skipping the execution/compilation of those drivers. (This is something close to what CONFIG_EXPERIMENTAL does)

Second problem deals more with actually hardening the driver and not skipping its compilation/execution.

Quoted 4 lines

I think people would like to change a driver's behaviour at run time. For example if they are booting in a unreliable environment they would like to reset the device otherwise they would skip that as BIOS has already done that for them.

In the general case the device reset does not hurt.

Yes there
is the case of the slow scsi probe. But a lot of that appears to be a poor implementation of the scsi probe. So I can see a kernel command line option to play fast and loose but we should be safe and thorough by default.

The more code paths you introduce the harder code is to maintain and test. The earlier discussion suggested you cannot harden some drivers. We can take action against drivers like that simply and easily.

Hacks in the driver initialization are a completely different story.

So it is matter of making a choice in case the device does not have a software reset capability.

- Either try to make driver work through some hacks based on crashboot option.

- Or mark the driver unusable in kdump scenarios.

Even if one decides to go for second option, at least "crashboot" or similar parameter will be required so that driver can choose whether to reset the device or not during initialization due to significant time penalty.

On Mon, Jun 26, 2006 at 10:00:55AM -0600, Eric W. Biederman wrote:

I think it does hurt.

- I have seen the case of MPT fusion drvier. It takes significantly more time to come up if we choose to reset the device during initialization. One of the reasons that we wait in a tight loop for the controller to come up after a reset message.

- Long back we fixed ips driver and I remember that the maintainer had a similar issue with the reset of device. He did not want to reset the device in normal boot because otherwise it took significantly longer for the driver to initialize.

- Just now Mike also confirmed that resetting the device definitely hurts in terms of time.

In the general case resets are trivial operations. In scsi land things are different. So a solution appropriate to that domain may be appropriate.

Quoted 3 lines · Eric W. Biederman

In the general case resets are trivial operations. In scsi land things are different. So a solution appropriate to that domain may be appropriate.

That's not necessarily true. You're talking about board level resets here. Some devices take quite a while to reboot after being reset ... particularly the complex ones with internal operating system type firmware ..

Quoted 7 lines · Vivek Goyal

On Mon, Jun 26, 2006 at 07:41:00AM +0530, Maneesh Soni wrote:

Maneesh, Keeping this code under a config option becomes a problem when we will have a relocatable kernel. At some point of time we got to have relocatable kernel so that people don't have to build two kernels. In fact this is becoming a pain area for distros. That's the reason I thought of making it a command line parameter.

Ok. Even if we do this with a command line, we need to have a clean concept. If the concept is ignore devices with a brittle init routine that is compre= hensible
and potentially useful for other reasons than crash dumps.

If the concept is crashdump it is a poorly defined concept and all of Andre= ws
objections apply.

Quoted 3 lines

I remember few months back, Eric had mentioned that he has got patches for relocatable kernel ready for review for i386 and x86_64. Eric, do you have any plans to post the patches for review?

I have some code that I keep intending to get to. It has probably bit rotted since I wrote it, but it shouldn't be too bad to clean up. Unfortunately the whole crashdump thing is fairly low on my priority list.

Not all pci busses support it but there is a standard pci bus reset bit in pci bridges.

I don't know if it would help but it might make sense to have a config option that can be used to mark drivers that are known to have problems, in these scenarios.

CONFIG_BRITTLE_INIT perhaps?

It would at least make it easier for people to see which drivers they don't want to use, and give people some incentive to fix things.

Quoted 8 lines · In article <[email protected]> you wrote:

Not all pci busses support it but there is a standard pci bus reset bit in pci bridges.

I don't know if it would help but it might make sense to have a config option that can be used to mark drivers that are known to have problems, in these scenarios.

Quoted 4 lines

CONFIG_BRITTLE_INIT perhaps?

It would at least make it easier for people to see which drivers they don't want to use, and give people some incentive to fix things.

I believe that MPT Fusion could go on that list.

http://permalink.gmane.org/gmane.linux.ports.ia64/14451

Quoted 9 lines · Horms

Quoted 8 lines · In article <[email protected]> you wrote:

Not all pci busses support it but there is a standard pci bus reset bit in pci bridges.

I don't know if it would help but it might make sense to have a config option that can be used to mark drivers that are known to have problems, in these scenarios.

Quoted 10 lines

Quoted 4 lines

CONFIG_BRITTLE_INIT perhaps?

It would at least make it easier for people to see which drivers they don't want to use, and give people some incentive to fix things.

I believe that MPT Fusion could go on that list.

http://permalink.gmane.org/gmane.linux.ports.ia64/14451

Above link does not give details of exact MPT fusion initialization problems faced on IA64. I faced MPT fusion initialization issues on i386/x86_64 and posted a fix.

http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=9bf0a28c9a24e2cee5deecf89d118254374c75ba

You might want to download and check.

init/main.c static int __init maxcpus(char *str) · line 168

Quoted 5 lines · Vivek Goyal

init/main.c renamed from init/main.c~introduce-crash-boot-command-line-parameter +5 -0

+static int __init set_crash_boot(char *str)
+{
+       crash_boot = 1;
+       return 1;
+}

what about a printk? Maybe a combined one which shows some other stuff (init level, init shell, apic settings) etc from command line.

o Add kernel command line option "crashboot"

o This option is an indication to the kernel that kernel is booting in an unreliable environment where possibly BIOS execution has been skipped and devices are left operational or in unknown state.

o Kernel, especially device drivers can use this option to take special actions like soft-resetting the device, relaxing some of the rules to make sure kernel can boot/device driver can initiliaze in this environment.

o As of today this option is useful to Kdump. Kdump will pass this option to second kernel to improve the reliability of successful kenrel boot/ device driver initializatoin.