add kdump_after_notifier

In latest kernel, we can't use panic_notifier_list if kdump is enabled. panic_notifier_list is very useful function for debug, failover, etc...

So this patch adds a control file /proc/sys/kernel/dump_after_notifier and resolves a problem users can not use both kdump and panic_notifier_list at the same time.

kdump_after_notifier = 0
-> panic()

    -> crash_kexec(NULL)

kdump_after_notifier = 1
-> panic()

    -> atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
    -> crash_kexec(NULL)

Quoted 12 lines · Takenori Nagano

In latest kernel, we can't use panic_notifier_list if kdump is enabled. panic_notifier_list is very useful function for debug, failover, etc...

So this patch adds a control file /proc/sys/kernel/dump_after_notifier and resolves a problem users can not use both kdump and panic_notifier_list at the same time.

kdump_after_notifier = 0
-> panic()

    -> crash_kexec(NULL)

Quoted 4 lines

kdump_after_notifier = 1
-> panic()

    -> atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
    -> crash_kexec(NULL)

What's problematic about this patch? I also would like to see that feature.

Quoted 18 lines · Bernhard Walle

Quoted 16 lines · Takenori Nagano

In latest kernel, we can't use panic_notifier_list if kdump is enabled. panic_notifier_list is very useful function for debug, failover, etc...

So this patch adds a control file /proc/sys/kernel/dump_after_notifier and resolves a problem users can not use both kdump and panic_notifier_list at the same time.

kdump_after_notifier = 0
-> panic()

    -> crash_kexec(NULL)

kdump_after_notifier = 1
-> panic()

    -> atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
    -> crash_kexec(NULL)

Quoted 2 lines

What's problematic about this patch? I also would like to see that feature.

I would like to see the code which will get executed after panic and before crash_kexec(). This potentially makes crash dump feature unreliable in the sense one can now register on panic_notifier_list and try to do whole lot of things and might get stuck there. After the system has crashed, one is not supposed to do a whole lot.

Quoted 22 lines · Vivek Goyal

Quoted 20 lines · Bernhard Walle

Quoted 8 lines · Takenori Nagano

In latest kernel, we can't use panic_notifier_list if kdump is enabled. panic_notifier_list is very useful function for debug, failover, etc...

So this patch adds a control file /proc/sys/kernel/dump_after_notifier and resolves a problem users can not use both kdump and panic_notifier_list at the same time.

Quoted 8 lines

kdump_after_notifier = 0
-> panic()

    -> crash_kexec(NULL)

kdump_after_notifier = 1
-> panic()

    -> atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
    -> crash_kexec(NULL)

What's problematic about this patch? I also would like to see that feature.

Quoted 5 lines

I would like to see the code which will get executed after panic and before crash_kexec(). This potentially makes crash dump feature unreliable in the sense one can now register on panic_notifier_list and try to do whole lot of things and might get stuck there. After the system has crashed, one is not supposed to do a whole lot.

Of course, but that's why the patch doesn't change this by default but gives the user the choice.

Thanks,
Bernhard

Quoted 4 lines

Of course, but that's why the patch doesn't change this by default but gives the user the choice.

What value will distro set it to by default?

Can we be more specific in terms of functionality and code that exactly what we are trying to do after panic?

Quoted 6 lines · Vivek Goyal

Quoted 4 lines

Of course, but that's why the patch doesn't change this by default but gives the user the choice.

What value will distro set it to by default?

0.

Quoted 2 lines

Can we be more specific in terms of functionality and code that exactly what we are trying to do after panic?

Well, KDB, but now everybody answers with “not mainline -- doesn't count”.

That's true. Its not mainline. We had similar discussion in the past also. I think we should allow only audited code to be run after panic(). Leaving it open to modules or unaudited code makes this solution something like LKCD where whole lot of code used to run after the crash, hence was unreliable.

If KDB goes mainline, then I think it is not a bad idea to call debugger first (if it is enabled) and then one can trigger crash dump from inside the debugger.

Quoted 7 lines · Vivek Goyal

That's true. Its not mainline. We had similar discussion in the past also. I think we should allow only audited code to be run after panic(). Leaving it open to modules or unaudited code makes this solution something like LKCD where whole lot of code used to run after the crash, hence was unreliable.

Quoted 3 lines

If KDB goes mainline, then I think it is not a bad idea to call debugger first (if it is enabled) and then one can trigger crash dump from inside the debugger.

Well, after thinking again about this, I think the best solution would to call kdump from KDB with a command as it was done by LKCD.

Quoted 4 lines · Bernhard Walle

Quoted 3 lines · Vivek Goyal

Quoted 3 lines

Of course, but that's why the patch doesn't change this by default but gives the user the choice.

Quoted 8 lines

Quoted 1 line

What value will distro set it to by default?

0.

Quoted 2 lines

Can we be more specific in terms of functionality and code that exactly what we are trying to do after panic?

Well, KDB, but now everybody answers with “not mainline -- doesn't count”.

That's true. Its not mainline. We had similar discussion in the past also. I think we should allow only audited code to be run after panic(). Leaving it open to modules or unaudited code makes this solution something like LKCD where whole lot of code used to run after the crash, hence was unreliable.

If KDB goes mainline, then I think it is not a bad idea to call debugger first (if it is enabled) and then one can trigger crash dump from inside the debugger.
Thanks
Vivek

Hi Vivek,

Quoted 14 lines · Vivek Goyal

Quoted 4 lines · Bernhard Walle

Quoted 3 lines · Vivek Goyal

Quoted 3 lines

Of course, but that's why the patch doesn't change this by default but gives the user the choice.

Quoted 8 lines

Quoted 1 line

What value will distro set it to by default?

0.

Quoted 2 lines

Can we be more specific in terms of functionality and code that exactly what we are trying to do after panic?

Well, KDB, but now everybody answers with “not mainline -- doesn't count”.

Quoted 5 lines

That's true. Its not mainline. We had similar discussion in the past also. I think we should allow only audited code to be run after panic(). Leaving it open to modules or unaudited code makes this solution something like LKCD where whole lot of code used to run after the crash, hence was unreliable.

It is *not* KDB specific problem. Please grep in mainline kernel. You can find some function using panic_notifier_list. (IPMI, softdog, heartbeat, etc...) My patch gives a chance to use kdump for panic_notifier user. It is good for kdump too, because kdump user goes to increase. :-)

Bernhard's idea (kdump uses panic_notifier) is very good for me. But it isn't good for kdump user, because they want to take a dump ASAP when panicked.

Quoted 3 lines

Bernhard's idea (kdump uses panic_notifier) is very good for me. But it isn't good for kdump user, because they want to take a dump ASAP when panicked.

This one is better than registering kdump as one of the users of a panic_notifier() list.

I think if there are any crash specific actions, they should be taken care in next kernel while it is booting.

If something is really very time critical, and has to be done immediately after panic (I am not sure how can one ensure that given the fact any number of users can register on panic_notifier_list and you are not sure about your order in the list and when one will get the control), then probably that piece of code should be in kernel and called before crash_kexec().

What is that specific piece of action which you can't do in second kernel?

Eric, do you have any thoughts on this. I think these guys are referring to failover problem where immediately after panic() they want to send message to other node.

Quoted 15 lines · Vivek Goyal

Quoted 3 lines

Bernhard's idea (kdump uses panic_notifier) is very good for me. But it isn't good for kdump user, because they want to take a dump ASAP when panicked.

This one is better than registering kdump as one of the users of a panic_notifier() list.

I think if there are any crash specific actions, they should be taken care in next kernel while it is booting.

If something is really very time critical, and has to be done immediately after panic (I am not sure how can one ensure that given the fact any number of users can register on panic_notifier_list and you are not sure about your order in the list and when one will get the control), then probably that piece of code should be in kernel and called before crash_kexec().

Quoted 5 lines

What is that specific piece of action which you can't do in second kernel?

Eric, do you have any thoughts on this. I think these guys are referring to failover problem where immediately after panic() they want to send message to other node.

My thoughts are roughly the same as they were last time this was suggested. I think adding a notifier to the kexec on panic path is a bad idea. This functionality sounds wrong, because it makes it hard to ensure reliability of the kexec on panic code path. We are still doing to much on it as it stands. The working assumption on that code path needs to be the kernel is broken. Anything else is just asking for trouble.

Currently we do have a hook in place for code to be called. It is called the purgatory section of /sbin/kexec. And it's user space so you can do whatever you want there. Or you can wait until the second kernel gets more fully booted.

If we really need to do something in the kernel we can patch the kernel to make a function call from crash_kexec. We don't need any notifiers to do this.

A further problem with notifiers is they mess up the state we would like to debug. Which again makes them a problem.

So at least until a specific case is made for a specific piece of code to get in I am totally opposed to the idea.

Hi all,

IMHO, most users don't use kdump, kdump users are only kernel developers and enterprise users.

think enterprise users want the notifier function, because they use some driver and software (hardware monitering driver, clustering software, heartbeat driver, etc...) to raise their system availability.

Some popular distributers added the dump function to their own kernel. We can use panic_notifier on LKCD (http://lkcd.sourceforge.net/), and diskdump
(http://sourceforge.net/projects/lkdump) provides own notifier function disk_dump_notifier.

Now, kdump was merged mainline kernel. Then some distributers chose kdump. I think kdump is greater than other dump function, but kdump has no notifier function. This is a large problem for enterprise users.

Solutions
1: my patch
2: Bernhard's idea
3: add kdump_notifier_list

I think my patch is better than other solutions, because it has only very few impact. Vivek, Eric, how do you think?

Quoted 4 lines · Takenori Nagano

Hi all,

IMHO, most users don't use kdump, kdump users are only kernel developers and enterprise users.

Not at all. So far the only kdump related bug report I have seen has been from fedora Core.

No. The problem with your patch is that it doesn't have a code impact. We need to see who is using this and why.

Both KDB and crash_kexec should be using the panic_notifier_chain, with KDB having a higher priority than crash_exec. The whole point of notifier chains is to handle cases like this, so we should not be adding more code to the panic routine.

The real problem here is the way that the crash_exec code is hard coded into various places instead of using notifier chains. The same issue exists in arch/ia64/kernel/mca.c because of bad coding practices from kexec.

Quoted 2 lines

No. The problem with your patch is that it doesn't have a code impact. We need to see who is using this and why.

My motivation is very simple. I want to use both kdb and kdump, but I think it is too weak to satisfy kexec guys. Then I brought up the example enterprise software. But it isn't a lie. I know some drivers which use panic_notifier. IMHO, they use only major distribution, and they has the workaround or they don't notice this problem yet. I think they will be in trouble if all distributions choose only kdump.

BTW, I use kdb and lkcd now, but I want to use kdb and kdump. I sent a patch to kdb community but it was rejected. kdb maintainer Keith Owens said,

Quoted 9 lines

Both KDB and crash_kexec should be using the panic_notifier_chain, with KDB having a higher priority than crash_exec. The whole point of notifier chains is to handle cases like this, so we should not be adding more code to the panic routine.

The real problem here is the way that the crash_exec code is hard coded into various places instead of using notifier chains. The same issue exists in arch/ia64/kernel/mca.c because of bad coding practices from kexec.

Then I gave up to merge my patch to kdb, and I tried to send another patch to kexec community. I can understand his opinion, but it is very difficult to modify that kdump is called from panic_notifier. Because it has a reason why kdump don't use panic_notifier. So, I made this patch.

Please do something about this problem.

Quoted 6 lines · Takenori Nagano

Then I gave up to merge my patch to kdb, and I tried to send another patch to kexec community. I can understand his opinion, but it is very difficult to modify that kdump is called from panic_notifier. Because it has a reason why kdump don't use panic_notifier. So, I made this patch.

Please do something about this problem.

Hmm. Tricky. These appear to be two code bases with a completely different philosophy on what errors are being avoided.

The kexec on panic assumption is that the kernel is broken and we better not touch it something horrible has gone wrong. And this is the reason why kexec on panic is replacing lkcd. Because the strong assumption results in more errors getting captured with less likely hood of messing up your system.

The kdb assumption appears to be that the kernel is mostly ok, and that there are just some specific thing that is wrong.

The easiest way I can think to resolve this is for kdb to simply set a break point at the entry point of panic() when it initializes. Then it wouldn't even need to be on the panic_list. That approach would probably even give better debug information because you would not have the effects of bust_spinlocks to undo.

Is there some reason why kdb doesn't want to hook panic with a some kind of break point?

Quoted 9 lines · Takenori Nagano

Quoted 2 lines

No. The problem with your patch is that it doesn't have a code impact. We need to see who is using this and why.

My motivation is very simple. I want to use both kdb and kdump, but I think it is too weak to satisfy kexec guys. Then I brought up the example enterprise software. But it isn't a lie. I know some drivers which use panic_notifier. IMHO, they use only major distribution, and they has the workaround or they don't notice this problem yet. I think they will be in trouble if all distributions choose only kdump.

Possibly.

Quoted 12 lines

BTW, I use kdb and lkcd now, but I want to use kdb and kdump. I sent a patch to kdb community but it was rejected. kdb maintainer Keith Owens said,

Quoted 9 lines

Both KDB and crash_kexec should be using the panic_notifier_chain, with KDB having a higher priority than crash_exec. The whole point of notifier chains is to handle cases like this, so we should not be adding more code to the panic routine.

The real problem here is the way that the crash_exec code is hard coded into various places instead of using notifier chains. The same issue exists in arch/ia64/kernel/mca.c because of bad coding practices from kexec.

I respectfully disagree with his opinion, as using notifier chains assumes more of the kernel works. Although following it's argument to it's logical conclusion we should call crash_kexec as the very first thing inside of panic. Given how much state something like bust_spinlocks messes up that might not be a bad idea.

It does make adding an alternative debug mechanism in there difficult. Does anyone know if this also affects kgdb?

Quoted 6 lines

Then I gave up to merge my patch to kdb, and I tried to send another patch to kexec community. I can understand his opinion, but it is very difficult to modify that kdump is called from panic_notifier. Because it has a reason why kdump don't use panic_notifier. So, I made this patch.

Please do something about this problem.

Hmm. Tricky. These appear to be two code bases with a completely different philosophy on what errors are being avoided.

The kexec on panic assumption is that the kernel is broken and we better not touch it something horrible has gone wrong. And this is the reason why kexec on panic is replacing lkcd. Because the strong assumption results in more errors getting captured with less likely hood of messing up your system.

The kdb assumption appears to be that the kernel is mostly ok, and that there are just some specific thing that is wrong.

Quoted 46 lines · Eric W. Biederman

Quoted 9 lines · Takenori Nagano

Quoted 2 lines

No. The problem with your patch is that it doesn't have a code impact. We need to see who is using this and why.

My motivation is very simple. I want to use both kdb and kdump, but I think it is too weak to satisfy kexec guys. Then I brought up the example enterprise software. But it isn't a lie. I know some drivers which use panic_notifier. IMHO, they use only major distribution, and they has the workaround or they don't notice this problem yet. I think they will be in trouble if all distributions choose only kdump.

Possibly.

Quoted 12 lines

BTW, I use kdb and lkcd now, but I want to use kdb and kdump. I sent a patch to kdb community but it was rejected. kdb maintainer Keith Owens said,

Quoted 9 lines

Both KDB and crash_kexec should be using the panic_notifier_chain, with KDB having a higher priority than crash_exec. The whole point of notifier chains is to handle cases like this, so we should not be adding more code to the panic routine.

The real problem here is the way that the crash_exec code is hard coded into various places instead of using notifier chains. The same issue exists in arch/ia64/kernel/mca.c because of bad coding practices from kexec.

I respectfully disagree with his opinion, as using notifier chains assumes more of the kernel works. Although following it's argument to it's logical conclusion we should call crash_kexec as the very first thing inside of panic. Given how much state something like bust_spinlocks messes up that might not be a bad idea.

It does make adding an alternative debug mechanism in there difficult. Does anyone know if this also affects kgdb?

Quoted 6 lines

Then I gave up to merge my patch to kdb, and I tried to send another patch to kexec community. I can understand his opinion, but it is very difficult to modify that kdump is called from panic_notifier. Because it has a reason why kdump don't use panic_notifier. So, I made this patch.

Please do something about this problem.

Hmm. Tricky. These appear to be two code bases with a completely different philosophy on what errors are being avoided.

Quoted 9 lines

The kexec on panic assumption is that the kernel is broken and we better not touch it something horrible has gone wrong. And this is the reason why kexec on panic is replacing lkcd. Because the strong assumption results in more errors getting captured with less likely hood of messing up your system.

The kdb assumption appears to be that the kernel is mostly ok, and that there are just some specific thing that is wrong.

Thinking more about it. So basically there are two kind of users. One who believe that despite the kernel has crashed something meaningful can be done. In fact kernel also thinks so. That's why we have created panic_notifier_list and even exported it to modules and now we have some users. These users most of the time do non-disruptive activities and can co-exist.

OTOH, we have kexec on panic, which thinks that once kernel is crashed nothing meaningful can be done and it is disruptive and can't co-exist with other users.

Some thoughts on possible solutions for this problem.

- Stop exporting panic_notifier_list list to modules. Audit the in kernel users of panic_notifier_list. Let crash_kexec() run once all other users of panic_notifier_list have been executed. This has fall side of breaking down external modules using panic_notifier_list and at the same time there is no gurantee that audited code will not run into the issues.

- Continue with existing policy. If kdump is configured, panic_notifier_list notifications will not be invoked. Any post panic action should be executed in second kernel. There might be 1-2 odd cases like in kernel debugger which still needs to be invoked in first kernel. These users should explicitly put hooks in panic() routine and refrain from using panic_notifier list.

One thing to keep in mind, doing things in second kernel might not be easy as we have lost all the config data of the first kernel. For example, if one wants to send a kernel crash event over network to a system management software, he might have to pack in lot of software in second kernel's initrd.

- Let the user decide if he wants to run panic_notifier_list after the crash or not with the help of a /proc option as suggested by the Takenori's patch. Fall side is, on what basis an enterprise user will take a decision whether he wants to run the notifiers or not. My gut feeling is that distro will end up setting this parameter as 1 by default, which would mean first run panic notifiers and then run crash_kexec().

- Make crash_kexec() a user of panic_notifier_list and let it run after all the callback handlers have run. This will invariably reduce the reliability of kdump.

Personally I believe that second solution should bring us best of both the worlds. Making sure post panic actions can be done more reliably at the same time making sure reliability of kdump is not compromised.

Keith, do you see a value in second solution and would there be any reason why kdb hook can not be explicitly placed in panic(). There will not be many users like kdb. Rest of the users should end up performing post panic actions in second kernel.

Solutoin 3, can prove to be a stop gap solution but I think this will make situation confusing for customers at the same time everybody will try to take short route of performing post panic operations in first kernel.

Thanks
Vivek

I have pretty well given up on RAS code in the Linux kernel. Everybody has different ideas, there is no overall plan and little interest from Linus in getting RAS tools into the kernel. We are just thrashing.

On Fri, 03 Aug 2007 14:05:47 +1000 Keith Owens <[email protected]> wrote:

Quoted 3 lines

I have pretty well given up on RAS code in the Linux kernel. Everybody has different ideas, there is no overall plan and little interest from Linus in getting RAS tools into the kernel. We are just thrashing.

Lots of different groups, little commonality in their desired funtionality, little interest in sharing infrastructure or concepts. Sometimes people need a bit of motivational help.

In this case that motivation would come from the understanding that all the RAS tools would be *required* to use such infrastructure if it was merged. Going off and open-coding your own stuff would henceforth not be acceptable. If it turns out that it really was unsuitable for a particular group's RAS feature, and we merged it anyway, well, that mismatch is that group's fault.

It was a sizeable mistake to send those patches to a few obscure mailing lists - this is the first I've heard of it, for example.

So. Please, send it all again, copy the correct lists and people, make sure that at least one client of the infrastructure is wired up and working (ideally, all such in-kernel clients should be wired up) and let's take a look at it.

Quoted 1 line · Andrew Morton (on Thu, 2 Aug 2007 23:25:02 -0700) wrote:

On Fri, 03 Aug 2007 14:05:47 +1000 Keith Owens <[email protected]> wrote:

Switching to kaos at ocs.com.au, I just resigned from SGI.

Quoted 15 lines

Quoted 3 lines

I have pretty well given up on RAS code in the Linux kernel. Everybody has different ideas, there is no overall plan and little interest from Linus in getting RAS tools into the kernel. We are just thrashing.

Lots of different groups, little commonality in their desired funtionality, little interest in sharing infrastructure or concepts. Sometimes people need a bit of motivational help.

In this case that motivation would come from the understanding that all the RAS tools would be *required* to use such infrastructure if it was merged. Going off and open-coding your own stuff would henceforth not be acceptable. If it turns out that it really was unsuitable for a particular group's RAS feature, and we merged it anyway, well, that mismatch is that group's fault.

Quoted 2 lines

It was a sizeable mistake to send those patches to a few obscure mailing lists - this is the first I've heard of it, for example.

linux-arch is obscure??

Where else do you send patches that affect
multiple architectures?

Quoted 3 lines

So. Please, send it all again, copy the correct lists and people, make sure that at least one client of the infrastructure is wired up and working (ideally, all such in-kernel clients should be wired up) and let's take a look at it.

Already tried that. The only RAS tool that is currently in the kernel is kexec/kdump and they insist on doing things their own way. That makes it impossible to put a common RAS structure in place, because kexec will not use it.

Sorry to keep beating on this drum, but kexec insist that their code must have priority and that they do not trust the rest of the kernel. Until that changes, there is no point is discussing how to make kexec coexist with other RAS tools. If kexec change their mind then we can look at using a common RAS interface, otherwise it is a waste of time and I have better things to do with my life.

Much of the onus is upon the various RAS tool developers to demonstrate why it is unsuitable for their use and, hopefully, to explain how it can be fixed for them.

Quoted 41 lines

Some thoughts on possible solutions for this problem.

- Stop exporting panic_notifier_list list to modules. Audit the in kernel users of panic_notifier_list. Let crash_kexec() run once all other users of panic_notifier_list have been executed. This has fall side of breaking down external modules using panic_notifier_list and at the same time there is no gurantee that audited code will not run into the issues.

- Continue with existing policy. If kdump is configured, panic_notifier_list notifications will not be invoked. Any post panic action should be executed in second kernel. There might be 1-2 odd cases like in kernel debugger which still needs to be invoked in first kernel. These users should explicitly put hooks in panic() routine and refrain from using panic_notifier list.

One thing to keep in mind, doing things in second kernel might not be easy as we have lost all the config data of the first kernel. For example, if one wants to send a kernel crash event over network to a system management software, he might have to pack in lot of software in second kernel's initrd.

- Let the user decide if he wants to run panic_notifier_list after the crash or not with the help of a /proc option as suggested by the Takenori's patch. Fall side is, on what basis an enterprise user will take a decision whether he wants to run the notifiers or not. My gut feeling is that distro will end up setting this parameter as 1 by default, which would mean first run panic notifiers and then run crash_kexec().

- Make crash_kexec() a user of panic_notifier_list and let it run after all the callback handlers have run. This will invariably reduce the reliability of kdump.

Personally I believe that second solution should bring us best of both the worlds. Making sure post panic actions can be done more reliably at the same time making sure reliability of kdump is not compromised.

Keith, do you see a value in second solution and would there be any reason why kdb hook can not be explicitly placed in panic(). There will not be many users like kdb. Rest of the users should end up performing post panic actions in second kernel.

Quoted 6 lines

Solutoin 3, can prove to be a stop gap solution but I think this will make situation confusing for customers at the same time everybody will try to take short route of performing post panic operations in first kernel.

Thanks
Vivek

Do not concentrate on kdb alone. The problem above applies to all the RAS tools, not just kdb.

My stance is that _all_ the RAS tools (kdb, kgdb, nlkd, netdump, lkcd, crash, kdump etc.) should be using a common interface that safely puts the entire system in a stopped state and saves the state of each cpu. Then each tool can do what it likes, instead of every RAS tool doing its own thing and they all conflict with each other, which is why this thread started.

It is not the kernel's job to decide which RAS tool runs first, second etc., it is the user's decision to set that policy. Different sites will want different orders, some will say "go straight to kdump", other sites will want to invoke a debugger first. Sites must be able to define that policy, but we hard code the policy into the kernel.

I proposed and wrote most of this common interface against 2.6.19-rc5. See http://marc.info/?l=linux-arch&w=2&r=1&s=crash_stop&q=b, look for crash_stop. The crash_stop interface stops all the cpus, saves the system state in a common format then runs an ordered list of RAS tools.

The order that the RAS tools are run depends on the priority value that each tool passes to register_die_notifier. Currently each RAS tool hard codes its priority but it is trivial to change the tools to make that priority a parameter, passing the policy decision back to the user, not the kernel.

On Fri, Aug 03, 2007 at 02:05:47PM +1000, Keith Owens wrote:

To sum up, couple of options come to mind.

- Register all the RAS tools on die notifier and panic notifier lists with fairly high priority. Export list of RAS tools to user space and allow users to decide the order of execution and priority of RAS tools.

- Create a separate RAS tool notifier list (ras_tool_notifer_list). All the RAS tools register on this list. This list gets priority over die or panic notifier list. User decides the oder of execution of RAS tools.

Here assumption is that above list will not be exported to modules. All the RAS tools will be in kernel and they always get a priority to inspect an event.

What do others think?

Quoted 19 lines · Vivek Goyal

On Fri, Aug 03, 2007 at 02:05:47PM +1000, Keith Owens wrote:

To sum up, couple of options come to mind.

- Register all the RAS tools on die notifier and panic notifier lists with fairly high priority. Export list of RAS tools to user space and allow users to decide the order of execution and priority of RAS tools.

- Create a separate RAS tool notifier list (ras_tool_notifer_list). All the RAS tools register on this list. This list gets priority over die or panic notifier list. User decides the oder of execution of RAS tools.

Here assumption is that above list will not be exported to modules. All the RAS tools will be in kernel and they always get a priority to inspect an event.

Quoted 1 line

What do others think?

Very good idea. But there is a problem how to give default priority to RAS tools.

How about priority changeable notifier_list? User can change list order dynamically if they want. Of course, we have to give highest priority to kdump by default. It is very useful for users who want to use some RAS tools.

Quoted 21 lines · Takenori Nagano

Quoted 20 lines · Vivek Goyal

On Fri, Aug 03, 2007 at 02:05:47PM +1000, Keith Owens wrote:

To sum up, couple of options come to mind.

- Register all the RAS tools on die notifier and panic notifier lists with fairly high priority. Export list of RAS tools to user space and allow users to decide the order of execution and priority of RAS tools.

- Create a separate RAS tool notifier list (ras_tool_notifer_list). All the RAS tools register on this list. This list gets priority over die or panic notifier list. User decides the oder of execution of RAS tools.

Here assumption is that above list will not be exported to modules. All the RAS tools will be in kernel and they always get a priority to inspect an event.

What do others think?

Quoted 5 lines

Very good idea. But there is a problem how to give default priority to RAS tools.

How about priority changeable notifier_list? User can change list order dynamically if they want. Of course, we have to give highest priority to kdump by default. It is very useful for users who want to use some RAS tools.

I think that was the idea of the first “-” (“export list of RAS tools to user space”).

Quoted 28 lines · Bernhard Walle

Quoted 26 lines · Takenori Nagano

Quoted 15 lines · Vivek Goyal

On Fri, Aug 03, 2007 at 02:05:47PM +1000, Keith Owens wrote:

To sum up, couple of options come to mind.

- Register all the RAS tools on die notifier and panic notifier lists with fairly high priority. Export list of RAS tools to user space and allow users to decide the order of execution and priority of RAS tools.

- Create a separate RAS tool notifier list (ras_tool_notifer_list). All the RAS tools register on this list. This list gets priority over die or panic notifier list. User decides the oder of execution of RAS tools.

Quoted 5 lines

Here assumption is that above list will not be exported to modules. All the RAS tools will be in kernel and they always get a priority to inspect an event.

What do others think?

Very good idea. But there is a problem how to give default priority to RAS tools.

How about priority changeable notifier_list? User can change list order dynamically if they want. Of course, we have to give highest priority to kdump by default. It is very useful for users who want to use some RAS tools.

Quoted 2 lines

I think that was the idea of the first “-” (“export list of RAS tools to user space”).

Ah, sorry.

I think first idea is very good. How export the list? (sysfs? procfs?)

Quoted 22 lines · Takenori Nagano

Quoted 20 lines · Vivek Goyal

On Fri, Aug 03, 2007 at 02:05:47PM +1000, Keith Owens wrote:

To sum up, couple of options come to mind.

- Register all the RAS tools on die notifier and panic notifier lists with fairly high priority. Export list of RAS tools to user space and allow users to decide the order of execution and priority of RAS tools.

- Create a separate RAS tool notifier list (ras_tool_notifer_list). All the RAS tools register on this list. This list gets priority over die or panic notifier list. User decides the oder of execution of RAS tools.

Here assumption is that above list will not be exported to modules. All the RAS tools will be in kernel and they always get a priority to inspect an event.

What do others think?

Quoted 5 lines

Very good idea. But there is a problem how to give default priority to RAS tools.

How about priority changeable notifier_list? User can change list order dynamically if they want. Of course, we have to give highest priority to kdump by default. It is very useful for users who want to use some RAS tools.

I think that was the idea of the first “-” (“export list of RAS tools to user space”).

Thanks,
Bernhard

So for the time being I think we can put RAS tools on die notifier list and if it runs into issues we can always think of creating a separate list.

Few things come to mind.

- Why there is a separate panic_notifier_list? Can't it be merged with die_chain? die_val already got one of the event type as PANIC. If there are no specific reasons then we should merge the two lists. Registering RAS tools on a single list is easier.

- Modify Kdump to register on die_chain list.
- Modify Kdb to register on die_chain list.
- Export all the registered members of die_chain through sysfs along with their priorities. Priorities should be modifiable. Most likely one shall have to introduce additional field in struct notifier_block. This field will be a string as an identifier of the user registerd. e.g "Kdump", "Kdb" etc.

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

Quoted 11 lines · Vivek Goyal

- Modify Kdump to register on die_chain list.
- Modify Kdb to register on die_chain list.
- Export all the registered members of die_chain through sysfs along with their priorities. Priorities should be modifiable. Most likely one shall have to introduce additional field in struct notifier_block. This field will be a string as an identifier of the user registerd. e.g "Kdump", "Kdb" etc.

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

This is my image of your proposal.

- Print current order

# cat /sys/class/misc/debug/panic_notifier_list
priority   name
1          IPMI
2          watchdog
3          Kdb
4          Kdump

Quoted 3 lines · Vivek Goyal

So for the time being I think we can put RAS tools on die notifier list and if it runs into issues we can always think of creating a separate list.

Quoted 6 lines

Few things come to mind.

- Why there is a separate panic_notifier_list? Can't it be merged with die_chain? die_val already got one of the event type as PANIC. If there are no specific reasons then we should merge the two lists. Registering RAS tools on a single list is easier.

I think it is difficult, because die_chain is defined by each architecture.

Quoted 11 lines

- Modify Kdump to register on die_chain list.
- Modify Kdb to register on die_chain list.
- Export all the registered members of die_chain through sysfs along with their priorities. Priorities should be modifiable. Most likely one shall have to introduce additional field in struct notifier_block. This field will be a string as an identifier of the user registerd. e.g "Kdump", "Kdb" etc.

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

This is my image of your proposal.

- Print current order

# cat /sys/class/misc/debug/panic_notifier_list
priority   name
1          IPMI
2          watchdog
3          Kdb
4          Kdump

Quoted 10 lines · Takenori Nagano

Quoted 9 lines · Vivek Goyal

So for the time being I think we can put RAS tools on die notifier list and if it runs into issues we can always think of creating a separate list.

Few things come to mind.

- Why there is a separate panic_notifier_list? Can't it be merged with die_chain? die_val already got one of the event type as PANIC. If there are no specific reasons then we should merge the two lists. Registering RAS tools on a single list is easier.

Quoted 2 lines

I think it is difficult, because die_chain is defined by each architecture.

I think die_chain is arch independent definition (kernel/die_notifier.c)? But anyway, to begin with it can be done only for panic_notifier.

I think Bernhard's suggestion looks better here. I noticed that /sys/kernel/debug is already present. So how about following.

/sys/kernel/debug/kdump/priority
/sys/kernel/debug/kdb/priority
/sys/kernel/debug/IPMI/priority

I think at some point of time we shall have to create another file say description.

/sys/kernel/debug/IPMI/description

Which can tell what does this tool do? Other a user might not have any clue how to prioritize various things.

Quoted 14 lines · Vivek Goyal

Quoted 12 lines · Takenori Nagano

Quoted 9 lines · Vivek Goyal

So for the time being I think we can put RAS tools on die notifier list and if it runs into issues we can always think of creating a separate list.

Few things come to mind.

- Why there is a separate panic_notifier_list? Can't it be merged with die_chain? die_val already got one of the event type as PANIC. If there are no specific reasons then we should merge the two lists. Registering RAS tools on a single list is easier.

I think it is difficult, because die_chain is defined by each architecture.

Quoted 2 lines

I think die_chain is arch independent definition (kernel/die_notifier.c)? But anyway, to begin with it can be done only for panic_notifier.

I think die_val (notify_die() argument) values are arch independent. They are defined in include/asm-<arch>/kdebug.h.

Your idea is good, but I think it has very large impact. It is very hard to fix them at the same time. So, how about putting off merging two lists?

Quoted 4 lines · Takenori Nagano

Quoted 3 lines · Vivek Goyal

So for the time being I think we can put RAS tools on die notifier list and if it runs into issues we can always think of creating a separate list.

Quoted 8 lines

Quoted 6 lines

Few things come to mind.

- Why there is a separate panic_notifier_list? Can't it be merged with die_chain? die_val already got one of the event type as PANIC. If there are no specific reasons then we should merge the two lists. Registering RAS tools on a single list is easier.

I think it is difficult, because die_chain is defined by each architecture.

I think die_chain is arch independent definition (kernel/die_notifier.c)? But anyway, to begin with it can be done only for panic_notifier.

Quoted 13 lines

Quoted 11 lines

- Modify Kdump to register on die_chain list.
- Modify Kdb to register on die_chain list.
- Export all the registered members of die_chain through sysfs along with their priorities. Priorities should be modifiable. Most likely one shall have to introduce additional field in struct notifier_block. This field will be a string as an identifier of the user registerd. e.g "Kdump", "Kdb" etc.

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

This is my image of your proposal.

Quoted 9 lines

- Print current order

# cat /sys/class/misc/debug/panic_notifier_list
priority   name
1          IPMI
2          watchdog
3          Kdb
4          Kdump

I think Bernhard's suggestion looks better here. I noticed that /sys/kernel/debug is already present. So how about following.

/sys/kernel/debug/kdump/priority
/sys/kernel/debug/kdb/priority
/sys/kernel/debug/IPMI/priority

I think at some point of time we shall have to create another file say description.

/sys/kernel/debug/IPMI/description

Which can tell what does this tool do? Other a user might not have any clue how to prioritize various things.

Thanks
Vivek

Quoted 43 lines · Vivek Goyal

Quoted 4 lines · Takenori Nagano

Quoted 3 lines · Vivek Goyal

So for the time being I think we can put RAS tools on die notifier list and if it runs into issues we can always think of creating a separate list.

Quoted 8 lines

Quoted 6 lines

Few things come to mind.

- Why there is a separate panic_notifier_list? Can't it be merged with die_chain? die_val already got one of the event type as PANIC. If there are no specific reasons then we should merge the two lists. Registering RAS tools on a single list is easier.

I think it is difficult, because die_chain is defined by each architecture.

I think die_chain is arch independent definition (kernel/die_notifier.c)? But anyway, to begin with it can be done only for panic_notifier.

Quoted 13 lines

Quoted 11 lines

- Modify Kdump to register on die_chain list.
- Modify Kdb to register on die_chain list.
- Export all the registered members of die_chain through sysfs along with their priorities. Priorities should be modifiable. Most likely one shall have to introduce additional field in struct notifier_block. This field will be a string as an identifier of the user registerd. e.g "Kdump", "Kdb" etc.

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

This is my image of your proposal.

Quoted 9 lines

- Print current order

# cat /sys/class/misc/debug/panic_notifier_list
priority   name
1          IPMI
2          watchdog
3          Kdb
4          Kdump

I think Bernhard's suggestion looks better here. I noticed that /sys/kernel/debug is already present. So how about following.

Quoted 3 lines

/sys/kernel/debug/kdump/priority
/sys/kernel/debug/kdb/priority
/sys/kernel/debug/IPMI/priority

Why separate priority files is better than a central file? At least i think you get a grand picture of priority being defined for all parties with a central file?

What do we decide priority if more than one component has the same priority value?

Quoted 18 lines

Quoted 5 lines

Quoted 3 lines

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

This is my image of your proposal.

Quoted 9 lines

- Print current order

# cat /sys/class/misc/debug/panic_notifier_list
priority   name
1          IPMI
2          watchdog
3          Kdb
4          Kdump

I think Bernhard's suggestion looks better here. I noticed that /sys/kernel/debug is already present. So how about following.

Quoted 3 lines

/sys/kernel/debug/kdump/priority
/sys/kernel/debug/kdb/priority
/sys/kernel/debug/IPMI/priority

Why separate priority files is better than a central file? At least i think you get a grand picture of priority being defined for all parties with a central file?

What do we decide priority if more than one component has the same priority value?

On Tue, Aug 21, 2007 at 06:18:31AM -0700, Jay Lan wrote:
[..]

Quoted 17 lines

Quoted 17 lines

Quoted 5 lines

Quoted 3 lines

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

This is my image of your proposal.

Quoted 9 lines

- Print current order

# cat /sys/class/misc/debug/panic_notifier_list
priority   name
1          IPMI
2          watchdog
3          Kdb
4          Kdump

I think Bernhard's suggestion looks better here. I noticed that /sys/kernel/debug is already present. So how about following.

Quoted 7 lines

Quoted 3 lines

/sys/kernel/debug/kdump/priority
/sys/kernel/debug/kdb/priority
/sys/kernel/debug/IPMI/priority

Why separate priority files is better than a central file? At least i think you get a grand picture of priority being defined for all parties with a central file?

I thought of couple of reasons.
- A very different syntax to modify the priority.
- Separate directories allow easy future extensions in terms of more files. For example, putting a small "description" file in each dir where each registered user can specify what does it do.

But I agree that a single file is good for consolidated view. As bernhard suggested, may be we should also implement a read only file where one will get a consolidated view.

Quoted 4 lines

What do we decide priority if more than one component has the same priority value?

I think first come first serve would be appropriate in this case instead of returning -EINVAL.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Quoted 26 lines · Vivek Goyal

On Tue, Aug 21, 2007 at 06:18:31AM -0700, Jay Lan wrote:
[..]

Quoted 17 lines

Quoted 17 lines

Quoted 5 lines

Quoted 3 lines

Now user will be able to view all the die_chain users through sysfs and be able to modify the order in which these should run by modifying their priority. Hence all the RAS tools can co-exist.

This is my image of your proposal.

Quoted 9 lines

- Print current order

# cat /sys/class/misc/debug/panic_notifier_list
priority   name
1          IPMI
2          watchdog
3          Kdb
4          Kdump

I think Bernhard's suggestion looks better here. I noticed that /sys/kernel/debug is already present. So how about following.

Quoted 7 lines

Quoted 3 lines

/sys/kernel/debug/kdump/priority
/sys/kernel/debug/kdb/priority
/sys/kernel/debug/IPMI/priority

Why separate priority files is better than a central file? At least i think you get a grand picture of priority being defined for all parties with a central file?

Quoted 5 lines

I thought of couple of reasons.
- A very different syntax to modify the priority.
- Separate directories allow easy future extensions in terms of more files. For example, putting a small "description" file in each dir where each registered user can specify what does it do.

The first can be easily resolved by providing a comment section in the file with real examples. Users can simply uncomment a line to activate. But future expansion is certainly is a good reason for this layout.

Quoted 3 lines

But I agree that a single file is good for consolidated view. As bernhard suggested, may be we should also implement a read only file where one will get a consolidated view.

Yep, this will help!

Quoted 5 lines

Quoted 3 lines

What do we decide priority if more than one component has the same priority value?

I think first come first serve would be appropriate in this case instead of returning -EINVAL.

How does the kernel process the configuration files? By alphabetic order of the filename? Either way, i think a clear failure/warning dmesg is very important.

Thanks,
- jay

Quoted 7 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec