writing OOPS/panic info to nvram?
hi
I just read in the OS X.2 technote
(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that they're writing the panic dump to nvram.
Is it hard to implement this on Linux?
roy
Signature · 6 lines
Roy Sigurd Karlsbakk, Datavaktmester ProntoTV AS - http://www.pronto.tv/ Tel: +47 9801 3356 Computers are like air conditioners. They stop working when you open Windows.
Hey, Roy
Quoted 3 lines · Roy Sigurd Karlsbakk
hi
I just read in the OS X.2 technote
So did I :)
Quoted 4 lines
(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that they're writing the panic dump to nvram.
Is it hard to implement this on Linux?
Doesnt look like too much work. As far as I can see, from a quick glance, we have everything we need at hand - just have to write a snippet of code in traps.c. Ill give it a whirl now.
Quoted 7 lines
roy
Signature · 3 lines
Roy Sigurd Karlsbakk, Datavaktmester ProntoTV AS - http://www.pronto.tv/ Tel: +47 9801 3356
== Morten
Signature · 8 lines
"Livet er ikke for nybegynnere" - sitat fra en klok person. mvh Morten Helgesen UNIX System Administrator & C Developer Nextframe AS [email protected] / 93445641 http://www.nextframe.net
Quoted 7 lines · Roy Sigurd Karlsbakk
hi
I just read in the OS X.2 technote
(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that they're writing the panic dump to nvram.Is it hard to implement this on Linux?
Its been done years ago. However on a PC you basically have no free nvram so its not terribly useful there.
Quoted 11 lines · Alan Cox
Quoted 7 lines · Roy Sigurd Karlsbakk
hi
I just read in the OS X.2 technote
(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that they're writing the panic dump to nvram.Is it hard to implement this on Linux?
Its been done years ago. However on a PC you basically have no free nvram so its not terribly useful there.
True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.
I didn`t realize we only had 114 bytes to work with.
== Morten
Signature · 8 lines
"Livet er ikke for nybegynnere" - sitat fra en klok person. mvh Morten Helgesen UNIX System Administrator & C Developer Nextframe AS [email protected] / 93445641 http://www.nextframe.net
Quoted 5 lines · Morten Helgesen
True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.
I didn`t realize we only had 114 bytes to work with.
We don't. They are all used by the BIOS
Quoted 8 lines · Alan Cox
Quoted 5 lines · Morten Helgesen
True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.
I didn`t realize we only had 114 bytes to work with.
We don't. They are all used by the BIOS
That makes it even less useful. Oh well.
== Morten
Signature · 8 lines
"Livet er ikke for nybegynnere" - sitat fra en klok person. mvh Morten Helgesen UNIX System Administrator & C Developer Nextframe AS [email protected] / 93445641 http://www.nextframe.net
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 6 lines · Morten Helgesen
Quoted 5 lines · Alan Cox
Quoted 4 lines · Morten Helgesen
True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
I didn`t realize we only had 114 bytes to work with.
We don't. They are all used by the BIOS
That makes it even less useful. Oh well.
For PC style hardware it does. For other platforms, it's stil nice to be able to see the oops info on an unattended crash (all crashes? ;) Dump to nvram, dump to file after boot.... Other option is to crash-dump to swap... Question is, do you really want to do that?
- ---
Met vriendelijke groeten,
Remco Post
SARA - Stichting Academisch Rekencentrum Amsterdam http://www.sara.nl
High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc
"I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (Darwin)
iD8DBQE9dhJkBIoCv9yTlOwRAgG3AJ0SFZCTxz01okoJjlD4QHqNEBJjuACgp6sI YJloQg8tfaE+677XsT7sq7M=
=p/b2
-----END PGP SIGNATURE-----
Quoted 17 lines · On 2002.09.04 Remco Post wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 11 lines · Morten Helgesen
Quoted 9 lines · Alan Cox
Quoted 6 lines · Morten Helgesen
True - the 'normal' size on a PC is apparently something like 114 bytes ...
I guess we could use it for something useful ... but maybe not for OOPSen/panics.I didn`t realize we only had 114 bytes to work with.
We don't. They are all used by the BIOS
Quoted 8 lines
Quoted 2 lines
That makes it even less useful. Oh well.
For PC style hardware it does. For other platforms, it's stil nice to be able to see the oops info on an unattended crash (all crashes? ;) Dump to nvram, dump to file after boot.... Other option is to crash-dump to swap... Question is, do you really want to do that?
Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.0 (Cooker) for i586 Linux 2.4.20-pre5-j0 (gcc 3.2 (Mandrake Linux 9.0 3.2-1mdk))
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 28 lines · J.A. Magallon
Quoted 17 lines · On 2002.09.04 Remco Post wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 11 lines · Morten Helgesen
Quoted 9 lines · Alan Cox
Quoted 6 lines · Morten Helgesen
True - the 'normal' size on a PC is apparently something like 114 bytes ...
I guess we could use it for something useful ... but maybe not for OOPSen/panics.I didn`t realize we only had 114 bytes to work with.
We don't. They are all used by the BIOS
Quoted 8 lines
Quoted 2 lines
That makes it even less useful. Oh well.
For PC style hardware it does. For other platforms, it's stil nice to be able to see the oops info on an unattended crash (all crashes? ;) Dump to nvram, dump to file after boot.... Other option is to crash-dump to swap... Question is, do you really want to do that?
Quoted 5 lines
Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".
Mjah,
swap can be destroyed during a crash dump, so why not use that? Everybody else does it (AIX, Solaris, IRIX) so if none of the commercial UNIX vendors see a problem in using swap for dump, why should we?
I like the nvram option best for hardware that allows a oops output to be stored in nvram, since that is also usable on diskless systems when the network is down (or the network driver oopses) or when the disk driver oopses... Maybe do something like:
if there is enough space on disk && ..., use that else
if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else if there is nvram available use that
and print on the console....
maybe this is all a bit overkill for what we need to do, maybe not... I guess it would make it possible to debug systems that are just sitting there by themselves, and have some curious problem...
- ---
Met vriendelijke groeten,
Remco Post
SARA - Stichting Academisch Rekencentrum Amsterdam http://www.sara.nl
High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc
"I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (Darwin)
iD8DBQE9dhdQBIoCv9yTlOwRAsm0AJ40VJx5QmhrI7ME4w02WyM3Vn9X6wCcDMyq M8Dp4XhDjSkKtEO3rbLlwxo=
=YtBZ
-----END PGP SIGNATURE-----
Hi!
Quoted 7 lines
driver oopses... Maybe do something like:
if there is enough space on disk && ..., use that else if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else
if there is nvram available use that
You just killed any data you had on the tape... too bad.
Pavel
Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt, details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
Quoted 4 lines · Pavel Machek
Hi!
Quoted 2 lines
driver oopses... Maybe do something like:
Quoted 8 lines
Quoted 5 lines
if there is enough space on disk && ..., use that else if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else
if there is nvram available use thatYou just killed any data you had on the tape... too bad.
Pavel
Yes, so, or you just saved that oops that has been bugging you for months... (And yes I'm probably one of those rare people that has tapedrives attached that are not used for anything usefull).
---
Met vriendelijke groeten,
Remco Post
SARA - Stichting Academisch Rekencentrum Amsterdam http://www.sara.nl
High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc
"I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
Hi!
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
driver oopses... Maybe do something like:
if there is enough space on disk && ..., use that else if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else
if there is nvram available use that
Quoted 6 lines
Quoted 2 lines
You just killed any data you had on the tape... too bad.
PavelYes, so, or you just saved that oops that has been bugging you for months... (And yes I'm probably one of those rare people that has tapedrives attached that are not used for anything usefull).
If it was bugging you for months, then you'd have probably copied it using paper and pencil already. cli(); while(1); at the end of oops handler is not *that* hard to do.
Pavel
Signature · 2 lines
Worst form of spam? Adding advertisment signatures ala sourceforge.net. What goes next? Inserting advertisment *into* email?
Quoted 3 lines · J.A. Magallon
Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".
With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat
Quoted 8 lines · On 2002.09.04 Alan Cox wrote:
Quoted 3 lines · J.A. Magallon
Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".
With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat
Ah, ther is no way to write raw blocks at a very low level to disk...?? LKCD at least writes to a floppy, doesn't it.?
Say you just need one block for the dump. Could you get the block location (H/C/S) on boot and tell the bios to write a chunk there on crash ?
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.0 (Cooker) for i586 Linux 2.4.20-pre5-j0 (gcc 3.2 (Mandrake Linux 9.0 3.2-1mdk))
Quoted 1 line · J.A. Magallon
Ah, ther is no way to write raw blocks at a very low level to disk...??
Not reliably; you _know_ your infrastructure has crashed, otherwise you wouldn't be inside the crash dump handler ;), so you can't possibly trust the normal block layer to write the crash dump (and not write it over your salary and customer database).
Parts of this could probably be circumvented by a checksum of the code which
is computed at boot time and checked before the crashdump takes place, but this doesn't deal with a crashed SCSI driver.
A network dump is much safer, though I would suggest running it over a dedicated card / driver combo and on a special ethernet protocol, because you might have lost your IP configuration...
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 2 lines
Immortality is an adequate definition of high availability for me. --- Gregory F. Pfister
Hi!
Quoted 6 lines
Quoted 1 line
Ah, ther is no way to write raw blocks at a very low level to disk...??
Not reliably; you _know_ your infrastructure has crashed, otherwise you wouldn't be inside the crash dump handler ;), so you can't possibly trust the normal block layer to write the crash dump (and not write it over your salary and customer database).
Floppy seems like safe choice. Verify its special "crash floppy" by checking signature, then write.
Quoted 3 lines
A network dump is much safer, though I would suggest running it over a dedicated card / driver combo and on a special ethernet protocol, because you might have lost your IP configuration...
Its enough for it to work 99% cases. Separate ethernet card is overkill, serial console is easier than that.
Pavel
Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt, details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
Quoted 2 lines · Alan Cox
With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat
Rusty has written a very basic polled-mode IDE driver for precisely this situation. He even tested it on x86 and powermac. :) He has a little user-space program that allocates a file and uses FIOBMAP to work out which disk blocks it is using. The program writes a signature to the blocks and then tells the kernel crashdump module the block numbers. When the kernel panics, it calls the crashdump module which first reads the blocks it was told and makes sure they have the right signature, then writes the oops information to those blocks and then reboots.
IDE was relatively straightforward since you can do basic block I/O with just the ATA-1 or ATA-2 registers and command set and PIO. In contrast, I believe SCSI defeated him. :)
Paul.
From: Paul Mackerras <[email protected]>
Date: Thu, 5 Sep 2002 09:34:52 +1000 (EST)
I believe SCSI defeated him. :)
If you can get at the interrupt handler, and the interrupt handler is coded well enough to handle sharing interrupts (%99 of PCI scsi drivers are) then it is doable. Just submit it as normal through scsi, mark it high priority somehow, and then keep calling the interrupt handler in a loop until your command completes or fails :-)
Quoted 3 lines · Paul Mackerras
IDE was relatively straightforward since you can do basic block I/O with just the ATA-1 or ATA-2 registers and command set and PIO. In contrast, I believe SCSI defeated him. :)
You have to reset and retune the interface/controller registers as well, otherwise bad things can happen. Rusty will be happy to know that next generation SATA drops the PIO interface for a demented PIO via DMA setup
8)
Quoted 7 lines · On 05 Sep 2002 01:07:01 +0100 Alan Cox <[email protected]> wrote:
Quoted 3 lines · Paul Mackerras
IDE was relatively straightforward since you can do basic block I/O with just the ATA-1 or ATA-2 registers and command set and PIO. In contrast, I believe SCSI defeated him. :)
You have to reset and retune the interface/controller registers as well, otherwise bad things can happen.
No, at this stage the code reboots the machine (well, I could call the ide reset code at this point I guess).
Note that my mini oops dumper object file is a leafnode which doesn't use any external code in the dump path (checked at build time). It is armed with the symbols and device & block offsets by userspace. I plan to update it in the next month, but it's trivial enough (a new driver with one hook in the oops code) to be done after the freeze if reqd.
The interesting bit becomes harvesting those reports: this is a higher level problem (userspace and privacy being the higher levels, respectively).
Rusty.
Signature · 2 lines
there are those who do and those who hang on and you don't see too many doers quoting their contemporaries. -- Larry McVoy
Quoted 9 lines · Alan Cox
Quoted 3 lines · J.A. Magallon
Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".
With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat
-
LKCD for 2.5 (WIP) has a dump driver interface through which different target types can be plugged in. For example Ingo's polled network dump code been integrated as one such dump driver target (generic type), block layer based i/o is available as another target (for those who chose to use it for their raw partition).Down the line specific dump drivers suited for the hardware concerned, e.g Rusty's polled IDE driver, could be plugged in as dump target too, as could NECs work on converting dump block i/o to polled mode.
Conceivably, one may perhaps have alternate targets available on the same system and failover to the suitable one based on the situation. (If the network interface code is the one in trouble, try to dump to the dedicated raw disk or vice versa).
And then, a little later there could be the option of memory save option abstracted as another driver target, to be followed by a soft-reboot (w/o clearing memory) for performing actual dump i/o to persistent storage on architectures where this option works out.
Regards
Suparna