Skip to contentSkip to conversation

writing OOPS/panic info to nvram?

DISCUSSION Roy Sigurd Karlsbakk 11 participants 21 messages Sep 4, 2002 – Sep 13, 2002

lore
Share & tools
lore
#1 lore raw

Hey, Roy

Quoted 3 lines · Roy Sigurd Karlsbakk

hi

I just read in the OS X.2 technote

So did I :)

Quoted 4 lines

(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that they're writing the panic dump to nvram.

Is it hard to implement this on Linux?

Doesnt look like too much work. As far as I can see, from a quick glance, we have everything we need at hand - just have to write a snippet of code in traps.c. Ill give it a whirl now.

Quoted 7 lines

roy

Signature · 3 lines
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356

== Morten

Signature · 8 lines
"Livet er ikke for nybegynnere" - sitat fra en klok person.

mvh
Morten Helgesen 
UNIX System Administrator & C Developer 
Nextframe AS
[email protected] / 93445641
http://www.nextframe.net
#3 lore raw
Quoted 11 lines · Alan Cox
Quoted 7 lines · Roy Sigurd Karlsbakk

hi

I just read in the OS X.2 technote
(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that they're writing the panic dump to nvram.

Is it hard to implement this on Linux?

Its been done years ago. However on a PC you basically have no free nvram so its not terribly useful there.

True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.

I didn`t realize we only had 114 bytes to work with.

== Morten

Signature · 8 lines
"Livet er ikke for nybegynnere" - sitat fra en klok person.

mvh
Morten Helgesen 
UNIX System Administrator & C Developer 
Nextframe AS
[email protected] / 93445641
http://www.nextframe.net
#4 lore raw
Quoted 5 lines · Morten Helgesen

True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.

I didn`t realize we only had 114 bytes to work with.

We don't. They are all used by the BIOS

#5 lore raw
Quoted 8 lines · Alan Cox
Quoted 5 lines · Morten Helgesen

True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.

I didn`t realize we only had 114 bytes to work with.

We don't. They are all used by the BIOS

That makes it even less useful. Oh well.

== Morten

Signature · 8 lines
"Livet er ikke for nybegynnere" - sitat fra en klok person.

mvh
Morten Helgesen 
UNIX System Administrator & C Developer 
Nextframe AS
[email protected] / 93445641
http://www.nextframe.net
#6 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 6 lines · Morten Helgesen
Quoted 5 lines · Alan Cox
Quoted 4 lines · Morten Helgesen

True - the 'normal' size on a PC is apparently something like 114 bytes ... I guess we could use it for something useful ... but maybe not for OOPSen/panics.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I didn`t realize we only had 114 bytes to work with.

We don't. They are all used by the BIOS

That makes it even less useful. Oh well.

For PC style hardware it does. For other platforms, it's stil nice to be able to see the oops info on an unattended crash (all crashes? ;) Dump to nvram, dump to file after boot.... Other option is to crash-dump to swap... Question is, do you really want to do that?

- ---
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam http://www.sara.nl
High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc

"I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (Darwin)

iD8DBQE9dhJkBIoCv9yTlOwRAgG3AJ0SFZCTxz01okoJjlD4QHqNEBJjuACgp6sI YJloQg8tfaE+677XsT7sq7M=
=p/b2
-----END PGP SIGNATURE-----

#7 lore raw
Quoted 17 lines · On 2002.09.04 Remco Post wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 11 lines · Morten Helgesen
Quoted 9 lines · Alan Cox
Quoted 6 lines · Morten Helgesen

True - the 'normal' size on a PC is apparently something like 114 bytes ...
I guess we could use it for something useful ... but maybe not for OOPSen/panics.

I didn`t realize we only had 114 bytes to work with.

We don't. They are all used by the BIOS

Quoted 8 lines
Quoted 2 lines

That makes it even less useful. Oh well.

For PC style hardware it does. For other platforms, it's stil nice to be able to see the oops info on an unattended crash (all crashes? ;) Dump to nvram, dump to file after boot.... Other option is to crash-dump to swap... Question is, do you really want to do that?

Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".

Signature · 4 lines
J.A. Magallon <[email protected]>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.0 (Cooker) for i586
Linux 2.4.20-pre5-j0 (gcc 3.2 (Mandrake Linux 9.0 3.2-1mdk))
#8 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 28 lines · J.A. Magallon
Quoted 17 lines · On 2002.09.04 Remco Post wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 11 lines · Morten Helgesen
Quoted 9 lines · Alan Cox
Quoted 6 lines · Morten Helgesen

True - the 'normal' size on a PC is apparently something like 114 bytes ...
I guess we could use it for something useful ... but maybe not for OOPSen/panics.

I didn`t realize we only had 114 bytes to work with.

We don't. They are all used by the BIOS

Quoted 8 lines
Quoted 2 lines

That makes it even less useful. Oh well.

For PC style hardware it does. For other platforms, it's stil nice to be able to see the oops info on an unattended crash (all crashes? ;) Dump to nvram, dump to file after boot.... Other option is to crash-dump to swap... Question is, do you really want to do that?

Quoted 5 lines

Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".

Mjah,

swap can be destroyed during a crash dump, so why not use that? Everybody else does it (AIX, Solaris, IRIX) so if none of the commercial UNIX vendors see a problem in using swap for dump, why should we?

I like the nvram option best for hardware that allows a oops output to be stored in nvram, since that is also usable on diskless systems when the network is down (or the network driver oopses) or when the disk driver oopses... Maybe do something like:

if there is enough space on disk && ..., use that else
if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else if there is nvram available use that

and print on the console....

maybe this is all a bit overkill for what we need to do, maybe not... I guess it would make it possible to debug systems that are just sitting there by themselves, and have some curious problem...

- ---
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam http://www.sara.nl
High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc

"I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (Darwin)

iD8DBQE9dhdQBIoCv9yTlOwRAsm0AJ40VJx5QmhrI7ME4w02WyM3Vn9X6wCcDMyq M8Dp4XhDjSkKtEO3rbLlwxo=
=YtBZ
-----END PGP SIGNATURE-----

#9 lore raw

Hi!

Quoted 7 lines

driver oopses... Maybe do something like:

if there is enough space on disk && ..., use that else if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else
if there is nvram available use that

You just killed any data you had on the tape... too bad.
Pavel

Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
#10 lore raw
Quoted 4 lines · Pavel Machek

Hi!

Quoted 2 lines

driver oopses... Maybe do something like:

Quoted 8 lines
Quoted 5 lines

if there is enough space on disk && ..., use that else if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else
if there is nvram available use that

You just killed any data you had on the tape... too bad.
Pavel

Yes, so, or you just saved that oops that has been bugging you for months... (And yes I'm probably one of those rare people that has tapedrives attached that are not used for anything usefull).

---
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam http://www.sara.nl
High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc

"I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams

#11 lore raw

Hi!

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

driver oopses... Maybe do something like:

if there is enough space on disk && ..., use that else if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is writeable... else
if there is nvram available use that

Quoted 6 lines
Quoted 2 lines

You just killed any data you had on the tape... too bad.
Pavel

Yes, so, or you just saved that oops that has been bugging you for months... (And yes I'm probably one of those rare people that has tapedrives attached that are not used for anything usefull).

If it was bugging you for months, then you'd have probably copied it using paper and pencil already. cli(); while(1); at the end of oops handler is not *that* hard to do.
Pavel

Signature · 2 lines
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?
#12 lore raw
In reply to J.A. Magallon
Quoted 3 lines · J.A. Magallon

Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".

With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat

#13 lore raw
Quoted 8 lines · On 2002.09.04 Alan Cox wrote:
Quoted 3 lines · J.A. Magallon

Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".

With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat

Ah, ther is no way to write raw blocks at a very low level to disk...?? LKCD at least writes to a floppy, doesn't it.?
Say you just need one block for the dump. Could you get the block location (H/C/S) on boot and tell the bios to write a chunk there on crash ?

Signature · 4 lines
J.A. Magallon <[email protected]>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.0 (Cooker) for i586
Linux 2.4.20-pre5-j0 (gcc 3.2 (Mandrake Linux 9.0 3.2-1mdk))
#14 lore raw
Quoted 1 line · J.A. Magallon

Ah, ther is no way to write raw blocks at a very low level to disk...??

Not reliably; you _know_ your infrastructure has crashed, otherwise you wouldn't be inside the crash dump handler ;), so you can't possibly trust the normal block layer to write the crash dump (and not write it over your salary and customer database).

Parts of this could probably be circumvented by a checksum of the code which
is computed at boot time and checked before the crashdump takes place, but this doesn't deal with a crashed SCSI driver.

A network dump is much safer, though I would suggest running it over a dedicated card / driver combo and on a special ethernet protocol, because you might have lost your IP configuration...

Sincerely,
Lars Marowsky-Brée <[email protected]>

Signature · 2 lines
Immortality is an adequate definition of high availability for me.
	--- Gregory F. Pfister
#15 lore raw

Hi!

Quoted 6 lines
Quoted 1 line

Ah, ther is no way to write raw blocks at a very low level to disk...??

Not reliably; you _know_ your infrastructure has crashed, otherwise you wouldn't be inside the crash dump handler ;), so you can't possibly trust the normal block layer to write the crash dump (and not write it over your salary and customer database).

Floppy seems like safe choice. Verify its special "crash floppy" by checking signature, then write.

Quoted 3 lines

A network dump is much safer, though I would suggest running it over a dedicated card / driver combo and on a special ethernet protocol, because you might have lost your IP configuration...

Its enough for it to work 99% cases. Separate ethernet card is overkill, serial console is easier than that.
Pavel

Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
#16 lore raw
In reply to Alan Cox
Quoted 2 lines · Alan Cox

With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat

Rusty has written a very basic polled-mode IDE driver for precisely this situation. He even tested it on x86 and powermac. :) He has a little user-space program that allocates a file and uses FIOBMAP to work out which disk blocks it is using. The program writes a signature to the blocks and then tells the kernel crashdump module the block numbers. When the kernel panics, it calls the crashdump module which first reads the blocks it was told and makes sure they have the right signature, then writes the oops information to those blocks and then reboots.

IDE was relatively straightforward since you can do basic block I/O with just the ATA-1 or ATA-2 registers and command set and PIO. In contrast, I believe SCSI defeated him. :)

Paul.

#17 lore raw

From: Paul Mackerras <[email protected]>
Date: Thu, 5 Sep 2002 09:34:52 +1000 (EST)

I believe SCSI defeated him. :)

If you can get at the interrupt handler, and the interrupt handler is coded well enough to handle sharing interrupts (%99 of PCI scsi drivers are) then it is doable. Just submit it as normal through scsi, mark it high priority somehow, and then keep calling the interrupt handler in a loop until your command completes or fails :-)

#18 lore raw
In reply to Paul Mackerras
Quoted 3 lines · Paul Mackerras

IDE was relatively straightforward since you can do basic block I/O with just the ATA-1 or ATA-2 registers and command set and PIO. In contrast, I believe SCSI defeated him. :)

You have to reset and retune the interface/controller registers as well, otherwise bad things can happen. Rusty will be happy to know that next generation SATA drops the PIO interface for a demented PIO via DMA setup
8)

#19 lore raw
Quoted 7 lines · On 05 Sep 2002 01:07:01 +0100 Alan Cox <[email protected]> wrote:
Quoted 3 lines · Paul Mackerras

IDE was relatively straightforward since you can do basic block I/O with just the ATA-1 or ATA-2 registers and command set and PIO. In contrast, I believe SCSI defeated him. :)

You have to reset and retune the interface/controller registers as well, otherwise bad things can happen.

No, at this stage the code reboots the machine (well, I could call the ide reset code at this point I guess).

Note that my mini oops dumper object file is a leafnode which doesn't use any external code in the dump path (checked at build time). It is armed with the symbols and device & block offsets by userspace. I plan to update it in the next month, but it's trivial enough (a new driver with one hook in the oops code) to be done after the freeze if reqd.

The interesting bit becomes harvesting those reports: this is a higher level problem (userspace and privacy being the higher levels, respectively).

Rusty.

Signature · 2 lines
   there are those who do and those who hang on and you don't see too
   many doers quoting their contemporaries.  -- Larry McVoy
#20 lore raw
In reply to Alan Cox
Quoted 9 lines · Alan Cox
Quoted 3 lines · J.A. Magallon

Instead of swap, let user specify a partition to raw dump there. If a user wants crash dumps, he has to leave some small disk space free and give an option like "dump=/dev/hda7".

With what will you write it - not the linux block layer thats for sure. Ingo has patches for doing network dumps which are kind of neat

-

LKCD for 2.5 (WIP) has a dump driver interface through which different target types can be plugged in. For example Ingo's polled network dump code been integrated as one such dump driver target (generic type), block layer based i/o is available as another target (for those who chose to use it for their raw partition).Down the line specific dump drivers suited for the hardware concerned, e.g Rusty's polled IDE driver, could be plugged in as dump target too, as could NECs work on converting dump block i/o to polled mode.

Conceivably, one may perhaps have alternate targets available on the same system and failover to the suitable one based on the situation. (If the network interface code is the one in trouble, try to dump to the dedicated raw disk or vice versa).

And then, a little later there could be the option of memory save option abstracted as another driver target, to be followed by a soft-reboot (w/o clearing memory) for performing actual dump i/o to persistent storage on architectures where this option works out.

Regards
Suparna