persistent store
Quoted 6 lines
Here's a patch based on some discussions I had with Thomas Gleixner at plumbers conference that implements a generic layer for persistent storage usable to pass tens or hundreds of kilobytes of data from the dying breath of a crashing kernel to its successor.
Quoted 6 lines
The usage model I'm envisioning is that a platform driver will register with this code to provide the actual storage. I've tried to make this interface general, but I'm working from a sample of one (the ACPI ERST code), so if anyone else has some persistent store that can't be handled by this code, speak up and we can put in the necessary tweaks.
I recently posted a patch set for powerpc to capture the most recent oops or panic message in NVRAM:
http://lists.ozlabs.org/pipermail/linuxppc-dev/2010-November/087032.html
It covers a lot of the same ground, and could be adapted to use your framework. See below for concerns and suggestions. I'd also be interested in feedback about the design decisions I mention below.
On powerpc, the amount of NVRAM available for this may be as little as 1-2 Kbytes. The minimal oops report (with essentially no backtrace) is about 1800 bytes. See below for implications.
We currently read our NVRAM contents via /dev/nvram and the nvram command. NVRAM is divided up into several "partitions" -- only one of which is used for the oops/panic report -- so the user-space code needs to know about how the partitions are laid out. It also needs to know how much text we actually wrote to the partition, and whether or not it's compressed. Since the kernel already knows how to determine all this, it would probably be more convenient to get at the oops/panic partition through your /sys interface.
Quoted 9 lines
...
2) "Why do you read in all the data from the device when it registers and save it in memory? Couldn't you just get the list of records and pick up the data from the device when the user reads the file?"
I don't think this is going to be very much data, just a few hundred kilobytes (i.e. less that $0.01 worth of memory, even expensive server memory). The memory is freed when the record is erased ... which is likely to be soon after boot.
Since the amount of text we capture is so tiny, this is unlikely to be an issue in my case.
Quoted 8 lines
...
6) "Is this widely useful? How many systems have persistent storage?" Although ERST was only added to the ACPI spec earlier this year, it merely documents existing functionality required for WHEA (Windows Hardware Error Architecture). So most modern server systems should have it (my test system has it, and it has a BIOS written in mid 2008). Sorry desktops & laptops - no love for you here.
Powerpc p Series does, obviously, and we're looking to exploit it in just this way.
Review
Quoted 11 lines
...
(quoted diff) +10
-0+static void +pstore_dump(struct kmsg_dumper *dumper, enum kmsg_dump_reason reason, + const char *s1, unsigned long l1, + const char *s2, unsigned long l2) +{ + unsigned long s1_start, s2_start; + unsigned long l1_cpy, l2_cpy; + char *dst = pstore_buf + psinfo->header_size; + + /* Don't dump oopses to persistent store */
Why not? In our case, we capture every oops and panic report, but keep only the most recent. Seems like catching the last oops could be useful if your system hangs thereafter and can't be made to panic. I suggest you pass along the reason (KMSG_DUMP_OOPS or whatever) and let the callback decide.
You'd have to serialize the oops handling, I guess, in case multiple CPUs oops simultaneously. (Gotta fix that in my code.)
Review
Quoted 13 lines
+ if (reason ==
KMSG_DUMP_OOPS)
+ return;(quoted diff) +11
-0+ + l2_cpy = min(l2, psinfo->data_size); + l1_cpy = min(l1, psinfo->data_size - l2_cpy); + + s2_start = l2 - l2_cpy; + s1_start = l1 - l1_cpy; + + memcpy(dst, s1 + s1_start, l1_cpy); + memcpy(dst + l1_cpy, s2 + s2_start, l2_cpy); + + psinfo->writer(PSTORE_DMESG, pstore_buf, l1_cpy + l2_cpy);
This assumes that you always want to capture the last psinfo->data_size bytes of the printk buffer. Given the small capacity of our NVRAM partition, I handle the case where the whole oops report doesn't fit. In that case, I sacrifice the end of the oops report to capture the beginning. Patch #3 in my set is about this.
Review
Quoted 6 lines
...
(quoted diff) +4
-0+static int +pstore_create_sysfs_entry(struct pstore_entry *new_pstore) +{ + new_pstore->attr.attr.mode = 0444;
/var/log/messages is typically not readable by everybody. This appears to circumvent that.
Quoted 1 line
...
Thanks.
Jim Keniston
IBM Linux Technology Center
Beaverton, OR
Quoted 7 lines · Jim Keniston
Quoted 1 line
+ /* Don't dump oopses to persistent store */
Why not? In our case, we capture every oops and panic report, but keep only the most recent. Seems like catching the last oops could be useful if your system hangs thereafter and can't be made to panic. I suggest you pass along the reason (
KMSG_DUMP_OOPSor whatever) and let the callback decide.
My thoughts were that Oops were non-fatal and ended up in /var/log/messages, so this would be unneeded (this bit of code was copied from one of mtdoops or ramoops - which does almost the same ... they do have an option to allow the copy - perhaps I should have copied that bit too?).
Quoted 2 lines
You'd have to serialize the oops handling, I guess, in case multiple CPUs oops simultaneously. (Gotta fix that in my code.)
Yup - I need to do this too (I only allocate one buffer).
Quoted 7 lines
Quoted 1 line
+
psinfo->writer(PSTORE_DMESG, pstore_buf, l1_cpy + l2_cpy);This assumes that you always want to capture the last
psinfo->data_sizebytes of the printk buffer. Given the small capacity of our NVRAM partition, I handle the case where the whole oops report doesn't fit. In that case, I sacrifice the end of the oops report to capture the beginning. Patch #3 in my set is about this.
Yes - I assume here that the last "data_size" bytes will be enough to be useful. But in your case it most likely won't be. You could lie about how much space you allow and then include some oops parsing code to get the vital bits out of what is passed to you. Not pretty - but it would work. If there are many such devices with limited capacity, then it would make sense to include this parsing code in this generic layer.
Quoted 4 lines
Quoted 1 line
+
new_pstore->attr.attr.mode= 0444;
/var/log/messagesis typically not readable by everybody. This appears to circumvent that.
But "dmesg(8)" typically *does* allow any user to see the most recent part of the console log - so we are not consistent about this.
-Tony
Quoted 7 lines · Tony Luck
Yes - I assume here that the last "
data_size" bytes will be enough to be useful. But in your case it most likely won't be. You could lie about how much space you allow and then include some oops parsing code to get the vital bits out of what is passed to you. Not pretty - but it would work. If there are many such devices with limited capacity, then it would make sense to include this parsing code in this generic layer.
Sorry - finding the good bits from the console log is architecture dependent ... so it can't be done (easily) in the generic code.
-Tony
Quoted 11 lines · Tony Luck
Quoted 7 lines · Tony Luck
Yes - I assume here that the last "
data_size" bytes will be enough to be useful. But in your case it most likely won't be. You could lie about how much space you allow and then include some oops parsing code to get the vital bits out of what is passed to you. Not pretty - but it would work. If there are many such devices with limited capacity, then it would make sense to include this parsing code in this generic layer.Sorry - finding the good bits from the console log is architecture dependent ... so it can't be done (easily) in the generic code.
FYI:
Now we use ramoops for this purpose. Even tough ramoops uses volatile memory, it's safe soft-reset and watchdog reset.
To overcome limiataion of ramoops. not persist, I'm working on mmcoops as persistent oops record.
As the discussed story. it covers the previous oops devices, mtdoops(drivers/mtd/mtdoops.c), ramoops (drivers/char/ramoops.c) and mmcoops (drivers/mmc/card/mmc_oops.c).
Actually I tried to enable the lcdoops, display oops message at LCD. but it doesn't work since not much time to display LCD.
Thank you,
Kyungmin Park
Quoted 8 lines
-Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines · Tony Luck
Quoted 7 lines · Jim Keniston
Quoted 1 line
+ /* Don't dump oopses to persistent store */
Why not? In our case, we capture every oops and panic report, but keep only the most recent. Seems like catching the last oops could be useful if your system hangs thereafter and can't be made to panic. I suggest you pass along the reason (
KMSG_DUMP_OOPSor whatever) and let the callback decide.
Quoted 4 lines
My thoughts were that Oops were non-fatal and ended up in
/var/log/messages, so this would be unneeded (this bit of code was copied from one of mtdoops or ramoops - which does almost the same ... they do have an option to allow the copy - perhaps I should have copied that bit too?).
Yes, I'd still vote for that, because:
1) it provides flexibility at very low cost;
2) it could be useful if syslogd and/or klogd and/or the filesystem holding /var/log are in trouble; and
3) it's helpful because I want to be sure -- even in the face of limited NVRAM -- to capture the start of an oops that causes a panic.
(3) requires a little more explanation: As far as I can tell, by the time you're in panic(), there's no way to know that you're panicking because of an oops. (The oops_in_progress flag doesn't seem to be intended for this.) But if I get notified at the time of the oops, I can check the panic_on_oops flag and know that we're GOING to panic, and set a panicking_on_oops flag for use when I get called back again during the panic. (No, my patch set doesn't do that yet, because I didn't figure it out 'til recently.) There's perhaps a more generic solution to this particular problem, but I may be your only client with such space constraints.
Quoted 13 lines
Quoted 3 lines
You'd have to serialize the oops handling, I guess, in case multiple CPUs oops simultaneously. (Gotta fix that in my code.)
Yup - I need to do this too (I only allocate one buffer).
Quoted 7 lines
Quoted 1 line
+
psinfo->writer(PSTORE_DMESG, pstore_buf, l1_cpy + l2_cpy);This assumes that you always want to capture the last
psinfo->data_sizebytes of the printk buffer. Given the small capacity of our NVRAM partition, I handle the case where the whole oops report doesn't fit. In that case, I sacrifice the end of the oops report to capture the beginning. Patch #3 in my set is about this.
Quoted 5 lines
Yes - I assume here that the last "
data_size" bytes will be enough to be useful. But in your case it most likely won't be. You could lie about how much space you allow and then include some oops parsing code to get the vital bits out of what is passed to you. Not pretty - but it would work.
Yeah, in the case of powerpc, a psinfo->data_size value of (say) 8K would almost certainly include the start of the oops. And then I could simplify my code quite a bit.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
+
new_pstore->attr.attr.mode= 0444;
/var/log/messagesis typically not readable by everybody. This appears to circumvent that.But "dmesg(8)" typically *does* allow any user to see the most recent part of the console log - so we are not consistent about this.
You're right, of course. It's the user-mode syslog messages that are being hidden.
Quoted 2 lines
-Tony
Jim