pstore improvements (pstore-next)
cover pstore improvements (pstore-next)
This is a posting of several patches I've been working on to improve pstore. Most of it is better comments, output, and naming, but one bug fix stands out to fix head-truncationg of compressed records. Details in the individual patches. Review appreciated! :)
-Kees
Kees Cook (8):
pstore/ram: Standardize module name in ramoops
pstore: Do not use crash buffer for decompression pstore/ram: Report backend assignments with finer granularity pstore/ram: Add kern-doc for struct persistent_ram_zone
pstore: Improve and update some comments and status output
pstore: Replace open-coded << with BIT()
pstore: Remove needless lock during console writes pstore/ram: Correctly calculate usable PRZ bytes
fs/pstore/platform.c | 92 ++++++++++++++------------------------ fs/pstore/ram.c | 19 ++++---- fs/pstore/ram_core.c | 25 +++++++++-- include/linux/pstore.h | 15 ++++--- include/linux/pstore_ram.h | 46 +++++++++++++++++-- 5 files changed, 116 insertions(+), 81 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/8 pstore/ram: Standardize module name in ramoops
With both ram.c and ram_core.c built into ramoops.ko, it doesn't make sense to have differing pr_fmt prefixes. This fixes ram_core.c to use the module name (as ram.c already does). Additionally improves region reservation error to include the region name.
--- fs/pstore/ram_core.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/pstore/ram_core.c +3 -2
@@ -12,7 +12,7 @@* */ -#define pr_fmt (fmt) "persistent_ram: " fmt +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <linux/device.h> #include <linux/err.h>
@@ -443,7 +443,8 @@ static void *persistent_ram_iomap(phys_addr_t start, size_t size,void *va;if (!request_mem_region (start, size, label ?:"ramoops" )) { -pr_err ("request mem region (0x%llx@0x%llx) failed\n" , +pr_err ("request mem region (%s 0x%llx@0x%llx) failed\n" , + label ?:"ramoops" , (unsigned long long )size, (unsigned long long )start);return NULL ; }
Signature · 1 line
2.17.1
PATCH 2/8 pstore: Do not use crash buffer for decompression
In reply to Kees CookThe pre-allocated compression buffer used for crash dumping was also being used for decompression. This isn't technically safe, since it's possible the kernel may attempt a crashdump while pstore is populating the pstore filesystem (and performing decompression). Instead, just allocate a separate buffer for decompression. Correctness is preferred over performance here.
--- fs/pstore/platform.c | 56 ++++++++++++++++++++------------------------ 1 file changed, 25 insertions(+), 31 deletions(-)
fs/pstore/platform.c +25 -31
@@ -258,20 +258,6 @@ static int pstore_compress(const void *in, void *out,return outlen; } -static int pstore_decompress (void *in,void *out, -unsigned int inlen,unsigned int outlen) -{ -int ret; - - ret =crypto_comp_decompress (tfm, in, inlen, out, &outlen); -if (ret) { -pr_err ("crypto_comp_decompress failed, ret = %d!\n" , ret); -return ret; - } - -return outlen; -} -static void allocate_buf_for_compression (void ) {struct crypto_comp *ctx;
@@ -656,8 +642,9 @@ EXPORT_SYMBOL_GPL(pstore_unregister);static void decompress_record (struct pstore_record *record) { +int ret;int unzipped_len; -char *decompressed; +char *unzipped, *workspace;if (!record->compressed)return ;
@@ -668,35 +655,42 @@ static void decompress_record(struct pstore_record *record)return ; } -/* No compression method has created the common buffer. */ +/* Missing compression buffer means compression was not initialized. */ if (!big_oops_buf) { -pr_warn ("no decompression buffer allocated\n" ); +pr_warn ("no decompression method initialized!\n" );return ; } - unzipped_len =pstore_decompress (record->buf, big_oops_buf, - record->size, big_oops_buf_sz); -if (unzipped_len <=0 ) { -pr_err ("decompression failed: %d\n" , unzipped_len); +/* Allocate enough space to hold max decompression and ECC. */ + unzipped_len = big_oops_buf_sz; + workspace =kmalloc (unzipped_len + record->ecc_notice_size, +GFP_KERNEL ); +if (!workspace)return ; - } -/* Build new buffer for decompressed contents. */ - decompressed =kmalloc (unzipped_len + record->ecc_notice_size, -GFP_KERNEL ); -if (!decompressed) { -pr_err ("decompression ran out of memory\n" ); +/* After decompression "unzipped_len" is almost certainly smaller. */ + ret =crypto_comp_decompress (tfm, record->buf, record->size, + workspace, &unzipped_len); +if (ret) { +pr_err ("crypto_comp_decompress failed, ret = %d!\n" , ret); +kfree (workspace);return ; } -memcpy (decompressed, big_oops_buf, unzipped_len);/* Append ECC notice to decompressed buffer. */ -memcpy (decompressed + unzipped_len, record->buf + record->size, +memcpy (workspace + unzipped_len, record->buf + record->size, record->ecc_notice_size); -/* Swap out compresed contents with decompressed contents. */ +/* Copy decompressed contents into an minimum-sized allocation. */ + unzipped =kmemdup (workspace, unzipped_len + record->ecc_notice_size, +GFP_KERNEL ); +kfree (workspace); +if (!unzipped) +return ; + +/* Swap out compressed contents with decompressed contents. */ kfree (record->buf); - record->buf = decompressed; + record->buf = unzipped; record->size = unzipped_len; record->compressed = false; }
Signature · 1 line
2.17.1
Re: [PATCH 2/8] pstore: Do not use crash buffer for decompression
Quoted 4 lines · Kees Cook
The pre-allocated compression buffer used for crash dumping was also being used for decompression. This isn't technically safe, since it's possible the kernel may attempt a crashdump while pstore is populating the pstore filesystem (and performing decompression). Instead, just allocate
Yeah, that would be bad if it happened ;)
Quoted 34 lines
a separate buffer for decompression. Correctness is preferred over performance here.
Signed-off-by Kees Cook <[email protected]>--- fs/pstore/platform.c | 56 ++++++++++++++++++++------------------------ 1 file changed, 25 insertions(+), 31 deletions(-)fs/pstore/platform.c +0
-14@@ -258,20 +258,6 @@ static int pstore_compress(const void *in, void *out,return outlen; } -static int pstore_decompress (void *in,void *out, -unsigned int inlen,unsigned int outlen) -{ -int ret; - - ret =crypto_comp_decompress (tfm, in, inlen, out, &outlen); -if (ret) { -pr_err ("crypto_comp_decompress failed, ret = %d!\n" , ret); -return ret; - } - -return outlen; -} -static void allocate_buf_for_compression (void ) {struct crypto_comp *ctx;
fs/pstore/platform.c EXPORT_SYMBOL_GPL(pstore_unregister); · line 646
Quoted 5 lines
fs/pstore/platform.c +1
-0@@ -656,8 +642,9 @@ EXPORT_SYMBOL_GPL(pstore_unregister);static void decompress_record (struct pstore_record *record) { +int ret;int unzipped_len;
nit: We could get rid of the unzipped_len variable now I think.
Quoted 14 lines
fs/pstore/platform.c +3
-3-char *decompressed; +char *unzipped, *workspace;if (!record->compressed)return ;@@ -668,35 +655,42 @@ static void decompress_record(struct pstore_record *record)return ; } -/* No compression method has created the common buffer. */ +/* Missing compression buffer means compression was not initialized. */ if (!big_oops_buf) { -pr_warn ("no decompression buffer allocated\n" ); +pr_warn ("no decompression method initialized!\n" );
fs/pstore/platform.c static void decompress_record(struct pstore_record *record) · line 666
Quoted 10 lines
fs/pstore/platform.c +3
-4return ; } - unzipped_len =pstore_decompress (record->buf, big_oops_buf, - record->size, big_oops_buf_sz); -if (unzipped_len <=0 ) { -pr_err ("decompression failed: %d\n" , unzipped_len); +/* Allocate enough space to hold max decompression and ECC. */ + unzipped_len = big_oops_buf_sz; + workspace = kmalloc(unzipped_len + record->ecc_notice_size,
Should tihs be unzipped_len + record->ecc_notice_size + 1. The extra byte being for the NULL character of the ecc notice?
This occurred to me when I saw the + 1 in ram.c. It could be better to just abstract the size as a macro.
Quoted 27 lines
fs/pstore/platform.c +10
-9+ GFP_KERNEL); +if (!workspace)return ; - } -/* Build new buffer for decompressed contents. */ - decompressed =kmalloc (unzipped_len + record->ecc_notice_size, -GFP_KERNEL ); -if (!decompressed) { -pr_err ("decompression ran out of memory\n" ); +/* After decompression "unzipped_len" is almost certainly smaller. */ + ret =crypto_comp_decompress (tfm, record->buf, record->size, + workspace, &unzipped_len); +if (ret) { +pr_err ("crypto_comp_decompress failed, ret = %d!\n" , ret); +kfree (workspace);return ; } -memcpy (decompressed, big_oops_buf, unzipped_len);/* Append ECC notice to decompressed buffer. */ -memcpy (decompressed + unzipped_len, record->buf + record->size, +memcpy (workspace + unzipped_len, record->buf + record->size, record->ecc_notice_size); -/* Swap out compresed contents with decompressed contents. */ +/* Copy decompressed contents into an minimum-sized allocation. */
fs/pstore/platform.c static void decompress_record(struct pstore_record *record) · line 693
Quoted 10 lines
fs/pstore/platform.c +8
-1+ unzipped =kmemdup (workspace, unzipped_len + record->ecc_notice_size, +GFP_KERNEL ); +kfree (workspace); +if (!unzipped) +return ; + +/* Swap out compressed contents with decompressed contents. */ kfree (record->buf); - record->buf = decompressed; + record->buf = unzipped;
Rest of it LGTM, thanks!
- Joel
Re: [PATCH 2/8] pstore: Do not use crash buffer for decompression
Quoted 7 lines · Joel Fernandes
Quoted 4 lines · Kees Cook
static void
decompress_record(struct pstore_record *record){+ int ret; int unzipped_len;nit: We could get rid of the
unzipped_lenvariable now I think.
I didn't follow this -- it gets used quite a bit. I don't see a clean way to remove it?
Quoted 7 lines
Quoted 1 line
+ workspace = kmalloc(unzipped_len +
record->ecc_notice_size,Should tihs be
unzipped_len+record->ecc_notice_size+ 1. The extra byte being for the NULL character of the ecc notice?This occurred to me when I saw the + 1 in
ram.c. It could be better to just abstract the size as a macro.
Ooh, yes, good catch. I'll get this fixed.
Thanks for the review!
Signature · 1 line
Kees Cook
Re: [PATCH 2/8] pstore: Do not use crash buffer for decompression
Quoted 11 lines · Kees Cook
Quoted 7 lines · Joel Fernandes
Quoted 4 lines · Kees Cook
static void
decompress_record(struct pstore_record *record){+ int ret; int unzipped_len;nit: We could get rid of the
unzipped_lenvariable now I think.I didn't follow this -- it gets used quite a bit. I don't see a clean way to remove it?
You are right. Sorry I missed that crpyto_comp_decompress actually uses it.
thanks,
- Joel
Re: [PATCH 2/8] pstore: Do not use crash buffer for decompression
In reply to Kees CookQuoted 11 lines · Kees Cook
Quoted 8 lines · Joel Fernandes
Quoted 1 line · Kees Cook
+ workspace = kmalloc(unzipped_len +
record->ecc_notice_size,Should tihs be
unzipped_len+record->ecc_notice_size+ 1. The extra byte being for the NULL character of the ecc notice?This occurred to me when I saw the + 1 in
ram.c. It could be better to just abstract the size as a macro.Ooh, yes, good catch. I'll get this fixed.
I spent more time looking at this, and it seems that only the initial creation of this string needs the +1, since all other operations are byte-based not NUL-terminated string based. It's a big odd, and I might try to clean it up differently, but as it stands, this is okay. (See inode.c which doesn't include the trailing NUL byte.)
-Kees
Signature · 1 line
Kees Cook
Re: [PATCH 2/8] pstore: Do not use crash buffer for decompression
Quoted 11 lines · Kees Cook
Quoted 10 lines · Kees Cook
Quoted 8 lines · Joel Fernandes
Quoted 1 line · Kees Cook
+ workspace = kmalloc(unzipped_len +
record->ecc_notice_size,Should tihs be
unzipped_len+record->ecc_notice_size+ 1. The extra byte being for the NULL character of the ecc notice?This occurred to me when I saw the + 1 in
ram.c. It could be better to just abstract the size as a macro.
Quoted 7 lines
Quoted 1 line
Ooh, yes, good catch. I'll get this fixed.
I spent more time looking at this, and it seems that only the initial creation of this string needs the +1, since all other operations are byte-based not NUL-terminated string based. It's a big odd, and I might try to clean it up differently, but as it stands, this is okay. (See
inode.cwhich doesn't include the trailing NUL byte.)
Ok. Yes it does seem a bit inconsistent but I agree its not an issue for this particular patch. Sorry to waste your time, thanks!
- Joel
PATCH 3/8 pstore/ram: Report backend assignments with finer granularity
In reply to Kees CookIn order to more easily perform automated regression testing, this adds pr_debug() calls to report each prz allocation which can then be verified against persistent storage. Specifically, seeing the dividing line between header, data, any ECC bytes. (And the general assignment output is updated to remove the bogus ECC blocksize which isn't actually recorded outside the prz instance.)
--- fs/pstore/ram.c | 4 ++-- fs/pstore/ram_core.c | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-)
fs/pstore/ram.c +2 -2
@@ -856,9 +856,9 @@ static int ramoops_probe(struct platform_device *pdev) ramoops_pmsg_size = pdata->pmsg_size; ramoops_ftrace_size = pdata->ftrace_size; -pr_info ("attached 0x%lx@0x%llx, ecc: %d/%d\n" , +pr_info ("using 0x%lx@0x%llx, ecc: %d\n" , cxt->size, (unsigned long long )cxt->phys_addr, - cxt->ecc_info.ecc_size, cxt->ecc_info.block_size); + cxt->ecc_info.ecc_size);return 0 ;
fs/pstore/ram_core.c +6 -0
@@ -576,6 +576,12 @@ struct persistent_ram_zone *persistent_ram_new(phys_addr_t start, size_t size,if (ret) goto err; +pr_debug ("attached %s 0x%lx@0x%llx: %lu header, %lu data, %lu ecc (%d/%d)\n" , + prz->label, prz->size, (unsigned long long )prz->paddr, +sizeof (*prz->buffer), prz->buffer_size, + prz->size -sizeof (*prz->buffer) - prz->buffer_size, + prz->ecc_info.ecc_size, prz->ecc_info.block_size); +return prz; err:persistent_ram_free (prz);
Signature · 1 line
2.17.1
PATCH 4/8 pstore/ram: Add kern-doc for struct persistent_ram_zone
In reply to Kees CookThe struct persistent_ram_zone wasn't well documented. This adds kern-doc for it.
--- fs/pstore/ram_core.c | 10 +++++++++ include/linux/pstore_ram.h | 46 +++++++++++++++++++++++++++++++++++--- 2 files changed, 53 insertions(+), 3 deletions(-)
fs/pstore/ram_core.c +10 -0
@@ -29,6 +29,16 @@#include <linux/vmalloc.h> #include <asm/page.h> +/** +* struct persistent_ram_buffer - persistent circular RAM buffer +* +* @sig: +* signature to indicate header (PERSISTENT_RAM_SIG xor PRZ-type value) +* @start: +* offset into @data where the beginning of the stored bytes begin +* @size: +* number of valid bytes stored in @data +*/ struct persistent_ram_buffer {uint32_t sig;atomic_t start;
include/linux/pstore_ram.h +43 -3
66 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 5/8 pstore: Improve and update some comments and status output
In reply to Kees CookThis improves and updates some comments:
- dump handler comment out of sync from calling convention
- fix kern-doc typo
and improves status output:
- reminder that only kernel crash dumps are compressed
- do not be silent about ECC infrastructure failures
--- fs/pstore/platform.c | 7 +++---- fs/pstore/ram_core.c | 4 +++- include/linux/pstore.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-)
fs/pstore/platform.c +3 -4
@@ -304,7 +304,7 @@ static void allocate_buf_for_compression(void) big_oops_buf_sz = size; big_oops_buf = buf; -pr_info ("Using compression: %s\n" , zbackend->name); +pr_info ("Using crash dump compression: %s\n" , zbackend->name); }static void free_buf_for_compression (void )
@@ -354,9 +354,8 @@ void pstore_record_init(struct pstore_record *record, }/* -* callback from kmsg_dump. (s2,l2) has the most recently -* written bytes, older bytes are in (s1,l1). Save as much -* as we can from the end of the buffer. +* callback from kmsg_dump. Save as much as we can (up to kmsg_bytes) from the +* end of the buffer. */ static void pstore_dump (struct kmsg_dumper *dumper,enum kmsg_dump_reason reason)
fs/pstore/ram_core.c +3 -1
@@ -503,8 +503,10 @@ static int persistent_ram_post_init(struct persistent_ram_zone *prz, u32 sig,bool zap = !!(prz->flags &PRZ_FLAG_ZAP_OLD ); ret =persistent_ram_init_ecc (prz, ecc_info); -if (ret) +if (ret) { +pr_warn ("ECC failed %s\n" , prz->label);return ret; + } sig ^=PERSISTENT_RAM_SIG ;
include/linux/pstore.h +1 -1
@@ -85,7 +85,7 @@ struct pstore_record {/** * struct pstore_info - backend pstore driver structure * -* @owner: module which is repsonsible for this backend driver +* @owner: module which is responsible for this backend driver * @name: name of the backend driver * * @buf_lock: spinlock to serialize access to @buf
Signature · 1 line
2.17.1
PATCH 6/8 pstore: Replace open-coded << with BIT()
In reply to Kees CookMinor clean-up to use BIT() (as already done in pstore_ram.h).
--- include/linux/pstore.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/pstore.h +4 -4
@@ -189,10 +189,10 @@ struct pstore_info { };/* Supported frontends */ -#define PSTORE_FLAGS_DMESG (1 << 0) -#define PSTORE_FLAGS_CONSOLE (1 << 1) -#define PSTORE_FLAGS_FTRACE (1 << 2) -#define PSTORE_FLAGS_PMSG (1 << 3) +#define PSTORE_FLAGS_DMESG BIT(0) +#define PSTORE_FLAGS_CONSOLE BIT(1) +#define PSTORE_FLAGS_FTRACE BIT(2) +#define PSTORE_FLAGS_PMSG BIT(3)extern int pstore_register (struct pstore_info *);extern void pstore_unregister (struct pstore_info *);
Signature · 1 line
2.17.1
PATCH 7/8 pstore: Remove needless lock during console writes
In reply to Kees CookSince commit 70ad35db3321 ("pstore: Convert console write to use ->write_buf"), the console writer does not use the preallocated crash dump buffer any more, so there is no reason to perform locking around it.
--- fs/pstore/platform.c | 29 ++++++----------------------- 1 file changed, 6 insertions(+), 23 deletions(-)
fs/pstore/platform.c +6 -23
@@ -461,31 +461,14 @@ static void pstore_unregister_kmsg(void)#ifdef CONFIG_PSTORE_CONSOLE static void pstore_console_write (struct console *con,const char *s,unsigned c) { -const char *e = s + c; +struct pstore_record record; -while (s < e) { -struct pstore_record record; -unsigned long flags; - -pstore_record_init (&record, psinfo); - record.type =PSTORE_TYPE_CONSOLE ; - -if (c > psinfo->bufsize) - c = psinfo->bufsize; +pstore_record_init (&record, psinfo); + record.type =PSTORE_TYPE_CONSOLE ; -if (oops_in_progress) { -if (!spin_trylock_irqsave (&psinfo->buf_lock, flags)) -break ; - }else { -spin_lock_irqsave (&psinfo->buf_lock, flags); - } - record.buf = (char *)s; - record.size = c; - psinfo->write (&record); -spin_unlock_irqrestore (&psinfo->buf_lock, flags); - s += c; - c = e - s; - } + record.buf = (char *)s; + record.size = c; + psinfo->write (&record); } static struct console pstore_console = {
Signature · 1 line
2.17.1
Re: [PATCH 7/8] pstore: Remove needless lock during console writes
Quoted 3 lines · Kees Cook
Since commit
70ad35db3321("pstore: Convert console write to use ->write_buf"), the console writer does not use the preallocated crash dump buffer any more, so there is no reason to perform locking around it.
Out of curiosity, what was the reason for having this preallocated crash buffer in the first place? I thought the 'console' type only did regular kernel console logging, not crash dumps.
I looked at all the patches and had some minor nits, with the nits addressed (if you agree with them), feel free to add my Reviewed-by on future respins:
Also I wonder if Namhyung is still poking around that virtio pstore driver he mentioned in the commit mentioned above. :)
thanks,
- Joel
Quoted 50 lines
Signed-off-by Kees Cook <[email protected]>--- fs/pstore/platform.c | 29 ++++++----------------------- 1 file changed, 6 insertions(+), 23 deletions(-)fs/pstore/platform.c +6
-23@@ -461,31 +461,14 @@ static void pstore_unregister_kmsg(void)#ifdef CONFIG_PSTORE_CONSOLE static void pstore_console_write (struct console *con,const char *s,unsigned c) { -const char *e = s + c; +struct pstore_record record; -while (s < e) { -struct pstore_record record; -unsigned long flags; - -pstore_record_init (&record, psinfo); - record.type =PSTORE_TYPE_CONSOLE ; - -if (c > psinfo->bufsize) - c = psinfo->bufsize; +pstore_record_init (&record, psinfo); + record.type =PSTORE_TYPE_CONSOLE ; -if (oops_in_progress) { -if (!spin_trylock_irqsave (&psinfo->buf_lock, flags)) -break ; - }else { -spin_lock_irqsave (&psinfo->buf_lock, flags); - } - record.buf = (char *)s; - record.size = c; - psinfo->write (&record); -spin_unlock_irqrestore (&psinfo->buf_lock, flags); - s += c; - c = e - s; - } + record.buf = (char *)s; + record.size = c; + psinfo->write (&record); } static struct console pstore_console = {Signature · 1 line
2.17.1
Re: [PATCH 7/8] pstore: Remove needless lock during console writes
Quoted 8 lines · Joel Fernandes
Quoted 3 lines · Kees Cook
Since commit
70ad35db3321("pstore: Convert console write to use ->write_buf"), the console writer does not use the preallocated crash dump buffer any more, so there is no reason to perform locking around it.Out of curiosity, what was the reason for having this preallocated crash buffer in the first place? I thought the 'console' type only did regular kernel console logging, not crash dumps.
The primary reason is that the dumper needs to write to somewhere and we don't know the state of the system (memory allocation may not work for example).
The other frontends tend to run at "sane" locations in the kernel. The dumper, however, is quite fragile.
Quoted 4 lines
I looked at all the patches and had some minor nits, with the nits addressed (if you agree with them), feel free to add my Reviewed-by on future respins:
Reviewed-by Joel Fernandes (Google) <[email protected]>
Thanks!
Quoted 2 lines
Also I wonder if Namhyung is still poking around that virtio pstore driver he mentioned in the commit mentioned above. :)
Did that never land? I thought it mostly had to happen at the qemu end?
With nvdimm emulation, we can just use ramoops. :)
-Kees
Signature · 1 line
Kees Cook
Re: [PATCH 7/8] pstore: Remove needless lock during console writes
Quoted 10 lines · Kees Cook
Quoted 8 lines · Joel Fernandes
Quoted 3 lines · Kees Cook
Since commit
70ad35db3321("pstore: Convert console write to use ->write_buf"), the console writer does not use the preallocated crash dump buffer any more, so there is no reason to perform locking around it.Out of curiosity, what was the reason for having this preallocated crash buffer in the first place? I thought the 'console' type only did regular kernel console logging, not crash dumps.
Quoted 6 lines
The primary reason is that the dumper needs to write to somewhere and we don't know the state of the system (memory allocation may not work for example).
The other frontends tend to run at "sane" locations in the kernel. The dumper, however, is quite fragile.
Makes sense. thanks.
Quoted 7 lines
Quoted 2 lines
Also I wonder if Namhyung is still poking around that virtio pstore driver he mentioned in the commit mentioned above. :)
Did that never land? I thought it mostly had to happen at the qemu end?
With nvdimm emulation, we can just use ramoops. :)
Yes it seems like it never landed: https://lwn.net/Articles/694742/
One of the nice thing about his virtio set though, vs the nvdimm way is that he actually gets a directory on the host instead of a backing memory file. Then he can just list this directory and see all the pstore files as he shows in his example.
I guess it should not be too hard to write a tool to post-process the nvdimm images and convert it to files anyway :)
thanks,
- Joel
PATCH 8/8 pstore/ram: Correctly calculate usable PRZ bytes
In reply to Kees CookThe actual number of bytes stored in a PRZ is smaller than the bytes requested by platform data, since there is a header on each PRZ. Additionally, if ECC is enabled, there are trailing bytes used as well. Normally this mismatch doesn't matter since PRZs are circular buffers and the leading "overflow" bytes are just thrown away. However, in the case of a compressed record, this rather badly corrupts the results.
This corruption was visible with "ramoops.mem_size=204800 ramoops.ecc=1". Any stored crashes would not be uncompressable (producing a pstorefs "dmesg-*.enc.z" file), and triggering errors at boot:
[ 2.790759] pstore: crypto_comp_decompress failed, ret = -22!
--- fs/pstore/ram.c | 15 ++++++--------- include/linux/pstore.h | 5 ++++- 2 files changed, 10 insertions(+), 10 deletions(-)
fs/pstore/ram.c +6 -9
@@ -814,17 +814,14 @@ static int ramoops_probe(struct platform_device *pdev) cxt->pstore.data = cxt;/* -* Console can handle any buffer size, so prefer LOG_LINE_MAX. If we -* have to handle dumps, we must have at least record_size buffer. And -* for ftrace, bufsize is irrelevant (if bufsize is 0, buf will be -* ZERO_SIZE_PTR). +* Since bufsize is only used for dmesg crash dumps, it +* must match the size of the dprz record (after PRZ header +* and ECC bytes have been accounted for). */ -if (cxt->console_size) - cxt->pstore.bufsize =1024 ;/* LOG_LINE_MAX */ - cxt->pstore.bufsize =max (cxt->record_size, cxt->pstore.bufsize); - cxt->pstore.buf =kmalloc (cxt->pstore.bufsize,GFP_KERNEL ); + cxt->pstore.bufsize = cxt->dprzs[0 ]->buffer_size; + cxt->pstore.buf =kzalloc (cxt->pstore.bufsize,GFP_KERNEL );if (!cxt->pstore.buf) { -pr_err ("cannot allocate pstore buffer\n" ); +pr_err ("cannot allocate pstore crash dump buffer\n" ); err = -ENOMEM ; goto fail_clear; }
include/linux/pstore.h +4 -1
@@ -90,7 +90,10 @@ struct pstore_record {* * @buf_lock: spinlock to serialize access to @buf * @buf: preallocated crash dump buffer -* @bufsize: size of @buf available for crash dump writes +* @bufsize: size of @buf available for crash dump bytes (must match +* smallest number of bytes available for writing to a +* backend entry, since compressed bytes don't take kindly +* to being truncated) * * @read_mutex: serializes @open, @read, @close, and @erase callbacks * @flags: bitfield of frontends the backend can accept writes for
Signature · 1 line
2.17.1
Re: [PATCH 8/8] pstore/ram: Correctly calculate usable PRZ bytes
Quoted 6 lines · Kees Cook
The actual number of bytes stored in a PRZ is smaller than the bytes requested by platform data, since there is a header on each PRZ. Additionally, if ECC is enabled, there are trailing bytes used as well. Normally this mismatch doesn't matter since PRZs are circular buffers and the leading "overflow" bytes are just thrown away. However, in the case of a compressed record, this rather badly corrupts the results.
Actually this would also mean some data loss for non-compressed records were also there before, but is now fixed?
Quoted 9 lines
This corruption was visible with "ramoops.mem_size=204800 ramoops.ecc=1". Any stored crashes would not be uncompressable (producing a pstorefs "dmesg-*.enc.z" file), and triggering errors at boot:
[ 2.790759] pstore: crypto_comp_decompress failed, ret = -22!Reported-by Joel Fernandes <[email protected]>Fixes b0aad7a99c1d ("pstore: Add compression support to pstore")Signed-off-by Kees Cook <[email protected]>
Thanks!
Also should this be fixed for other backends or are those good? AFAIR, I saw this for EFI too.
- Joel
Quoted 52 lines
--- fs/pstore/ram.c | 15 ++++++--------- include/linux/pstore.h | 5 ++++- 2 files changed, 10 insertions(+), 10 deletions(-)fs/pstore/ram.c +6
-9@@ -814,17 +814,14 @@ static int ramoops_probe(struct platform_device *pdev) cxt->pstore.data = cxt;/* -* Console can handle any buffer size, so prefer LOG_LINE_MAX. If we -* have to handle dumps, we must have at least record_size buffer. And -* for ftrace, bufsize is irrelevant (if bufsize is 0, buf will be -* ZERO_SIZE_PTR). +* Since bufsize is only used for dmesg crash dumps, it +* must match the size of the dprz record (after PRZ header +* and ECC bytes have been accounted for). */ -if (cxt->console_size) - cxt->pstore.bufsize =1024 ;/* LOG_LINE_MAX */ - cxt->pstore.bufsize =max (cxt->record_size, cxt->pstore.bufsize); - cxt->pstore.buf =kmalloc (cxt->pstore.bufsize,GFP_KERNEL ); + cxt->pstore.bufsize = cxt->dprzs[0 ]->buffer_size; + cxt->pstore.buf =kzalloc (cxt->pstore.bufsize,GFP_KERNEL );if (!cxt->pstore.buf) { -pr_err ("cannot allocate pstore buffer\n" ); +pr_err ("cannot allocate pstore crash dump buffer\n" ); err = -ENOMEM ; goto fail_clear; }include/linux/pstore.h +4
-1@@ -90,7 +90,10 @@ struct pstore_record {* * @buf_lock: spinlock to serialize access to @buf * @buf: preallocated crash dump buffer -* @bufsize: size of @buf available for crash dump writes +* @bufsize: size of @buf available for crash dump bytes (must match +* smallest number of bytes available for writing to a +* backend entry, since compressed bytes don't take kindly +* to being truncated) * * @read_mutex: serializes @open, @read, @close, and @erase callbacks * @flags: bitfield of frontends the backend can accept writes for Signature · 1 line
2.17.1
Re: [PATCH 8/8] pstore/ram: Correctly calculate usable PRZ bytes
Quoted 10 lines · Joel Fernandes
Quoted 6 lines · Kees Cook
The actual number of bytes stored in a PRZ is smaller than the bytes requested by platform data, since there is a header on each PRZ. Additionally, if ECC is enabled, there are trailing bytes used as well. Normally this mismatch doesn't matter since PRZs are circular buffers and the leading "overflow" bytes are just thrown away. However, in the case of a compressed record, this rather badly corrupts the results.
Actually this would also mean some data loss for non-compressed records were also there before, but is now fixed?
No, it's what I mentioned in the commit log: only the "tail" of any data was getting stored, which is consistent with the configuration given. The main problem is that ECC bytes weren't part of the calculation the ram backend provided to pstore for pstore to pick the correct amount of bytes to compress.
Quoted 4 lines
Quoted 4 lines
This corruption was visible with "ramoops.mem_size=204800 ramoops.ecc=1". Any stored crashes would not be uncompressable (producing a pstorefs "dmesg-*.enc.z" file), and triggering errors at boot:
Quoted 8 lines
Quoted 5 lines
[ 2.790759] pstore: crypto_comp_decompress failed, ret = -22!Reported-by Joel Fernandes <[email protected]>Fixes b0aad7a99c1d ("pstore: Add compression support to pstore")Signed-off-by Kees Cook <[email protected]>Thanks!
Reviewed-by Joel Fernandes (Google) <[email protected]>
Thanks!
Quoted 2 lines
Also should this be fixed for other backends or are those good? AFAIR, I saw this for EFI too.
It seemed like the other backends were doing it correctly (e.g. erst removes the header from calculation, etc). I did see that EFI allocates more memory than needed?
efi_pstore_info.buf = kmalloc(4096, GFP_KERNEL);
if (!efi_pstore_info.buf)
return -ENOMEM;
efi_pstore_info.bufsize = 1024;efi_pstore_write() does:
ret = efivar_entry_set_safe(efi_name, vendor, PSTORE_EFI_ATTRIBUTES,
!pstore_cannot_block_path(record->reason),
record->size, record->psi->buf);and efivar_entry_set_safe() says:
* Returns 0 on success, -ENOSPC if the firmware does not have enough
* space for set_variable() to succeed, or a converted EFI status code
* if set_variable() fails.
So I don't see how this could get truncated. (I'm not saying it didn't... just that I can't see it in an obvious place.)
-Kees
Signature · 1 line
Kees Cook
Re: [PATCH 8/8] pstore/ram: Correctly calculate usable PRZ bytes
Hi Kees,
On Fri, Nov 02, 2018 at 01:00:08PM -0700, Kees Cook wrote:
[..]
Quoted 34 lines
Quoted 4 lines
Quoted 4 lines
This corruption was visible with "ramoops.mem_size=204800 ramoops.ecc=1". Any stored crashes would not be uncompressable (producing a pstorefs "dmesg-*.enc.z" file), and triggering errors at boot:
Quoted 8 lines
Quoted 5 lines
[ 2.790759] pstore: crypto_comp_decompress failed, ret = -22!Reported-by Joel Fernandes <[email protected]>Fixes b0aad7a99c1d ("pstore: Add compression support to pstore")Signed-off-by Kees Cook <[email protected]>Thanks!
Reviewed-by Joel Fernandes (Google) <[email protected]>Thanks!
Quoted 2 lines
Also should this be fixed for other backends or are those good? AFAIR, I saw this for EFI too.
It seemed like the other backends were doing it correctly (e.g. erst removes the header from calculation, etc). I did see that EFI allocates more memory than needed?
efi_pstore_info.buf = kmalloc(4096, GFP_KERNEL); if (!efi_pstore_info.buf) return -ENOMEM; efi_pstore_info.bufsize = 1024;
efi_pstore_write()does:ret = efivar_entry_set_safe(efi_name, vendor, PSTORE_EFI_ATTRIBUTES, !pstore_cannot_block_path(record->reason), record->size, record->psi->buf);
Quoted 8 lines
and
efivar_entry_set_safe()says:* Returns 0 on success,
-ENOSPCif the firmware does not have enough
* space forset_variable()to succeed, or a converted EFI status code
* ifset_variable()fails.So I don't see how this could get truncated. (I'm not saying it didn't... just that I can't see it in an obvious place.)
So I *think* the issue is that the pstore had old compressed dmesg dumps in EFI on my laptop, after the crypto layer in the kernel probably changed enough to make the data non-decompressable, if that makes any sense. So older code did compression in certain way, and newer code is doing the decompress, or something like that.
I did some sysrq crashes on my laptop and the deflate decompress is working fine with pstore+EFI. Its interesting I see some .enc.z files which fail to decompress (which are older ones), and others which are decompressed fine (the newer ones) ;-)
Dumping the magic bytes of the non decompressable .enc.z files, I get this which shows a valid zlib compressed header:
Something like:
48 89 85 54 4d 6f 1a 31
The 0b1000 in the first byte means it is "deflate". The file tool indeed successfully shows "zlib compressed data" and I did the math for the header and it is indeed valid. So I don't think the data is insane. The buffer has enough room because even the very small dumps are not decompressable.
At this point we can park this issue I guess, but a scenario that is still broken is:
Say someone crashes the system on compress algo X and then recompiles with compress algo Y, then the decompress would fail no?
One way to fix that is to store the comrpession method in buffer as well, then initialize all algorithms at boot and choose the right one in the buffer ideally. Otherwise atleast we should print a message saying "buffer is encoded with algo X but compression selected is Y" or something. But I agree its a very low priority "doctor it hurts if I do this" kind of issue :)
Anyway, let me know what you think :)
thanks,
- Joel
Re: [PATCH 8/8] pstore/ram: Correctly calculate usable PRZ bytes
Quoted 10 lines · Joel Fernandes
Dumping the magic bytes of the non decompressable .enc.z files, I get this which shows a valid zlib compressed header:
Something like:
48 89 85 54 4d 6f 1a 31The 0b1000 in the first byte means it is "deflate". The file tool indeed successfully shows "zlib compressed data" and I did the math for the header and it is indeed valid. So I don't think the data is insane. The buffer has enough room because even the very small dumps are not decompressable.
Interesting. So the kernel wouldn't decompress it even though it's the right algo and untruncated? That seems worth fixing.
Quoted 10 lines
At this point we can park this issue I guess, but a scenario that is still broken is:
Say someone crashes the system on compress algo X and then recompiles with compress algo Y, then the decompress would fail no?One way to fix that is to store the comrpession method in buffer as well, then initialize all algorithms at boot and choose the right one in the buffer ideally. Otherwise atleast we should print a message saying "buffer is encoded with algo X but compression selected is Y" or something. But I agree its a very low priority "doctor it hurts if I do this" kind of issue :)
Right, this is fine: if algos change across a kernel version, I'm fine with it failing. pstore isn't expected to work sanely outside of a pretty narrow set of use-cases.
-Kees
Signature · 1 line
Kees Cook
Re: [PATCH 8/8] pstore/ram: Correctly calculate usable PRZ bytes
Quoted 7 lines · Kees Cook
Quoted 6 lines · Joel Fernandes
Dumping the magic bytes of the non decompressable .enc.z files, I get this which shows a valid zlib compressed header:
Something like:
48 89 85 54 4d 6f 1a 31
Quoted 7 lines
Quoted 4 lines
The 0b1000 in the first byte means it is "deflate". The file tool indeed successfully shows "zlib compressed data" and I did the math for the header and it is indeed valid. So I don't think the data is insane. The buffer has enough room because even the very small dumps are not decompressable.
Interesting. So the kernel wouldn't decompress it even though it's the right algo and untruncated? That seems worth fixing.
I just can't reproduce the issue though :) I am wondering if it was something to do with compression being done by an older version of the algorithm or a buggy algorithm, and the decompressor is newer. Or something.
Not sure if its worth spending more time on, I'll park it for now unless you want me to dig deeper into it.
- Joel