LZO Compression Support
Subsystems (3)
v2 cover LZO Compression Support
The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.
The RFC discussion was made here:
http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html
How to build:
1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:
1) Original author's website:
http://www.oberhumer.com/opensource/lzo/
2) yum framework on fedora. Older releases don't have the packages.
2. Apply the patch set to makedumpfile v1.4.2.
3. Do make as follows:
$ make USELZO=on
Note: In default, no LZO compression support is included.
How to use:
Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.
Example)
$ makedumpfile -l vmcore dumpfile
Performance evaluation:
- Kumagai-san's evaluation simulating working servers:
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html
- My evaluation focusing on the worst cases:
http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
LZO Support for crash:
I'll post LZO support patch for crash after makedumpfile merges these patches.
Changelog:
v1 => v2:
- Add build condition for LZO support. Enable LZO support if specifying USELZO=on to make command.
- Avoid LONG_MAX/ULONG_MAX redefinitions.
---
HATAYAMA Daisuke (4):
Add build condition for LZO support
Add help and manual messages about LZO compression support Avoid LONG_MAX/ULONG_MAX redefinitions
Add LZO Support
Makefile | 5 ++++ common.h | 4 +++ diskdump_mod.h | 3 ++- makedumpfile.8 | 6 +++-- makedumpfile.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++------ makedumpfile.h | 4 +++ print_info.c | 16 +++++++------ 7 files changed, 86 insertions(+), 19 deletions(-)
Signature · 6 lines
HATAYAMA Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 1/4 Add LZO Support
Add -l option as one of the command-line options users can specify. If -l option is specified, then makedumpfile generates dumpfile in kdump-compressed format with lzo compression by each page.
---
diskdump_mod.h | 3 ++- makedumpfile.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++--------- makedumpfile.h | 2 ++ 3 files changed, 52 insertions(+), 10 deletions(-)
diskdump_mod.h +2 -1
@@ -78,7 +78,8 @@ struct kdump_sub_header { };/* page flags */ -#define DUMP_DH_COMPRESSED 0x1/* page is compressed */ +#define DUMP_DH_COMPRESSED_ZLIB 0x1/* page is compressed with zlib */ +#define DUMP_DH_COMPRESSED_LZO 0x2/* paged is compressed with lzo */ /* descriptor of each page for vmcore */ typedef struct page_desc {
makedumpfile.c +48 -9
121 lines not on this page · raw message
makedumpfile.h +2 -0
@@ -31,6 +31,7 @@#include <libelf.h> #include <byteswap.h> #include <getopt.h> +#include <lzo/lzo1x.h> #include "common.h" #include "dwarf_info.h" #include "diskdump_mod.h"
@@ -767,6 +768,7 @@ struct DumpInfo {int num_dump_level;/* number of dump level */ int array_dump_level[NUM_ARRAY_DUMP_LEVEL ];int flag_compress;/* flag of compression */ +int flag_lzo_support;/* flag of LZO compression support */ int flag_elf_dumpfile;/* flag of creating ELF dumpfile */ int flag_generate_vmcoreinfo;/* flag of generating vmcoreinfo file */ int flag_read_vmcoreinfo;/* flag of reading vmcoreinfo file */
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 2/4 Avoid LONG_MAX/ULONG_MAX redefinitions
In reply to HATAYAMA Daisukecommon.h locally defines LONG_MAX and ULONG_MAX. But when importing lzo library's headers, ulimit.h that is imported implicitly is also imported so the two definitions collide, and the warning message below is displayed.
gcc -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DVERSION='"1.4.2"' -DRELEASE_DATE='"25 January 2012"' -D__x86_64__ -DUSELZO print_info.o dwarf_info.o elf_info.o erase_info.o sadump_info.o arch/arm.o arch/x86.o arch/x86_64.o arch/ia64.o arch/ppc64.o arch/s390x.o -o makedumpfile makedumpfile.c -llzo2 -ldw -lbz2 -lebl -ldl -lelf -lz In file included from makedumpfile.h:37:0,
from makedumpfile.c:16:
common.h:23:0: warning: "LONG_MAX" redefined/usr/lib/gcc/x86_64-redhat-linux/4.5.1/include/limits.h:132:0: note: this is the location of the previous definition
common.h:24:0: warning: "ULONG_MAX" redefined/usr/lib/gcc/x86_64-redhat-linux/4.5.1/include/limits.h:136:0: note: this is the location of the previous definition
To avoid this problem, check if each has already defiend before defining it.
---
common.h | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
common.h +4 -0
@@ -20,8 +20,12 @@#define FALSE (0)#define ERROR (-1) +#ifndef LONG_MAX #define LONG_MAX ((long)(~0UL>>1)) +#endif +#ifndef ULONG_MAX #define ULONG_MAX (~0UL) +#endif #define ULONGLONG_MAX (~0ULL)#define MAX (a,b) ((a) > (b) ? (a) : (b))
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 3/4 Add help and manual messages about LZO compression support
In reply to HATAYAMA Daisuke---
makedumpfile.8 | 6 +++--- print_info.c | 16 ++++++++-------- 2 files changed, 11 insertions(+), 11 deletions(-)
makedumpfile.8 +3 -3
@@ -121,8 +121,8 @@ configuration, you need to use --diskset option. .SH OPTIONS .TP -\fB\-c\fR -Compress dump data by each page. +\fB\-c,\-l\fR +Compress dump data by each page using zlib for -c option or lzo for -l option. .br A user cannot specify this option with \-E option, because the ELF format does not support compressed data.
@@ -196,7 +196,7 @@ by dump_level 11, makedumpfile retries it by dump_level 31. \fB\-E\fR Create \fIDUMPFILE\fR in the ELF format. .br -This option cannot be specified with \-c option, because the ELF format does not +This option cannot be specified with either of \-c option or \-l option, because the ELF format does not support compressed data. .br .B Example:
print_info.c +8 -8
@@ -35,15 +35,15 @@ print_usage(void)MSG ("\n" );MSG ("Usage:\n" );MSG (" Creating DUMPFILE:\n" ); -MSG (" # makedumpfile [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n" ); +MSG (" # makedumpfile [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n" );MSG ("\n" );MSG (" Creating DUMPFILE with filtered kernel data specified through filter config\n" );MSG (" file:\n" ); -MSG (" # makedumpfile [-c|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n" ); +MSG (" # makedumpfile [-c|-l|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n" );MSG (" DUMPFILE\n" );MSG ("\n" );MSG (" Outputting the dump data in the flattened format to the standard output:\n" ); -MSG (" # makedumpfile -F [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n" ); +MSG (" # makedumpfile -F [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n" );MSG ("\n" );MSG (" Rearranging the dump data in the flattened format to a readable DUMPFILE:\n" );MSG (" # makedumpfile -R DUMPFILE\n" );
@@ -70,14 +70,14 @@ print_usage(void)MSG ("\n" );MSG ("\n" );MSG (" Creating DUMPFILE from multiple VMCOREs generated on sadump diskset configuration:\n" ); -MSG (" # makedumpfile [-c] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n" ); +MSG (" # makedumpfile [-c|-l] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n" );MSG (" [--diskset=VMCORE3 ..] DUMPFILE\n" );MSG ("\n" );MSG ("\n" );MSG ("Available options:\n" ); -MSG (" [-c]:\n" ); -MSG (" Compress dump data by each page.\n" ); -MSG (" A user cannot specify this option with -E option, because the ELF format\n" ); +MSG (" [-c|-l]:\n" ); +MSG (" Compress dump data by each page using zlib for -c option and lzo for -l option.\n" ); +MSG (" A user cannot specify either of these options with -E option, because the ELF format\n" );MSG (" does not support compressed data.\n" );MSG (" THIS IS ONLY FOR THE CRASH UTILITY.\n" );MSG ("\n" );
@@ -103,7 +103,7 @@ print_usage(void)MSG ("\n" );MSG (" [-E]:\n" );MSG (" Create DUMPFILE in the ELF format.\n" ); -MSG (" This option cannot be specified with -c option, because the ELF\n" ); +MSG (" This option cannot be specified with either of -c option or -l option, because the ELF\n" );MSG (" format does not support compressed data.\n" );MSG ("\n" );MSG (" [-x VMLINUX]:\n" );
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 4/4 Add build condition for LZO support
In reply to HATAYAMA DaisukeTo enable lzo compression support, build makedumpfile as:
$ make USELZO
To disable, as:
$ make
In default, lzo compression support is disabled.
---
Makefile | 5 +++++ makedumpfile.c | 18 ++++++++++++++++-- makedumpfile.h | 2 ++ 3 files changed, 23 insertions(+), 2 deletions(-)
Makefile +5 -0
@@ -34,6 +34,11 @@ ifneq ($(LINKTYPE), dynamic) LIBS := -static $(LIBS) endif +ifeq ($(USELZO), on) +LIBS := -llzo2 $(LIBS) +CFLAGS += -DUSELZO +endif + all: makedumpfile $(OBJ_PART): $(SRC_PART)
makedumpfile.c +16 -2
@@ -266,6 +266,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size) goto error; }memcpy (bufptr, buf2 + page_offset, size); +#ifdef USELZO }else if (info->flag_lzo_support && (pd.flags &DUMP_DH_COMPRESSED_LZO )) { retlen = info->page_size;
@@ -277,6 +278,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size) goto error; }memcpy (bufptr, buf2 + page_offset, size); +#endif }else memcpy (bufptr, buf + page_offset, size);
@@ -2509,8 +2511,10 @@ initial(void)unsigned long size;int debug_info = FALSE; +#ifdef USELZO if (lzo_init () ==LZO_E_OK ) info->flag_lzo_support = TRUE; +#endif if (!is_xen_memory () && info->flag_exclude_xen_dom) {MSG ("'-X' option is disable," );
@@ -4679,11 +4683,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)off_t offset_data =0 ;struct disk_dump_header *dh = info->dump_header;unsigned char buf[info->page_size], *buf_out =NULL ; -unsigned long len_buf_out, len_buf_out_zlib, len_buf_out_lzo; +unsigned long len_buf_out;struct dump_bitmap bitmap2;struct timeval tv_start;const off_t failed = (off_t)-1 ; -lzo_bytep wrkmem =NULL ;int ret = FALSE;
@@ -4692,6 +4695,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)initialize_2nd_bitmap (&bitmap2); +#ifdef USELZO +unsigned long len_buf_out_zlib, len_buf_out_lzo; +lzo_bytep wrkmem; +if ((wrkmem =malloc (LZO1X_1_MEM_COMPRESS )) ==NULL ) {ERRMSG ("Can't allocate memory for the working memory. %s\n" ,strerror (errno));
@@ -4701,6 +4708,9 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) len_buf_out_zlib =compressBound (info->page_size); len_buf_out_lzo = info->page_size + info->page_size /16 +64 +3 ; len_buf_out =MAX (len_buf_out_zlib, len_buf_out_lzo); +#else + len_buf_out =compressBound (info->page_size); +#endif if ((buf_out =malloc (len_buf_out)) ==NULL ) { ERRMSG("Can't allocate memory for the compression buffer. %s\n",
@@ -4791,6 +4801,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) pd.flags =DUMP_DH_COMPRESSED_ZLIB ; pd.size = size_out;memcpy (buf, buf_out, pd.size); +#ifdef USELZO } else if (info->flag_lzo_support && (info->flag_compress & DUMP_DH_COMPRESSED_LZO) && ((size_out = info->page_size),
@@ -4800,6 +4811,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) pd.flags =DUMP_DH_COMPRESSED_LZO ; pd.size = size_out;memcpy (buf, buf_out, pd.size); +#endif }else { pd.flags =0 ; pd.size = info->page_size;
@@ -4840,8 +4852,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) out:if (buf_out !=NULL )free (buf_out); +#ifdef USELZO if (wrkmem !=NULL )free (wrkmem); +#endif return ret; }
makedumpfile.h +2 -0
@@ -31,7 +31,9 @@#include <libelf.h> #include <byteswap.h> #include <getopt.h> +#ifdef USELZO #include <lzo/lzo1x.h> +#endif #include "common.h" #include "dwarf_info.h" #include "diskdump_mod.h"
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Hatayama-san,
Quoted 49 lines · On Thu, 23 Feb 2012 10:33:54 +0900 HATAYAMA Daisuke…
The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.
The RFC discussion was made here:
http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.htmlHow to build:
1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:
1) Original author's website:
http://www.oberhumer.com/opensource/lzo/2) yum framework on fedora. Older releases don't have the packages.
2. Apply the patch set to makedumpfile v1.4.2.
3. Do make as follows:
$ make USELZO=onNote: In default, no LZO compression support is included.
How to use:
Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.
Example)
$ makedumpfile -l vmcore dumpfilePerformance evaluation:
- Kumagai-san's evaluation simulating working servers:
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html- My evaluation focusing on the worst cases:
http://lists.infradead.org/pipermail/kexec/2011-November/005783.htmlLZO Support for crash:
I'll post LZO support patch for crash after makedumpfile merges these patches.
Quoted 8 lines
Changelog:
v1 => v2:
- Add build condition for LZO support. Enable LZO support if specifying USELZO=on to make command.
- Avoid LONG_MAX/ULONG_MAX redefinitions.
Thank you for your work.
I will review your patches for v1.4.4.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Helllo Kumagai-san,
From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Thu, 23 Feb 2012 17:16:46 +0900
Quoted 2 lines
Thank you for your work.
I will review your patches for v1.4.4.
Could you tell me when you plan to release version 1.4.4? Not necessarily exact date, rough schedule is enough. I need to post the corresponding patch to the crash utility's side after LZO feature is merged into makedumpfile.
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello HATAYAMA-san,
Quoted 6 lines · On Thu, 10 May 2012 09:38:29 +0900 (JST) HATAYAMA Daisuke…
Helllo Kumagai-san,
From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Thu, 23 Feb 2012 17:16:46 +0900
Quoted 7 lines
Quoted 2 lines
Thank you for your work.
I will review your patches for v1.4.4.Could you tell me when you plan to release version 1.4.4? Not necessarily exact date, rough schedule is enough. I need to post the corresponding patch to the crash utility's side after LZO feature is merged into makedumpfile.
At first, I planned to merge features below into v1.4.4.
- vmalloc translation support for PPC32 (by Suzuki K. Poulose)
- LZO Compression Support (by HATAYAMA Daisuke)
- sadump: check if given cpu is online in per-cpu related helper functions (by HATAYAMA Daisuke)
However, the first one is being discussed and the schedule is not clear. So, I would like to postpone the feature for v1.4.5 and start testing the other two for v1.4.4.
If I can start testing now, I will release v1.4.4 in this month.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Thu, 10 May 2012 12:40:27 +0900
Quoted 8 lines
At first, I planned to merge features below into v1.4.4.
- vmalloc translation support for PPC32 (by Suzuki K. Poulose)
- LZO Compression Support (by HATAYAMA Daisuke)
- sadump: check if given cpu is online in per-cpu related helper functions (by HATAYAMA Daisuke)
Quoted 6 lines
However, the first one is being discussed and the schedule is not clear. So, I would like to postpone the feature for v1.4.5 and start testing the other two for v1.4.4.
If I can start testing now, I will release v1.4.4 in this month.
Thanks. So the release would be before the release of crash utility's next version. I'll post the patch for crash utility soon.
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Hatayama-san,
Quoted 4 lines · On Thu, 23 Feb 2012 10:33:54 +0900 HATAYAMA Daisuke…
The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.
Sorry for late reply.
[PATCH v2 4/4] allows to use -l option even without USELZO but lzo compression doesn't work without USELZO.
I think that warning messages should be displayed when -l option is used on makedumpfile built without USELZO.
So I made v3 patches attached to this mail.
([PATCH v2 1/4] and [PATCH v2 2/4] look good and I didn't change them.)
v2 => v3:
- Add warning messages for invalid -l option.
- Show LZO support status with help messages.
- Add description of USELZO in README and man page.
- (and I will do some cleanups.)
What do you think, Hatayama-san ?
Thanks
Atsushi Kumagai
Quoted 62 lines
The RFC discussion was made here:
http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.htmlHow to build:
1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:
1) Original author's website:
http://www.oberhumer.com/opensource/lzo/2) yum framework on fedora. Older releases don't have the packages.
2. Apply the patch set to makedumpfile v1.4.2.
3. Do make as follows:
$ make USELZO=onNote: In default, no LZO compression support is included.
How to use:
Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.
Example)
$ makedumpfile -l vmcore dumpfilePerformance evaluation:
- Kumagai-san's evaluation simulating working servers:
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html- My evaluation focusing on the worst cases:
http://lists.infradead.org/pipermail/kexec/2011-November/005783.htmlLZO Support for crash:
I'll post LZO support patch for crash after makedumpfile merges these patches.
Changelog:
v1 => v2:
- Add build condition for LZO support. Enable LZO support if specifying USELZO=on to make command.
- Avoid LONG_MAX/ULONG_MAX redefinitions.
---
HATAYAMA Daisuke (4):
Add build condition for LZO support
Add help and manual messages about LZO compression support Avoid LONG_MAX/ULONG_MAX redefinitions
Add LZO Support
Quoted 16 lines
Makefile | 5 ++++ common.h | 4 +++ diskdump_mod.h | 3 ++- makedumpfile.8 | 6 +++-- makedumpfile.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++------ makedumpfile.h | 4 +++ print_info.c | 16 +++++++------ 7 files changed, 86 insertions(+), 19 deletions(-)Signature · 6 lines
HATAYAMA Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
From e3cb084f316cda87810d1289b57fd5d8240abb6c Mon Sep 17 00:00:00 2001
From: HATAYAMA Daisuke <[email protected]>
Date: Fri, 23 Mar 2012 10:27:35 +0900
Subject: [PATCH 1/2] [PATCH v3 3/4] Add help and manual messages about LZO
compression support.
--- makedumpfile.8 | 8 ++++---- print_info.c | 21 ++++++++++++--------- 2 files changed, 16 insertions(+), 13 deletions(-)
makedumpfile.8 +4 -4
@@ -121,8 +121,8 @@ configuration, you need to use --diskset option. .SH OPTIONS .TP -\fB\-c\fR -Compress dump data by each page. +\fB\-c,\-l\fR +Compress dump data by each page using zlib for -c option or lzo for -l option. .br A user cannot specify this option with \-E option, because the ELF format does not support compressed data.
@@ -196,7 +196,7 @@ by dump_level 11, makedumpfile retries it by dump_level 31. \fB\-E\fR Create \fIDUMPFILE\fR in the ELF format. .br -This option cannot be specified with \-c option, because the ELF format does not +This option cannot be specified with either of \-c option or \-l option, because the ELF format does not support compressed data. .br .B Example:
@@ -491,7 +491,7 @@ Print debugging message. .TP \fB\-h\fR -Show help message. +Show help message and LZO support status (enabled/disabled). .TP \fB\-v\fR
print_info.c +12 -9
@@ -33,17 +33,20 @@ voidprint_usage (void) {MSG ("\n" ); +MSG ("LZO support:\n" ); +MSG (" enabled\n" ); +MSG ("\n" );MSG ("Usage:\n" );MSG (" Creating DUMPFILE:\n" ); -MSG (" # makedumpfile [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n" ); +MSG (" # makedumpfile [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n" );MSG ("\n" );MSG (" Creating DUMPFILE with filtered kernel data specified through filter config\n" );MSG (" file:\n" ); -MSG (" # makedumpfile [-c|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n" ); +MSG (" # makedumpfile [-c|-l|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n" );MSG (" DUMPFILE\n" );MSG ("\n" );MSG (" Outputting the dump data in the flattened format to the standard output:\n" ); -MSG (" # makedumpfile -F [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n" ); +MSG (" # makedumpfile -F [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n" );MSG ("\n" );MSG (" Rearranging the dump data in the flattened format to a readable DUMPFILE:\n" );MSG (" # makedumpfile -R DUMPFILE\n" );
@@ -70,14 +73,14 @@ print_usage(void)MSG ("\n" );MSG ("\n" );MSG (" Creating DUMPFILE from multiple VMCOREs generated on sadump diskset configuration:\n" ); -MSG (" # makedumpfile [-c] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n" ); +MSG (" # makedumpfile [-c|-l] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n" );MSG (" [--diskset=VMCORE3 ..] DUMPFILE\n" );MSG ("\n" );MSG ("\n" );MSG ("Available options:\n" ); -MSG (" [-c]:\n" ); -MSG (" Compress dump data by each page.\n" ); -MSG (" A user cannot specify this option with -E option, because the ELF format\n" ); +MSG (" [-c|-l]:\n" ); +MSG (" Compress dump data by each page using zlib for -c option and lzo for -l option.\n" ); +MSG (" A user cannot specify either of these options with -E option, because the ELF format\n" );MSG (" does not support compressed data.\n" );MSG (" THIS IS ONLY FOR THE CRASH UTILITY.\n" );MSG ("\n" );
@@ -103,7 +106,7 @@ print_usage(void)MSG ("\n" );MSG (" [-E]:\n" );MSG (" Create DUMPFILE in the ELF format.\n" ); -MSG (" This option cannot be specified with -c option, because the ELF\n" ); +MSG (" This option cannot be specified with either of -c option or -l option, because the ELF\n" );MSG (" format does not support compressed data.\n" );MSG ("\n" );MSG (" [-x VMLINUX]:\n" );
@@ -214,7 +217,7 @@ print_usage(void)MSG (" Overwrite DUMPFILE even if it already exists.\n" );MSG ("\n" );MSG (" [-h]:\n" ); -MSG (" Show help message.\n" ); +MSG (" Show help message and LZO support status (enabled/disabled).\n" );MSG ("\n" );MSG (" [-b <order>]\n" );MSG (" Specify the cache 2^order pages in ram when generating vmcore info\n" );
Signature · 1 line
1.7.9.2
From 14308f5a7881104e3acf8bfa20b141edde22965e Mon Sep 17 00:00:00 2001
From: HATAYAMA Daisuke <[email protected]>
Date: Fri, 23 Mar 2012 10:23:04 +0900
Subject: [PATCH 2/2] [PATCH v3 4/4] Add build condition for LZO support.
To enable lzo compression support, build makedumpfile as:
$ make USELZO=on
To disable, as:
$ make
In default, lzo compression support is disabled.
--- Makefile | 5 +++++ README | 3 +++ makedumpfile.8 | 1 + makedumpfile.c | 24 ++++++++++++++++++++++-- makedumpfile.h | 2 ++ print_info.c | 4 ++++ 6 files changed, 37 insertions(+), 2 deletions(-)
Makefile +5 -0
@@ -50,6 +50,11 @@ ifneq ($(LINKTYPE), dynamic) LIBS := -static $(LIBS) endif +ifeq ($(USELZO), on) +LIBS := -llzo2 $(LIBS) +CFLAGS += -DUSELZO +endif + all: makedumpfile $(OBJ_PART): $(SRC_PART)
README +3 -0
@@ -42,6 +42,9 @@ where <arch> is the 'uname -m' of the target architecture. The user has to set the environment variable CC to appropriate compiler for the target architecture. + 6.Build with lzo support: + # make USELZO=on ; make install + The user has to prepare lzo library. * SUPPORTED KERNELS This makedumpfile supports the following kernels.
makedumpfile.8 +1 -0
@@ -123,6 +123,7 @@ configuration, you need to use --diskset option. .TP \fB\-c,\-l\fR Compress dump data by each page using zlib for -c option or lzo for -l option. +(-l option needs USELZO=on when building.) .br A user cannot specify this option with \-E option, because the ELF format does not support compressed data.
makedumpfile.c +22 -2
@@ -266,6 +266,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size) goto error; }memcpy (bufptr, buf2 + page_offset, size); +#ifdef USELZO }else if (info->flag_lzo_support && (pd.flags &DUMP_DH_COMPRESSED_LZO )) { retlen = info->page_size;
@@ -277,6 +278,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size) goto error; }memcpy (bufptr, buf2 + page_offset, size); +#endif }else memcpy (bufptr, buf + page_offset, size);
@@ -2525,8 +2527,16 @@ initial(void)unsigned long size;int debug_info = FALSE; +#ifdef USELZO if (lzo_init () ==LZO_E_OK ) info->flag_lzo_support = TRUE; +#else +if (info->flag_compress ==DUMP_DH_COMPRESSED_LZO ) { +MSG ("'-l' option is disabled, " ); +MSG ("because this binary doesn't support lzo compression.\n" ); +MSG ("Try `make USELZO=on` when building.\n" ); + } +#endif if (!is_xen_memory () && info->flag_exclude_xen_dom) {MSG ("'-X' option is disable," );
@@ -4695,11 +4705,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)off_t offset_data =0 ;struct disk_dump_header *dh = info->dump_header;unsigned char buf[info->page_size], *buf_out =NULL ; -unsigned long len_buf_out, len_buf_out_zlib, len_buf_out_lzo; +unsigned long len_buf_out;struct dump_bitmap bitmap2;struct timeval tv_start;const off_t failed = (off_t)-1 ; -lzo_bytep wrkmem =NULL ;int ret = FALSE;
@@ -4708,6 +4717,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)initialize_2nd_bitmap (&bitmap2); +#ifdef USELZO +unsigned long len_buf_out_zlib, len_buf_out_lzo; +lzo_bytep wrkmem; +if ((wrkmem =malloc (LZO1X_1_MEM_COMPRESS )) ==NULL ) {ERRMSG ("Can't allocate memory for the working memory. %s\n" ,strerror (errno));
@@ -4717,6 +4730,9 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) len_buf_out_zlib =compressBound (info->page_size); len_buf_out_lzo = info->page_size + info->page_size /16 +64 +3 ; len_buf_out =MAX (len_buf_out_zlib, len_buf_out_lzo); +#else + len_buf_out =compressBound (info->page_size); +#endif if ((buf_out =malloc (len_buf_out)) ==NULL ) { ERRMSG("Can't allocate memory for the compression buffer. %s\n",
@@ -4807,6 +4823,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) pd.flags =DUMP_DH_COMPRESSED_ZLIB ; pd.size = size_out;memcpy (buf, buf_out, pd.size); +#ifdef USELZO } else if (info->flag_lzo_support && (info->flag_compress & DUMP_DH_COMPRESSED_LZO) && ((size_out = info->page_size),
@@ -4816,6 +4833,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) pd.flags =DUMP_DH_COMPRESSED_LZO ; pd.size = size_out;memcpy (buf, buf_out, pd.size); +#endif }else { pd.flags =0 ; pd.size = info->page_size;
@@ -4856,8 +4874,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page) out:if (buf_out !=NULL )free (buf_out); +#ifdef USELZO if (wrkmem !=NULL )free (wrkmem); +#endif return ret; }
makedumpfile.h +2 -0
@@ -31,7 +31,9 @@#include <libelf.h> #include <byteswap.h> #include <getopt.h> +#ifdef USELZO #include <lzo/lzo1x.h> +#endif #include "common.h" #include "dwarf_info.h" #include "diskdump_mod.h"
print_info.c +4 -0
@@ -34,7 +34,11 @@ print_usage(void) {MSG ("\n" );MSG ("LZO support:\n" ); +#ifdef USELZO MSG (" enabled\n" ); +#else +MSG (" disabled ('-l' option will be ignored.)\n" ); +#endif MSG ("\n" );MSG ("Usage:\n" );MSG (" Creating DUMPFILE:\n" );
Signature · 1 line
1.7.9.2
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Kumagai-san,
From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Fri, 23 Mar 2012 16:33:03 +0900
Quoted 18 lines
Quoted 4 lines · On Thu, 23 Feb 2012 10:33:54 +0900 HATAYAMA Daisuke…
The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.
Sorry for late reply.
[PATCH v2 4/4] allows to use -l option even without USELZO but lzo compression doesn't work without USELZO.
I think that warning messages should be displayed when -l option is used on makedumpfile built without USELZO.So I made v3 patches attached to this mail.
([PATCH v2 1/4] and [PATCH v2 2/4] look good and I didn't change them.)
Quoted 8 lines
v2 => v3:
- Add warning messages for invalid -l option.
- Show LZO support status with help messages.
- Add description of USELZO in README and man page.
- (and I will do some cleanups.)What do you think, Hatayama-san ?
Thanks. I think the warning message very helpful for users.
BTW, I have a question about future build option of LZO library. You said previously you're going to introduce autotools. Then, do you consider default enable if LZO library is present on the environment?
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Hatayama-san,
Quoted 3 lines · On Fri, 23 Mar 2012 17:26:08 +0900 ( ) HATAYAMA Daisuke…
BTW, I have a question about future build option of LZO library. You said previously you're going to introduce autotools. Then, do you consider default enable if LZO library is present on the environment?
I'm afraid I decide not to introduce autotools for the following reasons.
First, because autotools cannot know whether LZO library is prepared in 2nd kernel environment, it cannot decide whether it should link LZO library dynamically even if LINKTYPE=dynamic is specified.
Second, if autotools behaves differently depending on LINKTYPE, it is difficult for users to understand.
So, I think that current method is simpler and better than autotools.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Kumagai-san,
From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Wed, 28 Mar 2012 14:18:09 +0900
Quoted 11 lines
Hello Hatayama-san,
Quoted 3 lines · On Fri, 23 Mar 2012 17:26:08 +0900 ( ) HATAYAMA Daisuke…
BTW, I have a question about future build option of LZO library. You said previously you're going to introduce autotools. Then, do you consider default enable if LZO library is present on the environment?
I'm afraid I decide not to introduce autotools for the following reasons.
It's OK if you don't have plan to use autotools.
Quoted 4 lines
First, because autotools cannot know whether LZO library is prepared in 2nd kernel environment, it cannot decide whether it should link LZO library dynamically even if LINKTYPE=dynamic is specified.
Yes, I of course agree with the fact that makedumpfile for kdump 2nd kernel should always be built in statically linked way.
But I think two problems are different, that is, the problem making users easily able to choose build option by autotools, and the problem guranteeing that the makedumpfile for kdump 2nd kernel be built always statically.
Quoted 2 lines
Second, if autotools behaves differently depending on LINKTYPE, it is difficult for users to understand.
If autotools were introduced, then I think we would no longer use LINKTYPE and we would choose building option through autotools features. For example, through new options in configure script such as --enable-lzo-{static,dynamic} for example?
Quoted 3 lines
So, I think that current method is simpler and better than autotools.
I agree. I think simpler one is better if it's enough for the purpose.
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello,
Quoted 9 lines · HATAYAMA Daisuke
Yes, I of course agree with the fact that makedumpfile for kdump 2nd kernel should always be built in statically linked way.
But I think two problems are different, that is, the problem making users easily able to choose build option by autotools, and the problem guranteeing that the makedumpfile for kdump 2nd kernel be built always statically.
I don't see why it should be statically linked. Debian & Ubuntu both use a dynamically linked makedumpfile for the 2nd kdump kernel.
Debian executes makedumpfile on the 'real' root FS while Ubuntu runs it in the initramfs. It is simply a chroot on the root FS being mounted on /root.
HTH,
...Louis
Signature · 9 lines
Louis Bouchard Backline Support Analyst Canonical Ltd Ubuntu support: http://landscape.canonical.com _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Hello,
From: Bouchard Louis <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Wed, 28 Mar 2012 09:59:04 +0200
Quoted 13 lines
Hello,
Quoted 9 lines · HATAYAMA Daisuke
Yes, I of course agree with the fact that makedumpfile for kdump 2nd kernel should always be built in statically linked way.
But I think two problems are different, that is, the problem making users easily able to choose build option by autotools, and the problem guranteeing that the makedumpfile for kdump 2nd kernel be built always statically.
Quoted 7 lines
I don't see why it should be statically linked. Debian & Ubuntu both use a dynamically linked makedumpfile for the 2nd kdump kernel.
Debian executes makedumpfile on the 'real' root FS while Ubuntu runs it in the initramfs. It is simply a chroot on the root FS being mounted on /root.
Thanks. I've noticed the fact on Debian for the first time.
Root filesystem can be broken due to filesystem related bugs or disk crash. So, on fedora/RHEL environments, root filesystem is not mounted in kdump initrd/initramfs. Necessary resources in 2nd kernel, including makedumpfile, are all packed in kdump initrd/initramfs.
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec