Skip to contentSkip to conversation

LZO Compression Support

PATCH In review HATAYAMA Daisuke 3 participants 15 messages Feb 23, 2012 – May 10, 2012

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover LZO Compression Support

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.

The RFC discussion was made here:

http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

How to build:

1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:

1) Original author's website:
http://www.oberhumer.com/opensource/lzo/

2) yum framework on fedora. Older releases don't have the packages.

2. Apply the patch set to makedumpfile v1.4.2.

3. Do make as follows:

    $ make USELZO=on

Note: In default, no LZO compression support is included.

How to use:

Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.

Example)

  $ makedumpfile -l vmcore dumpfile

Performance evaluation:

- Kumagai-san's evaluation simulating working servers:

    http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

- My evaluation focusing on the worst cases:

    http://lists.infradead.org/pipermail/kexec/2011-November/005783.html

LZO Support for crash:

I'll post LZO support patch for crash after makedumpfile merges these patches.

Changelog:

v1 => v2:

- Add build condition for LZO support. Enable LZO support if specifying USELZO=on to make command.

- Avoid LONG_MAX/ULONG_MAX redefinitions.

---

HATAYAMA Daisuke (4):
Add build condition for LZO support
Add help and manual messages about LZO compression support Avoid LONG_MAX/ULONG_MAX redefinitions
Add LZO Support

 Makefile       |    5 ++++
 common.h       |    4 +++
 diskdump_mod.h |    3 ++-
 makedumpfile.8 |    6 +++--
 makedumpfile.c |   67 ++++++++++++++++++++++++++++++++++++++++++++++++++------
 makedumpfile.h |    4 +++
 print_info.c   |   16 +++++++------
 7 files changed, 86 insertions(+), 19 deletions(-)
Signature · 6 lines
HATAYAMA Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

PATCH v2 1/4 Add LZO Support

Add -l option as one of the command-line options users can specify. If -l option is specified, then makedumpfile generates dumpfile in kdump-compressed format with lzo compression by each page.

---

 diskdump_mod.h |    3 ++-
 makedumpfile.c |   57 +++++++++++++++++++++++++++++++++++++++++++++++---------
 makedumpfile.h |    2 ++
 3 files changed, 52 insertions(+), 10 deletions(-)
diskdump_mod.h +2 -1
index c1de972..e672485 100644
@@ -78,7 +78,8 @@ struct kdump_sub_header {
 };
 
 /* page flags */
-#define DUMP_DH_COMPRESSED	0x1	/* page is compressed               */
+#define DUMP_DH_COMPRESSED_ZLIB	0x1	/* page is compressed with zlib */
+#define DUMP_DH_COMPRESSED_LZO	0x2	/* paged is compressed with lzo */
 
 /* descriptor of each page for vmcore */
 typedef struct page_desc {
makedumpfile.c +48 -9

121 lines not on this page · raw message

makedumpfile.h +2 -0
index ebb8929..9594ca7 100644
@@ -31,6 +31,7 @@
 #include <libelf.h>
 #include <byteswap.h>
 #include <getopt.h>
+#include <lzo/lzo1x.h>
 #include "common.h"
 #include "dwarf_info.h"
 #include "diskdump_mod.h"
@@ -767,6 +768,7 @@ struct DumpInfo {
 	int		num_dump_level;      /* number of dump level */
 	int		array_dump_level[NUM_ARRAY_DUMP_LEVEL];
 	int		flag_compress;       /* flag of compression */
+	int		flag_lzo_support;    /* flag of LZO compression support */
 	int		flag_elf_dumpfile;   /* flag of creating ELF dumpfile */
 	int		flag_generate_vmcoreinfo;/* flag of generating vmcoreinfo file */
 	int		flag_read_vmcoreinfo;    /* flag of reading vmcoreinfo file */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

PATCH v2 2/4 Avoid LONG_MAX/ULONG_MAX redefinitions

In reply to HATAYAMA Daisuke

common.h locally defines LONG_MAX and ULONG_MAX. But when importing lzo library's headers, ulimit.h that is imported implicitly is also imported so the two definitions collide, and the warning message below is displayed.

gcc -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DVERSION='"1.4.2"' -DRELEASE_DATE='"25 January 2012"' -D__x86_64__ -DUSELZO print_info.o dwarf_info.o elf_info.o erase_info.o sadump_info.o arch/arm.o arch/x86.o arch/x86_64.o arch/ia64.o arch/ppc64.o arch/s390x.o -o makedumpfile makedumpfile.c -llzo2 -ldw -lbz2 -lebl -ldl -lelf -lz In file included from makedumpfile.h:37:0,
from makedumpfile.c:16:
common.h:23:0: warning: "LONG_MAX" redefined
/usr/lib/gcc/x86_64-redhat-linux/4.5.1/include/limits.h:132:0: note: this is the location of the previous definition
common.h:24:0: warning: "ULONG_MAX" redefined
/usr/lib/gcc/x86_64-redhat-linux/4.5.1/include/limits.h:136:0: note: this is the location of the previous definition

To avoid this problem, check if each has already defiend before defining it.

---

 common.h |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
common.h +4 -0
index a4cc2f7..6ad3ca7 100644
@@ -20,8 +20,12 @@
 #define FALSE		(0)
 #define ERROR		(-1)
 
+#ifndef LONG_MAX
 #define LONG_MAX	((long)(~0UL>>1))
+#endif
+#ifndef ULONG_MAX
 #define ULONG_MAX	(~0UL)
+#endif
 #define ULONGLONG_MAX	(~0ULL)
 
 #define MAX(a,b)	((a) > (b) ? (a) : (b))

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

PATCH v2 3/4 Add help and manual messages about LZO compression support

In reply to HATAYAMA Daisuke

---

 makedumpfile.8 |    6 +++---
 print_info.c   |   16 ++++++++--------
 2 files changed, 11 insertions(+), 11 deletions(-)
makedumpfile.8 +3 -3
index 4733420..f607bf5 100644
@@ -121,8 +121,8 @@ configuration, you need to use --diskset option.
 .SH OPTIONS
 
 .TP
-\fB\-c\fR
-Compress dump data by each page.
+\fB\-c,\-l\fR
+Compress dump data by each page using zlib for -c option or lzo for -l option.
 .br
 A user cannot specify this option with \-E option, because the ELF format does
 not support compressed data.
@@ -196,7 +196,7 @@ by dump_level 11, makedumpfile retries it by dump_level 31.
 \fB\-E\fR
 Create \fIDUMPFILE\fR in the ELF format.
 .br
-This option cannot be specified with \-c option, because the ELF format does not
+This option cannot be specified with either of \-c option or \-l option, because the ELF format does not
 support compressed data.
 .br
 .B Example:
print_info.c +8 -8
index 31460b7..237892e 100644
@@ -35,15 +35,15 @@ print_usage(void)
 	MSG("\n");
 	MSG("Usage:\n");
 	MSG("  Creating DUMPFILE:\n");
-	MSG("  # makedumpfile    [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n");
+	MSG("  # makedumpfile    [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n");
 	MSG("\n");
 	MSG("  Creating DUMPFILE with filtered kernel data specified through filter config\n");
 	MSG("  file:\n");
-	MSG("  # makedumpfile    [-c|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n");
+	MSG("  # makedumpfile    [-c|-l|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n");
 	MSG("    DUMPFILE\n");
 	MSG("\n");
 	MSG("  Outputting the dump data in the flattened format to the standard output:\n");
-	MSG("  # makedumpfile -F [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n");
+	MSG("  # makedumpfile -F [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n");
 	MSG("\n");
 	MSG("  Rearranging the dump data in the flattened format to a readable DUMPFILE:\n");
 	MSG("  # makedumpfile -R DUMPFILE\n");
@@ -70,14 +70,14 @@ print_usage(void)
 	MSG("\n");
 	MSG("\n");
 	MSG("  Creating DUMPFILE from multiple VMCOREs generated on sadump diskset configuration:\n");
-	MSG("  # makedumpfile [-c] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n");
+	MSG("  # makedumpfile [-c|-l] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n");
 	MSG("    [--diskset=VMCORE3 ..] DUMPFILE\n");
 	MSG("\n");
 	MSG("\n");
 	MSG("Available options:\n");
-	MSG("  [-c]:\n");
-	MSG("      Compress dump data by each page.\n");
-	MSG("      A user cannot specify this option with -E option, because the ELF format\n");
+	MSG("  [-c|-l]:\n");
+	MSG("      Compress dump data by each page using zlib for -c option and lzo for -l option.\n");
+	MSG("      A user cannot specify either of these options with -E option, because the ELF format\n");
 	MSG("      does not support compressed data.\n");
 	MSG("      THIS IS ONLY FOR THE CRASH UTILITY.\n");
 	MSG("\n");
@@ -103,7 +103,7 @@ print_usage(void)
 	MSG("\n");
 	MSG("  [-E]:\n");
 	MSG("      Create DUMPFILE in the ELF format.\n");
-	MSG("      This option cannot be specified with -c option, because the ELF\n");
+	MSG("      This option cannot be specified with either of -c option or -l option, because the ELF\n");
 	MSG("      format does not support compressed data.\n");
 	MSG("\n");
 	MSG("  [-x VMLINUX]:\n");

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw

PATCH v2 4/4 Add build condition for LZO support

In reply to HATAYAMA Daisuke

To enable lzo compression support, build makedumpfile as:

  $ make USELZO

To disable, as:

  $ make

In default, lzo compression support is disabled.

---

 Makefile       |    5 +++++
 makedumpfile.c |   18 ++++++++++++++++--
 makedumpfile.h |    2 ++
 3 files changed, 23 insertions(+), 2 deletions(-)
Makefile +5 -0
index e6b7b89..55082f7 100644
@@ -34,6 +34,11 @@ ifneq ($(LINKTYPE), dynamic)
 LIBS := -static $(LIBS)
 endif
 
+ifeq ($(USELZO), on)
+LIBS := -llzo2 $(LIBS)
+CFLAGS += -DUSELZO
+endif
+
 all: makedumpfile
 
 $(OBJ_PART): $(SRC_PART)
makedumpfile.c +16 -2
index e0079b8..717519a 100644
@@ -266,6 +266,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size)
 			goto error;
 		}
 		memcpy(bufptr, buf2 + page_offset, size);
+#ifdef USELZO
 	} else if (info->flag_lzo_support
 		   && (pd.flags & DUMP_DH_COMPRESSED_LZO)) {
 		retlen = info->page_size;
@@ -277,6 +278,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size)
 			goto error;
 		}
 		memcpy(bufptr, buf2 + page_offset, size);
+#endif
 	} else
 		memcpy(bufptr, buf + page_offset, size);
 
@@ -2509,8 +2511,10 @@ initial(void)
 	unsigned long size;
 	int debug_info = FALSE;
 
+#ifdef USELZO
 	if (lzo_init() == LZO_E_OK)
 		info->flag_lzo_support = TRUE;
+#endif
 
 	if (!is_xen_memory() && info->flag_exclude_xen_dom) {
 		MSG("'-X' option is disable,");
@@ -4679,11 +4683,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 	off_t offset_data = 0;
 	struct disk_dump_header *dh = info->dump_header;
 	unsigned char buf[info->page_size], *buf_out = NULL;
-	unsigned long len_buf_out, len_buf_out_zlib, len_buf_out_lzo;
+	unsigned long len_buf_out;
 	struct dump_bitmap bitmap2;
 	struct timeval tv_start;
 	const off_t failed = (off_t)-1;
-	lzo_bytep wrkmem = NULL;
 
 	int ret = FALSE;
 
@@ -4692,6 +4695,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 
 	initialize_2nd_bitmap(&bitmap2);
 
+#ifdef USELZO
+	unsigned long len_buf_out_zlib, len_buf_out_lzo;
+	lzo_bytep wrkmem;
+
 	if ((wrkmem = malloc(LZO1X_1_MEM_COMPRESS)) == NULL) {
 		ERRMSG("Can't allocate memory for the working memory. %s\n",
 		       strerror(errno));
@@ -4701,6 +4708,9 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 	len_buf_out_zlib = compressBound(info->page_size);
 	len_buf_out_lzo = info->page_size + info->page_size / 16 + 64 + 3;
 	len_buf_out = MAX(len_buf_out_zlib, len_buf_out_lzo);
+#else
+	len_buf_out = compressBound(info->page_size);
+#endif
 
 	if ((buf_out = malloc(len_buf_out)) == NULL) {
 		ERRMSG("Can't allocate memory for the compression buffer. %s\n",
@@ -4791,6 +4801,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 			pd.flags = DUMP_DH_COMPRESSED_ZLIB;
 			pd.size  = size_out;
 			memcpy(buf, buf_out, pd.size);
+#ifdef USELZO
 		} else if (info->flag_lzo_support
 			   && (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
 			   && ((size_out = info->page_size),
@@ -4800,6 +4811,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 			pd.flags = DUMP_DH_COMPRESSED_LZO;
 			pd.size  = size_out;
 			memcpy(buf, buf_out, pd.size);
+#endif
 		} else {
 			pd.flags = 0;
 			pd.size  = info->page_size;
@@ -4840,8 +4852,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 out:
 	if (buf_out != NULL)
 		free(buf_out);
+#ifdef USELZO
 	if (wrkmem != NULL)
 		free(wrkmem);
+#endif
 
 	return ret;
 }
makedumpfile.h +2 -0
index 9594ca7..170ac82 100644
@@ -31,7 +31,9 @@
 #include <libelf.h>
 #include <byteswap.h>
 #include <getopt.h>
+#ifdef USELZO
 #include <lzo/lzo1x.h>
+#endif
 #include "common.h"
 #include "dwarf_info.h"
 #include "diskdump_mod.h"

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw
In reply to HATAYAMA Daisuke

Hello Hatayama-san,

Quoted 49 lines · On Thu, 23 Feb 2012 10:33:54 +0900 HATAYAMA Daisuke…

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.

The RFC discussion was made here:

http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

How to build:

1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:

1) Original author's website:
http://www.oberhumer.com/opensource/lzo/

2) yum framework on fedora. Older releases don't have the packages.

2. Apply the patch set to makedumpfile v1.4.2.

3. Do make as follows:

    $ make USELZO=on

Note: In default, no LZO compression support is included.

How to use:

Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.

Example)

  $ makedumpfile -l vmcore dumpfile

Performance evaluation:

- Kumagai-san's evaluation simulating working servers:

    http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

- My evaluation focusing on the worst cases:

    http://lists.infradead.org/pipermail/kexec/2011-November/005783.html

LZO Support for crash:

I'll post LZO support patch for crash after makedumpfile merges these patches.

Quoted 8 lines

Changelog:

v1 => v2:

- Add build condition for LZO support. Enable LZO support if specifying USELZO=on to make command.

- Avoid LONG_MAX/ULONG_MAX redefinitions.

Thank you for your work.
I will review your patches for v1.4.4.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

Helllo Kumagai-san,

From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Thu, 23 Feb 2012 17:16:46 +0900

Quoted 2 lines

Thank you for your work.
I will review your patches for v1.4.4.

Could you tell me when you plan to release version 1.4.4? Not necessarily exact date, rough schedule is enough. I need to post the corresponding patch to the crash utility's side after LZO feature is merged into makedumpfile.

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw

Hello HATAYAMA-san,

Quoted 6 lines · On Thu, 10 May 2012 09:38:29 +0900 (JST) HATAYAMA Daisuke…

Helllo Kumagai-san,

From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Thu, 23 Feb 2012 17:16:46 +0900

Quoted 7 lines
Quoted 2 lines

Thank you for your work.
I will review your patches for v1.4.4.

Could you tell me when you plan to release version 1.4.4? Not necessarily exact date, rough schedule is enough. I need to post the corresponding patch to the crash utility's side after LZO feature is merged into makedumpfile.

At first, I planned to merge features below into v1.4.4.

- vmalloc translation support for PPC32 (by Suzuki K. Poulose)
- LZO Compression Support (by HATAYAMA Daisuke)
- sadump: check if given cpu is online in per-cpu related helper functions (by HATAYAMA Daisuke)

However, the first one is being discussed and the schedule is not clear. So, I would like to postpone the feature for v1.4.5 and start testing the other two for v1.4.4.

If I can start testing now, I will release v1.4.4 in this month.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw

From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Thu, 10 May 2012 12:40:27 +0900

Quoted 8 lines

At first, I planned to merge features below into v1.4.4.

- vmalloc translation support for PPC32 (by Suzuki K. Poulose)
- LZO Compression Support (by HATAYAMA Daisuke)
- sadump: check if given cpu is online in per-cpu related helper functions (by HATAYAMA Daisuke)

Quoted 6 lines

However, the first one is being discussed and the schedule is not clear. So, I would like to postpone the feature for v1.4.5 and start testing the other two for v1.4.4.

If I can start testing now, I will release v1.4.4 in this month.

Thanks. So the release would be before the release of crash utility's next version. I'll post the patch for crash utility soon.

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
In reply to HATAYAMA Daisuke

Hello Hatayama-san,

Quoted 4 lines · On Thu, 23 Feb 2012 10:33:54 +0900 HATAYAMA Daisuke…

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.

Sorry for late reply.

[PATCH v2 4/4] allows to use -l option even without USELZO but lzo compression doesn't work without USELZO.
I think that warning messages should be displayed when -l option is used on makedumpfile built without USELZO.

So I made v3 patches attached to this mail.
([PATCH v2 1/4] and [PATCH v2 2/4] look good and I didn't change them.)

v2 => v3:

- Add warning messages for invalid -l option.
- Show LZO support status with help messages.
- Add description of USELZO in README and man page.
- (and I will do some cleanups.)

What do you think, Hatayama-san ?

Thanks
Atsushi Kumagai

Quoted 62 lines

The RFC discussion was made here:

http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

How to build:

1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:

1) Original author's website:
http://www.oberhumer.com/opensource/lzo/

2) yum framework on fedora. Older releases don't have the packages.

2. Apply the patch set to makedumpfile v1.4.2.

3. Do make as follows:

    $ make USELZO=on

Note: In default, no LZO compression support is included.

How to use:

Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.

Example)

  $ makedumpfile -l vmcore dumpfile

Performance evaluation:

- Kumagai-san's evaluation simulating working servers:

    http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

- My evaluation focusing on the worst cases:

    http://lists.infradead.org/pipermail/kexec/2011-November/005783.html

LZO Support for crash:

I'll post LZO support patch for crash after makedumpfile merges these patches.

Changelog:

v1 => v2:

- Add build condition for LZO support. Enable LZO support if specifying USELZO=on to make command.

- Avoid LONG_MAX/ULONG_MAX redefinitions.

---

HATAYAMA Daisuke (4):
Add build condition for LZO support
Add help and manual messages about LZO compression support Avoid LONG_MAX/ULONG_MAX redefinitions
Add LZO Support

Quoted 16 lines
 Makefile       |    5 ++++
 common.h       |    4 +++
 diskdump_mod.h |    3 ++-
 makedumpfile.8 |    6 +++--
 makedumpfile.c |   67 ++++++++++++++++++++++++++++++++++++++++++++++++++------
 makedumpfile.h |    4 +++
 print_info.c   |   16 +++++++------
 7 files changed, 86 insertions(+), 19 deletions(-)
Signature · 6 lines
HATAYAMA Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

From e3cb084f316cda87810d1289b57fd5d8240abb6c Mon Sep 17 00:00:00 2001
From: HATAYAMA Daisuke <[email protected]>
Date: Fri, 23 Mar 2012 10:27:35 +0900
Subject: [PATCH 1/2] [PATCH v3 3/4] Add help and manual messages about LZO
compression support.

---
 makedumpfile.8 |    8 ++++----
 print_info.c   |   21 ++++++++++++---------
 2 files changed, 16 insertions(+), 13 deletions(-)
makedumpfile.8 +4 -4
index db0705c..42e36b6 100644
@@ -121,8 +121,8 @@ configuration, you need to use --diskset option.
 .SH OPTIONS
 
 .TP
-\fB\-c\fR
-Compress dump data by each page.
+\fB\-c,\-l\fR
+Compress dump data by each page using zlib for -c option or lzo for -l option.
 .br
 A user cannot specify this option with \-E option, because the ELF format does
 not support compressed data.
@@ -196,7 +196,7 @@ by dump_level 11, makedumpfile retries it by dump_level 31.
 \fB\-E\fR
 Create \fIDUMPFILE\fR in the ELF format.
 .br
-This option cannot be specified with \-c option, because the ELF format does not
+This option cannot be specified with either of \-c option or \-l option, because the ELF format does not
 support compressed data.
 .br
 .B Example:
@@ -491,7 +491,7 @@ Print debugging message.
 
 .TP
 \fB\-h\fR
-Show help message.
+Show help message and LZO support status (enabled/disabled).
 
 .TP
 \fB\-v\fR
print_info.c +12 -9
index 31460b7..580741a 100644
@@ -33,17 +33,20 @@ void
 print_usage(void)
 {
 	MSG("\n");
+	MSG("LZO support:\n");
+	MSG("  enabled\n");
+	MSG("\n");
 	MSG("Usage:\n");
 	MSG("  Creating DUMPFILE:\n");
-	MSG("  # makedumpfile    [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n");
+	MSG("  # makedumpfile    [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n");
 	MSG("\n");
 	MSG("  Creating DUMPFILE with filtered kernel data specified through filter config\n");
 	MSG("  file:\n");
-	MSG("  # makedumpfile    [-c|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n");
+	MSG("  # makedumpfile    [-c|-l|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n");
 	MSG("    DUMPFILE\n");
 	MSG("\n");
 	MSG("  Outputting the dump data in the flattened format to the standard output:\n");
-	MSG("  # makedumpfile -F [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n");
+	MSG("  # makedumpfile -F [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n");
 	MSG("\n");
 	MSG("  Rearranging the dump data in the flattened format to a readable DUMPFILE:\n");
 	MSG("  # makedumpfile -R DUMPFILE\n");
@@ -70,14 +73,14 @@ print_usage(void)
 	MSG("\n");
 	MSG("\n");
 	MSG("  Creating DUMPFILE from multiple VMCOREs generated on sadump diskset configuration:\n");
-	MSG("  # makedumpfile [-c] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n");
+	MSG("  # makedumpfile [-c|-l] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n");
 	MSG("    [--diskset=VMCORE3 ..] DUMPFILE\n");
 	MSG("\n");
 	MSG("\n");
 	MSG("Available options:\n");
-	MSG("  [-c]:\n");
-	MSG("      Compress dump data by each page.\n");
-	MSG("      A user cannot specify this option with -E option, because the ELF format\n");
+	MSG("  [-c|-l]:\n");
+	MSG("      Compress dump data by each page using zlib for -c option and lzo for -l option.\n");
+	MSG("      A user cannot specify either of these options with -E option, because the ELF format\n");
 	MSG("      does not support compressed data.\n");
 	MSG("      THIS IS ONLY FOR THE CRASH UTILITY.\n");
 	MSG("\n");
@@ -103,7 +106,7 @@ print_usage(void)
 	MSG("\n");
 	MSG("  [-E]:\n");
 	MSG("      Create DUMPFILE in the ELF format.\n");
-	MSG("      This option cannot be specified with -c option, because the ELF\n");
+	MSG("      This option cannot be specified with either of -c option or -l option, because the ELF\n");
 	MSG("      format does not support compressed data.\n");
 	MSG("\n");
 	MSG("  [-x VMLINUX]:\n");
@@ -214,7 +217,7 @@ print_usage(void)
 	MSG("      Overwrite DUMPFILE even if it already exists.\n");
 	MSG("\n");
 	MSG("  [-h]:\n");
-	MSG("      Show help message.\n");
+	MSG("      Show help message and LZO support status (enabled/disabled).\n");
 	MSG("\n");
 	MSG("  [-b <order>]\n");
 	MSG("      Specify the cache 2^order pages in ram when generating vmcore info\n");
Signature · 1 line
1.7.9.2

From 14308f5a7881104e3acf8bfa20b141edde22965e Mon Sep 17 00:00:00 2001
From: HATAYAMA Daisuke <[email protected]>
Date: Fri, 23 Mar 2012 10:23:04 +0900
Subject: [PATCH 2/2] [PATCH v3 4/4] Add build condition for LZO support.

To enable lzo compression support, build makedumpfile as:

  $ make USELZO=on

To disable, as:

  $ make

In default, lzo compression support is disabled.

---
 Makefile       |    5 +++++
 README         |    3 +++
 makedumpfile.8 |    1 +
 makedumpfile.c |   24 ++++++++++++++++++++++--
 makedumpfile.h |    2 ++
 print_info.c   |    4 ++++
 6 files changed, 37 insertions(+), 2 deletions(-)
Makefile +5 -0
index a06b2db..28c881a 100644
@@ -50,6 +50,11 @@ ifneq ($(LINKTYPE), dynamic)
 LIBS := -static $(LIBS)
 endif
 
+ifeq ($(USELZO), on)
+LIBS := -llzo2 $(LIBS)
+CFLAGS += -DUSELZO
+endif
+
 all: makedumpfile
 
 $(OBJ_PART): $(SRC_PART)
README +3 -0
index 62ba413..c313a78 100644
@@ -42,6 +42,9 @@
     where <arch> is the 'uname -m' of the target architecture. 
     The user has to set the environment variable CC to appropriate
     compiler for the target architecture.
+  6.Build with lzo support:
+    # make USELZO=on ; make install
+    The user has to prepare lzo library.
 
 * SUPPORTED KERNELS
   This makedumpfile supports the following kernels.
makedumpfile.8 +1 -0
index 42e36b6..7ce04ed 100644
@@ -123,6 +123,7 @@ configuration, you need to use --diskset option.
 .TP
 \fB\-c,\-l\fR
 Compress dump data by each page using zlib for -c option or lzo for -l option.
+(-l option needs USELZO=on when building.)
 .br
 A user cannot specify this option with \-E option, because the ELF format does
 not support compressed data.
makedumpfile.c +22 -2
index a7fec52..1342b25 100644
@@ -266,6 +266,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size)
 			goto error;
 		}
 		memcpy(bufptr, buf2 + page_offset, size);
+#ifdef USELZO
 	} else if (info->flag_lzo_support
 		   && (pd.flags & DUMP_DH_COMPRESSED_LZO)) {
 		retlen = info->page_size;
@@ -277,6 +278,7 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size)
 			goto error;
 		}
 		memcpy(bufptr, buf2 + page_offset, size);
+#endif
 	} else
 		memcpy(bufptr, buf + page_offset, size);
 
@@ -2525,8 +2527,16 @@ initial(void)
 	unsigned long size;
 	int debug_info = FALSE;
 
+#ifdef USELZO
 	if (lzo_init() == LZO_E_OK)
 		info->flag_lzo_support = TRUE;
+#else
+	if (info->flag_compress == DUMP_DH_COMPRESSED_LZO) {
+		MSG("'-l' option is disabled, ");
+		MSG("because this binary doesn't support lzo compression.\n");
+		MSG("Try `make USELZO=on` when building.\n");
+	}
+#endif
 
 	if (!is_xen_memory() && info->flag_exclude_xen_dom) {
 		MSG("'-X' option is disable,");
@@ -4695,11 +4705,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 	off_t offset_data = 0;
 	struct disk_dump_header *dh = info->dump_header;
 	unsigned char buf[info->page_size], *buf_out = NULL;
-	unsigned long len_buf_out, len_buf_out_zlib, len_buf_out_lzo;
+	unsigned long len_buf_out;
 	struct dump_bitmap bitmap2;
 	struct timeval tv_start;
 	const off_t failed = (off_t)-1;
-	lzo_bytep wrkmem = NULL;
 
 	int ret = FALSE;
 
@@ -4708,6 +4717,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 
 	initialize_2nd_bitmap(&bitmap2);
 
+#ifdef USELZO
+	unsigned long len_buf_out_zlib, len_buf_out_lzo;
+	lzo_bytep wrkmem;
+
 	if ((wrkmem = malloc(LZO1X_1_MEM_COMPRESS)) == NULL) {
 		ERRMSG("Can't allocate memory for the working memory. %s\n",
 		       strerror(errno));
@@ -4717,6 +4730,9 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 	len_buf_out_zlib = compressBound(info->page_size);
 	len_buf_out_lzo = info->page_size + info->page_size / 16 + 64 + 3;
 	len_buf_out = MAX(len_buf_out_zlib, len_buf_out_lzo);
+#else
+	len_buf_out = compressBound(info->page_size);
+#endif
 
 	if ((buf_out = malloc(len_buf_out)) == NULL) {
 		ERRMSG("Can't allocate memory for the compression buffer. %s\n",
@@ -4807,6 +4823,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 			pd.flags = DUMP_DH_COMPRESSED_ZLIB;
 			pd.size  = size_out;
 			memcpy(buf, buf_out, pd.size);
+#ifdef USELZO
 		} else if (info->flag_lzo_support
 			   && (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
 			   && ((size_out = info->page_size),
@@ -4816,6 +4833,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 			pd.flags = DUMP_DH_COMPRESSED_LZO;
 			pd.size  = size_out;
 			memcpy(buf, buf_out, pd.size);
+#endif
 		} else {
 			pd.flags = 0;
 			pd.size  = info->page_size;
@@ -4856,8 +4874,10 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 out:
 	if (buf_out != NULL)
 		free(buf_out);
+#ifdef USELZO
 	if (wrkmem != NULL)
 		free(wrkmem);
+#endif
 
 	return ret;
 }
makedumpfile.h +2 -0
index 0a6f8bb..592f122 100644
@@ -31,7 +31,9 @@
 #include <libelf.h>
 #include <byteswap.h>
 #include <getopt.h>
+#ifdef USELZO
 #include <lzo/lzo1x.h>
+#endif
 #include "common.h"
 #include "dwarf_info.h"
 #include "diskdump_mod.h"
print_info.c +4 -0
index 580741a..3c9151d 100644
@@ -34,7 +34,11 @@ print_usage(void)
 {
 	MSG("\n");
 	MSG("LZO support:\n");
+#ifdef USELZO
 	MSG("  enabled\n");
+#else
+	MSG("  disabled ('-l' option will be ignored.)\n");
+#endif
 	MSG("\n");
 	MSG("Usage:\n");
 	MSG("  Creating DUMPFILE:\n");
Signature · 1 line
1.7.9.2

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw

Hello Kumagai-san,

From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Fri, 23 Mar 2012 16:33:03 +0900

Quoted 18 lines
Quoted 4 lines · On Thu, 23 Feb 2012 10:33:54 +0900 HATAYAMA Daisuke…

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.

Sorry for late reply.

[PATCH v2 4/4] allows to use -l option even without USELZO but lzo compression doesn't work without USELZO.
I think that warning messages should be displayed when -l option is used on makedumpfile built without USELZO.

So I made v3 patches attached to this mail.
([PATCH v2 1/4] and [PATCH v2 2/4] look good and I didn't change them.)

Quoted 8 lines

v2 => v3:

- Add warning messages for invalid -l option.
- Show LZO support status with help messages.
- Add description of USELZO in README and man page.
- (and I will do some cleanups.)

What do you think, Hatayama-san ?

Thanks. I think the warning message very helpful for users.

BTW, I have a question about future build option of LZO library. You said previously you're going to introduce autotools. Then, do you consider default enable if LZO library is present on the environment?

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw

Hello Hatayama-san,

Quoted 3 lines · On Fri, 23 Mar 2012 17:26:08 +0900 ( ) HATAYAMA Daisuke…

BTW, I have a question about future build option of LZO library. You said previously you're going to introduce autotools. Then, do you consider default enable if LZO library is present on the environment?

I'm afraid I decide not to introduce autotools for the following reasons.

First, because autotools cannot know whether LZO library is prepared in 2nd kernel environment, it cannot decide whether it should link LZO library dynamically even if LINKTYPE=dynamic is specified.

Second, if autotools behaves differently depending on LINKTYPE, it is difficult for users to understand.

So, I think that current method is simpler and better than autotools.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw

Hello Kumagai-san,

From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Wed, 28 Mar 2012 14:18:09 +0900

Quoted 11 lines

Hello Hatayama-san,

Quoted 3 lines · On Fri, 23 Mar 2012 17:26:08 +0900 ( ) HATAYAMA Daisuke…

BTW, I have a question about future build option of LZO library. You said previously you're going to introduce autotools. Then, do you consider default enable if LZO library is present on the environment?

I'm afraid I decide not to introduce autotools for the following reasons.

It's OK if you don't have plan to use autotools.

Quoted 4 lines

First, because autotools cannot know whether LZO library is prepared in 2nd kernel environment, it cannot decide whether it should link LZO library dynamically even if LINKTYPE=dynamic is specified.

Yes, I of course agree with the fact that makedumpfile for kdump 2nd kernel should always be built in statically linked way.

But I think two problems are different, that is, the problem making users easily able to choose build option by autotools, and the problem guranteeing that the makedumpfile for kdump 2nd kernel be built always statically.

Quoted 2 lines

Second, if autotools behaves differently depending on LINKTYPE, it is difficult for users to understand.

If autotools were introduced, then I think we would no longer use LINKTYPE and we would choose building option through autotools features. For example, through new options in configure script such as --enable-lzo-{static,dynamic} for example?

Quoted 3 lines

So, I think that current method is simpler and better than autotools.

I agree. I think simpler one is better if it's enough for the purpose.

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#13 lore raw

Hello,

Quoted 9 lines · HATAYAMA Daisuke

Yes, I of course agree with the fact that makedumpfile for kdump 2nd kernel should always be built in statically linked way.

But I think two problems are different, that is, the problem making users easily able to choose build option by autotools, and the problem guranteeing that the makedumpfile for kdump 2nd kernel be built always statically.

I don't see why it should be statically linked. Debian & Ubuntu both use a dynamically linked makedumpfile for the 2nd kdump kernel.

Debian executes makedumpfile on the 'real' root FS while Ubuntu runs it in the initramfs. It is simply a chroot on the root FS being mounted on /root.

HTH,

...Louis

Signature · 9 lines
Louis Bouchard
Backline Support Analyst
Canonical Ltd
Ubuntu support: http://landscape.canonical.com

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#14 lore raw

Hello,

From: Bouchard Louis <[email protected]>
Subject: Re: [PATCH makedumpfile v2 0/4] LZO Compression Support
Date: Wed, 28 Mar 2012 09:59:04 +0200

Quoted 13 lines

Hello,

Quoted 9 lines · HATAYAMA Daisuke

Yes, I of course agree with the fact that makedumpfile for kdump 2nd kernel should always be built in statically linked way.

But I think two problems are different, that is, the problem making users easily able to choose build option by autotools, and the problem guranteeing that the makedumpfile for kdump 2nd kernel be built always statically.

Quoted 7 lines

I don't see why it should be statically linked. Debian & Ubuntu both use a dynamically linked makedumpfile for the 2nd kdump kernel.

Debian executes makedumpfile on the 'real' root FS while Ubuntu runs it in the initramfs. It is simply a chroot on the root FS being mounted on /root.

Thanks. I've noticed the fact on Debian for the first time.

Root filesystem can be broken due to filesystem related bugs or disk crash. So, on fedora/RHEL environments, root filesystem is not mounted in kdump initrd/initramfs. Necessary resources in 2nd kernel, including makedumpfile, are all packed in kdump initrd/initramfs.

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec