Skip to contentSkip to conversation

LZO Compression Support

PATCH Superseded HATAYAMA Daisuke 2 participants 7 messages Feb 20, 2012 – Feb 22, 2012

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover LZO Compression Support

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.

The RFC discussion:

http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

How to build:

1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:

1) Original author's website:
http://www.oberhumer.com/opensource/lzo/

2) yum framework on fedora. Older releases don't have the packages.

2. Apply the patch set to makedumpfile v1.4.2.

3. Do make.

How to use:

Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.

Example)

  $ makedumpfile -l vmcore dumpfile

Performance evaluation:

- Kumagai-san's evaluation simulating actually working servers:

    http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

- My evaluation focusing on the worst cases:

    http://lists.infradead.org/pipermail/kexec/2011-November/005783.html

LZO Support for crash:

I'll post LZO support patch for crash after makedumpfile merges these patches.

---

HATAYAMA Daisuke (2):
Add help and manual messages about LZO compression support Add LZO Support

 Makefile       |    2 +-
 diskdump_mod.h |    3 ++-
 makedumpfile.8 |    6 +++---
 makedumpfile.c |   57 +++++++++++++++++++++++++++++++++++++++++++++++---------
 makedumpfile.h |    2 ++
 print_info.c   |   16 ++++++++--------
 6 files changed, 64 insertions(+), 22 deletions(-)
Signature · 6 lines
HATAYAMA Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

PATCH 1/2 Add LZO Support

Add -l option as one of the command-line options users can specify. If -l option is specified, then makedumpfile generates dumpfile in kdump-compressed format with lzo compression by each page.

---

 Makefile       |    2 +-
 diskdump_mod.h |    3 ++-
 makedumpfile.c |   57 +++++++++++++++++++++++++++++++++++++++++++++++---------
 makedumpfile.h |    2 ++
 4 files changed, 53 insertions(+), 11 deletions(-)
Makefile +1 -1
index e6b7b89..21b3058 100644
@@ -29,7 +29,7 @@ OBJ_PART = print_info.o dwarf_info.o elf_info.o erase_info.o sadump_info.o
 SRC_ARCH = arch/arm.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c
 OBJ_ARCH = arch/arm.o arch/x86.o arch/x86_64.o arch/ia64.o arch/ppc64.o arch/s390x.o
 
-LIBS = -ldw -lbz2 -lebl -ldl -lelf -lz
+LIBS = -ldw -lbz2 -lebl -ldl -lelf -lz -llzo2
 ifneq ($(LINKTYPE), dynamic)
 LIBS := -static $(LIBS)
 endif
diskdump_mod.h +2 -1
index c1de972..e672485 100644
@@ -78,7 +78,8 @@ struct kdump_sub_header {
 };
 
 /* page flags */
-#define DUMP_DH_COMPRESSED	0x1	/* page is compressed               */
+#define DUMP_DH_COMPRESSED_ZLIB	0x1	/* page is compressed with zlib */
+#define DUMP_DH_COMPRESSED_LZO	0x2	/* paged is compressed with lzo */
 
 /* descriptor of each page for vmcore */
 typedef struct page_desc {
makedumpfile.c +48 -9

121 lines not on this page · raw message

makedumpfile.h +2 -0
index ebb8929..9594ca7 100644
@@ -31,6 +31,7 @@
 #include <libelf.h>
 #include <byteswap.h>
 #include <getopt.h>
+#include <lzo/lzo1x.h>
 #include "common.h"
 #include "dwarf_info.h"
 #include "diskdump_mod.h"
@@ -767,6 +768,7 @@ struct DumpInfo {
 	int		num_dump_level;      /* number of dump level */
 	int		array_dump_level[NUM_ARRAY_DUMP_LEVEL];
 	int		flag_compress;       /* flag of compression */
+	int		flag_lzo_support;    /* flag of LZO compression support */
 	int		flag_elf_dumpfile;   /* flag of creating ELF dumpfile */
 	int		flag_generate_vmcoreinfo;/* flag of generating vmcoreinfo file */
 	int		flag_read_vmcoreinfo;    /* flag of reading vmcoreinfo file */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

PATCH 2/2 Add help and manual messages about LZO compression support

In reply to HATAYAMA Daisuke

---

 makedumpfile.8 |    6 +++---
 print_info.c   |   16 ++++++++--------
 2 files changed, 11 insertions(+), 11 deletions(-)
makedumpfile.8 +3 -3
index 4733420..f607bf5 100644
@@ -121,8 +121,8 @@ configuration, you need to use --diskset option.
 .SH OPTIONS
 
 .TP
-\fB\-c\fR
-Compress dump data by each page.
+\fB\-c,\-l\fR
+Compress dump data by each page using zlib for -c option or lzo for -l option.
 .br
 A user cannot specify this option with \-E option, because the ELF format does
 not support compressed data.
@@ -196,7 +196,7 @@ by dump_level 11, makedumpfile retries it by dump_level 31.
 \fB\-E\fR
 Create \fIDUMPFILE\fR in the ELF format.
 .br
-This option cannot be specified with \-c option, because the ELF format does not
+This option cannot be specified with either of \-c option or \-l option, because the ELF format does not
 support compressed data.
 .br
 .B Example:
print_info.c +8 -8
index 31460b7..237892e 100644
@@ -35,15 +35,15 @@ print_usage(void)
 	MSG("\n");
 	MSG("Usage:\n");
 	MSG("  Creating DUMPFILE:\n");
-	MSG("  # makedumpfile    [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n");
+	MSG("  # makedumpfile    [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE DUMPFILE\n");
 	MSG("\n");
 	MSG("  Creating DUMPFILE with filtered kernel data specified through filter config\n");
 	MSG("  file:\n");
-	MSG("  # makedumpfile    [-c|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n");
+	MSG("  # makedumpfile    [-c|-l|-E] [-d DL] -x VMLINUX --config FILTERCONFIGFILE VMCORE\n");
 	MSG("    DUMPFILE\n");
 	MSG("\n");
 	MSG("  Outputting the dump data in the flattened format to the standard output:\n");
-	MSG("  # makedumpfile -F [-c|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n");
+	MSG("  # makedumpfile -F [-c|-l|-E] [-d DL] [-x VMLINUX|-i VMCOREINFO] VMCORE\n");
 	MSG("\n");
 	MSG("  Rearranging the dump data in the flattened format to a readable DUMPFILE:\n");
 	MSG("  # makedumpfile -R DUMPFILE\n");
@@ -70,14 +70,14 @@ print_usage(void)
 	MSG("\n");
 	MSG("\n");
 	MSG("  Creating DUMPFILE from multiple VMCOREs generated on sadump diskset configuration:\n");
-	MSG("  # makedumpfile [-c] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n");
+	MSG("  # makedumpfile [-c|-l] [-d DL] -x VMLINUX --diskset=VMCORE1 --diskset=VMCORE2\n");
 	MSG("    [--diskset=VMCORE3 ..] DUMPFILE\n");
 	MSG("\n");
 	MSG("\n");
 	MSG("Available options:\n");
-	MSG("  [-c]:\n");
-	MSG("      Compress dump data by each page.\n");
-	MSG("      A user cannot specify this option with -E option, because the ELF format\n");
+	MSG("  [-c|-l]:\n");
+	MSG("      Compress dump data by each page using zlib for -c option and lzo for -l option.\n");
+	MSG("      A user cannot specify either of these options with -E option, because the ELF format\n");
 	MSG("      does not support compressed data.\n");
 	MSG("      THIS IS ONLY FOR THE CRASH UTILITY.\n");
 	MSG("\n");
@@ -103,7 +103,7 @@ print_usage(void)
 	MSG("\n");
 	MSG("  [-E]:\n");
 	MSG("      Create DUMPFILE in the ELF format.\n");
-	MSG("      This option cannot be specified with -c option, because the ELF\n");
+	MSG("      This option cannot be specified with either of -c option or -l option, because the ELF\n");
 	MSG("      format does not support compressed data.\n");
 	MSG("\n");
 	MSG("  [-x VMLINUX]:\n");

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
In reply to HATAYAMA Daisuke

Hello Hatayama-san,

Quoted 37 lines · On Mon, 20 Feb 2012 18:10:00 +0900 HATAYAMA Daisuke…

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.

The RFC discussion:

http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

How to build:

1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:

1) Original author's website:
http://www.oberhumer.com/opensource/lzo/

2) yum framework on fedora. Older releases don't have the packages.

2. Apply the patch set to makedumpfile v1.4.2.

3. Do make.

How to use:

Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.

Example)

  $ makedumpfile -l vmcore dumpfile

Performance evaluation:

- Kumagai-san's evaluation simulating actually working servers:

    http://lists.infradead.org/pipermail/kexec/2011-December/005868.html
Quoted 7 lines

- My evaluation focusing on the worst cases:

    http://lists.infradead.org/pipermail/kexec/2011-November/005783.html

LZO Support for crash:

I'll post LZO support patch for crash after makedumpfile merges these patches.

Thank you for your patches.

I'll review and merge them into v1.4.4 with making LZO support optional. I'll post the patch to make it optional after v1.4.3 is released.

Please wait for a while.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw

Hello Kumagai-san,

From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile 0/2] LZO Compression Support
Date: Tue, 21 Feb 2012 16:50:41 +0900

Quoted 52 lines

Hello Hatayama-san,

Quoted 37 lines · On Mon, 20 Feb 2012 18:10:00 +0900 HATAYAMA Daisuke…

The following series implements LZO compression support to makedumpfile. LZO is as good as in size but by far better in speed than ZLIB, readucing down time during generation of crash dump and refiltering.

The RFC discussion:

http://lists.infradead.org/pipermail/kexec/2011-November/005783.html
http://lists.infradead.org/pipermail/kexec/2011-December/005868.html

How to build:

1. Get lzo libraries: lzo, lzo-devel and lzo-minilzo from either of the following:

1) Original author's website:
http://www.oberhumer.com/opensource/lzo/

2) yum framework on fedora. Older releases don't have the packages.

2. Apply the patch set to makedumpfile v1.4.2.

3. Do make.

How to use:

Introduce new -l option. If a user specify this, makedumpfile generates dumpfile compressed by pages with lzo compression.

Example)

  $ makedumpfile -l vmcore dumpfile

Performance evaluation:

- Kumagai-san's evaluation simulating actually working servers:

    http://lists.infradead.org/pipermail/kexec/2011-December/005868.html
Quoted 7 lines

- My evaluation focusing on the worst cases:

    http://lists.infradead.org/pipermail/kexec/2011-November/005783.html

LZO Support for crash:

I'll post LZO support patch for crash after makedumpfile merges these patches.

Thank you for your patches.

Quoted 8 lines

I'll review and merge them into v1.4.4 with making LZO support optional. I'll post the patch to make it optional after v1.4.3 is released.

Please wait for a while.

Thanks
Atsushi Kumagai

Thanks. But I'm OK if I'll repost the patch after you have decided how you change Makefile to support optional building. Or is it more effective for you to do them by yourself at the same time?

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw

Hello Hatayama-san,

Quoted 7 lines · On Tue, 21 Feb 2012 17:01:25 +0900 ( ) HATAYAMA Daisuke…
Quoted 7 lines

Thank you for your patches.

I'll review and merge them into v1.4.4 with making LZO support optional. I'll post the patch to make it optional after v1.4.3 is released.

Please wait for a while.

Quoted 7 lines
Quoted 3 lines

Thanks
Atsushi Kumagai

Thanks. But I'm OK if I'll repost the patch after you have decided how you change Makefile to support optional building. Or is it more effective for you to do them by yourself at the same time?

I wonder if you could repost the patch with implementing my idea. My idea is that using build option (like "LINKTYPE" at v1.4.2).

Example)
1. enable lzo support

     $ make USELZO=on

2. disable lzo support (default)

     $ make

In future, I would like to use automake but don't worry about it now.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

From: Atsushi Kumagai <[email protected]>
Subject: Re: [PATCH makedumpfile 0/2] LZO Compression Support
Date: Wed, 22 Feb 2012 11:42:09 +0900

Quoted 25 lines

Hello Hatayama-san,

Quoted 7 lines · On Tue, 21 Feb 2012 17:01:25 +0900 ( ) HATAYAMA Daisuke…
Quoted 7 lines

Thank you for your patches.

I'll review and merge them into v1.4.4 with making LZO support optional. I'll post the patch to make it optional after v1.4.3 is released.

Please wait for a while.

Quoted 7 lines
Quoted 3 lines

Thanks
Atsushi Kumagai

Thanks. But I'm OK if I'll repost the patch after you have decided how you change Makefile to support optional building. Or is it more effective for you to do them by yourself at the same time?

I wonder if you could repost the patch with implementing my idea. My idea is that using build option (like "LINKTYPE" at v1.4.2).

Example)
1. enable lzo support

Quoted 7 lines
     $ make USELZO=on

2. disable lzo support (default)

     $ make

In future, I would like to use automake but don't worry about it now.

I see. I'll repost the patch soon.

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec