Skip to contentSkip to conversation

makedumpfile 1.5.4, 734G kdump tests

DISCUSSION Cliff Wickman 3 participants 8 messages Jul 9, 2013 – Jul 16, 2013 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I have run some tests with the latest makedumpfile and kexec and the results (below) look very good to me.

This particular test machine has a megaraid root, which had been a problem with previous kexec-tools (I did have to allocate a lot of low memory).

My 3.10 kernel does have Hatayama's vmcore mmap patches.

My only remaining concern for makedumpfile is the filtering of huge pages. I believe that patch is in progress; but I don't see it in 1.5.4.

-Cliff

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
booted with crashkernel=1G,high crashkernel=192M,low
non-cyclic mode

write to       option            init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                19          91      11.7G
megaraid disk  zlib compression              20         209       1.4G
megaraid disk  snappy compression            20          46       2.4G
megaraid disk  snappy compression no mmap    30          72       2.4G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              19         206          -
/dev/null      snappy compression            19          41          -

Notes and observations
- Snappy compression is a big win over zlib compression; over 4 times faster with a cost of relatively little disk space.
- The vmcore mmap kernel patch cuts off about 1/3 of both page scan time and data copy time.
I hope those patches reach Linus' tree shortly, as you expect.
- Data copy time is dominated by compression time; writing compressed data to /dev/null uses almost the same time as writing to disk.
I hope your efforts to multi-thread the crash kernel go forward.

Signature · 9 lines
Cliff Wickman
SGI
[email protected]
(651) 683-3824

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

A message that is not in the archive (sent off-list or not yet received): CAJGZr0JPrBB3cVyVdwJdd6cEUfnXNMuRijb9EOoSy+XmRupv7A@mail.gmail.com

#2 lore raw
Quoted 2 lines · Maxim Uvarov

does crash tool read snappy compressed files?

Yes, but you need to specify libraries at build.

http://people.redhat.com/anderson/crash.changelog.html

6.0.9 - Fix for building on host machines that have glibc-2.15.90 installed,
<cut>

- Add support for reading compressed kdump dumpfiles that were compressed by the snappy compressor. This feature is disabled by default. To enable this feature, build the crash utility in the following manner:
(1) Install the snappy libraries by using the host system's package manager or by directly downloading libraries from author's website. The packages required are:
- snappy
- snappy-devel
The author's website is: http://code.google.com/p/snappy
(2) Create a CFLAGS.extra file and an LDFLAGS.extra file in top-level crash sources directory:
- enter -DSNAPPY in the CFLAGS.extra file
- enter -lsnappy in the LDFLAGS.extra file.
(3) Build crash with "make" as always.

6.0.7 - Enhanced the "search" command to allow the searched-for value
<cut>
- Add support to for reading dumpfiles compressed by LZO using makedumpfile version 1.4.4 or later. This feature is disabled by default. To enable this feature, build the crash utility in the following manner:
(1) Install the LZO libraries by using the host system's package manager or by directly downloading libraries from author's website. The packages required are:
- lzo
- lzo-minilzo
- lzo-devel
The author's website is: http://www.oberhumer.com/opensource/lzo
(2) Create a CFLAGS.extra file and an LDFLAGS.extra file in top-level crash sources directory:
- enter -DLZO in the CFLAGS.extra file
- enter -llzo2 in the LDFLAGS.extra file.
(3) Build crash with "make" as always.

Signature · 8 lines
Thanks.
HATAYAMA, Daisuke


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

A message that is not in the archive (sent off-list or not yet received): CAJGZr0L7GBqJHaPzgpFxhpF_jmAZfDJYU_=MFxATxnLk13ni4g@mail.gmail.com

#4 lore raw
Quoted 10 lines · Maxim Uvarov

thanks. I also have a question about static support of snappy in makedumpfile.

Dynamic linking works ok. But static link confuses snappy C++ library and C, and fails with:
undefined reference to `__gxx_personality_v0' and undefined reference to new[].

Is there workaround for that?
gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)

I think you'll have to install the libstdc++ devel rpm, such as libstdc++-devel-4.4.7-3.el6.x86_64.rpm, to get libstdc++.a

And probably add -lstdc to the Makefile
LIBS := -lsnappy -lstdc++ $(LIBS)

Quoted 68 lines

Maxim.

2013/7/10 HATAYAMA Daisuke <[email protected]>

Quoted 59 lines
Quoted 3 lines · Maxim Uvarov

does crash tool read snappy compressed files?

Yes, but you need to specify libraries at build.

http://people.redhat.com/**anderson/crash.changelog.html<http://people.redhat.com/anderson/crash.changelog.html>

6.0.9 - Fix for building on host machines that have glibc-2.15.90
installed,
<cut>

- Add support for reading compressed kdump dumpfiles that were
compressed by the snappy compressor. This feature is disabled
by
default. To enable this feature, build the crash utility in the
following manner:
(1) Install the snappy libraries by using the host system's
package
manager or by directly downloading libraries from author's
website. The packages required are:
- snappy
- snappy-devel
The author's website is: http://code.google.com/p/**snappy<http://code.google.com/p/snappy>
(2) Create a CFLAGS.extra file and an LDFLAGS.extra file in
top-level
crash sources directory:
- enter -DSNAPPY in the CFLAGS.extra file
- enter -lsnappy in the LDFLAGS.extra file.
(3) Build crash with "make" as always.

6.0.7 - Enhanced the "search" command to allow the searched-for value
<cut>
- Add support to for reading dumpfiles compressed by LZO using makedumpfile version 1.4.4 or later. This feature is disabled
by
default. To enable this feature, build the crash utility in the following manner:
(1) Install the LZO libraries by using the host system's package manager or by directly downloading libraries from author's website. The packages required are:
- lzo
- lzo-minilzo
- lzo-devel
The author's website is: http://www.oberhumer.com/**
opensource/lzo <http://www.oberhumer.com/opensource/lzo>
(2) Create a CFLAGS.extra file and an LDFLAGS.extra file in
top-level
crash sources directory:
- enter -DLZO in the CFLAGS.extra file
- enter -llzo2 in the LDFLAGS.extra file.
(3) Build crash with "make" as always.

--
Thanks.
HATAYAMA, Daisuke

Signature · 2 lines
Best regards,
Maxim Uvarov
Signature · 9 lines
Cliff Wickman
SGI
[email protected]
(651) 683-3824

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#5 lore raw
In reply to Cliff Wickman

On Tue, Jul 09, 2013 at 11:24:03AM -0500, Cliff Wickman wrote:

[..]

Quoted 16 lines

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
booted with crashkernel=1G,high crashkernel=192M,low
non-cyclic mode

write to       option            init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                19          91      11.7G
megaraid disk  zlib compression              20         209       1.4G
megaraid disk  snappy compression            20          46       2.4G
megaraid disk  snappy compression no mmap    30          72       2.4G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              19         206          -
/dev/null      snappy compression            19          41          -
Quoted 3 lines

Notes and observations
- Snappy compression is a big win over zlib compression; over 4 times faster with a cost of relatively little disk space.

Thanks for the results Cliff. If it is not too much of trouble, can you please also test with lzo compression on same configuration. I am curious to know how much better snappy performs as compared to lzo.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw
Quoted 23 lines · Vivek Goyal

On Tue, Jul 09, 2013 at 11:24:03AM -0500, Cliff Wickman wrote:

[..]

Quoted 16 lines

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
booted with crashkernel=1G,high crashkernel=192M,low
non-cyclic mode

write to       option            init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                19          91      11.7G
megaraid disk  zlib compression              20         209       1.4G
megaraid disk  snappy compression            20          46       2.4G
megaraid disk  snappy compression no mmap    30          72       2.4G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              19         206          -
/dev/null      snappy compression            19          41          -
Quoted 3 lines

Notes and observations
- Snappy compression is a big win over zlib compression; over 4 times faster with a cost of relatively little disk space.

Quoted 6 lines

Thanks for the results Cliff. If it is not too much of trouble, can you please also test with lzo compression on same configuration. I am curious to know how much better snappy performs as compared to lzo.

Thanks
Vivek

Ok. I repeated the tests and included LZO compression.

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4 non-cyclic mode
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
3.10 kernel with vmcore mmap patches
booted with crashkernel=1G,high crashkernel=192M,low

write to       compression       init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                20          86      11.6G
megaraid disk  zlib compression              19         209       1.4G
megaraid disk  snappy compression            20          47       2.4G
megaraid disk  lzo compression               19          54       2.8G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              20         206          -
/dev/null      snappy compression            19          42          -
/dev/null      lzo compression               20          47          -

Notes:
- Snappy compression is still be fastest (and more compressed than LZO), but LZO is close.
- Compression and I/O seem pretty well overlapped, so I am not sure that multithreading the crash kernel (to speed compression) will speed the dump as much I was hoping, unless perhaps the I/O device is an SSD.

-Cliff

Signature · 9 lines
Cliff Wickman
SGI
[email protected]
(651) 683-3824

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#7 lore raw
Quoted 51 lines · Cliff Wickman
Quoted 23 lines · Vivek Goyal

On Tue, Jul 09, 2013 at 11:24:03AM -0500, Cliff Wickman wrote:

[..]

Quoted 16 lines

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
booted with crashkernel=1G,high crashkernel=192M,low
non-cyclic mode

write to       option            init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                19          91      11.7G
megaraid disk  zlib compression              20         209       1.4G
megaraid disk  snappy compression            20          46       2.4G
megaraid disk  snappy compression no mmap    30          72       2.4G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              19         206          -
/dev/null      snappy compression            19          41          -
Quoted 3 lines

Notes and observations
- Snappy compression is a big win over zlib compression; over 4 times faster with a cost of relatively little disk space.

Quoted 6 lines

Thanks for the results Cliff. If it is not too much of trouble, can you please also test with lzo compression on same configuration. I am curious to know how much better snappy performs as compared to lzo.

Thanks
Vivek

Ok. I repeated the tests and included LZO compression.

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4 non-cyclic mode
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
3.10 kernel with vmcore mmap patches
booted with crashkernel=1G,high crashkernel=192M,low

write to       compression       init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                20          86      11.6G
megaraid disk  zlib compression              19         209       1.4G
megaraid disk  snappy compression            20          47       2.4G
megaraid disk  lzo compression               19          54       2.8G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              20         206          -
/dev/null      snappy compression            19          42          -
/dev/null      lzo compression               20          47          -
Quoted 6 lines

Notes:
- Snappy compression is still be fastest (and more compressed than LZO), but LZO is close.
- Compression and I/O seem pretty well overlapped, so I am not sure that multithreading the crash kernel (to speed compression) will speed the dump as much I was hoping, unless perhaps the I/O device is an SSD.

Thanks Cliff. So LZO is pretty close to snappy in this case.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw
Quoted 52 lines · Vivek Goyal
Quoted 51 lines · Cliff Wickman
Quoted 23 lines · Vivek Goyal

On Tue, Jul 09, 2013 at 11:24:03AM -0500, Cliff Wickman wrote:

[..]

Quoted 16 lines

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
booted with crashkernel=1G,high crashkernel=192M,low
non-cyclic mode

write to       option            init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                19          91      11.7G
megaraid disk  zlib compression              20         209       1.4G
megaraid disk  snappy compression            20          46       2.4G
megaraid disk  snappy compression no mmap    30          72       2.4G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              19         206          -
/dev/null      snappy compression            19          41          -
Quoted 3 lines

Notes and observations
- Snappy compression is a big win over zlib compression; over 4 times faster with a cost of relatively little disk space.

Quoted 6 lines

Thanks for the results Cliff. If it is not too much of trouble, can you please also test with lzo compression on same configuration. I am curious to know how much better snappy performs as compared to lzo.

Thanks
Vivek

Ok. I repeated the tests and included LZO compression.

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4 non-cyclic mode
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
3.10 kernel with vmcore mmap patches
booted with crashkernel=1G,high crashkernel=192M,low

write to       compression       init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                20          86      11.6G
megaraid disk  zlib compression              19         209       1.4G
megaraid disk  snappy compression            20          47       2.4G
megaraid disk  lzo compression               19          54       2.8G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              20         206          -
/dev/null      snappy compression            19          42          -
/dev/null      lzo compression               20          47          -
Quoted 9 lines
Quoted 6 lines

Notes:
- Snappy compression is still be fastest (and more compressed than LZO), but LZO is close.
- Compression and I/O seem pretty well overlapped, so I am not sure that multithreading the crash kernel (to speed compression) will speed the dump as much I was hoping, unless perhaps the I/O device is an SSD.

Thanks Cliff. So LZO is pretty close to snappy in this case.

This benchmarks lack considering randamized part ratio of data. On my benchmark, LZO was slower than snappy from 50% to 100% randomized.

The attached is a graph of benchmark result that compares LZO and snappy on a variety of ratio of randomized data. The benchmark detail is that

- block size is 4KiB
- sample data is 4MiB
- so 4K blocks in total
- x value is percentage of amount of randomized data
- y value is performance of compression, i.e. 4MiB / (the time consumed for compressing the 4MiB sample data)
- processor is Xeon E7540
- randomizing data is done per a single byte. The 1-byte randomized data is chosen from /dev/urandom. Other part is filled with '\000'.

On this result, LZO remains 100 [MiB/sec] on data whose more than 50 percent is randomized, while snappy shows better performance on more randomized ratio.

On the worst case of this 100 [MiB/sec], 1TiB system memory needs about 3 hours to take crash dump.

While I don't think it's typical case, it's problematic that crash dump requires some more hours depending on contents of memory at crash time. It should always complete in as stable time as possible.

Signature · 2 lines
Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
Quoted 83 lines · HATAYAMA Daisuke
Quoted 52 lines · Vivek Goyal
Quoted 51 lines · Cliff Wickman
Quoted 23 lines · Vivek Goyal

On Tue, Jul 09, 2013 at 11:24:03AM -0500, Cliff Wickman wrote:

[..]

Quoted 16 lines

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
booted with crashkernel=1G,high crashkernel=192M,low
non-cyclic mode

write to       option            init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                19          91      11.7G
megaraid disk  zlib compression              20         209       1.4G
megaraid disk  snappy compression            20          46       2.4G
megaraid disk  snappy compression no mmap    30          72       2.4G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              19         206          -
/dev/null      snappy compression            19          41          -
Quoted 3 lines

Notes and observations
- Snappy compression is a big win over zlib compression; over 4 times faster with a cost of relatively little disk space.

Quoted 6 lines

Thanks for the results Cliff. If it is not too much of trouble, can you please also test with lzo compression on same configuration. I am curious to know how much better snappy performs as compared to lzo.

Thanks
Vivek

Ok. I repeated the tests and included LZO compression.

UV2000 memory: 734G
makedumpfile: makedumpfile-1.5.4 non-cyclic mode
kexec: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
3.10 kernel with vmcore mmap patches
booted with crashkernel=1G,high crashkernel=192M,low

write to       compression       init&scan sec.   copy sec.  dump size
-------------  -----------------           ----   ---------  ---------
megaraid disk  no compression                20          86      11.6G
megaraid disk  zlib compression              19         209       1.4G
megaraid disk  snappy compression            20          47       2.4G
megaraid disk  lzo compression               19          54       2.8G
/dev/null      no compression                19          28          -
/dev/null      zlib compression              20         206          -
/dev/null      snappy compression            19          42          -
/dev/null      lzo compression               20          47          -
Quoted 9 lines
Quoted 6 lines

Notes:
- Snappy compression is still be fastest (and more compressed than LZO), but LZO is close.
- Compression and I/O seem pretty well overlapped, so I am not sure that multithreading the crash kernel (to speed compression) will speed the dump as much I was hoping, unless perhaps the I/O device is an SSD.

Thanks Cliff. So LZO is pretty close to snappy in this case.

This benchmarks lack considering randamized part ratio of data. On my benchmark, LZO was slower than snappy from 50% to 100% randomized.

The attached is a graph of benchmark result that compares LZO and snappy on a variety of ratio of randomized data. The benchmark detail is that

- block size is 4KiB
- sample data is 4MiB
- so 4K blocks in total
- x value is percentage of amount of randomized data
- y value is performance of compression, i.e. 4MiB / (the time consumed for compressing the 4MiB sample data)
- processor is Xeon E7540
- randomizing data is done per a single byte. The 1-byte randomized data is chosen from /dev/urandom. Other part is filled with '\000'.

On this result, LZO remains 100 [MiB/sec] on data whose more than 50 percent is randomized, while snappy shows better performance on more randomized ratio.

Quoted 6 lines

On the worst case of this 100 [MiB/sec], 1TiB system memory needs about 3 hours to take crash dump.

While I don't think it's typical case, it's problematic that crash dump requires some more hours depending on contents of memory at crash time. It should always complete in as stable time as possible.

As per your performance graphs, both lzo and snappy vary in performance based on randomized data in the system. So that means total dump time will vary based on contents in memory at crash time (until and unless there is a fast compression algorithm which does not get impacted much due to randomness of data). So being able to dump in constant time irresepctive of randomness of data in memory is probably not the goal here.

Instead being able to dump faster in most of the scenarios is the goal. And your graph does show that snappy performs much better at higher ranomness ratios.

So based on your graph, I agree that lzo is not a replacement for snappy and snappy can be much faster depending on randomness of data.

Thanks
Vivek

be faster in

Quoted 4 lines
Signature · 3 lines
-- 
Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec