makedumpfile: stream compress flat ELF format with libz
Subsystems (2)
Hi,
I created an enhancement request for makedumpfile here,
https://github.com/makedumpfile/makedumpfile/issues/1
I found that compressing a flat core with gzip significantly reduces the size of the core. Here were my findings,
32G flat elf core -E -F -d 0
33G kdump core -d 0
16G kdump compressed -c -d 0
1.9G flat elf core stream compressed with gzip -E -F -d 0
My feature request was to implement an option inside makedumpfile to gzip compress the core output. This can already be accomplished by piping the core thru the gzip tool, however, because makedumpfile already links against libz having the option for makedumpfile to do it allows the gzip tools to be removed from the crash kernels initramfs there by reducing the size.
Kazuhito Hagio had suggest adding the -C option instead of repurposing -c to do this.
So a resulting command line might looks like this,
makedumpfile -C -F -E -d 31 /proc/vmcore core.gz
Thanks,
Daniel
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 19 lines · Daniel Walker (danielwa)
Hi,
I created an enhancement request for makedumpfile here,
I found that compressing a flat core with gzip significantly reduces the size of the core. Here were my findings,
32G flat elf core -E -F -d 0
33G kdump core -d 0
16G kdump compressed -c -d 0
1.9G flat elf core stream compressed with gzip -E -F -d 0
Quoted 9 lines
My feature request was to implement an option inside makedumpfile to gzip compress the core output. This can already be accomplished by piping the core thru the gzip tool, however, because makedumpfile already links against libz having the option for makedumpfile to do it allows the gzip tools to be removed from the crash kernels initramfs there by reducing the size.
Kazuhito Hagio had suggest adding the -C option instead of repurposing -c to do this.
Hi Daniel.
-z happens to be used by tar and rsync to indicate compression ;-) .
Quoted 10 lines
So a resulting command line might looks like this,
makedumpfile -C -F -E -d 31
/proc/vmcorecore.gzThanks,
Daniel
_______________________________________________
kexec mailing list
[email protected]
https://urldefense.com/v3/__http://lists.infradead.org/mailman/listinfo/kexec__;!!GqivPVa7Brio!LJIWfQ8ged-9RjjV00zqmBGbL2-UU0baDJmxqVXo5AxYcofHzP8oxfWHZZ4ijsPXKz5A$
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 6 lines · John Donnelly
Hi Daniel.
-z happens to be used by tar and rsync to indicate compression ;-) .
This would be fine with me,
makedumpfile -z -F -E -d 31 /proc/vmcore core.gz
Daniel
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
It's a quite interesting feature Daniel, thanks for the effort! I'm curious about memory usage - did you somehow measure the consumption with your patches vs. the regular kdump compression?
Thanks,
Guilherme
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 3 lines · Guilherme Piccoli
It's a quite interesting feature Daniel, thanks for the effort! I'm curious about memory usage - did you somehow measure the consumption with your patches vs. the regular kdump compression?
It's not really a memory consumption issue, it's a core size issue. The core size is much smaller when compressed with gzip. I measured the core size with different formats and gzip compressed.
Daniel
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Sure, I understood that. I'm curious though about the memory consumption of the compression operation, or put in other words: do we need to increase crashkernel reserved memory if we plan to use your gzip approach?
It's good to be sure about this, to evaluate the trade-off of core file size vs. necessary reserved memory.
Thanks!
Quoted 11 lines · Daniel Walker (danielwa)
Quoted 3 lines · Guilherme Piccoli
It's a quite interesting feature Daniel, thanks for the effort! I'm curious about memory usage - did you somehow measure the consumption with your patches vs. the regular kdump compression?
It's not really a memory consumption issue, it's a core size issue. The core size is much smaller when compressed with gzip. I measured the core size with different formats and gzip compressed.
Daniel
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 6 lines · Guilherme Piccoli
Sure, I understood that. I'm curious though about the memory consumption of the compression operation, or put in other words: do we need to increase crashkernel reserved memory if we plan to use your gzip approach?
It's good to be sure about this, to evaluate the trade-off of core file size vs. necessary reserved memory.
I'm not sure. I didn't measure this, but from my experience makedumpfile uses more memory during dumping than gzip uses for compression. If our dump kernels run out of memory it's usually something inside makedumpfile when it's removing pages or something along those lines.
Daniel
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines
-----Original Message-----
Hi,
I created an enhancement request for makedumpfile here,
Quoted 7 lines
I found that compressing a flat core with gzip significantly reduces the size of the core. Here were my findings,
32G flat elf core -E -F -d 0
33G kdump core -d 0
16G kdump compressed -c -d 0
1.9G flat elf core stream compressed with gzip -E -F -d 0
The kdump-compressed format does per-page compression with compression level 1, so depending on data in vmcore, gzip (with -6 by default) can compress one much smaller like this. Although I don't have a vmcore that shows such a drop in size with gzip as far as I tested.
Quoted 6 lines
My feature request was to implement an option inside makedumpfile to gzip compress the core output. This can already be accomplished by piping the core thru the gzip tool, however, because makedumpfile already links against libz having the option for makedumpfile to do it allows the gzip tools to be removed from the crash kernels initramfs there by reducing the size.
(personally I prefer piping it through gzip, but apparently there are cases where the initramfs size needs to be minimized..)
Quoted 7 lines
Kazuhito Hagio had suggest adding the -C option instead of repurposing -c to do this.
So a resulting command line might looks like this,
makedumpfile -C -F -E -d 31
/proc/vmcorecore.gz
Yes, the existing -c option is for per-page compression and creates a dumpfile that can be opened directly with crash utility, but stream compression is different. So I'd like to separate it from the -c.
(as John commented, -z also might be good, but I wonder if someone might start to want the same function with lzo or snappy..)
Anyway, I'll wait for patches for this enhancement.
Thanks,
Kazu
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec