Reducing the size of the dump file/speeding up collection
Subsystems (2)
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Please advise
Regards,
Nikolay
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 13 lines · Nikolay Borisov
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Quoted 1 line
Please advise
Hi nikolay,
Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.
And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:
https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
Signature · 2 lines
Regards Qiao Nuohan
Quoted 9 lines
Regards,
Nikolay_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi Qiao,
Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:
- Reading from /proc/vmcore - that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?
Regards,
Nikolay
Quoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Quoted 1 line
Please advise
Hi nikolay,
Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.
Quoted 7 lines
And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:
https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Just to follow up: which version of kexec/makedumpfile are required to have the parallel dump feature? The thread you referenced adds documentation how to use that and I grepped the source of makedumpfile 1.5.8 and couldn't find any references to the num-threads option? So what version of the code do I need to test that ?
Quoted 13 lines · Nikolay Borisov
Hi Qiao,
Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:
- Reading from
/proc/vmcore- that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?
Quoted 33 lines
Regards,
NikolayQuoted 29 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Quoted 1 line
Please advise
Hi nikolay,
Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.
And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:
https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines · Nikolay Borisov
Hi Qiao,
Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:
Quoted 4 lines
- Reading from
/proc/vmcore- that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?
I cannot recall percentage of each part. But writing/compression takes most of the time
1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently committed patch, so you cannot find it in the master branch, checkout devel branch or find it here:
http://sourceforge.net/p/makedumpfile/code/commit_browser
Make makedumpfile available to read and compress pages parallelly.
Quoted 36 lines
Regards,
NikolayQuoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Quoted 1 line
Please advise
Hi nikolay,
Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.
Quoted 7 lines
And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:
https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
.
Signature · 7 lines
Regards Qiao Nuohan _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Yeah, I did see the commit browser. But in my case I haven't even tested the split option so I guess there are things to try. Am I correct in my understanding as to how --split is supposed to work ( i tried that to no avail though):
My core_collector line is this:
core_collector makedumpfile --message-level 1 -d 3 --split dump1 dump2 dump3 dump4 dump5 dump6
And then in /etc/sysconfig/kdump I have:
KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6 reset_devices cgroup_disable=memory mce=off"
(the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical cores in total). Do I need to do something else to utilize the --split option?
Quoted 28 lines · qiaonuohan
Quoted 9 lines · Nikolay Borisov
Hi Qiao,
Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:
Quoted 4 lines
- Reading from
/proc/vmcore- that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?I cannot recall percentage of each part. But writing/compression takes most of the time
1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing
and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently
committed patch, so you cannot find it in the master branch, checkout
devel branch
or find it here:
Quoted 43 lines
http://sourceforge.net/p/makedumpfile/code/commit_browser
Make makedumpfile available to read and compress pages parallelly.
Quoted 37 lines
Regards,
NikolayQuoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Quoted 1 line
Please advise
Hi nikolay,
Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.
Quoted 8 lines
And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:
https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 10 lines · Nikolay Borisov
Yeah, I did see the commit browser. But in my case I haven't even tested the split option so I guess there are things to try. Am I correct in my understanding as to how --split is supposed to work ( i tried that to no avail though):
My
core_collectorline is this:
core_collectormakedumpfile --message-level 1 -d 3 --split dump1 dump2 dump3 dump4 dump5 dump6
Quoted 8 lines
And then in
/etc/sysconfig/kdumpI have:KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6
reset_devicescgroup_disable=memory mce=off"(the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical cores in total). Do I need to do something else to utilize the --split option?
As I recall, it is OK to go on the test.
Quoted 75 lines · qiaonuohan
Quoted 28 lines · Nikolay Borisov
Quoted 9 lines
Hi Qiao,
Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:
Quoted 4 lines
- Reading from
/proc/vmcore- that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?I cannot recall percentage of each part. But writing/compression takes most of the time
1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing
and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently
committed patch, so you cannot find it in the master branch, checkout
devel branch
or find it here:Quoted 43 lines
http://sourceforge.net/p/makedumpfile/code/commit_browser
Make makedumpfile available to read and compress pages parallelly.
Quoted 37 lines
Regards,
NikolayQuoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Quoted 1 line
Please advise
Hi nikolay,
Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.
Quoted 8 lines
And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:
https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
.
.
Signature · 7 lines
Regards Qiao Nuohan _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 7 lines · Nikolay Borisov
Yeah, I did see the commit browser. But in my case I haven't even tested the split option so I guess there are things to try. Am I correct in my understanding as to how --split is supposed to work ( i tried that to no avail though):
My
core_collectorline is this:
Quoted 7 lines
core_collectormakedumpfile --message-level 1 -d 3 --split dump1 dump2 dump3 dump4 dump5 dump6And then in
/etc/sysconfig/kdumpI have:KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6
reset_devicescgroup_disable=memory mce=off"
disable_cpu_apicid need be specified if nr_cpus > 1. Otherwise it could be hang.
Quoted 78 lines
(the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical cores in total). Do I need to do something else to utilize the --split option?
Quoted 28 lines · qiaonuohan
Quoted 9 lines · Nikolay Borisov
Hi Qiao,
Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:
Quoted 4 lines
- Reading from
/proc/vmcore- that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?I cannot recall percentage of each part. But writing/compression takes most of the time
1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing
and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently
committed patch, so you cannot find it in the master branch, checkout
devel branch
or find it here:Quoted 43 lines
http://sourceforge.net/p/makedumpfile/code/commit_browser
Make makedumpfile available to read and compress pages parallelly.
Quoted 37 lines
Regards,
NikolayQuoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov
Hello,
I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.
Quoted 1 line
Please advise
Hi nikolay,
Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.
Quoted 8 lines
And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:
https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
.
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec