Skip to contentSkip to conversation

Reducing the size of the dump file/speeding up collection

DISCUSSION Nikolay Borisov 3 participants 8 messages Sep 16, 2015 – Sep 23, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Please advise

Regards,
Nikolay

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw
Quoted 13 lines · Nikolay Borisov

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Quoted 1 line

Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.

And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html

Signature · 2 lines
Regards
Qiao Nuohan
Quoted 9 lines

Regards,
Nikolay

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

Hi Qiao,

Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:

- Reading from /proc/vmcore - that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?

Regards,
Nikolay

Quoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Quoted 1 line

Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.

Quoted 7 lines

And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

Just to follow up: which version of kexec/makedumpfile are required to have the parallel dump feature? The thread you referenced adds documentation how to use that and I grepped the source of makedumpfile 1.5.8 and couldn't find any references to the num-threads option? So what version of the code do I need to test that ?

Quoted 13 lines · Nikolay Borisov

Hi Qiao,

Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:

- Reading from /proc/vmcore - that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?

Quoted 33 lines

Regards,
Nikolay

Quoted 29 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Quoted 1 line

Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.

And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw
In reply to Nikolay Borisov
Quoted 8 lines · Nikolay Borisov

Hi Qiao,

Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:

Quoted 4 lines

- Reading from /proc/vmcore - that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?

I cannot recall percentage of each part. But writing/compression takes most of the time

1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently committed patch, so you cannot find it in the master branch, checkout devel branch or find it here:

http://sourceforge.net/p/makedumpfile/code/commit_browser

Make makedumpfile available to read and compress pages parallelly.

Quoted 36 lines

Regards,
Nikolay

Quoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Quoted 1 line

Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.

Quoted 7 lines

And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html

.

Signature · 7 lines
Regards
Qiao Nuohan

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#5 lore raw

Yeah, I did see the commit browser. But in my case I haven't even tested the split option so I guess there are things to try. Am I correct in my understanding as to how --split is supposed to work ( i tried that to no avail though):

My core_collector line is this:

core_collector makedumpfile --message-level 1 -d 3 --split dump1 dump2 dump3 dump4 dump5 dump6

And then in /etc/sysconfig/kdump I have:

KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6 reset_devices cgroup_disable=memory mce=off"

(the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical cores in total). Do I need to do something else to utilize the --split option?

Quoted 28 lines · qiaonuohan
Quoted 9 lines · Nikolay Borisov

Hi Qiao,

Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:

Quoted 4 lines

- Reading from /proc/vmcore - that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?

I cannot recall percentage of each part. But writing/compression takes most of the time

1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing
and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently
committed patch, so you cannot find it in the master branch, checkout
devel branch
or find it here:

Quoted 43 lines

http://sourceforge.net/p/makedumpfile/code/commit_browser

Make makedumpfile available to read and compress pages parallelly.

Quoted 37 lines

Regards,
Nikolay

Quoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Quoted 1 line

Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.

Quoted 8 lines

And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw
Quoted 10 lines · Nikolay Borisov

Yeah, I did see the commit browser. But in my case I haven't even tested the split option so I guess there are things to try. Am I correct in my understanding as to how --split is supposed to work ( i tried that to no avail though):

My core_collector line is this:

core_collector makedumpfile --message-level 1 -d 3 --split dump1 dump2 dump3 dump4 dump5 dump6

Quoted 8 lines

And then in /etc/sysconfig/kdump I have:

KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6 reset_devices cgroup_disable=memory mce=off"

(the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical cores in total). Do I need to do something else to utilize the --split option?

As I recall, it is OK to go on the test.

Quoted 75 lines · qiaonuohan
Quoted 28 lines · Nikolay Borisov
Quoted 9 lines

Hi Qiao,

Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:

Quoted 4 lines

- Reading from /proc/vmcore - that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?

I cannot recall percentage of each part. But writing/compression takes most of the time

1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing
and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently
committed patch, so you cannot find it in the master branch, checkout
devel branch
or find it here:

Quoted 43 lines

http://sourceforge.net/p/makedumpfile/code/commit_browser

Make makedumpfile available to read and compress pages parallelly.

Quoted 37 lines

Regards,
Nikolay

Quoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Quoted 1 line

Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.

Quoted 8 lines

And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html

.

.

Signature · 7 lines
Regards
Qiao Nuohan

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#7 lore raw
In reply to Nikolay Borisov
Quoted 7 lines · Nikolay Borisov

Yeah, I did see the commit browser. But in my case I haven't even tested the split option so I guess there are things to try. Am I correct in my understanding as to how --split is supposed to work ( i tried that to no avail though):

My core_collector line is this:

Quoted 7 lines

core_collector makedumpfile --message-level 1 -d 3 --split dump1 dump2 dump3 dump4 dump5 dump6

And then in /etc/sysconfig/kdump I have:

KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6 reset_devices cgroup_disable=memory mce=off"

disable_cpu_apicid need be specified if nr_cpus > 1. Otherwise it could be hang.

Quoted 78 lines

(the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical cores in total). Do I need to do something else to utilize the --split option?

Quoted 28 lines · qiaonuohan
Quoted 9 lines · Nikolay Borisov

Hi Qiao,

Thanks for the reply. So far I haven't been using the the compression feature of makedumpfile. But I want to ask if anything wouldn't compression make the dump process slower since in addition to having to write the dump to disk it also has to compress it which would put more strain on the cpu. Also, which part of the dump process is the bottleneck:

Quoted 4 lines

- Reading from /proc/vmcore - that has mmap support so should be fairly fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?

I cannot recall percentage of each part. But writing/compression takes most of the time

1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing
and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently
committed patch, so you cannot find it in the master branch, checkout
devel branch
or find it here:

Quoted 43 lines

http://sourceforge.net/p/makedumpfile/code/commit_browser

Make makedumpfile available to read and compress pages parallelly.

Quoted 37 lines

Regards,
Nikolay

Quoted 22 lines · qiaonuohan
Quoted 13 lines · Nikolay Borisov

Hello,

I've been using makedumpfile as the crash collector with the -d31 parameter. The machine this is being run on usually have 128-256GB of ram and the resulting crash dumps are in the range of 14-20gb which is very bug for the type of analysis I'm usually performing on crashed machine. I was wondering whether there is a way to further reduce the size and the time to take the dump (now it takes around 25 minutes to collect such a dump). I've seen reports where people with TBs of ram take that long, meaning for a machine with 256gb it should be even faster. I've been running this configuration on kernels 3.12.28 and 4.1 where mmap for the vmcore file is supported.

Quoted 1 line

Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot. About the current situation, try --split, it will save time.

Quoted 8 lines

And lzo/snappy instead of zlib, these two compression format are faster but need more space to save. Or if you still want zlib (to save space), try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html

.

Quoted 4 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec