makedumpfile memory usage grows with system memory size

I was talking to Vivek about kdump memory requirements and he mentioned that they vary based on how much system memory is used.

I was interested in knowing why that was and again he mentioned that makedumpfile needed lots of memory if it was running on a large machine (for example 1TB of system memory).

Looking through the makedumpfile README and using what Vivek remembered of makedumpfile, we gathered that as the number of pages grows, the more makedumpfile has to temporarily store the information in memory. The possible reason was to calculate the size of the file before it was copied to its final destination?

I was curious if that was true and if it was, would it be possible to only process memory in chunks instead of all at once.

The idea is that a machine with 4Gigs of memory should consume the same the amount of kdump runtime memory as a 1TB memory system.

Just trying to research ways to keep the memory requirements consistent across all memory ranges.

On Wed, 28 Mar 2012 17:22:04 -0400
Don Zickus <[email protected]> wrote:

Quoted 8 lines

I was curious if that was true and if it was, would it be possible to only process memory in chunks instead of all at once.

The idea is that a machine with 4Gigs of memory should consume the same the amount of kdump runtime memory as a 1TB memory system.

Just trying to research ways to keep the memory requirements consistent across all memory ranges.

Hi Don-san,

Quoted 8 lines · On Wed, 28 Mar 2012 17:22:04 -0400 Don Zickus <[email protected]> wrote:

I was talking to Vivek about kdump memory requirements and he mentioned that they vary based on how much system memory is used.

I was interested in knowing why that was and again he mentioned that makedumpfile needed lots of memory if it was running on a large machine (for example 1TB of system memory).

Quoted 5 lines

Looking through the makedumpfile README and using what Vivek remembered of makedumpfile, we gathered that as the number of pages grows, the more makedumpfile has to temporarily store the information in memory. The possible reason was to calculate the size of the file before it was copied to its final destination?

makedumpfile uses the system memory of 2nd-kernel for a bitmap if RHEL. The bitmap represents each page of 1st-kernel is excluded or not. So the bitmap size depends on 1st-kernel's system memory.

makedumpfile creates a file /tmp/kdump_bitmapXXXXXX as the bitmap, and the file is created on 2nd-kernel's memory if RHEL, because RHEL does not mount a root filesystem when 2nd-kernel is running.

Quoted 8 lines

I was curious if that was true and if it was, would it be possible to only process memory in chunks instead of all at once.

The idea is that a machine with 4Gigs of memory should consume the same the amount of kdump runtime memory as a 1TB memory system.

Quoted 2 lines

Just trying to research ways to keep the memory requirements consistent across all memory ranges.

I think the above purpose is good, and I don't have any idea for reducing the bitmap size. And now I am out of makedumpfile development. Kumagai-san is the makedumpfile maintainer now, and he will help you.

Hello Don,

I'm missing your mail somehow so replying Oomichi-san's mail...

From: "Ken'ichi Ohmichi" <[email protected]>
Subject: Re: makedumpfile memory usage grows with system memory size
Date: Thu, 29 Mar 2012 17:09:18 +0900

Quoted 10 lines

Quoted 6 lines · On Wed, 28 Mar 2012 17:22:04 -0400 Don Zickus <[email protected]> wrote:

I was curious if that was true and if it was, would it be possible to only process memory in chunks instead of all at once.

The idea is that a machine with 4Gigs of memory should consume the same the amount of kdump runtime memory as a 1TB memory system.

Quoted 2 lines

Quoted 2 lines

Just trying to research ways to keep the memory requirements consistent across all memory ranges.

I think this is possible in constant memory space by creating bitmaps and writing pages in a certain amount of memory. That is, if choosing 4GB, do [0, 4GB) space processing, [4GB, 8GB) space processing, [8GB, 12GB) ... in order. The key is to restrict the target memory range of filtering.

Quoted 14 lines

From: "Ken'ichi Ohmichi" <[email protected]>
Subject: Re: makedumpfile memory usage grows with system memory size
Date: Thu, 29 Mar 2012 17:09:18 +0900

Quoted 9 lines

Quoted 6 lines · On Wed, 28 Mar 2012 17:22:04 -0400 Don Zickus <[email protected]> wrote:

I was curious if that was true and if it was, would it be possible to only process memory in chunks instead of all at once.

The idea is that a machine with 4Gigs of memory should consume the same the amount of kdump runtime memory as a 1TB memory system.

Quoted 8 lines

Quoted 2 lines

Quoted 2 lines

Just trying to research ways to keep the memory requirements consistent across all memory ranges.

I think this is possible in constant memory space by creating bitmaps and writing pages in a certain amount of memory. That is, if choosing 4GB, do [0, 4GB) space processing, [4GB, 8GB) space processing, [8GB, 12GB) ... in order. The key is to restrict the target memory range of filtering.

Yes, that was what I was thinking. I am glad to hear that is possible. Is there some place in the code that I can help try out that idea? I would also be curious if there is a 'time' impact on how long it takes to process this (for example, would it add a couple of milliseconds overhead or seconds overhead).

Quoted 29 lines

Quoted 26 lines · On Thu, 29 Mar 2012 09:05:14 -0400 Don Zickus <[email protected]> wrote:

Hi Ken'ichi-san,

Quoted 23 lines · Ken'ichi Ohmichi

Hi Don-san,

Quoted 8 lines · On Wed, 28 Mar 2012 17:22:04 -0400 Don Zickus <[email protected]> wrote:

I was talking to Vivek about kdump memory requirements and he mentioned that they vary based on how much system memory is used.

I was interested in knowing why that was and again he mentioned that makedumpfile needed lots of memory if it was running on a large machine (for example 1TB of system memory).

Quoted 5 lines

Looking through the makedumpfile README and using what Vivek remembered of makedumpfile, we gathered that as the number of pages grows, the more makedumpfile has to temporarily store the information in memory. The possible reason was to calculate the size of the file before it was copied to its final destination?

makedumpfile uses the system memory of 2nd-kernel for a bitmap if RHEL. The bitmap represents each page of 1st-kernel is excluded or not. So the bitmap size depends on 1st-kernel's system memory.

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

makedumpfile creates a file /tmp/kdump_bitmapXXXXXX as the bitmap, and the file is created on 2nd-kernel's memory if RHEL, because RHEL does not mount a root filesystem when 2nd-kernel is running.

Ok.

Hello Don,
Does setting TMPDIR solve your problem ? Please refer to the man page.

ENVIRONMENT VARIABLES
TMPDIR This environment variable is for a temporary memory bitmap
file. If your machine has a lots of memory and you use tmpfs on /tmp, makedumpfile can fail for a little memory in the 2nd kernel because makedumpfile makes a very large temporary memory bitmap file in this case. To avoid this failure, you can set a TMPDIR environment variable. If you do not set a TMPDIR envi- ronment variable, makedumpfile uses /tmp directory for a tempo- rary bitmap file as a default.

On the other hand, I'm considering the enhancement suggested by Hatayama-san now.

For the processing of writing pages per range of memory, it's useful to reuse the code for --split's splitting features that split a single dumpfile into a multiple dumpfiles, which has prepared data strucutre to have start and end page frame numbers of the corresponding dumped memory. For example, see part below in write_kdump_pages().

        if (info->flag_split) {
                start_pfn = info->split_start_pfn;
                end_pfn   = info->split_end_pfn;
        }
        else {
                start_pfn = 0;
                end_pfn   = info->max_mapnr;
        }

        for (pfn = start_pfn; pfn < end_pfn; pfn++) {

For the processing of creating and referencing bitmaps per range of memory, there's no functions that do that. The ones for a whole memory
only: create_bitmap() and is_dumpable(). Also, creating bitmap depends on source dumpfile format. Trying with ELF to kdump-compressed format case first seems most handy; or if usecase is on the 2nd kernel only, this case is enough?)

For performance impact, I don't know that exactly. But I guess iterating filtering processing is most significant. I don't know exact data structure for each kind of memory, but if there's the ones needing linear order to look up the data for a given page frame number, there would be necessary to add some special handling not to reduce performance.

On Wed, 28 Mar 2012 17:22:04 -0400
makedumpfile uses the system memory of 2nd-kernel for a bitmap if RHEL. The bitmap represents each page of 1st-kernel is excluded or not. So the bitmap size depends on 1st-kernel's system memory.

Quoted 6 lines · On Fri, 30 Mar 2012 09:51:43 +0900 ( ) HATAYAMA Daisuke…

For performance impact, I don't know that exactly. But I guess iterating filtering processing is most significant. I don't know exact data structure for each kind of memory, but if there's the ones needing linear order to look up the data for a given page frame number, there would be necessary to add some special handling not to reduce performance.

Thank you for your idea.

I think this is an important issue and I have no idea except iterating filtering processes for each memory range.

But as you said, we should consider the issue related to performance. For example, makedumpfile must parse free_list repeatedly to distinguish whether each pfn is a free page or not, because each range may be inside the same zone. It will be overhead.

* Bad performance is free pages only. Cache, cache private, user and zero pages are processed per range of memory in good performance.

[..]

Quoted 2 lines

* Bad performance is free pages only. Cache, cache private, user and zero pages are processed per range of memory in good performance.

Hi Daisuke-san,

I am wondering why can't we walk through the memmap array and look into struct page for figuring out if page is free or not. Looks like that in the past we used to have PG_buddy flag and same information possibly could be retrieved by looking at page->_count field.

So I am just curious that why do we walk through free pages list to figure out free pages instead of looking at "struct page".

Hello Ken'ichi,

Quoted 4 lines · Ken'ichi Ohmichi

On Wed, 28 Mar 2012 17:22:04 -0400
makedumpfile uses the system memory of 2nd-kernel for a bitmap if RHEL. The bitmap represents each page of 1st-kernel is excluded or not. So the bitmap size depends on 1st-kernel's system memory.

Does this mean that makedumpfile's memory demand linearly grows with 1 bit per page of 1-st kernel's memory?

Is that the exact factor, if /tmp is in memory? Or is there any other memory allocation that is not constant regarding the 1-st kernel memory size?

Thanks. To be honest, I have just beginning with reading around here and known PG_buddy just now. I have small checked this fact on 2.6.18 with the patch in the bottom of this mail and free pages found from free_list and by PG_buddy check are coincide.

As Vivek says, more recent kernel has change around PG_buddy and the patch says we should check _mapcount; I have yet to check this.

Author: Andrea Arcangeli <[email protected]>
Date:   Thu Jan 13 15:47:00 2011 -0800

thp: remove PG_buddy

PG_buddy can be converted to _mapcount == -2. So the PG_compound_lock can be added to page->flags without overflowing (because of the sparse section bits increasing) with CONFIG_X86_PAE=y and CONFIG_X86_PAT=y. This also has to move the memory hotplug code from _mapcount to lru.next to avoid any risk of clashes. We can't use lru.next for PG_buddy removal, but memory hotplug can use lru.next even more easily than the mapcount instead.

Signed-off-by Andrea Arcangeli <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

Signed-off-by Linus Torvalds <[email protected]>

$ git describe 5f24ce5fd34c3ca1b3d10d30da754732da64d5c0
v2.6.37-7012-g5f24ce5

So now we can walk on the memmap array also for free pages like other kinds of memory. The question I have now is why the current implementation was chosen. Is there any difference between two ways?

Hello Hatayama-san,

Quoted 25 lines · On Fri, 06 Apr 2012 10:12:12 +0900 ( ) HATAYAMA Daisuke…

Thanks. To be honest, I have just beginning with reading around here and known PG_buddy just now. I have small checked this fact on 2.6.18 with the patch in the bottom of this mail and free pages found from free_list and by PG_buddy check are coincide.

As Vivek says, more recent kernel has change around PG_buddy and the patch says we should check _mapcount; I have yet to check this.

Author: Andrea Arcangeli <[email protected]>
Date:   Thu Jan 13 15:47:00 2011 -0800

thp: remove PG_buddy

PG_buddy can be converted to _mapcount == -2. So the PG_compound_lock can be added to page->flags without overflowing (because of the sparse section bits increasing) with CONFIG_X86_PAE=y and CONFIG_X86_PAT=y. This also has to move the memory hotplug code from _mapcount to lru.next to avoid any risk of clashes. We can't use lru.next for PG_buddy removal, but memory hotplug can use lru.next even more easily than the mapcount instead.

Signed-off-by Andrea Arcangeli <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

Signed-off-by Linus Torvalds <[email protected]>

Quoted 6 lines

$ git describe 5f24ce5fd34c3ca1b3d10d30da754732da64d5c0
v2.6.37-7012-g5f24ce5

So now we can walk on the memmap array also for free pages like other kinds of memory. The question I have now is why the current implementation was chosen. Is there any difference between two ways?

We just referred to the implementation of disk_dump.

Now, I'm checking the validity of using _count field to figure out free pages. I would like to use _count rather than PG_buddy because I would like to avoid changing behavior based on versions as long as possible.

So now we can walk on the memmap array also for free pages like other kinds of memory. The question I have now is why the current implementation was chosen. Is there any difference between two ways?

I agree. On the other hand, there is one more thing to consider. The value of order is in private member of the page descripter. Now there's no information for private member in VMCOREINFO. If we choose this method and delete the current one, it's necessary to prepare vmlinux file for old kernels.

[..]

Quoted 5 lines

I agree. On the other hand, there is one more thing to consider. The value of order is in private member of the page descripter. Now there's no information for private member in VMCOREINFO. If we choose this method and delete the current one, it's necessary to prepare vmlinux file for old kernels.

What information do you need to access "private" member of "struct page". offset? Can't we extend VMCOREINFO to export this info too?

From: Vivek Goyal <[email protected]>
Subject: Re: makedumpfile memory usage grows with system memory size
Date: Mon, 9 Apr 2012 14:57:28 -0400

Quoted 12 lines · HATAYAMA Daisuke

[..]

Quoted 5 lines

I agree. On the other hand, there is one more thing to consider. The value of order is in private member of the page descripter. Now there's no information for private member in VMCOREINFO. If we choose this method and delete the current one, it's necessary to prepare vmlinux file for old kernels.

What information do you need to access "private" member of "struct page". offset? Can't we extend VMCOREINFO to export this info too?

Yes, I mean offset of private member in page structure. The member contains order of the buddy. Extending VMCOREINFO is easy, but we cannot do that for old kernels, for which vmlinux is needed separately.

This might be the same as what Kumagai-san says he doesn' want to change behaviour on kernel versions.

Hello Michael,

Quoted 10 lines · On Mon, 02 Apr 2012 19:15:33 +0200 Michael Holzheu…

Hello Ken'ichi,

Quoted 4 lines · Ken'ichi Ohmichi

On Wed, 28 Mar 2012 17:22:04 -0400
makedumpfile uses the system memory of 2nd-kernel for a bitmap if RHEL. The bitmap represents each page of 1st-kernel is excluded or not. So the bitmap size depends on 1st-kernel's system memory.

Does this mean that makedumpfile's memory demand linearly grows with 1 bit per page of 1-st kernel's memory?

Yes, you are right. (Precisely, 2 bit per page.)

Quoted 3 lines

Is that the exact factor, if /tmp is in memory? Or is there any other memory allocation that is not constant regarding the 1-st kernel memory size?

bitmap file is main cause of memory consuming if 2nd kernel uses initramfs only. There are other parts where the size of allocated memory varies based on 1-st kernel memory size, but they don't have big influence.

Quoted 17 lines · HATAYAMA Daisuke

From: Vivek Goyal <[email protected]>
Subject: Re: makedumpfile memory usage grows with system memory size
Date: Mon, 9 Apr 2012 14:57:28 -0400

Quoted 12 lines · HATAYAMA Daisuke

[..]

Quoted 5 lines

I agree. On the other hand, there is one more thing to consider. The value of order is in private member of the page descripter. Now there's no information for private member in VMCOREINFO. If we choose this method and delete the current one, it's necessary to prepare vmlinux file for old kernels.

What information do you need to access "private" member of "struct page". offset? Can't we extend VMCOREINFO to export this info too?

Quoted 7 lines

Yes, I mean offset of private member in page structure. The member contains order of the buddy. Extending VMCOREINFO is easy, but we cannot do that for old kernels, for which vmlinux is needed separately.

This might be the same as what Kumagai-san says he doesn' want to change behaviour on kernel versions.

We can retain both the mechanisms. For newer kernels which export page->private offset, we can walk through memmap array and prepare a chunk of bitmap and discard it. For older kernels we can continue to walk through free pages list and prepare big bitmap in userspace.

It is desirable to keep mechanism same across kernel versions, but change is unavoidable as things evolve in newer kernels. So at max we can provide backward compatibility with old kernels.

Hello,

Quoted 26 lines · On Tue, 10 Apr 2012 08:52:05 -0400 Vivek Goyal <[email protected]> wrote:

Quoted 17 lines · HATAYAMA Daisuke

From: Vivek Goyal <[email protected]>
Subject: Re: makedumpfile memory usage grows with system memory size
Date: Mon, 9 Apr 2012 14:57:28 -0400

Quoted 12 lines · HATAYAMA Daisuke

[..]

Quoted 5 lines

I agree. On the other hand, there is one more thing to consider. The value of order is in private member of the page descripter. Now there's no information for private member in VMCOREINFO. If we choose this method and delete the current one, it's necessary to prepare vmlinux file for old kernels.

What information do you need to access "private" member of "struct page". offset? Can't we extend VMCOREINFO to export this info too?

Quoted 7 lines

Yes, I mean offset of private member in page structure. The member contains order of the buddy. Extending VMCOREINFO is easy, but we cannot do that for old kernels, for which vmlinux is needed separately.

This might be the same as what Kumagai-san says he doesn' want to change behaviour on kernel versions.

Quoted 8 lines

We can retain both the mechanisms. For newer kernels which export page->private offset, we can walk through memmap array and prepare a chunk of bitmap and discard it. For older kernels we can continue to walk through free pages list and prepare big bitmap in userspace.

It is desirable to keep mechanism same across kernel versions, but change is unavoidable as things evolve in newer kernels. So at max we can provide backward compatibility with old kernels.

I said I want to avoid changing behavior based on kernel versions, but it seems difficult as Vivek said. So, I will accept the changing if it is necessary.

Now, I will make two prototypes to consider the method to figure out free pages.

- a prototype based on _count
- a prototype based on PG_buddy (or _mapcount)

If prototypes work fine, then we can select the method.

Quoted 3 lines

bitmap file is main cause of memory consuming if 2nd kernel uses initramfs only. There are other parts where the size of allocated memory varies based on 1-st kernel memory size, but they don't have big influence.

Thanks for the explanation.

I ask because I want to exactly calculate the required size for the crashkernel parameter. On s390 the kdump kernel memory consumption is fix and not dependent on the 1st kernel memory size. So based on your explanation I, will use:

crashkernel=<base size> + <variable size>

where

<variable size> = <pages of 1st kernel> * (2 + x) / 8

where "x" is the variable makedumpfile memory allocation that is on top of the bitmap allocation. What would be a good value for "x"?

Hello,

On Thu, 12 Apr 2012 16:47:14 +0900 (JST) HATAYAMA Daisuke <[email protected]> wrote:

[..]

Quoted 15 lines

Quoted 11 lines

I said I want to avoid changing behavior based on kernel versions, but it seems difficult as Vivek said. So, I will accept the changing if it is necessary.

Now, I will make two prototypes to consider the method to figure out free pages.

- a prototype based on _count
- a prototype based on PG_buddy (or _mapcount)

If prototypes work fine, then we can select the method.

I think the first one would work well and it's more accurate in meaning of free page.

Quoted 6 lines

Although this might be not problematic in practice, new method that walks on page tables can lead to different result from the previous one that looks up free_list: looking at __free_pages(), it first decreases page->_count and then add the page to free_list, and looking at __alloc_pages(), it first retrieves a page from free_list and then set page->_count to 1.

I tested the prototype based on _count and the other based on _mapcount. So, the former didn't work as expected while the latter worked fine. (The former excluded some used pages as free pages.)

As a next step, I measured performance of the prototype based on _mapcount, please see below.

Performance Comparison:

Explanation:
- The new method supports 2.6.39 and later, and it needs vmlinux.

- Now, the prototype doesn't support PG_buddy because the value of PG_buddy is different depending on kernel configuration and it isn't stored into VMCOREINFO. However, I'll extend get_length_of_free_pages() for PG_buddy when the value of PG_buddy is stored into VMCOREINFO.

- The prototype has dump_level "32" to use new method, but I don't think to extend dump_level for official version.

How to measure:
I measured execution times with vmcore of 5GB in below cases with attached patches.

- dump_level 16: exclude only free pages with the current method
- dump_level 31: exclude all excludable pages with the current method
- dump_level 32: exclude only free pages with the new method
- dump_level 47: exclude all excludable pages with the new method

Result:

     ------------------------------------------------------------------------
     dump_level	     size [Bytes]    total time	   d_all_time     d_new_time	
     ------------------------------------------------------------------------
     	16		431864384	28.6s	     4.19s	      0s
     	31		111808568	14.5s	      0.9s	      0s
     	32		431864384	41.2s	     16.8s	   0.05s
     	47		111808568	31.5s	     16.6s	   0.05s
     ------------------------------------------------------------------------

Discussion:
I think the new method can be used instead of the current method in many cases. (However, the result of dump_level 31 looks too fast, I'm researching why the case can execute so fast.)

I would like to get your opinion.

Hello,

On Thu, 12 Apr 2012 16:47:14 +0900 (JST) HATAYAMA Daisuke <[email protected]> wrote:

[..]

Quoted 15 lines

Quoted 11 lines

I said I want to avoid changing behavior based on kernel versions, but it seems difficult as Vivek said. So, I will accept the changing if it is necessary.

Now, I will make two prototypes to consider the method to figure out free pages.

- a prototype based on _count
- a prototype based on PG_buddy (or _mapcount)

If prototypes work fine, then we can select the method.

I think the first one would work well and it's more accurate in meaning of free page.

Quoted 6 lines

Although this might be not problematic in practice, new method that walks on page tables can lead to different result from the previous one that looks up free_list: looking at __free_pages(), it first decreases page->_count and then add the page to free_list, and looking at __alloc_pages(), it first retrieves a page from free_list and then set page->_count to 1.

I tested the prototype based on _count and the other based on _mapcount. So, the former didn't work as expected while the latter worked fine. (The former excluded some used pages as free pages.)

As a next step, I measured performance of the prototype based on _mapcount, please see below.

Performance Comparison:

Explanation:
- The new method supports 2.6.39 and later, and it needs vmlinux.

- Now, the prototype doesn't support PG_buddy because the value of PG_buddy is different depending on kernel configuration and it isn't stored into VMCOREINFO. However, I'll extend get_length_of_free_pages() for PG_buddy when the value of PG_buddy is stored into VMCOREINFO.

- The prototype has dump_level "32" to use new method, but I don't think to extend dump_level for official version.

Quoted 6 lines

I tested the prototype based on _count and the other based on _mapcount. So, the former didn't work as expected while the latter worked fine. (The former excluded some used pages as free pages.)

As a next step, I measured performance of the prototype based on _mapcount, please see below.

Thanks for this work. I assume this work just switches the free page referencing and does not attempt to try and cut down on the memory usage (I guess that would be the next step if using mapcount is acceptable)?

Quoted 35 lines

Performance Comparison:

Explanation:
- The new method supports 2.6.39 and later, and it needs vmlinux.

- Now, the prototype doesn't support PG_buddy because the value of PG_buddy is different depending on kernel configuration and it isn't stored into VMCOREINFO. However, I'll extend get_length_of_free_pages() for PG_buddy when the value of PG_buddy is stored into VMCOREINFO.

- The prototype has dump_level "32" to use new method, but I don't think to extend dump_level for official version.

How to measure:
I measured execution times with vmcore of 5GB in below cases with attached patches.

- dump_level 16: exclude only free pages with the current method
- dump_level 31: exclude all excludable pages with the current method
- dump_level 32: exclude only free pages with the new method
- dump_level 47: exclude all excludable pages with the new method

Result:

     ------------------------------------------------------------------------
     dump_level	     size [Bytes]    total time	   d_all_time     d_new_time	
     ------------------------------------------------------------------------
     	16		431864384	28.6s	     4.19s	      0s
     	31		111808568	14.5s	      0.9s	      0s
     	32		431864384	41.2s	     16.8s	   0.05s
     	47		111808568	31.5s	     16.6s	   0.05s
     ------------------------------------------------------------------------

Quoted 6 lines

Discussion:
I think the new method can be used instead of the current method in many cases. (However, the result of dump_level 31 looks too fast, I'm researching why the case can execute so fast.)

I would like to get your opinion.

I am curious. Looking through your patches, it seems d_all_time's increase in time should be from the new method because the if-statement is setup to only accept the new method. Therefore I was expecting d_new_time for the new method when added to d_all_time for the current method would come close to d_all_time for the new method. IOW I would have expected the extra 10-12 seconds from the new method to be found in d_new_time.

However, I do not see that. d_new_time hardly increases at all. So what is accounting for the increase in d_all_time for the new method?

Thanks,
Don

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Hello,

On Fri, 27 Apr 2012 08:52:14 -0400
Don Zickus <[email protected]> wrote:

[..]

Quoted 10 lines

Quoted 6 lines

I tested the prototype based on _count and the other based on _mapcount. So, the former didn't work as expected while the latter worked fine. (The former excluded some used pages as free pages.)

As a next step, I measured performance of the prototype based on _mapcount, please see below.

Thanks for this work. I assume this work just switches the free page referencing and does not attempt to try and cut down on the memory usage (I guess that would be the next step if using mapcount is acceptable)?

Thank you for your reply, Don, Vivek.

As Don said, I tried to change the method to exclude free pages and planed to resolve the memory consumption issue after it, because parsing free list repeatedly may cause a performance issue.

However, I'm thinking that to fix the size of memory consumption is more important than to resolve a performance issue for large system.

So I'm afraid that I would like to change the plan as:

1. Implement "iterating filtering processing" to fix the size of memory consumption. At this stage, makedumpfile will parse free list repeatedly even though it may cause a performance issue.

2. Take care of the performance issue after the 1st step.

Thanks
Atsushi Kumagai

Quoted 59 lines

Quoted 35 lines

Performance Comparison:

Explanation:
- The new method supports 2.6.39 and later, and it needs vmlinux.

- Now, the prototype doesn't support PG_buddy because the value of PG_buddy is different depending on kernel configuration and it isn't stored into VMCOREINFO. However, I'll extend get_length_of_free_pages() for PG_buddy when the value of PG_buddy is stored into VMCOREINFO.

- The prototype has dump_level "32" to use new method, but I don't think to extend dump_level for official version.

How to measure:
I measured execution times with vmcore of 5GB in below cases with attached patches.

- dump_level 16: exclude only free pages with the current method
- dump_level 31: exclude all excludable pages with the current method
- dump_level 32: exclude only free pages with the new method
- dump_level 47: exclude all excludable pages with the new method

Result:

     ------------------------------------------------------------------------
     dump_level	     size [Bytes]    total time	   d_all_time     d_new_time	
     ------------------------------------------------------------------------
     	16		431864384	28.6s	     4.19s	      0s
     	31		111808568	14.5s	      0.9s	      0s
     	32		431864384	41.2s	     16.8s	   0.05s
     	47		111808568	31.5s	     16.6s	   0.05s
     ------------------------------------------------------------------------

Quoted 6 lines

Discussion:
I think the new method can be used instead of the current method in many cases. (However, the result of dump_level 31 looks too fast, I'm researching why the case can execute so fast.)

I would like to get your opinion.

I am curious. Looking through your patches, it seems d_all_time's increase in time should be from the new method because the if-statement is setup to only accept the new method. Therefore I was expecting d_new_time for the new method when added to d_all_time for the current method would come close to d_all_time for the new method. IOW I would have expected the extra 10-12 seconds from the new method to be found in d_new_time.

However, I do not see that. d_new_time hardly increases at all. So what is accounting for the increase in d_all_time for the new method?

Thanks,
Don

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

- Now, the prototype doesn't support PG_buddy because the value of PG_buddy is different depending on kernel configuration and it isn't stored into VMCOREINFO. However, I'll extend get_length_of_free_pages() for PG_buddy when the value of PG_buddy is stored into VMCOREINFO.

Quoted 15 lines

Thank you for your reply, Don, Vivek.

As Don said, I tried to change the method to exclude free pages and planed to resolve the memory consumption issue after it, because parsing free list repeatedly may cause a performance issue.

However, I'm thinking that to fix the size of memory consumption is more important than to resolve a performance issue for large system.

So I'm afraid that I would like to change the plan as:

1. Implement "iterating filtering processing" to fix the size of memory consumption. At this stage, makedumpfile will parse free list repeatedly even though it may cause a performance issue.

Quoted 1 line

2. Take care of the performance issue after the 1st step.

Hello Atsushi-san,

Hmm. The problem with the free list is that the addresses are in random order, hence the reason to parse it repeatedly, correct?

I figured, now that you have a solution to parse the addresses in a linear way (the changes you made a couple of weeks ago), you would just continue with that. With that complete, we can look at the performance issues and solve them then.

But it is up to you. You are willing to do the work, so I will defer to your judgement on how best to proceed. :-)

Hello Don,

Quoted 15 lines · On Fri, 11 May 2012 09:26:01 -0400 Don Zickus <[email protected]> wrote:

Quoted 15 lines

Thank you for your reply, Don, Vivek.

As Don said, I tried to change the method to exclude free pages and planed to resolve the memory consumption issue after it, because parsing free list repeatedly may cause a performance issue.

However, I'm thinking that to fix the size of memory consumption is more important than to resolve a performance issue for large system.

So I'm afraid that I would like to change the plan as:

1. Implement "iterating filtering processing" to fix the size of memory consumption. At this stage, makedumpfile will parse free list repeatedly even though it may cause a performance issue.

Quoted 6 lines

Quoted 1 line

2. Take care of the performance issue after the 1st step.

Hello Atsushi-san,

Hmm. The problem with the free list is that the addresses are in random order, hence the reason to parse it repeatedly, correct?

Yes.

Quoted 5 lines

I figured, now that you have a solution to parse the addresses in a linear way (the changes you made a couple of weeks ago), you would just continue with that. With that complete, we can look at the performance issues and solve them then.

Quoted 2 lines

But it is up to you. You are willing to do the work, so I will defer to your judgement on how best to proceed. :-)

What I wanted to tell you was I want to resolve the memory consumption issue as soon as possible. In other words, I think the method to exclude free pages is not so important.
I'll continue to work with the method which is easy to implement.

From: Atsushi Kumagai <[email protected]>
Subject: Re: makedumpfile memory usage grows with system memory size
Date: Fri, 27 Apr 2012 16:46:49 +0900

Quoted 4 lines

- Now, the prototype doesn't support PG_buddy because the value of PG_buddy is different depending on kernel configuration and it isn't stored into VMCOREINFO. However, I'll extend get_length_of_free_pages() for PG_buddy when the value of PG_buddy is stored into VMCOREINFO.

Hello Kumagai san,

I'm now investigating how to perform filtering free pages without kernel debuginfo. For this, I've investigated which of PG_buddy and _mapcount to use in kernel versions. In the current conclusion, it's reasonable to do that as shown in the following table.

| kernel version   |  Use PG_buddy? or _mapcount?                             |
|------------------+----------------------------------------------------------|
| 2.6.15 -- 2.6.16 | offsetof(page,_mapcount):=sizeof(ulong)+sizeof(atomic_t) |
| 2.6.17 -- 2.6.26 |        PG_buddy := 19                                    |
| 2.6.27 -- 2.6.36 |        PG_buddy := 18                                    |
| 2.6.37 and later | offsetof(page,_mapcount):= under investigation           |                                           |

In summary: PG_buddy was first introduced at 2.6.17 as 19 to fix some race bug leading to lru list corruptions, and from 2.6.17 to 2.6.26, it had been defined using macro preprocessor. At 2.6.27 enum pageflags was introduced for ease of page flags maintainance and its value changed to 18. At 2.6.37, it was removed, and it no longer exists in later kernel versions.

My quick feeling is that solving dependency of PG_buddy is simler than that of _mapcount from 2.6.17 to 2.6.36.

From 2.6.15 to 2.6.16, PG_buddy has not been introduced so we need to rely on _mapcount. It's very complex to solve _mapcount dependency in general on all supported kernel versions, but only on both kernel versions, definition of struct page begins with the following layout. I think it's not so much complex to hardcode offset of _mapcount for these two kernel versions only: that is, sizeof(unsigned long) + sizeof(atomic_t) which is in fact struct { volatile int counter } on all platforms.

struct page {
unsigned long flags; /* Atomic flags, some possibly
* updated asynchronously */
atomic_t _count; /* Usage count, see below. */
atomic_t _mapcount; /* Count of ptes mapped in mms,
...

In the period of PG_buddy is defined as enumeration value, PG_buddy value depends on CONFIG_PAGEFLAGS_EXTENDED. At commit e20b8cca760ed2a6abcfe37ef56f2306790db648, PG_head and PG_tail were introduced and they are positioned before PG_buddy if CONFIG_PAGEFLAGS_EXTENDED is set; then PG_buddy value becomes
19. However, its users are mips, um and xtensa only as:

  $ git grep "CONFIG_PAGEFLAGS_EXTENDED"
  arch/mips/configs/db1300_defconfig:CONFIG_PAGEFLAGS_EXTENDED=y
  arch/um/defconfig:CONFIG_PAGEFLAGS_EXTENDED=y
  arch/xtensa/configs/iss_defconfig:CONFIG_PAGEFLAGS_EXTENDED=y
  arch/xtensa/configs/s6105_defconfig:CONFIG_PAGEFLAGS_EXTENDED=y
  include/linux/page-flags.h:#ifdef CONFIG_PAGEFLAGS_EXTENDED
  include/linux/page-flags.h:#ifdef CONFIG_PAGEFLAGS_EXTENDED
  mm/memory-failure.c:#ifdef CONFIG_PAGEFLAGS_EXTENDED
  mm/page_alloc.c:#ifdef CONFIG_PAGEFLAGS_EXTENDED

and makedumpfile doesn't support any of these platforms now. So we don't need to consider this case more.

On 2.6.37 and the later kernels, we must use _mapcount. I'm now looking into how to get offset of _mapcount in each kernel version without kernel debug information. But page structure has changed considerably on recent kernels so I guess the way hardcoding them gets more complicated.

Anyway, I think it better to add _mapcount information to VMCOREINFO on upstream as soon as possible.

Quoted 2 lines · On Mon, 14 May 2012 14:44:28 +0900 (JST) HATAYAMA Daisuke…

Anyway, I think it better to add _mapcount information to VMCOREINFO on upstream as soon as possible.

I think it's better way to use _mapcount.
But we don't certainly decide to use _mapcount and even if we decide to use it, we still have problems to use it.
For example, the upstream kernel(v3.4-rc7) has _mapcount in union, we need a information to judge whether the found data is _mapcount or not. So, more investigation is needed and I think it's too early to send the request to upstream kernel.

I plan to finish working to reduce memory consumption by the end of June, and I will continue to discuss performance issues.
Therefore, the request will be delayed until July or August.