Skip to contentSkip to conversation

makedumpfile memmory usage seems high with option -E

PATCH In review Vivek Goyal 5 participants 38 messages Apr 10, 2014 – May 27, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Atsushi/Hatayama,

We noticed in our testing that makedumpfile gets OOM killed if we happen to use -E option. Saving to compressed kdump format works just fine.

Also we noticed that with -E if we disable cyclic mode then it works just fine.

So looks like something is going on with -E and cyclic mode enabled. I am not sure what it is.

Do you suspect something?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw

Hello Vivek,

Quoted 5 lines

Hi Atsushi/Hatayama,

We noticed in our testing that makedumpfile gets OOM killed if we happen to use -E option. Saving to compressed kdump format works just fine.

Quoted 7 lines

Also we noticed that with -E if we disable cyclic mode then it works just fine.

So looks like something is going on with -E and cyclic mode enabled. I am not sure what it is.

Do you suspect something?

At first, I supposed the function to calculate cyclic buffer size may be related to this issue, but I haven't found the answer yet...

int
calculate_cyclic_buffer_size() {

        if (info->flag_elf_dumpfile) {
                free_size = get_free_memory_size() * 0.4;
                needed_size = (info->max_mapnr * 2) / BITPERBYTE;
        } else {
                free_size = get_free_memory_size() * 0.8;
                needed_size = info->max_mapnr / BITPERBYTE;
        }
        [...]
        info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

I've found this function has an issue about memory allocation. When -E is specified, info->bufsize_cyclic will be the total size of the 1st and 2nd bitmap if free memory is enough. Then, info->bufsize_cyclic will be used to allocate each bitmap in prepare_bitmap_buffer_cyclic() like below:

        if ((info->partial_bitmap1 = (char *)malloc(info->bufsize_cyclic)) == NULL) {
                ERRMSG("Can't allocate memory for the 1st-bitmap. %s\n",
                       strerror(errno));
                return FALSE;
        }
        if ((info->partial_bitmap2 = (char *)malloc(info->bufsize_cyclic)) == NULL) {
                ERRMSG("Can't allocate memory for the 2nd-bitmap. %s\n",
                       strerror(errno));
                return FALSE;
        }

It's a too much allocation definitely, but it mustn't exceed 80% of free memory due to the condition check in calculate_cyclic_buffer_size(), so I think the OOM issue will not happen by this issue.
I'll fix this too much allocation with the patch below, but it will not resolve your OOM issue...

BTW, what are your version of makedumpfile and crashkernel= size and the system memory size? and does the issue happen even if you specify --cyclic-buffer which is small enough to fit the available memory ? I'm curious to know the details of the condition which cause the issue.

Thanks
Atsushi Kumagai

makedumpfile.c +6 -7
index 75092a8..ae9e69a 100644
@@ -8996,7 +8996,7 @@ out:
  */
 int
 calculate_cyclic_buffer_size(void) {
-	unsigned long long free_size, needed_size;
+	unsigned long long limit_size, bitmap_size;
 
 	if (info->max_mapnr <= 0) {
 		ERRMSG("Invalid max_mapnr(%llu).\n", info->max_mapnr);
@@ -9009,18 +9009,17 @@ calculate_cyclic_buffer_size(void) {
 	 * within 80% of free memory.
 	 */
 	if (info->flag_elf_dumpfile) {
-		free_size = get_free_memory_size() * 0.4;
-		needed_size = (info->max_mapnr * 2) / BITPERBYTE;
+		limit_size = get_free_memory_size() * 0.4;
 	} else {
-		free_size = get_free_memory_size() * 0.8;
-		needed_size = info->max_mapnr / BITPERBYTE;
+		limit_size = get_free_memory_size() * 0.8;
 	}
+	bitmap_size = info->max_mapnr / BITPERBYTE;
 
 	/* if --split was specified cyclic buffer allocated per dump file */
 	if (info->num_dumpfile > 1)
-		needed_size /= info->num_dumpfile;
+		bitmap_size /= info->num_dumpfile;
 
-	info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;
+	info->bufsize_cyclic = (limit_size <= bitmap_size) ? limit_size : bitmap_size;
 
 	return TRUE;
 }
Signature · 6 lines
1.8.0.2

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#2 lore raw

----- Original Message -----

Quoted 58 lines

Hello Vivek,

Quoted 5 lines

Hi Atsushi/Hatayama,

We noticed in our testing that makedumpfile gets OOM killed if we happen to use -E option. Saving to compressed kdump format works just fine.

Quoted 7 lines

Also we noticed that with -E if we disable cyclic mode then it works just fine.

So looks like something is going on with -E and cyclic mode enabled. I am not sure what it is.

Do you suspect something?

At first, I supposed the function to calculate cyclic buffer size may be related to this issue, but I haven't found the answer yet...

int
calculate_cyclic_buffer_size() {

        if (info->flag_elf_dumpfile) {
                free_size = get_free_memory_size() * 0.4;
                needed_size = (info->max_mapnr * 2) / BITPERBYTE;
        } else {
                free_size = get_free_memory_size() * 0.8;
                needed_size = info->max_mapnr / BITPERBYTE;
        }
        [...]
        info->bufsize_cyclic = (free_size <= needed_size) ? free_size :
        needed_size;

I've found this function has an issue about memory allocation. When -E is specified, info->bufsize_cyclic will be the total size of the 1st and 2nd bitmap if free memory is enough. Then, info->bufsize_cyclic will be used to allocate each bitmap in prepare_bitmap_buffer_cyclic() like below:

        if ((info->partial_bitmap1 = (char *)malloc(info->bufsize_cyclic)) ==
        NULL) {
                ERRMSG("Can't allocate memory for the 1st-bitmap. %s\n",
                       strerror(errno));
                return FALSE;
        }
        if ((info->partial_bitmap2 = (char *)malloc(info->bufsize_cyclic)) ==
        NULL) {
                ERRMSG("Can't allocate memory for the 2nd-bitmap. %s\n",
                       strerror(errno));
                return FALSE;
        }

It's a too much allocation definitely, but it mustn't exceed 80% of free memory due to the condition check in calculate_cyclic_buffer_size(), so I think the OOM issue will not happen by this issue.
I'll fix this too much allocation with the patch below, but it will not resolve your OOM issue...

Quoted 6 lines

BTW, what are your version of makedumpfile and crashkernel= size and the system memory size? and does the issue happen even if you specify --cyclic-buffer which is small enough to fit the available memory ? I'm curious to know the details of the condition which cause the issue.

Hi Atsushi Kumagai
The makedumpfile version is 1.5.4. crashkernel=auto which actually is 166M.
the system memory size is about 96G and the arch is x86_64 which kernel version 3.10.110. But the same issue happened when using the newest makedumpfile in devel branch. After add --cyclic-buffer=100, dump succeed.

Thanks
arthur

Quoted 46 lines

Thanks
Atsushi Kumagai

makedumpfile.c +6 -7
index 75092a8..ae9e69a 100644
@@ -8996,7 +8996,7 @@ out:
  */
 int
 calculate_cyclic_buffer_size(void) {
-	unsigned long long free_size, needed_size;
+	unsigned long long limit_size, bitmap_size;
 
 	if (info->max_mapnr <= 0) {
 		ERRMSG("Invalid max_mapnr(%llu).\n", info->max_mapnr);
@@ -9009,18 +9009,17 @@ calculate_cyclic_buffer_size(void) {
 	 * within 80% of free memory.
 	 */
 	if (info->flag_elf_dumpfile) {
-		free_size = get_free_memory_size() * 0.4;
-		needed_size = (info->max_mapnr * 2) / BITPERBYTE;
+		limit_size = get_free_memory_size() * 0.4;
 	} else {
-		free_size = get_free_memory_size() * 0.8;
-		needed_size = info->max_mapnr / BITPERBYTE;
+		limit_size = get_free_memory_size() * 0.8;
 	}
+	bitmap_size = info->max_mapnr / BITPERBYTE;
 
 	/* if --split was specified cyclic buffer allocated per dump file */
 	if (info->num_dumpfile > 1)
-		needed_size /= info->num_dumpfile;
+		bitmap_size /= info->num_dumpfile;
 
-	info->bufsize_cyclic = (free_size <= needed_size) ? free_size :
 needed_size;
+	info->bufsize_cyclic = (limit_size <= bitmap_size) ? limit_size :
 bitmap_size;
 

return TRUE;
}
--
1.8.0.2

Quoted 5 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

PATCH 1/1 makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for bufsize_cyclic need be double. Hence free size should be 40% of free memory, however the needed_size which free_size is compared with should be info->max_mapnr / BITPERBYTE, but not 2 times of it.

Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.

---
 makedumpfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
makedumpfile.c +1 -1
index 75092a8..01ec516 100644
@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {
 	 */
 	if (info->flag_elf_dumpfile) {
 		free_size = get_free_memory_size() * 0.4;
-		needed_size = (info->max_mapnr * 2) / BITPERBYTE;
+		needed_size = info->max_mapnr / BITPERBYTE;
 	} else {
 		free_size = get_free_memory_size() * 0.8;
 		needed_size = info->max_mapnr / BITPERBYTE;
Signature · 7 lines
1.8.5.3


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#4 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Hi Atsushi,

Since both of our test machines where this bug was reproduced are reserved, I haven't tested this patch. But after several times of the code flow of elf dump, finally this only this suspect is found.

Could you also help check this?

Also below change is OK too, and its meaning is more obvious. But my previous patch is more readable for code.

int

calculate_cyclic_buffer_size(void) {
        unsigned long long free_size, needed_size;
 
        if (info->max_mapnr <= 0) {
                ERRMSG("Invalid max_mapnr(%llu).\n", info->max_mapnr);
                return FALSE;
        }
 
        /*
         * free_size will be used to allocate 1st and 2nd bitmap, so it
         * should be 40% of free memory to keep the size of cyclic
         * buffer
         * within 80% of free memory.
         */
        if (info->flag_elf_dumpfile) {
                free_size = get_free_memory_size() * 0.4 * 2;
                needed_size = (info->max_mapnr * 2) / BITPERBYTE;
        } else {
                free_size = get_free_memory_size() * 0.8;
                needed_size = info->max_mapnr / BITPERBYTE;
        }
 
        /* if --split was specified cyclic buffer allocated per dump
 * file */
        if (info->num_dumpfile > 1)                                                                                                                                                     
                needed_size /= info->num_dumpfile;
 
        info->bufsize_cyclic = (free_size <= needed_size) ? free_size/2 :

needed_size/2;

        return TRUE;
}

Thanks
Baoquan

Quoted 35 lines · Baoquan He

In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for bufsize_cyclic need be double. Hence free size should be 40% of free memory, however the needed_size which free_size is compared with should be info->max_mapnr / BITPERBYTE, but not 2 times of it.

Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.

---
 makedumpfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
makedumpfile.c +1 -1
index 75092a8..01ec516 100644
@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {
 	 */
 	if (info->flag_elf_dumpfile) {
 		free_size = get_free_memory_size() * 0.4;
-		needed_size = (info->max_mapnr * 2) / BITPERBYTE;
+		needed_size = info->max_mapnr / BITPERBYTE;
 	} else {
 		free_size = get_free_memory_size() * 0.8;
 		needed_size = info->max_mapnr / BITPERBYTE;
Signature · 7 lines
1.8.5.3


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Baoquan He

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Thanks
Baoquan

Quoted 35 lines · Baoquan He

In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for bufsize_cyclic need be double. Hence free size should be 40% of free memory, however the needed_size which free_size is compared with should be info->max_mapnr / BITPERBYTE, but not 2 times of it.

Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.

---
 makedumpfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
makedumpfile.c +1 -1
index 75092a8..01ec516 100644
@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {
 	 */
 	if (info->flag_elf_dumpfile) {
 		free_size = get_free_memory_size() * 0.4;
-		needed_size = (info->max_mapnr * 2) / BITPERBYTE;
+		needed_size = info->max_mapnr / BITPERBYTE;
 	} else {
 		free_size = get_free_memory_size() * 0.8;
 		needed_size = info->max_mapnr / BITPERBYTE;
Signature · 7 lines
1.8.5.3


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Hello Baoquan,

Quoted 4 lines

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Great, thanks for your help!
However, I still have questions.

First, what is the difference between yours and mine?

http://lists.infradead.org/pipermail/kexec/2014-April/011535.html

My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,

Quoted 9 lines · Baoquan He
Quoted 8 lines

In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for bufsize_cyclic need be double. Hence free size should be 40% of free memory, however the needed_size which free_size is compared with should be info->max_mapnr / BITPERBYTE, but not 2 times of it.

Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.

but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:

    info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.

I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?

Thanks
Atsushi Kumagai

Quoted 22 lines
Quoted 22 lines
---
 makedumpfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
makedumpfile.c +1 -2
index 75092a8..01ec516 100644
@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {
 	 */
 	if (info->flag_elf_dumpfile) {
 		free_size = get_free_memory_size() * 0.4;
-		needed_size = (info->max_mapnr * 2) / BITPERBYTE;
+		needed_size = info->max_mapnr / BITPERBYTE;
 	} else {
 		free_size = get_free_memory_size() * 0.8;
 		needed_size = info->max_mapnr / BITPERBYTE;
--

1.8.5.3

Quoted 4 lines
Quoted 4 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 7 lines · Atsushi Kumagai

Hello Baoquan,

Quoted 4 lines

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Quoted 6 lines

Great, thanks for your help!
However, I still have questions.

First, what is the difference between yours and mine?

http://lists.infradead.org/pipermail/kexec/2014-April/011535.html

Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.

Quoted 19 lines

My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,

Quoted 9 lines · Baoquan He
Quoted 8 lines

In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for bufsize_cyclic need be double. Hence free size should be 40% of free memory, however the needed_size which free_size is compared with should be info->max_mapnr / BITPERBYTE, but not 2 times of it.

Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.

but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:

Quoted 7 lines
    info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.

I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

Quoted 27 lines

Thanks
Atsushi Kumagai

Quoted 22 lines
Quoted 22 lines
---
 makedumpfile.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
makedumpfile.c +1 -2
index 75092a8..01ec516 100644
@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {
 	 */
 	if (info->flag_elf_dumpfile) {
 		free_size = get_free_memory_size() * 0.4;
-		needed_size = (info->max_mapnr * 2) / BITPERBYTE;
+		needed_size = info->max_mapnr / BITPERBYTE;
 	} else {
 		free_size = get_free_memory_size() * 0.8;
 		needed_size = info->max_mapnr / BITPERBYTE;
--

1.8.5.3

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 11 lines · Baoquan He
Quoted 10 lines · Atsushi Kumagai

Hello Baoquan,

Quoted 4 lines

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Great, thanks for your help!
However, I still have questions.

Quoted 6 lines
Quoted 3 lines

First, what is the difference between yours and mine?

http://lists.infradead.org/pipermail/kexec/2014-April/011535.html

Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.

                                                          must<--
Quoted 16 lines
Quoted 16 lines

My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,

Quoted 1 line · Baoquan He

but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:

    info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.

Quoted 9 lines
Quoted 2 lines

I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines

Hello Baoquan,

Quoted 4 lines

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Great, thanks for your help!
However, I still have questions.

Quoted 6 lines
Quoted 3 lines

First, what is the difference between yours and mine?

http://lists.infradead.org/pipermail/kexec/2014-April/011535.html

Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.

                                                         must<--
Quoted 16 lines
Quoted 16 lines

My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,

Quoted 1 line · Baoquan He

but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:

    info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.

Quoted 9 lines
Quoted 2 lines

I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

Quoted 2 lines

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.

What I think is:

1. Use a constant value as safe limit to calculate bufsize_cyclic instead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.

2. If free memory is smaller than the value, makedumpfile gives up to work early.

This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.

Any comments are welcome.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Quoted 7 lines

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (base_value + linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.

Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of base_value is needed. So a constant value as you said may not be needed.

Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.

Quoted 17 lines

I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.

What I think is:

1. Use a constant value as safe limit to calculate bufsize_cyclic instead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.

2. If free memory is smaller than the value, makedumpfile gives up to work early.

This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.

Quoted 5 lines

Any comments are welcome.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 25 lines · On Fri, 18 Apr 2014 22:29:12 +0800 "[email protected]" <[email protected]> wrote:
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Quoted 7 lines

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (base_value + linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.

Quoted 6 lines

Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of base_value is needed. So a constant value as you said may not be needed.

Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.

I believe these 80% come from the default value of vm.dirty_ratio, which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.

I figured out that the actual requirements also depend on the target device driver, filesystem (e.g. NFS is considerably more memory-hungry than ext3) and (not the least) number of online CPUs.

The calculation is quite complex, and that's probably why nobody has done it properly yet.

Petr T

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#17 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Atsushi Kumagai
Quoted 48 lines · Atsushi Kumagai
Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines

Hello Baoquan,

Quoted 4 lines

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Great, thanks for your help!
However, I still have questions.

Quoted 6 lines
Quoted 3 lines

First, what is the difference between yours and mine?

http://lists.infradead.org/pipermail/kexec/2014-April/011535.html

Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.

                                                         must<--
Quoted 16 lines
Quoted 16 lines

My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,

Quoted 1 line · Baoquan He

but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:

    info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.

Quoted 9 lines
Quoted 2 lines

I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

Quoted 2 lines

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Quoted 7 lines

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?

Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.

Quoted 4 lines

I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.

Quoted 8 lines

What I think is:

1. Use a constant value as safe limit to calculate bufsize_cyclic instead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.

2. If free memory is smaller than the value, makedumpfile gives up to work early.

What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.

Quoted 4 lines

This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.

I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.

Secondly, I think using absolute values is not a good idea. It will be very hard to keep track and udpate that value.

At max we can warn saying makedumpfile needs X MB of memory and only Y MB of free memory is available.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#15 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to [email protected]
Quoted 25 lines · On Fri, Apr 18, 2014 at 10:29:12PM +0800, [email protected] wrote:
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Quoted 7 lines

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (base_value + linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.

Quoted 3 lines

Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of base_value is needed. So a constant value as you said may not be needed.

That logic is old and we probably should get rid of at some point. We don't want makedumpfile's memory usage to go up because system has more physical RAM. That's why cyclic mode was introduced.

Quoted 3 lines

Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.

I had come up with this 80% number randomly. So you think that's the problem?

I am still scratching my head that why 30MB is not sufficient for makdumpfile.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Petr Tesarik
Quoted 28 lines · Petr Tesarik
Quoted 25 lines · On Fri, 18 Apr 2014 22:29:12 +0800 "[email protected]" <[email protected]> wrote:
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Quoted 7 lines

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (base_value + linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.

Quoted 8 lines
Quoted 6 lines

Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of base_value is needed. So a constant value as you said may not be needed.

Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.

I believe these 80% come from the default value of vm.dirty_ratio,

Actually I had suggested this 80% number when --cyclic feature was implemented. And I did not base it on dirty_ratio. Just a random suggestion.

Quoted 5 lines

which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.

We can start playig with reducing dirty_raio too and see how does it go.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#13 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 4 lines · On Mon, 21 Apr 2014 11:19:14 -0400 Vivek Goyal <[email protected]> wrote:
Quoted 2 lines · Petr Tesarik

On Fri, 18 Apr 2014 22:29:12 +0800
"[email protected]" <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.

I believe these 80% come from the default value of vm.dirty_ratio,

Actually I had suggested this 80% number when --cyclic feature was implemented. And I did not base it on dirty_ratio. Just a random suggestion.

Hm. OK, you seem to have very good suggestions, even if they are random. ;-)

Quoted 7 lines
Quoted 5 lines

which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.

We can start playig with reducing dirty_raio too and see how does it go.

Or use direct I/O... Is it still broken?

Petr T

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#14 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 25 lines · Petr Tesarik
Quoted 4 lines · On Mon, 21 Apr 2014 11:19:14 -0400 Vivek Goyal <[email protected]> wrote:
Quoted 2 lines · Petr Tesarik

On Fri, 18 Apr 2014 22:29:12 +0800
"[email protected]" <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.

I believe these 80% come from the default value of vm.dirty_ratio,

Actually I had suggested this 80% number when --cyclic feature was implemented. And I did not base it on dirty_ratio. Just a random suggestion.

Hm. OK, you seem to have very good suggestions, even if they are random. ;-)

Quoted 6 lines
Quoted 5 lines

which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.

Quoted 3 lines
Quoted 1 line

We can start playig with reducing dirty_raio too and see how does it go.

Or use direct I/O... Is it still broken?

May be if that turns out to be an issue. Or just reduce dirty_ratio to say 5 instead of default 20. Different things can be tried once we know what's the root cause of the issue.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#18 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Vivek Goyal
Quoted 52 lines · Atsushi Kumagai
Quoted 51 lines
Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines

Hello Baoquan,

Quoted 4 lines

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Great, thanks for your help!
However, I still have questions.

Quoted 6 lines
Quoted 3 lines

First, what is the difference between yours and mine?

http://lists.infradead.org/pipermail/kexec/2014-April/011535.html

Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.

                                                         must<--
Quoted 16 lines
Quoted 16 lines

My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,

Quoted 1 line · Baoquan He

but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:

    info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.

Quoted 9 lines
Quoted 2 lines

I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

Quoted 2 lines

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Thanks for your description, I understand that situation and the nature of the problem.

Quoted 7 lines
Quoted 4 lines

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?

Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

Quoted 2 lines

Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.

According to the table above and Baoquan's report, 23.6MB was enough but 17.2 MB was short for the other works, it sounds too much requirement also to me.

Quoted 9 lines
Quoted 9 lines

I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.

What I think is:

1. Use a constant value as safe limit to calculate bufsize_cyclic instead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.

Quoted 5 lines
Quoted 2 lines

2. If free memory is smaller than the value, makedumpfile gives up to work early.

What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.

Oh, you got a point.

Quoted 7 lines
Quoted 4 lines

This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.

I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.

I agree. I'm trying to get the peak size of memory footprint except cyclic_buffer with memory cgroup (memory.max_usage_in_bytes), but I don't make it yet.

Quoted 5 lines

Secondly, I think using absolute values is not a good idea. It will be very hard to keep track and udpate that value.

At max we can warn saying makedumpfile needs X MB of memory and only Y MB of free memory is available.

Yes, I don't want to choose that way, and to maintain the constant value just to inform possibility of OOM sounds worthless.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#16 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Vivek Goyal
Quoted 19 lines
Quoted 11 lines

Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (base_value + linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.

Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of base_value is needed. So a constant value as you said may not be needed.

That logic is old and we probably should get rid of at some point. We don't want makedumpfile's memory usage to go up because system has more physical RAM. That's why cyclic mode was introduced.

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.

I had come up with this 80% number randomly. So you think that's the problem?

As Astushi said, when memory amount goes up, the bitmap also goes up. The cyclic should solve this problem, but seems it doesn't. In the bug we found, it always get the whole bitmap size to cyclic_bugsize. And that is because 80% number.

We are try to add debug code to see what is going on during makedumpfile running.

Quoted 11 lines

I am still scratching my head that why 30MB is not sufficient for makdumpfile.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#19 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Atsushi Kumagai
Quoted 25 lines
Quoted 10 lines
Quoted 7 lines

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?

Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

Thanks Atsushi for this detailed table.

In fact in cyclic mode, the cyclic bugsize is the dynamic number which need be grieved, not the remaining memory which always got to be decreased. Since user set cyclic mode, they would like to accept the fact that dumping could cost a longer time. But we give them a OOM, that's not acceptable.

So I think we need think about how the cyclic_bufsize can be calculated better. Maybe dynamically adjust the 80% number?

Thanks
Baoquan

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#20 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Atsushi Kumagai
Quoted 57 lines · Atsushi Kumagai
Quoted 57 lines · Atsushi Kumagai
Quoted 55 lines
Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines

Hello Baoquan,

Quoted 4 lines

Hi Atsushi,

I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.

Great, thanks for your help!
However, I still have questions.

Quoted 6 lines
Quoted 3 lines

First, what is the difference between yours and mine?

http://lists.infradead.org/pipermail/kexec/2014-April/011535.html

Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.

                                                         must<--
Quoted 16 lines
Quoted 16 lines

My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,

Quoted 1 line · Baoquan He

but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:

    info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;

So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.

Quoted 9 lines
Quoted 2 lines

I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

Quoted 2 lines

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Quoted 6 lines
Quoted 2 lines

Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?

Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:

What I meant was memory allocation is of the order O(1) and not O(N).

Quoted 11 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

Quoted 30 lines
Quoted 3 lines

Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.

According to the table above and Baoquan's report, 23.6MB was enough but 17.2 MB was short for the other works, it sounds too much requirement also to me.

Quoted 9 lines
Quoted 9 lines

I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.

What I think is:

1. Use a constant value as safe limit to calculate bufsize_cyclic instead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.

Quoted 5 lines
Quoted 2 lines

2. If free memory is smaller than the value, makedumpfile gives up to work early.

What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.

Oh, you got a point.

Quoted 5 lines
Quoted 4 lines

This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.

Quoted 6 lines
Quoted 2 lines

I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.

I agree. I'm trying to get the peak size of memory footprint except cyclic_buffer with memory cgroup (memory.max_usage_in_bytes), but I don't make it yet.

That sounds like a good idea.

I am wondering if we are doing anything which requires kernel to allocate memory and that leads to OOM.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#21 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 20 lines · Vivek Goyal
Quoted 11 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Quoted 5 lines

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.

Thanks
Baoquan

Quoted 2 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#22 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 28 lines · Baoquan He
Quoted 20 lines · Vivek Goyal
Quoted 11 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Quoted 5 lines

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

Quoted 4 lines

How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.

Oh, this is what Astushi suggested earlier in his comments.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#23 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to [email protected]
Quoted 28 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 20 lines · Vivek Goyal
Quoted 11 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Quoted 5 lines

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

Quoted 4 lines

How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.

I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.

Tricky bit here is to figure out how much memory does makedumpfile need.

A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#36 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Vivek Goyal
Quoted 87 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines

It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.

BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.

Thanks for your description, I understand that situation and the nature of the problem.

That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.

Quoted 6 lines
Quoted 2 lines

Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?

Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:

What I meant was memory allocation is of the order O(1) and not O(N).

Quoted 11 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

Quoted 33 lines
Quoted 3 lines

Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.

According to the table above and Baoquan's report, 23.6MB was enough but 17.2 MB was short for the other works, it sounds too much requirement also to me.

Quoted 9 lines
Quoted 9 lines

I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.

What I think is:

1. Use a constant value as safe limit to calculate bufsize_cyclic instead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.

Quoted 5 lines
Quoted 2 lines

2. If free memory is smaller than the value, makedumpfile gives up to work early.

What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.

Oh, you got a point.

Quoted 7 lines
Quoted 4 lines

This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.

I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.

Quoted 8 lines
Quoted 3 lines

I agree. I'm trying to get the peak size of memory footprint except cyclic_buffer with memory cgroup (memory.max_usage_in_bytes), but I don't make it yet.

That sounds like a good idea.

I am wondering if we are doing anything which requires kernel to allocate memory and that leads to OOM.

max_usage_in_bytes seems to include page cache, so I used simpler way like the patch at the bottom of this mail instead.

Here is the result:

parameter | result
dump_lv | buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------

    d31	 |	1	|	 on	|	5872
   Ed31	 |	1	|	 on	|	5804
    d31	 |	1	|	off	|	1784
   Ed31	 |	1	|	off	|	1720
    d31	 |   1000	|	 on	|	8884  (*1)
   Ed31  |   1000	|	 on	|	7812
    d31	 |   1000	|	off	|	4788  (*1)
   Ed31  |   1000	|	off	|	3720

*1. 2nd bitmap is allocated twice by the bug found by Arthur Zou.

According to this result and the estimation below produced from the design of makedumpfile,

base size + (buffer size * 2) + (for mmap size) = VmHWM

the base size may be about 2 MB, so about 6MB (base + mmap) will be the deadline. If 20% of the available memory is less than 6MB, OOM will happen.
As for Baoquan's case, remained 17.2MB sounds enough even if makedumpfile consumes 6MB as for the other works.
So I don't know why OOM happened yet, but the current safety limit looks still reasonable to me at least.

By the way, I noticed it's better to remove 4MB(for mmap) from the available memory before calculate the bitmap buffer size. I'll do this anyway.

Thanks
Atsushi Kumagai

makedumpfile.c +10 -0
index 0b31932..4b565f1 100644
@@ -37,6 +37,14 @@ struct DumpInfo		*info = NULL;
 
 char filename_stdout[] = FILENAME_STDOUT;
 
+void
+print_VmHWM(void)
+{
+	char cmd[64];
+	sprintf(cmd, "grep VmHWM /proc/%d/status", getpid());
+	system(cmd);
+}
+
 static void first_cycle(mdf_pfn_t start, mdf_pfn_t max, struct cycle *cycle)
 {
 	cycle->start_pfn = round(start, info->pfn_cyclic);
@@ -9357,5 +9365,7 @@ out:
 	}
 	free_elf_info();
 
+	print_VmHWM();
+
 	return retcd;
 }

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#24 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Vivek Goyal
Quoted 34 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 28 lines · Vivek Goyal
Quoted 21 lines
Quoted 12 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Quoted 5 lines

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

Quoted 4 lines

How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.

Quoted 6 lines

I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.

Tricky bit here is to figure out how much memory does makedumpfile need.

Did you said using such value is bad idea since it's hard to update it? If we got the needed memory size, it would be changing every version. At least I think this may be an ideal way but not practical.

Quoted 2 lines

A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.

That's just specific tuning for you and Baoquan.

Now, I think this case is just lack of free memory caused by inappropriate parameter setting for your environment. You should increase crashkernel= to get enough free memory, 166M may be too small for your environment.

By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.

Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#25 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 39 lines · Atsushi Kumagai
Quoted 39 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 28 lines · Vivek Goyal
Quoted 21 lines
Quoted 12 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Quoted 5 lines

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

Quoted 4 lines

How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.

I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.

Quoted 5 lines
Quoted 1 line

Tricky bit here is to figure out how much memory does makedumpfile need.

Did you said using such value is bad idea since it's hard to update it? If we got the needed memory size, it would be changing every version. At least I think this may be an ideal way but not practical.

Yep, I am not too convinced about fixing makedumpfile memory usage at a particular value.

Quoted 10 lines
Quoted 3 lines

A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.

That's just specific tuning for you and Baoquan.

Now, I think this case is just lack of free memory caused by inappropriate parameter setting for your environment. You should increase crashkernel= to get enough free memory, 166M may be too small for your environment.

I don't think it is bad tuning from our side. makedumpfile has 30MB free memory when it was launched and still OOM happened.

30MB should be more than enough to save dump.

Quoted 3 lines

By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.

Sure, talk to you more about this once you are back.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#37 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Atsushi Kumagai
Quoted 35 lines
Quoted 2 lines

I am wondering if we are doing anything which requires kernel to allocate memory and that leads to OOM.

max_usage_in_bytes seems to include page cache, so I used simpler way like the patch at the bottom of this mail instead.

Here is the result:

parameter | result
dump_lv | buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------

   d31  |       1     |       on      |        5872
  Ed31  |       1     |       on      |        5804
   d31  |       1     |      off      |        1784
  Ed31  |       1     |      off      |        1720
   d31  |    1000     |       on      |        8884  (*1)
  Ed31  |    1000     |       on      |        7812
   d31  |    1000     |      off      |        4788  (*1)
  Ed31  |    1000     |      off      |        3720

*1. 2nd bitmap is allocated twice by the bug found by Arthur Zou.

According to this result and the estimation below produced from the design of makedumpfile,

base size + (buffer size * 2) + (for mmap size) = VmHWM

the base size may be about 2 MB, so about 6MB (base + mmap) will be the deadline. If 20% of the available memory is less than 6MB, OOM will happen.
As for Baoquan's case, remained 17.2MB sounds enough even if makedumpfile consumes 6MB as for the other works.
So I don't know why OOM happened yet, but the current safety limit looks still reasonable to me at least.

Quoted 3 lines

By the way, I noticed it's better to remove 4MB(for mmap) from the available memory before calculate the bitmap buffer size. I'll do this anyway.

Sorry, I did my test on a 1st kernel environment.
The dump file was on a disk and I used mmap() for it, so __do_fault() occurred and the VmHWM (MAX(MM_FILEPAGES + MM_ANONPAGES)) increased by 4MB.

Here is the result on a 2nd kernel environment:

parameter | result
dump_lv | buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------

     d31  |       1     |       on      |         776
    Ed31  |       1     |       on      |         712
     d31  |       1     |      off      |         704
    Ed31  |       1     |      off      |         708
     d31  |    1000     |       on      |        1776
    Ed31  |    1000     |       on      |        2716
     d31  |    1000     |      off      |        1660
    Ed31  |    1000     |      off      |        2556

According to this, I think there is no need to take care of the memory usage of mmap() because the actual memory usage is small enough.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#26 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Vivek Goyal
Quoted 60 lines · Atsushi Kumagai
Quoted 39 lines
Quoted 39 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 28 lines · Vivek Goyal
Quoted 21 lines
Quoted 12 lines

- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)

pattern | bitmap size | remains
----------------------------------------------+---------------+-------------

  A. 100G memory with the too allocation bug  |    12.8 MB    |   17.2 MB
  B. 100G memory with fixed makedumpfile      |     6.4 MB    |   23.6 MB
  C. 200G memory with fixed makedumpfile      |    12.8 MB    |   17.2 MB
  D. 300G memory with fixed makedumpfile      |    19.2 MB    |   10.8 MB
  E. 400G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
  F. 500G memory with fixed makedumpfile      |    24.0 MB    |    6.0 MB
Quoted 4 lines

...

Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.

ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.

Quoted 5 lines

Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.

Quoted 4 lines

How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.

I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.

Quoted 5 lines
Quoted 1 line

Tricky bit here is to figure out how much memory does makedumpfile need.

Did you said using such value is bad idea since it's hard to update it? If we got the needed memory size, it would be changing every version. At least I think this may be an ideal way but not practical.

Yep, I am not too convinced about fixing makedumpfile memory usage at a particular value.

Quoted 10 lines
Quoted 3 lines

A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.

That's just specific tuning for you and Baoquan.

Now, I think this case is just lack of free memory caused by inappropriate parameter setting for your environment. You should increase crashkernel= to get enough free memory, 166M may be too small for your environment.

Quoted 4 lines

I don't think it is bad tuning from our side. makedumpfile has 30MB free memory when it was launched and still OOM happened.

30MB should be more than enough to save dump.

There were more than 10 MB remain memory in your case (A pattern), and probably they were used as page cache mostly, so it sounds OOM couldn't happen since such pages are reclaimable.

I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:

/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8000 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [100.0 %] |

The dumpfile is saved to /mnt/tmp/dumpfile.E.

makedumpfile Completed.
VmHWM: 16456 kB
/ #

As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).

Of course, OOM happened when the memory usage exceeded the limit:

/ # free
total used free shared buffers
Mem: 37544 21608 15936 0 368
Swap: 0 0 0
Total: 37544 21608 15936
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8192 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [ 0.0 %] /Out of memory: Kill process 1389 (makedumpfile_st) score 428 or sacrifice child
Killed process 1389 (makedumpfile_st) total-vm:22196kB, anon-rss:16524kB, file-rss:8kB KILL
/ #

I think we should investigate why OOM happened in your environment, otherwise we can't decide a safety limit of a user process's memory usage.

Thanks
Atsushi Kumagai

Quoted 6 lines
Quoted 3 lines

By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.

Sure, talk to you more about this once you are back.

Quoted 7 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#27 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

On Fri, May 09, 2014 at 05:36:13AM +0000, Atsushi Kumagai wrote:

[..]

Quoted 14 lines

I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:

/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8000 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [100.0 %] |

The dumpfile is saved to /mnt/tmp/dumpfile.E.

Quoted 8 lines

makedumpfile Completed.
VmHWM: 16456 kB
/ #

As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).

Of course, OOM happened when the memory usage exceeded the limit:

Hi Atsushi,

I think this is the key point. How did makedumpfile exceed the limit. So if you don't specify --cyclic-buffer=X, then makedumpfile will allocate 80% of available memory. That would be roughly 16MB of cyclic buffer (instead of 8MB).

In your test case looks like makeudmpfile used 16MB of memory. Out of which 8MB must have been used for bitmaps and rest 8MB for other purposes.

So clearly our calculation of using 80% of available memory for bitmaps is not right. Rest of the 20% memory might not be enough for fulfilling the needs of makeudmpfile.

We don't have a good way to determine how much *non-bitmap* memory allocation makedumpfile requires. I am wondering how about making some assumtions and modify the formula for bitmap memory.

- Assume makedumpfile requires 16MB of free memory for various purposes.
- Subtract 16MB from available memory and take 80% of remaining memory to calculate the bitmap memory size.
- Keep 4MB as minimum for bitmap buffer size.

One more person has reported makedumpfile OOM (even without -E option), on a machine. There also 30MB of memory is free. I think it has the same issue that 24MB must have been allocated in bitmaps and remainig 6MB is not sufficient to satisfy makedumpfile's needs.

What do you think?

Also, does anybody know what's the impact of bitmap buffer size on dump speed. Does dump speed change significantly with bitmap buffer size?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#35 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Atsushi Kumagai
Quoted 16 lines · Atsushi Kumagai

There were more than 10 MB remain memory in your case (A pattern), and probably they were used as page cache mostly, so it sounds OOM couldn't happen since such pages are reclaimable.

I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:

/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8000 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [100.0 %] |

Quoted 4 lines

The dumpfile is saved to /mnt/tmp/dumpfile.E.

makedumpfile Completed.
VmHWM: 16456 kB

Hi Atsushi,

What is the dump target in your test? The OOM bugs happened if and only if it nfs target. Other dump target including ssh/local fs/ didn't happen ever though those cyclic buffer size code bugs existed.

We added some debug code and made test, the result shows the OOM happened when the left memory is only about 2M. But when we drop page caches every 1000 times of write, it's OK to complete the dump. And the OOM happened when page cache need allocate page for writing.

So could you adjust your test to nfs dump with elf format or lzo compression? I think nfs dump have heavy page cache affect which is different with others.

I will take machines with 100G memory and 10G memory to test separately, will paste the result with our configuration.

Thanks
Baoquan

Quoted 19 lines

/ #

As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).

Of course, OOM happened when the memory usage exceeded the limit:

/ # free
total used free shared buffers
Mem: 37544 21608 15936 0 368
Swap: 0 0 0
Total: 37544 21608 15936
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8192 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [ 0.0 %] /Out of memory: Kill process 1389 (makedumpfile_st) score 428 or sacrifice child
Killed process 1389 (makedumpfile_st) total-vm:22196kB, anon-rss:16524kB, file-rss:8kB KILL
/ #

Quoted 11 lines

I think we should investigate why OOM happened in your environment, otherwise we can't decide a safety limit of a user process's memory usage.

Thanks
Atsushi Kumagai

Quoted 4 lines
Quoted 3 lines

By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#28 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Vivek Goyal
Quoted 24 lines · Vivek Goyal

On Fri, May 09, 2014 at 05:36:13AM +0000, Atsushi Kumagai wrote:

[..]

Quoted 21 lines

I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:

/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8000 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [100.0 %] |

The dumpfile is saved to /mnt/tmp/dumpfile.E.

makedumpfile Completed.
VmHWM: 16456 kB
/ #

As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).

Quoted 8 lines
Quoted 1 line

Of course, OOM happened when the memory usage exceeded the limit:

Hi Atsushi,

I think this is the key point. How did makedumpfile exceed the limit. So if you don't specify --cyclic-buffer=X, then makedumpfile will allocate 80% of available memory. That would be roughly 16MB of cyclic buffer (instead of 8MB).

Hi,

Here I think Vivek is right. In Astushi's test, when --cyclic-buffer=8000 and elf dump, the info->bufsize_cyclic is only 4M, because it need be divided by 2 for elf case.

I took tests on my local machine with 16G memory, the result suprised me. In my tests, even 70% is not safe. I didn't specify --cyclic-buffer=X like Astushi did, just change code like below, I think it's more accurate.

@@ -9049,8 +9055,9 @@ calculate_cyclic_buffer_size(void) {
* should be 40% of free memory to keep the size of cyclic
* buffer
* within 80% of free memory.
*/
if (info->flag_elf_dumpfile) {
- limit_size = get_free_memory_size() * 0.4;

+               limit_size = get_free_memory_size() * 0.30;
        } else {
                limit_size = get_free_memory_size() * 0.8;
        }
@@ -9060,7 +9067,8 @@ calculate_cyclic_buffer_size(void) {
        if (info->num_dumpfile > 1)
                bitmap_size /= info->num_dumpfile;

- info->bufsize_cyclic = MIN(limit_size, bitmap_size);
+ info->bufsize_cyclic = limit_size;

        return TRUE;

}

All dump is elf format, and target is nfs and local separately. And the crashkernel=168M

Then in nfs dump, when bufsize_cyclic is 80% of memory, OOM happened 3 times in all 5 times. when bufsize_cyclic is 70% of memory, OOM happened 2 times in all 7 times. I paste the output message in the following (aaaa) section, it's 80% of memory for bufsize_cyclic.

Then I thought 70% is not safe, maybe local fs dump is better. So I took local fs dump, when bufsize_cyclic is 80%, OOM happened 3 times in all 5 times of test. When bufsize_cyclic is 70%, 2 of 5. when 60%, finally, it's safe now. I paste the output message in the following (bbbb) section, it's 80% of memory for bufsize_cyclic.

I can't get a conclusion, just give the test result for your reference.

(aaaa)

pre-pivot:/# cat /etc/kdump.conf 

nfs 10.66.16.116:/var/crash
path /var/crash
core_collector makedumpfile -E --message-level 1 -d 31 default shell
pre-pivot:/#

pre-pivot:/# exit
kdump: dump target is 10.66.16.116:/var/crash
kdump: saving to
/sysroot/mnt//var/crash/10.66.16.106-2014.05.15-09:08:03/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore

calculate_cyclic_buffer_size, free memory 34795520

Buffer size for the cyclic mode: 13918208
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,before write_elf_headerfree, memory 6819840
Excluding unnecessary pages : [100.0 %] \
writeout_dumpfile,before write_elf_pages_cyclic, memory 6791168

In calculate_cyclic_buffer_size, free memory 6791168
Copying data : [ 1.1 %] /
In calculate_cyclic_buffer_size, free memory 2412544
Excluding unnecessary pages : [100.0 %] |
In calculate_cyclic_buffer_size, free memory 2691072
Excluding unnecessary pages : [100.0 %] \
In calculate_cyclic_buffer_size, free memory 2756608
Copying data : [ 2.8 %] |
In calculate_cyclic_buffer_size, free memory 2363392
Copying data : [ 17.2 %] \[ 56.567080] usb
2-1.1: USB disconnect, device number 3

[   58.025437] usb 2-1.1: new low-speed USB device number 5 using
ehci-pci
[   58.125797] usb 2-1.1: New USB device found, idVendor=03f0,
idProduct=0b4a
[   58.132907] usb 2-1.1: New USB device strings: Mfr=1, Product=2,
SerialNumber=0
[   58.140431] usb 2-1.1: Product: USB Optical Mouse
[   58.145345] usb 2-1.1: Manufacturer: Logitech
[   58.153006] input: Logitech USB Optical Mouse as
/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1:1.0/input/input4
[   58.164942] hid-generic 0003:03F0:0B4A.0003: input,hidraw0: USB HID
v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:1d.0-1.1/inp0
[   58.646175] makedumpfile invoked oom-killer: gfp_mask=0x10200da,
order=0, oom_score_adj=0
[   58.654581] makedumpfile cpuset=/ mems_allowed=0
[   58.659411] CPU: 0 PID: 517 Comm: makedumpfile Not tainted
3.10.0-114.el7.x86_64 #1
[   58.667275] Hardware name: Hewlett-Packard HP Z420 Workstation/1589,
BIOS J61 v01.02 03/09/2012
[   58.676177]  ffff880032db4440 000000003ed297af ffff88002e4b3880
ffffffff815ec0ca
[   58.683882]  ffff88002e4b3910 ffffffff815e773d ffffffff810b6918
ffff880033aca0c0
[   58.691584]  ffffffff00000206 ffffffff00000000 0000000000000000
ffffffff81102e03
[   58.699293] Call Trace:
[   58.701959]  [<ffffffff815ec0ca>] dump_stack+0x19/0x1b
[   58.707309]  [<ffffffff815e773d>] dump_header+0x8e/0x214
[   58.712827]  [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0
[   58.718529]  [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0
[   58.724310]  [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0
[   58.730339]  [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30
[   58.736895]  [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0
[   58.742665]  [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0
[   58.749222]  [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170
[   58.755440]  [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0
[   58.761479]  [<ffffffff81142566>]
grab_cache_page_write_begin+0x76/0xd0
[   58.768304]  [<ffffffffa03f4767>] nfs_write_begin+0x77/0x210 [nfs]
[   58.774694]  [<ffffffff8114158e>]
generic_file_buffered_write+0x11e/0x290
[   58.781693]  [<ffffffff81050f00>] ?
rbt_memtype_copy_nth_element+0xa0/0xa0
[   58.788779]  [<ffffffff811436e5>]
__generic_file_aio_write+0x1d5/0x3e0
[   58.795512]  [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0
[   58.801903]  [<ffffffffa03f3c1b>] nfs_file_write+0xbb/0x1d0 [nfs]
[   58.808206]  [<ffffffff811af21d>] do_sync_write+0x8d/0xd0
[   58.813796]  [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0
[   58.819141]  [<ffffffff811af4a4>] ? generic_file_llseek+0x24/0x30
[   58.825434]  [<ffffffff811b0408>] SyS_write+0x58/0xb0
[   58.830676]  [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b
[   58.836874] Mem-Info:
[   58.839349] Node 0 DMA per-cpu:
[   58.842714] CPU    0: hi:    0, btch:   1 usd:   0
[   58.847704] Node 0 DMA32 per-cpu:
[   58.851244] CPU    0: hi:   42, btch:   7 usd:   0
[   58.856235] active_anon:11294 inactive_anon:925 isolated_anon:0
[   58.856235]  active_file:1032 inactive_file:1653 isolated_file:32
[   58.856235]  unevictable:11030 dirty:0 writeback:1800 unstable:0
[   58.856235]  free:473 slab_reclaimable:2202 slab_unreclaimable:3849
[   58.856235]  mapped:1245 shmem:1103 pagetables:283 bounce:0
[   58.856235]  free_cma:0
[   58.891859] Node 0 DMA free:492kB min:4kB low:4kB high:4kB
active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:8kB
unevictabs
[   58.931901] lowmem_reserve[]: 0 122 122 122
[   58.936393] Node 0 DMA32 free:1400kB min:1412kB low:1764kB
high:2116kB active_anon:45176kB inactive_anon:3700kB active_file:4124kB
inacts
[   58.981849] lowmem_reserve[]: 0 0 0 0
[   58.985815] Node 0 DMA: 3*4kB (UM) 2*8kB (U) 1*16kB (M) 0*32kB 1*64kB
(U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4B
[   58.999026] Node 0 DMA32: 38*4kB (EM) 78*8kB (UEMR) 11*16kB (UER)
0*32kB 3*64kB (R) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096B
[   59.012831] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[   59.021469] 14852 total pagecache pages
[   59.025522] 0 pages in swap cache
[   59.029048] Swap cache stats: add 0, delete 0, find 0/0
[   59.034491] Free swap  = 0kB
[   59.037583] Total swap = 0kB
[   59.041703] 90211 pages RAM
[   59.044712] 53920 pages reserved
[   59.048157] 267612 pages shared
[   59.051514] 31588 pages non-shared
[   59.055133] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
[   59.063173] [   64]     0    64    13020      570      24        0
0 systemd-journal
[   59.072172] [  170]    32   170     9977      282      23        0
0 rpcbind
[   59.080474] [  174]     0   174    11120      416      25        0
0 rpc.statd
[   59.088960] [  179]     0   179     6366      169      16        0
0 rpc.idmapd
[   59.097551] [  201]     0   201     8729      425      22        0
-1000 systemd-udevd
[   59.106402] [  324]     0   324    25589     3179      47        0
0 dhclient
[   59.114815] [  383]     0   383     3101      553      16        0
0 dracut-pre-pivo
[   59.123832] [  517]     0   517    13333     7103      42        0
0 makedumpfile
[   59.132594] [  518]     0   518     8728      272      21        0
0 systemd-udevd
[   59.141442] [  519]     0   519     8728      214      21        0
0 systemd-udevd
[   59.150287] Out of memory: Kill process 517 (makedumpfile) score 190
or sacrifice child
[   59.158527] Killed process 517 (makedumpfile) total-vm:53332kB,

anon-rss:27524kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"

(bbbb)
kdump: dump target is /dev/sda2
kdump: saving t[ 13.492083] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
o /sysroot//var/crash/127.0.0.1-2014.05.15-11:24:51/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore

calculate_cyclic_buffer_size, free memory 77033472

Buffer size for the cyclic mode: 30813388
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,before write_elf_headerfree, memory 15233024
Excluding unnecessary pages : [100.0 %] -
writeout_dumpfile,before write_elf_pages_cyclic, memory 15204352

In calculate_cyclic_buffer_size, free memory 15204352
Excluding unnecessary pages : [100.0 %] /
In calculate_cyclic_buffer_size, free memory 2932736
Excluding unnecessary pages : [100.0 %] |
In calculate_cyclic_buffer_size, free memory 2387968
Excluding unnecessary pages : [100.0 %] \
In calculate_cyclic_buffer_size, free memory 2555904
Excluding unnecessary pages : [100.0 %] -
In calculate_cyclic_buffer_size, free memory 2838528
Copying data : [ 3.8 %] |[ 15.406789]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0

[   15.415222] makedumpfile cpuset=/ mems_allowed=0
[   15.420191] CPU: 0 PID: 287 Comm: makedumpfile Not tainted
3.10.0-114.el7.x86_64 #1
[   15.428092] Hardware name: Hewlett-Packard HP Z420 Workstation/1589,
BIOS J61 v01.02 03/09/2012
[   15.437280]  ffff88002f8196c0 0000000043db25d9 ffff88002ff2b7e0
ffffffff815ec0ca
[   15.445105]  ffff88002ff2b870 ffffffff815e773d ffffffff810b6918
ffff880030018bb0
[   15.452935]  ffffffff00000206 ffffffff00000000 0000000000000000
ffffffff81102e03
[   15.460693] Call Trace:
[   15.463375]  [<ffffffff815ec0ca>] dump_stack+0x19/0x1b
[   15.468743]  [<ffffffff815e773d>] dump_header+0x8e/0x214
[   15.474288]  [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0
[   15.479999]  [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0
[   15.485790]  [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0
[   15.491848]  [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30
[   15.498430]  [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0
[   15.504222]  [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0
[   15.510793]  [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170
[   15.517020]  [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0
[   15.523061]  [<ffffffff81142566>]
grab_cache_page_write_begin+0x76/0xd0
[   15.529898]  [<ffffffffa02ab133>] ext4_da_write_begin+0xa3/0x330
[ext4]
[   15.536728]  [<ffffffff8114158e>]
generic_file_buffered_write+0x11e/0x290
[   15.543724]  [<ffffffff811436e5>]
__generic_file_aio_write+0x1d5/0x3e0
[   15.550462]  [<ffffffff81050f00>] ?
rbt_memtype_copy_nth_element+0xa0/0xa0
[   15.557551]  [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0
[   15.563952]  [<ffffffffa02a1189>] ext4_file_write+0xa9/0x450 [ext4]
[   15.570434]  [<ffffffff811797fc>] ? free_vmap_area_noflush+0x7c/0x90
[   15.577009]  [<ffffffff811af21d>] do_sync_write+0x8d/0xd0
[   15.582616]  [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0
[   15.587967]  [<ffffffff811b0408>] SyS_write+0x58/0xb0
[   15.593223]  [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b
[   15.599437] Mem-Info:
[   15.601912] Node 0 DMA per-cpu:
[   15.605283] CPU    0: hi:    0, btch:   1 usd:   0
[   15.610279] Node 0 DMA32 per-cpu:
[   15.613825] CPU    0: hi:   42, btch:   7 usd:  29
[   15.618827] active_anon:16072 inactive_anon:924 isolated_anon:0
[   15.618827]  active_file:2075 inactive_file:2232 isolated_file:0
[   15.618827]  unevictable:6464 dirty:0 writeback:3 unstable:0
[   15.618827]  free:511 slab_reclaimable:2077 slab_unreclaimable:3528
[   15.618827]  mapped:1055 shmem:1088 pagetables:146 bounce:0
[   15.618827]  free_cma:0
[   15.654098] Node 0 DMA free:508kB min:4kB low:4kB high:4kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictabs
[   15.693926] lowmem_reserve[]: 0 129 129 129
[   15.698429] Node 0 DMA32 free:1536kB min:1452kB low:1812kB
high:2176kB active_anon:64288kB inactive_anon:3696kB active_file:8300kB
inacto
[   15.742999] lowmem_reserve[]: 0 0 0 0
[   15.746981] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U)
1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B
[   15.760499] Node 0 DMA32: 6*4kB (M) 7*8kB (R) 3*16kB (R) 2*32kB (M)
1*64kB (M) 0*128kB 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB
[   15.774255] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[   15.782928] 11860 total pagecache pages
[   15.786986] 0 pages in swap cache
[   15.790524] Swap cache stats: add 0, delete 0, find 0/0
[   15.795976] Free swap  = 0kB
[   15.799080] Total swap = 0kB
[   15.803169] 90211 pages RAM
[   15.806188] 53919 pages reserved
[   15.809635] 6131 pages shared
[   15.812820] 29843 pages non-shared
[   15.816437] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
[   15.824495] [   64]     0    64    13020      559      23        0
0 systemd-journal
[   15.833506] [  119]     0   119     8864      549      23        0
-1000 systemd-udevd
[   15.842347] [  204]     0   204     3071      531      15        0
0 dracut-pre-pivo
[   15.851353] [  287]     0   287    21581    15351      58        0
0 makedumpfile
[   15.860103] Out of memory: Kill process 287 (makedumpfile) score 411
or sacrifice child
[   15.868333] Killed process 287 (makedumpfile) total-vm:86324kB,

anon-rss:60516kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/999
Generating "/run/initramfs/rdsosreport.txt"

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#29 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Quoted 34 lines · Vivek Goyal
Quoted 24 lines · Atsushi Kumagai

[..]

Quoted 21 lines

I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:

/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8000 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [100.0 %] |

The dumpfile is saved to /mnt/tmp/dumpfile.E.

makedumpfile Completed.
VmHWM: 16456 kB
/ #

As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).

Quoted 8 lines
Quoted 1 line

Of course, OOM happened when the memory usage exceeded the limit:

Hi Atsushi,

I think this is the key point. How did makedumpfile exceed the limit. So if you don't specify --cyclic-buffer=X, then makedumpfile will allocate 80% of available memory. That would be roughly 16MB of cyclic buffer (instead of 8MB).

Quoted 5 lines

Hi,

Here I think Vivek is right. In Astushi's test, when --cyclic-buffer=8000 and elf dump, the info->bufsize_cyclic is only 4M, because it need be divided by 2 for elf case.

No, 16MB was used for bitmaps also in my case.
--cyclic-buffer option means specifying each bitmap size, so allocated bitmap size is the double of --cyclic-buffer.
(But now, this behavior is only for ELF case, just the specified size is allocated in kdump case. Yes, it's confusing...)

--cyclic-buffer buffer_size
Specify the buffer size in kilo bytes for analysis in the cyclic mode. Actually, the double of buffer_size kilo bytes will be allocated in memory. In the cyclic mode, the number of cycles is represented as:

So only 456KB was the requirement for other purposes, 20% of available memory (3.5MB) was enough, thus the current assumption is safe even when available memory is only 17MB like my test.

According to my test:

http://lists.infradead.org/pipermail/kexec/2014-May/011784.html

Quoted 5 lines

Here is the result on a 2nd kernel environment:

parameter | result
dump_lv | buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------

Quoted 8 lines
     d31  |       1     |       on      |         776    // about 700
    Ed31  |       1     |       on      |         712
     d31  |       1     |      off      |         704
    Ed31  |       1     |      off      |         708
     d31  |    1000     |       on      |        1776    // about 700 + 1000
    Ed31  |    1000     |       on      |        2716    // about 700 + 1000 * 2
     d31  |    1000     |      off      |        1660
    Ed31  |    1000     |      off      |        2556

The requirement memory size for other purposes seems about 700KB when the dump level is 31, it's so small that we can ignore it.
The size will increase by KB order based on the system memory size in practice (this assumption comes from my code investigation), but 6MB (20% of 30MB) still sounds much enough as safety limit.
This is why I was wondering why OOM happened on your machine.

Now that I heard that OOM happened only on nfs, I guess just nfs requires lots of memory. (I did my test only on local fs.)
If we can get the required size, we could reflect it in the safety limit size.

I haven't checked your test result below yet, so I'll mention it later.

BTW, I prepared a temporal branch "oom" to investigate this issue, let's use this after this:

http://sourceforge.net/p/makedumpfile/code/ci/oom/tree/

Thanks
Atsushi Kumagai

Quoted 309 lines

I took tests on my local machine with 16G memory, the result suprised me. In my tests, even 70% is not safe. I didn't specify --cyclic-buffer=X like Astushi did, just change code like below, I think it's more accurate.

(quoted diff) +2 -2
@@ -9049,8 +9055,9 @@ calculate_cyclic_buffer_size(void) {
        * should be 40% of free memory to keep the size of cyclic
        * buffer
        * within 80% of free memory.
        */
       if (info->flag_elf_dumpfile) {
-               limit_size = get_free_memory_size() * 0.4;
+               limit_size = get_free_memory_size() * 0.30;
       } else {
               limit_size = get_free_memory_size() * 0.8;
       }
@@ -9060,7 +9067,8 @@ calculate_cyclic_buffer_size(void) {
       if (info->num_dumpfile > 1)
               bitmap_size /= info->num_dumpfile;
 
-       info->bufsize_cyclic = MIN(limit_size, bitmap_size);
+       info->bufsize_cyclic = limit_size;
 
       return TRUE;
 }

All dump is elf format, and target is nfs and local separately. And the crashkernel=168M

Then in nfs dump, when bufsize_cyclic is 80% of memory, OOM happened 3 times in all 5 times. when bufsize_cyclic is 70% of memory, OOM happened 2 times in all 7 times. I paste the output message in the following (aaaa) section, it's 80% of memory for bufsize_cyclic.

Then I thought 70% is not safe, maybe local fs dump is better. So I took local fs dump, when bufsize_cyclic is 80%, OOM happened 3 times in all 5 times of test. When bufsize_cyclic is 70%, 2 of 5. when 60%, finally, it's safe now. I paste the output message in the following (bbbb) section, it's 80% of memory for bufsize_cyclic.

I can't get a conclusion, just give the test result for your reference.

(aaaa)

pre-pivot:/# cat /etc/kdump.conf

nfs 10.66.16.116:/var/crash
path /var/crash
core_collector makedumpfile -E --message-level 1 -d 31 default shell
pre-pivot:/#

pre-pivot:/# exit
kdump: dump target is 10.66.16.116:/var/crash
kdump: saving to
/sysroot/mnt//var/crash/10.66.16.106-2014.05.15-09:08:03/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore

calculate_cyclic_buffer_size, free memory 34795520

Buffer size for the cyclic mode: 13918208
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,before write_elf_headerfree, memory 6819840
Excluding unnecessary pages : [100.0 %] \
writeout_dumpfile,before write_elf_pages_cyclic, memory 6791168

In calculate_cyclic_buffer_size, free memory 6791168
Copying data : [ 1.1 %] /
In calculate_cyclic_buffer_size, free memory 2412544
Excluding unnecessary pages : [100.0 %] |
In calculate_cyclic_buffer_size, free memory 2691072
Excluding unnecessary pages : [100.0 %] \
In calculate_cyclic_buffer_size, free memory 2756608
Copying data : [ 2.8 %] |
In calculate_cyclic_buffer_size, free memory 2363392
Copying data : [ 17.2 %] \[ 56.567080] usb
2-1.1: USB disconnect, device number 3

[   58.025437] usb 2-1.1: new low-speed USB device number 5 using
ehci-pci
[   58.125797] usb 2-1.1: New USB device found, idVendor=03f0,
idProduct=0b4a
[   58.132907] usb 2-1.1: New USB device strings: Mfr=1, Product=2,
SerialNumber=0
[   58.140431] usb 2-1.1: Product: USB Optical Mouse
[   58.145345] usb 2-1.1: Manufacturer: Logitech
[   58.153006] input: Logitech USB Optical Mouse as
/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1:1.0/input/input4
[   58.164942] hid-generic 0003:03F0:0B4A.0003: input,hidraw0: USB HID
v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:1d.0-1.1/inp0
[   58.646175] makedumpfile invoked oom-killer: gfp_mask=0x10200da,
order=0, oom_score_adj=0
[   58.654581] makedumpfile cpuset=/ mems_allowed=0
[   58.659411] CPU: 0 PID: 517 Comm: makedumpfile Not tainted
3.10.0-114.el7.x86_64 #1
[   58.667275] Hardware name: Hewlett-Packard HP Z420 Workstation/1589,
BIOS J61 v01.02 03/09/2012
[   58.676177]  ffff880032db4440 000000003ed297af ffff88002e4b3880
ffffffff815ec0ca
[   58.683882]  ffff88002e4b3910 ffffffff815e773d ffffffff810b6918
ffff880033aca0c0
[   58.691584]  ffffffff00000206 ffffffff00000000 0000000000000000
ffffffff81102e03
[   58.699293] Call Trace:
[   58.701959]  [<ffffffff815ec0ca>] dump_stack+0x19/0x1b
[   58.707309]  [<ffffffff815e773d>] dump_header+0x8e/0x214
[   58.712827]  [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0
[   58.718529]  [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0
[   58.724310]  [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0
[   58.730339]  [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30
[   58.736895]  [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0
[   58.742665]  [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0
[   58.749222]  [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170
[   58.755440]  [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0
[   58.761479]  [<ffffffff81142566>]
grab_cache_page_write_begin+0x76/0xd0
[   58.768304]  [<ffffffffa03f4767>] nfs_write_begin+0x77/0x210 [nfs]
[   58.774694]  [<ffffffff8114158e>]
generic_file_buffered_write+0x11e/0x290
[   58.781693]  [<ffffffff81050f00>] ?
rbt_memtype_copy_nth_element+0xa0/0xa0
[   58.788779]  [<ffffffff811436e5>]
__generic_file_aio_write+0x1d5/0x3e0
[   58.795512]  [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0
[   58.801903]  [<ffffffffa03f3c1b>] nfs_file_write+0xbb/0x1d0 [nfs]
[   58.808206]  [<ffffffff811af21d>] do_sync_write+0x8d/0xd0
[   58.813796]  [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0
[   58.819141]  [<ffffffff811af4a4>] ? generic_file_llseek+0x24/0x30
[   58.825434]  [<ffffffff811b0408>] SyS_write+0x58/0xb0
[   58.830676]  [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b
[   58.836874] Mem-Info:
[   58.839349] Node 0 DMA per-cpu:
[   58.842714] CPU    0: hi:    0, btch:   1 usd:   0
[   58.847704] Node 0 DMA32 per-cpu:
[   58.851244] CPU    0: hi:   42, btch:   7 usd:   0
[   58.856235] active_anon:11294 inactive_anon:925 isolated_anon:0
[   58.856235]  active_file:1032 inactive_file:1653 isolated_file:32
[   58.856235]  unevictable:11030 dirty:0 writeback:1800 unstable:0
[   58.856235]  free:473 slab_reclaimable:2202 slab_unreclaimable:3849
[   58.856235]  mapped:1245 shmem:1103 pagetables:283 bounce:0
[   58.856235]  free_cma:0
[   58.891859] Node 0 DMA free:492kB min:4kB low:4kB high:4kB
active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:8kB
unevictabs
[   58.931901] lowmem_reserve[]: 0 122 122 122
[   58.936393] Node 0 DMA32 free:1400kB min:1412kB low:1764kB
high:2116kB active_anon:45176kB inactive_anon:3700kB active_file:4124kB
inacts
[   58.981849] lowmem_reserve[]: 0 0 0 0
[   58.985815] Node 0 DMA: 3*4kB (UM) 2*8kB (U) 1*16kB (M) 0*32kB 1*64kB
(U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4B
[   58.999026] Node 0 DMA32: 38*4kB (EM) 78*8kB (UEMR) 11*16kB (UER)
0*32kB 3*64kB (R) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096B
[   59.012831] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[   59.021469] 14852 total pagecache pages
[   59.025522] 0 pages in swap cache
[   59.029048] Swap cache stats: add 0, delete 0, find 0/0
[   59.034491] Free swap  = 0kB
[   59.037583] Total swap = 0kB
[   59.041703] 90211 pages RAM
[   59.044712] 53920 pages reserved
[   59.048157] 267612 pages shared
[   59.051514] 31588 pages non-shared
[   59.055133] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
[   59.063173] [   64]     0    64    13020      570      24        0
0 systemd-journal
[   59.072172] [  170]    32   170     9977      282      23        0
0 rpcbind
[   59.080474] [  174]     0   174    11120      416      25        0
0 rpc.statd
[   59.088960] [  179]     0   179     6366      169      16        0
0 rpc.idmapd
[   59.097551] [  201]     0   201     8729      425      22        0
-1000 systemd-udevd
[   59.106402] [  324]     0   324    25589     3179      47        0
0 dhclient
[   59.114815] [  383]     0   383     3101      553      16        0
0 dracut-pre-pivo
[   59.123832] [  517]     0   517    13333     7103      42        0
0 makedumpfile
[   59.132594] [  518]     0   518     8728      272      21        0
0 systemd-udevd
[   59.141442] [  519]     0   519     8728      214      21        0
0 systemd-udevd
[   59.150287] Out of memory: Kill process 517 (makedumpfile) score 190
or sacrifice child
[   59.158527] Killed process 517 (makedumpfile) total-vm:53332kB,

anon-rss:27524kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"

(bbbb)
kdump: dump target is /dev/sda2
kdump: saving t[ 13.492083] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
o /sysroot//var/crash/127.0.0.1-2014.05.15-11:24:51/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore

calculate_cyclic_buffer_size, free memory 77033472

Buffer size for the cyclic mode: 30813388
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,before write_elf_headerfree, memory 15233024
Excluding unnecessary pages : [100.0 %] -
writeout_dumpfile,before write_elf_pages_cyclic, memory 15204352

In calculate_cyclic_buffer_size, free memory 15204352
Excluding unnecessary pages : [100.0 %] /
In calculate_cyclic_buffer_size, free memory 2932736
Excluding unnecessary pages : [100.0 %] |
In calculate_cyclic_buffer_size, free memory 2387968
Excluding unnecessary pages : [100.0 %] \
In calculate_cyclic_buffer_size, free memory 2555904
Excluding unnecessary pages : [100.0 %] -
In calculate_cyclic_buffer_size, free memory 2838528
Copying data : [ 3.8 %] |[ 15.406789]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0

[   15.415222] makedumpfile cpuset=/ mems_allowed=0
[   15.420191] CPU: 0 PID: 287 Comm: makedumpfile Not tainted
3.10.0-114.el7.x86_64 #1
[   15.428092] Hardware name: Hewlett-Packard HP Z420 Workstation/1589,
BIOS J61 v01.02 03/09/2012
[   15.437280]  ffff88002f8196c0 0000000043db25d9 ffff88002ff2b7e0
ffffffff815ec0ca
[   15.445105]  ffff88002ff2b870 ffffffff815e773d ffffffff810b6918
ffff880030018bb0
[   15.452935]  ffffffff00000206 ffffffff00000000 0000000000000000
ffffffff81102e03
[   15.460693] Call Trace:
[   15.463375]  [<ffffffff815ec0ca>] dump_stack+0x19/0x1b
[   15.468743]  [<ffffffff815e773d>] dump_header+0x8e/0x214
[   15.474288]  [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0
[   15.479999]  [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0
[   15.485790]  [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0
[   15.491848]  [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30
[   15.498430]  [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0
[   15.504222]  [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0
[   15.510793]  [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170
[   15.517020]  [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0
[   15.523061]  [<ffffffff81142566>]
grab_cache_page_write_begin+0x76/0xd0
[   15.529898]  [<ffffffffa02ab133>] ext4_da_write_begin+0xa3/0x330
[ext4]
[   15.536728]  [<ffffffff8114158e>]
generic_file_buffered_write+0x11e/0x290
[   15.543724]  [<ffffffff811436e5>]
__generic_file_aio_write+0x1d5/0x3e0
[   15.550462]  [<ffffffff81050f00>] ?
rbt_memtype_copy_nth_element+0xa0/0xa0
[   15.557551]  [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0
[   15.563952]  [<ffffffffa02a1189>] ext4_file_write+0xa9/0x450 [ext4]
[   15.570434]  [<ffffffff811797fc>] ? free_vmap_area_noflush+0x7c/0x90
[   15.577009]  [<ffffffff811af21d>] do_sync_write+0x8d/0xd0
[   15.582616]  [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0
[   15.587967]  [<ffffffff811b0408>] SyS_write+0x58/0xb0
[   15.593223]  [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b
[   15.599437] Mem-Info:
[   15.601912] Node 0 DMA per-cpu:
[   15.605283] CPU    0: hi:    0, btch:   1 usd:   0
[   15.610279] Node 0 DMA32 per-cpu:
[   15.613825] CPU    0: hi:   42, btch:   7 usd:  29
[   15.618827] active_anon:16072 inactive_anon:924 isolated_anon:0
[   15.618827]  active_file:2075 inactive_file:2232 isolated_file:0
[   15.618827]  unevictable:6464 dirty:0 writeback:3 unstable:0
[   15.618827]  free:511 slab_reclaimable:2077 slab_unreclaimable:3528
[   15.618827]  mapped:1055 shmem:1088 pagetables:146 bounce:0
[   15.618827]  free_cma:0
[   15.654098] Node 0 DMA free:508kB min:4kB low:4kB high:4kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictabs
[   15.693926] lowmem_reserve[]: 0 129 129 129
[   15.698429] Node 0 DMA32 free:1536kB min:1452kB low:1812kB
high:2176kB active_anon:64288kB inactive_anon:3696kB active_file:8300kB
inacto
[   15.742999] lowmem_reserve[]: 0 0 0 0
[   15.746981] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U)
1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B
[   15.760499] Node 0 DMA32: 6*4kB (M) 7*8kB (R) 3*16kB (R) 2*32kB (M)
1*64kB (M) 0*128kB 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB
[   15.774255] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[   15.782928] 11860 total pagecache pages
[   15.786986] 0 pages in swap cache
[   15.790524] Swap cache stats: add 0, delete 0, find 0/0
[   15.795976] Free swap  = 0kB
[   15.799080] Total swap = 0kB
[   15.803169] 90211 pages RAM
[   15.806188] 53919 pages reserved
[   15.809635] 6131 pages shared
[   15.812820] 29843 pages non-shared
[   15.816437] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
[   15.824495] [   64]     0    64    13020      559      23        0
0 systemd-journal
[   15.833506] [  119]     0   119     8864      549      23        0
-1000 systemd-udevd
[   15.842347] [  204]     0   204     3071      531      15        0
0 dracut-pre-pivo
[   15.851353] [  287]     0   287    21581    15351      58        0
0 makedumpfile
[   15.860103] Out of memory: Kill process 287 (makedumpfile) score 411
or sacrifice child
[   15.868333] Killed process 287 (makedumpfile) total-vm:86324kB,
Quoted 3 lines

anon-rss:60516kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/999
Generating "/run/initramfs/rdsosreport.txt"

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#30 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

Hi Astushi,

Quoted 11 lines

No, 16MB was used for bitmaps also in my case.
--cyclic-buffer option means specifying each bitmap size, so allocated bitmap size is the double of --cyclic-buffer.
(But now, this behavior is only for ELF case, just the specified size is allocated in kdump case. Yes, it's confusing...)

--cyclic-buffer buffer_size
Specify the buffer size in kilo bytes for analysis in the cyclic mode. Actually, the double of buffer_size kilo bytes will be allocated in memory. In the cyclic mode, the number of cycles is represented as:

Yeah, I was wrong about this. If specified --cyclic-buffer doesn't exceed the memory limit, it is equal to info->bufsize_cyclic. But for elf dump, it need be double. E.g if free memory is 30M, --cyclic-buffer is 16M, it will absolutely fail. But the code check here should be done separately. Since it's meaningless for elf dump if 2 x --cyclic-buffer exceeds the free memory.

	if (info->bufsize_cyclic > free_memory) {
	.......
	{

I think code here need be changed.

Quoted 34 lines

So only 456KB was the requirement for other purposes, 20% of available memory (3.5MB) was enough, thus the current assumption is safe even when available memory is only 17MB like my test.

According to my test:

http://lists.infradead.org/pipermail/kexec/2014-May/011784.html

Quoted 5 lines

Here is the result on a 2nd kernel environment:

parameter | result
dump_lv | buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------

Quoted 8 lines
     d31  |       1     |       on      |         776    // about 700
    Ed31  |       1     |       on      |         712
     d31  |       1     |      off      |         704
    Ed31  |       1     |      off      |         708
     d31  |    1000     |       on      |        1776    // about 700 + 1000
    Ed31  |    1000     |       on      |        2716    // about 700 + 1000 * 2
     d31  |    1000     |      off      |        1660
    Ed31  |    1000     |      off      |        2556

The requirement memory size for other purposes seems about 700KB when the dump level is 31, it's so small that we can ignore it.
The size will increase by KB order based on the system memory size in practice (this assumption comes from my code investigation), but 6MB (20% of 30MB) still sounds much enough as safety limit.
This is why I was wondering why OOM happened on your machine.

Now that I heard that OOM happened only on nfs, I guess just nfs requires lots of memory. (I did my test only on local fs.)
If we can get the required size, we could reflect it in the safety limit size.

Quoted 6 lines

I haven't checked your test result below yet, so I'll mention it later.

BTW, I prepared a temporal branch "oom" to investigate this issue, let's use this after this:

http://sourceforge.net/p/makedumpfile/code/ci/oom/tree/

About OOM, I don't know why the difference between yours and mine is so big. My test machine has 16G memory, and the reserved crashkernel=161M. The OOM happened in a very high frequency. You can check the last section of kdump failed log, OOM failed when use 80% for info->bufsize_buffer though left free memory is 13M.

I changed the code like below.

---------------------------------------------------

makedumpfile.c +1 -0
index 8dc1181..cd710a3 100644
@@ -3157,6 +3157,7 @@ out:
                return FALSE;
 
        if (info->flag_cyclic) {
+               printf("\ncalculate_cyclic_buffer_size,
get_free_memory_size: %llu\n", get_free_memory_size());
                if (info->bufsize_cyclic == 0) {
                        if (!calculate_cyclic_buffer_size())
                                return FALSE;

@@ -3204,6 +3205,7 @@ out:

DEBUG_MSG("\n");
DEBUG_MSG("Buffer size for the cyclic mode: %ld\n",
info->bufsize_cyclic);
+ printf("\n Buffer size for the cyclic mode: %ld\n",

info->bufsize_cyclic);
        }
 
        if (!is_xen_memory() && !cache_init())
@@ -9060,7 +9062,7 @@ calculate_cyclic_buffer_size(void) {
        if (info->num_dumpfile > 1)
                bitmap_size /= info->num_dumpfile;

- info->bufsize_cyclic = MIN(limit_size, bitmap_size);
+ info->bufsize_cyclic = limit_size;

-------------------------------------------------
bhe# cat /etc/kdump.conf
path /var/crash
core_collector makedumpfile -E --message-level 1 -d 31

------------------------------------------
kdump: dump target is /dev/sda2
kdump: saving [ 9.595153] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
to /sysroot//var/crash/127.0.0.1-2014.05.19-18:50:18/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore

calculate_cyclic_buffer_size, get_free_memory_size: 68857856

Buffer size for the cyclic mode: 27543142
Copying data : [ 15.9 %] -[ 14.955468]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0

[   14.963876] makedumpfile cpuset=/ mems_allowed=0
[   14.968723] CPU: 0 PID: 286 Comm: makedumpfile Not tainted
3.10.0-123.el7.x86_64 #1
[   14.976606] Hardware name: Hewlett-Packard HP Z420 Workstation/1589,
BIOS J61 v01.02 03/09/2012
[   14.985567]  ffff88002fedc440 00000000f650c592 ffff88002fcb57d0
ffffffff815e19ba
[   14.993291]  ffff88002fcb5860 ffffffff815dd02d ffffffff810b68f8
ffff8800359dc0c0
[   15.001013]  ffffffff00000206 ffffffff00000000 0000000000000000
ffffffff81102e03
[   15.008733] Call Trace:
[   15.011413]  [<ffffffff815e19ba>] dump_stack+0x19/0x1b
[   15.016778]  [<ffffffff815dd02d>] dump_header+0x8e/0x214
[   15.022321]  [<ffffffff810b68f8>] ? ktime_get_ts+0x48/0xe0
[   15.028036]  [<ffffffff81102e03>] ? proc_do_uts_string+0xe3/0x130
[   15.034383]  [<ffffffff8114520e>] oom_kill_process+0x24e/0x3b0
[   15.040446]  [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30
[   15.047068]  [<ffffffff81145a36>] out_of_memory+0x4b6/0x4f0
[   15.052864]  [<ffffffff8114b579>] __alloc_pages_nodemask+0xa09/0xb10
[   15.059482]  [<ffffffff81188779>] alloc_pages_current+0xa9/0x170
[   15.065711]  [<ffffffff811419f7>] __page_cache_alloc+0x87/0xb0
[   15.071804]  [<ffffffff81142606>]
grab_cache_page_write_begin+0x76/0xd0
[   15.078646]  [<ffffffffa02aa133>] ext4_da_write_begin+0xa3/0x330
[ext4]
[   15.085495]  [<ffffffff8114162e>]
generic_file_buffered_write+0x11e/0x290
[   15.092504]  [<ffffffff81143785>]
__generic_file_aio_write+0x1d5/0x3e0
[   15.099294]  [<ffffffff81050f00>] ?
rbt_memtype_copy_nth_element+0xa0/0xa0
[   15.106385]  [<ffffffff811439ed>] generic_file_aio_write+0x5d/0xc0
[   15.112841]  [<ffffffffa02a0189>] ext4_file_write+0xa9/0x450 [ext4]
[   15.119321]  [<ffffffff8117997c>] ? free_vmap_area_noflush+0x7c/0x90
[   15.125884]  [<ffffffff811af36d>] do_sync_write+0x8d/0xd0
[   15.131492]  [<ffffffff811afb0d>] vfs_write+0xbd/0x1e0
[   15.136839]  [<ffffffff811b0558>] SyS_write+0x58/0xb0
[   15.142091]  [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b
[   15.148293] Mem-Info:
[   15.150770] Node 0 DMA per-cpu:
[   15.154138] CPU    0: hi:    0, btch:   1 usd:   0
[   15.159133] Node 0 DMA32 per-cpu:
[   15.162741] CPU    0: hi:   42, btch:   7 usd:  12
[   15.167732] active_anon:14395 inactive_anon:1034 isolated_anon:0
[   15.167732]  active_file:2406 inactive_file:2533 isolated_file:0
[   15.167732]  unevictable:7137 dirty:2 writeback:1511 unstable:0
[   15.167732]  free:488 slab_reclaimable:2371 slab_unreclaimable:3533
[   15.167732]  mapped:1110 shmem:1065 pagetables:166 bounce:0
[   15.167732]  free_cma:0
[   15.203076] Node 0 DMA free:508kB min:4kB low:4kB high:4kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictabs
[   15.242882] lowmem_reserve[]: 0 128 128 128
[   15.247447] Node 0 DMA32 free:1444kB min:1444kB low:1804kB
high:2164kB active_anon:57580kB inactive_anon:4136kB active_file:9624kB
inacts
[   15.292683] lowmem_reserve[]: 0 0 0 0
[   15.296761] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U)
1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B
[   15.310372] Node 0 DMA32: 78*4kB (UEM) 52*8kB (UEM) 17*16kB (UM)
12*32kB (UM) 2*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*40B
[   15.324412] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[   15.333088] 13144 total pagecache pages
[   15.337161] 0 pages in swap cache
[   15.340708] Swap cache stats: add 0, delete 0, find 0/0
[   15.346165] Free swap  = 0kB
[   15.349280] Total swap = 0kB
[   15.353385] 90211 pages RAM
[   15.356420] 53902 pages reserved
[   15.359880] 6980 pages shared
[   15.363088] 29182 pages non-shared
[   15.366719] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
[   15.374788] [   85]     0    85    13020      553      24        0
0 systemd-journal
[   15.383818] [  134]     0   134     8860      547      22        0
-1000 systemd-udevd
[   15.392664] [  146]     0   146     5551      245      23        0
0 plymouthd
[   15.401167] [  230]     0   230     3106      537      16        0
0 dracut-pre-pivo
[   15.410181] [  286]     0   286    19985    13756      55        0
0 makedumpfile
[   15.418942] Out of memory: Kill process 286 (makedumpfile) score 368
or sacrifice child
[   15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,

anon-rss:54132kB, file-rss:892kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"

Quoted 5 lines

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#31 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

On Mon, May 19, 2014 at 07:15:38PM +0800, [email protected] wrote:

[..]

Quoted 14 lines

-------------------------------------------------
bhe# cat /etc/kdump.conf
path /var/crash
core_collector makedumpfile -E --message-level 1 -d 31

------------------------------------------
kdump: dump target is /dev/sda2
kdump: saving [ 9.595153] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
to /sysroot//var/crash/127.0.0.1-2014.05.19-18:50:18/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore

Quoted 3 lines

calculate_cyclic_buffer_size, get_free_memory_size: 68857856

Buffer size for the cyclic mode: 27543142

Bao,

So 68857856 is 65MB. So we have around 65MB free when makedumpfile started.

27543142 is 26MB. So we reserved 26MB for bitmaps or we reserved 52MB for bitmaps?

Looking at the backtrace, larry pointed out few things.

- makedumpfile has already allocated around 52MB of anonymous memory. I guess this primarily comes from bitmaps and looks like we are reserving 52MB in bitmaps and not 26MB. I think this could be consistent with current 80% logic as 80% of 65MB is around 52MB.

	[   15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,
			anon-rss:54132kB, file-rss:892kB

- So we are left with 65-52 = 13MB of total memory for kernel as well as makedumpfile.

- We have around 1500 pages in page cache which are in writeback stage. That means around 6MB of pages are dirty and being written back to disk. That means makedumpfile might not require lot of memory but kernel does require free memory in dirty/writeback pages when dump file is being written.

	[   15.167732]  unevictable:7137 dirty:2 writeback:1511 unstable:0

- Larry mentioend that there are around 5000 pages (20MB of memory) sitting in file pages in page cache which ideally should be reclaimable. It is not clear why that memory is not being reclaimed fast enough.

	[   15.167732]  active_file:2406 inactive_file:2533 isolated_file:0

So to me bottom line is that once the write out starts, kernel needs memory for holding dirty and writeback pages in cache too. So we probably are being too aggresive in allocating 80% of free memory for bitmaps. May be we should drop it down to 50-60% of free memory for bitmaps.

Thanks
Vivek

Quoted 91 lines

Copying data : [ 15.9 %] -[ 14.955468]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0

[   14.963876] makedumpfile cpuset=/ mems_allowed=0
[   14.968723] CPU: 0 PID: 286 Comm: makedumpfile Not tainted
3.10.0-123.el7.x86_64 #1
[   14.976606] Hardware name: Hewlett-Packard HP Z420 Workstation/1589,
BIOS J61 v01.02 03/09/2012
[   14.985567]  ffff88002fedc440 00000000f650c592 ffff88002fcb57d0
ffffffff815e19ba
[   14.993291]  ffff88002fcb5860 ffffffff815dd02d ffffffff810b68f8
ffff8800359dc0c0
[   15.001013]  ffffffff00000206 ffffffff00000000 0000000000000000
ffffffff81102e03
[   15.008733] Call Trace:
[   15.011413]  [<ffffffff815e19ba>] dump_stack+0x19/0x1b
[   15.016778]  [<ffffffff815dd02d>] dump_header+0x8e/0x214
[   15.022321]  [<ffffffff810b68f8>] ? ktime_get_ts+0x48/0xe0
[   15.028036]  [<ffffffff81102e03>] ? proc_do_uts_string+0xe3/0x130
[   15.034383]  [<ffffffff8114520e>] oom_kill_process+0x24e/0x3b0
[   15.040446]  [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30
[   15.047068]  [<ffffffff81145a36>] out_of_memory+0x4b6/0x4f0
[   15.052864]  [<ffffffff8114b579>] __alloc_pages_nodemask+0xa09/0xb10
[   15.059482]  [<ffffffff81188779>] alloc_pages_current+0xa9/0x170
[   15.065711]  [<ffffffff811419f7>] __page_cache_alloc+0x87/0xb0
[   15.071804]  [<ffffffff81142606>]
grab_cache_page_write_begin+0x76/0xd0
[   15.078646]  [<ffffffffa02aa133>] ext4_da_write_begin+0xa3/0x330
[ext4]
[   15.085495]  [<ffffffff8114162e>]
generic_file_buffered_write+0x11e/0x290
[   15.092504]  [<ffffffff81143785>]
__generic_file_aio_write+0x1d5/0x3e0
[   15.099294]  [<ffffffff81050f00>] ?
rbt_memtype_copy_nth_element+0xa0/0xa0
[   15.106385]  [<ffffffff811439ed>] generic_file_aio_write+0x5d/0xc0
[   15.112841]  [<ffffffffa02a0189>] ext4_file_write+0xa9/0x450 [ext4]
[   15.119321]  [<ffffffff8117997c>] ? free_vmap_area_noflush+0x7c/0x90
[   15.125884]  [<ffffffff811af36d>] do_sync_write+0x8d/0xd0
[   15.131492]  [<ffffffff811afb0d>] vfs_write+0xbd/0x1e0
[   15.136839]  [<ffffffff811b0558>] SyS_write+0x58/0xb0
[   15.142091]  [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b
[   15.148293] Mem-Info:
[   15.150770] Node 0 DMA per-cpu:
[   15.154138] CPU    0: hi:    0, btch:   1 usd:   0
[   15.159133] Node 0 DMA32 per-cpu:
[   15.162741] CPU    0: hi:   42, btch:   7 usd:  12
[   15.167732] active_anon:14395 inactive_anon:1034 isolated_anon:0
[   15.167732]  active_file:2406 inactive_file:2533 isolated_file:0
[   15.167732]  unevictable:7137 dirty:2 writeback:1511 unstable:0
[   15.167732]  free:488 slab_reclaimable:2371 slab_unreclaimable:3533
[   15.167732]  mapped:1110 shmem:1065 pagetables:166 bounce:0
[   15.167732]  free_cma:0
[   15.203076] Node 0 DMA free:508kB min:4kB low:4kB high:4kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictabs
[   15.242882] lowmem_reserve[]: 0 128 128 128
[   15.247447] Node 0 DMA32 free:1444kB min:1444kB low:1804kB
high:2164kB active_anon:57580kB inactive_anon:4136kB active_file:9624kB
inacts
[   15.292683] lowmem_reserve[]: 0 0 0 0
[   15.296761] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U)
1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B
[   15.310372] Node 0 DMA32: 78*4kB (UEM) 52*8kB (UEM) 17*16kB (UM)
12*32kB (UM) 2*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*40B
[   15.324412] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[   15.333088] 13144 total pagecache pages
[   15.337161] 0 pages in swap cache
[   15.340708] Swap cache stats: add 0, delete 0, find 0/0
[   15.346165] Free swap  = 0kB
[   15.349280] Total swap = 0kB
[   15.353385] 90211 pages RAM
[   15.356420] 53902 pages reserved
[   15.359880] 6980 pages shared
[   15.363088] 29182 pages non-shared
[   15.366719] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
[   15.374788] [   85]     0    85    13020      553      24        0
0 systemd-journal
[   15.383818] [  134]     0   134     8860      547      22        0
-1000 systemd-udevd
[   15.392664] [  146]     0   146     5551      245      23        0
0 plymouthd
[   15.401167] [  230]     0   230     3106      537      16        0
0 dracut-pre-pivo
[   15.410181] [  286]     0   286    19985    13756      55        0
0 makedumpfile
[   15.418942] Out of memory: Kill process 286 (makedumpfile) score 368
or sacrifice child
[   15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,
Quoted 9 lines

anon-rss:54132kB, file-rss:892kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"

Quoted 5 lines

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#34 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to [email protected]

Hello Bao,

Quoted 22 lines

Hi Astushi,

Quoted 11 lines

No, 16MB was used for bitmaps also in my case.
--cyclic-buffer option means specifying each bitmap size, so allocated bitmap size is the double of --cyclic-buffer.
(But now, this behavior is only for ELF case, just the specified size is allocated in kdump case. Yes, it's confusing...)

--cyclic-buffer buffer_size
Specify the buffer size in kilo bytes for analysis in the cyclic mode. Actually, the double of buffer_size kilo bytes will be allocated in memory. In the cyclic mode, the number of cycles is represented as:

Yeah, I was wrong about this. If specified --cyclic-buffer doesn't exceed the memory limit, it is equal to info->bufsize_cyclic. But for elf dump, it need be double. E.g if free memory is 30M, --cyclic-buffer is 16M, it will absolutely fail. But the code check here should be done separately. Since it's meaningless for elf dump if 2 x --cyclic-buffer exceeds the free memory.

Quoted 5 lines
	if (info->bufsize_cyclic > free_memory) {
	.......
	{

I think code here need be changed.

Exactly, this is a bug.
I can fix it simply by modification to the check above, but I think it's better to change the ELF path to get rid of the 1st bitmap buffer since it's unused. Then the required buffer size will be just <--cyclic-buffer> in both cases, so I'll post the patch to do it later.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#32 lore raw

RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

In reply to Vivek Goyal
Quoted 21 lines · On Mon, May 19, 2014 at 07:15:38PM +0800, [email protected] wrote:

[..]

Quoted 14 lines

-------------------------------------------------
bhe# cat /etc/kdump.conf
path /var/crash
core_collector makedumpfile -E --message-level 1 -d 31

------------------------------------------
kdump: dump target is /dev/sda2
kdump: saving [ 9.595153] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
to /sysroot//var/crash/127.0.0.1-2014.05.19-18:50:18/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore

Quoted 3 lines

calculate_cyclic_buffer_size, get_free_memory_size: 68857856

Buffer size for the cyclic mode: 27543142

Quoted 7 lines

Bao,

So 68857856 is 65MB. So we have around 65MB free when makedumpfile started.

27543142 is 26MB. So we reserved 26MB for bitmaps or we reserved 52MB for bitmaps?

52MB is correct, so the larry's view below looks right.

Quoted 25 lines

Looking at the backtrace, larry pointed out few things.

- makedumpfile has already allocated around 52MB of anonymous memory. I guess this primarily comes from bitmaps and looks like we are reserving 52MB in bitmaps and not 26MB. I think this could be consistent with current 80% logic as 80% of 65MB is around 52MB.

	[   15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,
			anon-rss:54132kB, file-rss:892kB

- So we are left with 65-52 = 13MB of total memory for kernel as well as makedumpfile.

- We have around 1500 pages in page cache which are in writeback stage. That means around 6MB of pages are dirty and being written back to disk. That means makedumpfile might not require lot of memory but kernel does require free memory in dirty/writeback pages when dump file is being written.

	[   15.167732]  unevictable:7137 dirty:2 writeback:1511 unstable:0

- Larry mentioend that there are around 5000 pages (20MB of memory) sitting in file pages in page cache which ideally should be reclaimable. It is not clear why that memory is not being reclaimed fast enough.

Quoted 6 lines
	[   15.167732]  active_file:2406 inactive_file:2533 isolated_file:0

So to me bottom line is that once the write out starts, kernel needs memory for holding dirty and writeback pages in cache too. So we probably are being too aggresive in allocating 80% of free memory for bitmaps. May be we should drop it down to 50-60% of free memory for bitmaps.

I don't disagree to changing the 80% limit but I prefer to remove such a percentage threshold because it's dependent on the environment. Actually, I think it makes this problem more complex.

Now, thanks to page_is_buddy(), the performance degradation caused by multi-cycle processing looks very small according to the benchmark on 2TB memory:

https://lkml.org/lkml/2013/3/26/914

This result means we don't need to make an effort to allocate the bitmap buffer as large as possible. So how about just setting a small fixed value like 5MB as a safety limit?
It may be safer, and it will be easier to estimate the total memory usage of makedumpfile, so I think it's better way if the most users especially large machine users accept it.

Thanks
Atsushi Kumagai

Quoted 97 lines
Quoted 97 lines

Copying data : [ 15.9 %] -[ 14.955468]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0

[   14.963876] makedumpfile cpuset=/ mems_allowed=0
[   14.968723] CPU: 0 PID: 286 Comm: makedumpfile Not tainted
3.10.0-123.el7.x86_64 #1
[   14.976606] Hardware name: Hewlett-Packard HP Z420 Workstation/1589,
BIOS J61 v01.02 03/09/2012
[   14.985567]  ffff88002fedc440 00000000f650c592 ffff88002fcb57d0
ffffffff815e19ba
[   14.993291]  ffff88002fcb5860 ffffffff815dd02d ffffffff810b68f8
ffff8800359dc0c0
[   15.001013]  ffffffff00000206 ffffffff00000000 0000000000000000
ffffffff81102e03
[   15.008733] Call Trace:
[   15.011413]  [<ffffffff815e19ba>] dump_stack+0x19/0x1b
[   15.016778]  [<ffffffff815dd02d>] dump_header+0x8e/0x214
[   15.022321]  [<ffffffff810b68f8>] ? ktime_get_ts+0x48/0xe0
[   15.028036]  [<ffffffff81102e03>] ? proc_do_uts_string+0xe3/0x130
[   15.034383]  [<ffffffff8114520e>] oom_kill_process+0x24e/0x3b0
[   15.040446]  [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30
[   15.047068]  [<ffffffff81145a36>] out_of_memory+0x4b6/0x4f0
[   15.052864]  [<ffffffff8114b579>] __alloc_pages_nodemask+0xa09/0xb10
[   15.059482]  [<ffffffff81188779>] alloc_pages_current+0xa9/0x170
[   15.065711]  [<ffffffff811419f7>] __page_cache_alloc+0x87/0xb0
[   15.071804]  [<ffffffff81142606>]
grab_cache_page_write_begin+0x76/0xd0
[   15.078646]  [<ffffffffa02aa133>] ext4_da_write_begin+0xa3/0x330
[ext4]
[   15.085495]  [<ffffffff8114162e>]
generic_file_buffered_write+0x11e/0x290
[   15.092504]  [<ffffffff81143785>]
__generic_file_aio_write+0x1d5/0x3e0
[   15.099294]  [<ffffffff81050f00>] ?
rbt_memtype_copy_nth_element+0xa0/0xa0
[   15.106385]  [<ffffffff811439ed>] generic_file_aio_write+0x5d/0xc0
[   15.112841]  [<ffffffffa02a0189>] ext4_file_write+0xa9/0x450 [ext4]
[   15.119321]  [<ffffffff8117997c>] ? free_vmap_area_noflush+0x7c/0x90
[   15.125884]  [<ffffffff811af36d>] do_sync_write+0x8d/0xd0
[   15.131492]  [<ffffffff811afb0d>] vfs_write+0xbd/0x1e0
[   15.136839]  [<ffffffff811b0558>] SyS_write+0x58/0xb0
[   15.142091]  [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b
[   15.148293] Mem-Info:
[   15.150770] Node 0 DMA per-cpu:
[   15.154138] CPU    0: hi:    0, btch:   1 usd:   0
[   15.159133] Node 0 DMA32 per-cpu:
[   15.162741] CPU    0: hi:   42, btch:   7 usd:  12
[   15.167732] active_anon:14395 inactive_anon:1034 isolated_anon:0
[   15.167732]  active_file:2406 inactive_file:2533 isolated_file:0
[   15.167732]  unevictable:7137 dirty:2 writeback:1511 unstable:0
[   15.167732]  free:488 slab_reclaimable:2371 slab_unreclaimable:3533
[   15.167732]  mapped:1110 shmem:1065 pagetables:166 bounce:0
[   15.167732]  free_cma:0
[   15.203076] Node 0 DMA free:508kB min:4kB low:4kB high:4kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictabs
[   15.242882] lowmem_reserve[]: 0 128 128 128
[   15.247447] Node 0 DMA32 free:1444kB min:1444kB low:1804kB
high:2164kB active_anon:57580kB inactive_anon:4136kB active_file:9624kB
inacts
[   15.292683] lowmem_reserve[]: 0 0 0 0
[   15.296761] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U)
1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B
[   15.310372] Node 0 DMA32: 78*4kB (UEM) 52*8kB (UEM) 17*16kB (UM)
12*32kB (UM) 2*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*40B
[   15.324412] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[   15.333088] 13144 total pagecache pages
[   15.337161] 0 pages in swap cache
[   15.340708] Swap cache stats: add 0, delete 0, find 0/0
[   15.346165] Free swap  = 0kB
[   15.349280] Total swap = 0kB
[   15.353385] 90211 pages RAM
[   15.356420] 53902 pages reserved
[   15.359880] 6980 pages shared
[   15.363088] 29182 pages non-shared
[   15.366719] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
[   15.374788] [   85]     0    85    13020      553      24        0
0 systemd-journal
[   15.383818] [  134]     0   134     8860      547      22        0
-1000 systemd-udevd
[   15.392664] [  146]     0   146     5551      245      23        0
0 plymouthd
[   15.401167] [  230]     0   230     3106      537      16        0
0 dracut-pre-pivo
[   15.410181] [  286]     0   286    19985    13756      55        0
0 makedumpfile
[   15.418942] Out of memory: Kill process 286 (makedumpfile) score 368
or sacrifice child
[   15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,

anon-rss:54132kB, file-rss:892kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"

Quoted 2 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#33 lore raw

Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump

On Tue, May 27, 2014 at 05:34:05AM +0000, Atsushi Kumagai wrote:

[..]

Quoted 13 lines
Quoted 4 lines

So to me bottom line is that once the write out starts, kernel needs memory for holding dirty and writeback pages in cache too. So we probably are being too aggresive in allocating 80% of free memory for bitmaps. May be we should drop it down to 50-60% of free memory for bitmaps.

I don't disagree to changing the 80% limit but I prefer to remove such a percentage threshold because it's dependent on the environment. Actually, I think it makes this problem more complex.

Now, thanks to page_is_buddy(), the performance degradation caused by multi-cycle processing looks very small according to the benchmark on 2TB memory:

Quoted 8 lines

https://lkml.org/lkml/2013/3/26/914

This result means we don't need to make an effort to allocate the bitmap buffer as large as possible. So how about just setting a small fixed value like 5MB as a safety limit?
It may be safer, and it will be easier to estimate the total memory usage of makedumpfile, so I think it's better way if the most users especially large machine users accept it.

Hi Atsushi,

If increasing buffer size does not have any significant increase in dump time, then it is reasonable to have fixed buffer size for bitmaps. (instead of trying to maximize bitmap size).

We can probably go for 4MB as bitmap size (instead of 5MB).

Also can we modify the logic a bit so that we automatically shrhink the size of bitmap if sufficient memory is not available. Say assume that 60% of available memory can be used for bitmaps. If it is less then 4MB then we drop the buffer size and that hopefully still makes makedumpfile successful (instead of being OOMed).

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec