makedumpfile memmory usage seems high with option -E
Subsystems (2)
Hi Atsushi/Hatayama,
We noticed in our testing that makedumpfile gets OOM killed if we happen to use -E option. Saving to compressed kdump format works just fine.
Also we noticed that with -E if we disable cyclic mode then it works just fine.
So looks like something is going on with -E and cyclic mode enabled. I am not sure what it is.
Do you suspect something?
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Vivek,
Quoted 5 lines
Hi Atsushi/Hatayama,
We noticed in our testing that makedumpfile gets OOM killed if we happen to use -E option. Saving to compressed kdump format works just fine.
Quoted 7 lines
Also we noticed that with -E if we disable cyclic mode then it works just fine.
So looks like something is going on with -E and cyclic mode enabled. I am not sure what it is.
Do you suspect something?
At first, I supposed the function to calculate cyclic buffer size may be related to this issue, but I haven't found the answer yet...
intcalculate_cyclic_buffer_size() {
if (info->flag_elf_dumpfile) {
free_size = get_free_memory_size() * 0.4;
needed_size = (info->max_mapnr * 2) / BITPERBYTE;
} else {
free_size = get_free_memory_size() * 0.8;
needed_size = info->max_mapnr / BITPERBYTE;
}
[...]
info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;I've found this function has an issue about memory allocation. When -E is specified, info->bufsize_cyclic will be the total size of the 1st and 2nd bitmap if free memory is enough. Then, info->bufsize_cyclic will be used to allocate each bitmap in prepare_bitmap_buffer_cyclic() like below:
if ((info->partial_bitmap1 = (char *)malloc(info->bufsize_cyclic)) == NULL) {
ERRMSG("Can't allocate memory for the 1st-bitmap. %s\n",
strerror(errno));
return FALSE;
}
if ((info->partial_bitmap2 = (char *)malloc(info->bufsize_cyclic)) == NULL) {
ERRMSG("Can't allocate memory for the 2nd-bitmap. %s\n",
strerror(errno));
return FALSE;
}It's a too much allocation definitely, but it mustn't exceed 80% of free memory due to the condition check in calculate_cyclic_buffer_size(), so I think the OOM issue will not happen by this issue.
I'll fix this too much allocation with the patch below, but it will not resolve your OOM issue...
BTW, what are your version of makedumpfile and crashkernel= size and the system memory size? and does the issue happen even if you specify --cyclic-buffer which is small enough to fit the available memory ? I'm curious to know the details of the condition which cause the issue.
Thanks
Atsushi Kumagai
makedumpfile.c +6 -7
@@ -8996,7 +8996,7 @@ out:*/ int calculate_cyclic_buffer_size (void ) { -unsigned long long free_size, needed_size; +unsigned long long limit_size, bitmap_size;if (info->max_mapnr <=0 ) {ERRMSG ("Invalid max_mapnr(%llu).\n" , info->max_mapnr);
@@ -9009,18 +9009,17 @@ calculate_cyclic_buffer_size(void) {* within 80% of free memory. */ if (info->flag_elf_dumpfile) { - free_size =get_free_memory_size () *0.4 ; - needed_size = (info->max_mapnr *2 ) /BITPERBYTE ; + limit_size =get_free_memory_size () *0.4 ; }else { - free_size =get_free_memory_size () *0.8 ; - needed_size = info->max_mapnr /BITPERBYTE ; + limit_size =get_free_memory_size () *0.8 ; } + bitmap_size = info->max_mapnr /BITPERBYTE ;/* if --split was specified cyclic buffer allocated per dump file */ if (info->num_dumpfile >1 ) - needed_size /= info->num_dumpfile; + bitmap_size /= info->num_dumpfile; - info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size; + info->bufsize_cyclic = (limit_size <= bitmap_size) ? limit_size : bitmap_size;return TRUE; }
Signature · 6 lines
1.8.0.2 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
----- Original Message -----
Quoted 58 lines
Hello Vivek,
Quoted 5 lines
Hi Atsushi/Hatayama,
We noticed in our testing that makedumpfile gets OOM killed if we happen to use -E option. Saving to compressed kdump format works just fine.
Quoted 7 lines
Also we noticed that with -E if we disable cyclic mode then it works just fine.
So looks like something is going on with -E and cyclic mode enabled. I am not sure what it is.
Do you suspect something?
At first, I supposed the function to calculate cyclic buffer size may be related to this issue, but I haven't found the answer yet...
int
calculate_cyclic_buffer_size(){if (info->flag_elf_dumpfile) { free_size = get_free_memory_size() * 0.4; needed_size = (info->max_mapnr * 2) / BITPERBYTE; } else { free_size = get_free_memory_size() * 0.8; needed_size = info->max_mapnr / BITPERBYTE; } [...] info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;I've found this function has an issue about memory allocation. When -E is specified,
info->bufsize_cyclicwill be the total size of the 1st and 2nd bitmap if free memory is enough. Then,info->bufsize_cyclicwill be used to allocate each bitmap inprepare_bitmap_buffer_cyclic()like below:if ((info->partial_bitmap1 = (char *)malloc(info->bufsize_cyclic)) == NULL) { ERRMSG("Can't allocate memory for the 1st-bitmap. %s\n", strerror(errno)); return FALSE; } if ((info->partial_bitmap2 = (char *)malloc(info->bufsize_cyclic)) == NULL) { ERRMSG("Can't allocate memory for the 2nd-bitmap. %s\n", strerror(errno)); return FALSE; }It's a too much allocation definitely, but it mustn't exceed 80% of free memory due to the condition check in
calculate_cyclic_buffer_size(), so I think the OOM issue will not happen by this issue.
I'll fix this too much allocation with the patch below, but it will not resolve your OOM issue...
Quoted 6 lines
BTW, what are your version of makedumpfile and crashkernel= size and the system memory size? and does the issue happen even if you specify --cyclic-buffer which is small enough to fit the available memory ? I'm curious to know the details of the condition which cause the issue.
Hi Atsushi Kumagai
The makedumpfile version is 1.5.4. crashkernel=auto which actually is 166M.
the system memory size is about 96G and the arch is x86_64 which kernel version 3.10.110. But the same issue happened when using the newest makedumpfile in devel branch. After add --cyclic-buffer=100, dump succeed.
Thanks
arthur
Quoted 46 lines
Thanks
Atsushi Kumagaimakedumpfile.c +6
-7@@ -8996,7 +8996,7 @@ out:*/ int calculate_cyclic_buffer_size (void ) { -unsigned long long free_size, needed_size; +unsigned long long limit_size, bitmap_size;if (info->max_mapnr <=0 ) {ERRMSG ("Invalid max_mapnr(%llu).\n" , info->max_mapnr);@@ -9009,18 +9009,17 @@ calculate_cyclic_buffer_size(void) {* within 80% of free memory. */ if (info->flag_elf_dumpfile) { - free_size =get_free_memory_size () *0.4 ; - needed_size = (info->max_mapnr *2 ) /BITPERBYTE ; + limit_size =get_free_memory_size () *0.4 ; }else { - free_size =get_free_memory_size () *0.8 ; - needed_size = info->max_mapnr /BITPERBYTE ; + limit_size =get_free_memory_size () *0.8 ; } + bitmap_size = info->max_mapnr /BITPERBYTE ;/* if --split was specified cyclic buffer allocated per dump file */ if (info->num_dumpfile >1 ) - needed_size /= info->num_dumpfile; + bitmap_size /= info->num_dumpfile; - info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size; + info->bufsize_cyclic = (limit_size <= bitmap_size) ? limit_size : bitmap_size;return TRUE;
}
--
1.8.0.2
Quoted 5 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH 1/1 makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for bufsize_cyclic need be double. Hence free size should be 40% of free memory, however the needed_size which free_size is compared with should be info->max_mapnr / BITPERBYTE, but not 2 times of it.
Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.
--- makedumpfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
makedumpfile.c +1 -1
@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {*/ if (info->flag_elf_dumpfile) { free_size =get_free_memory_size () *0.4 ; - needed_size = (info->max_mapnr *2 ) /BITPERBYTE ; + needed_size = info->max_mapnr /BITPERBYTE ; }else { free_size =get_free_memory_size () *0.8 ; needed_size = info->max_mapnr /BITPERBYTE ;
Signature · 7 lines
1.8.5.3 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Hi Atsushi,
Since both of our test machines where this bug was reproduced are reserved, I haven't tested this patch. But after several times of the code flow of elf dump, finally this only this suspect is found.
Could you also help check this?
Also below change is OK too, and its meaning is more obvious. But my previous patch is more readable for code.
int
calculate_cyclic_buffer_size(void) {
unsigned long long free_size, needed_size;
if (info->max_mapnr <= 0) {
ERRMSG("Invalid max_mapnr(%llu).\n", info->max_mapnr);
return FALSE;
}
/*
* free_size will be used to allocate 1st and 2nd bitmap, so it
* should be 40% of free memory to keep the size of cyclic
* buffer
* within 80% of free memory.
*/
if (info->flag_elf_dumpfile) {
free_size = get_free_memory_size() * 0.4 * 2;
needed_size = (info->max_mapnr * 2) / BITPERBYTE;
} else {
free_size = get_free_memory_size() * 0.8;
needed_size = info->max_mapnr / BITPERBYTE;
}
/* if --split was specified cyclic buffer allocated per dump
* file */
if (info->num_dumpfile > 1)
needed_size /= info->num_dumpfile;
info->bufsize_cyclic = (free_size <= needed_size) ? free_size/2 :needed_size/2;
return TRUE; }
Thanks
Baoquan
Quoted 35 lines · Baoquan He
In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for
bufsize_cyclicneed be double. Hence free size should be 40% of free memory, however theneeded_sizewhichfree_sizeis compared with should beinfo->max_mapnr/ BITPERBYTE, but not 2 times of it.Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.
Signed-off-by Baoquan He <[email protected]>--- makedumpfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)makedumpfile.c +1
-1@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {*/ if (info->flag_elf_dumpfile) { free_size =get_free_memory_size () *0.4 ; - needed_size = (info->max_mapnr *2 ) /BITPERBYTE ; + needed_size = info->max_mapnr /BITPERBYTE ; }else { free_size =get_free_memory_size () *0.8 ; needed_size = info->max_mapnr /BITPERBYTE ;Signature · 7 lines
1.8.5.3 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Baoquan HeHi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Thanks
Baoquan
Quoted 35 lines · Baoquan He
In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for
bufsize_cyclicneed be double. Hence free size should be 40% of free memory, however theneeded_sizewhichfree_sizeis compared with should beinfo->max_mapnr/ BITPERBYTE, but not 2 times of it.Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.
Signed-off-by Baoquan He <[email protected]>--- makedumpfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)makedumpfile.c +1
-1@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {*/ if (info->flag_elf_dumpfile) { free_size =get_free_memory_size () *0.4 ; - needed_size = (info->max_mapnr *2 ) /BITPERBYTE ; + needed_size = info->max_mapnr /BITPERBYTE ; }else { free_size =get_free_memory_size () *0.8 ; needed_size = info->max_mapnr /BITPERBYTE ;Signature · 7 lines
1.8.5.3 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Hello Baoquan,
Quoted 4 lines
Hi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Great, thanks for your help!
However, I still have questions.
First, what is the difference between yours and mine?
http://lists.infradead.org/pipermail/kexec/2014-April/011535.html
My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,
Quoted 9 lines · Baoquan He
Quoted 8 lines
In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for
bufsize_cyclicneed be double. Hence free size should be 40% of free memory, however theneeded_sizewhichfree_sizeis compared with should beinfo->max_mapnr/ BITPERBYTE, but not 2 times of it.Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.
but I still don't think this bug causes OOM.
Even if needed_size is calculated as so much size wrongly, bufsize_cyclic will not exceed 40% of free memory by the check below:
info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;
So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.
I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?
Thanks
Atsushi Kumagai
Quoted 22 lines
Quoted 22 lines
Signed-off-by Baoquan He <[email protected]>--- makedumpfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)makedumpfile.c +1
-2@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {*/ if (info->flag_elf_dumpfile) { free_size =get_free_memory_size () *0.4 ; - needed_size = (info->max_mapnr *2 ) /BITPERBYTE ; + needed_size = info->max_mapnr /BITPERBYTE ; }else { free_size =get_free_memory_size () *0.8 ; needed_size = info->max_mapnr /BITPERBYTE ; --1.8.5.3
Quoted 4 lines
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 7 lines · Atsushi Kumagai
Hello Baoquan,
Quoted 4 lines
Hi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Quoted 6 lines
Great, thanks for your help!
However, I still have questions.First, what is the difference between yours and mine?
http://lists.infradead.org/pipermail/kexec/2014-April/011535.html
Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.
Quoted 19 lines
My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,Quoted 9 lines · Baoquan He
Quoted 8 lines
In case elf dump, the code to calculate the cyclic buffer size is not correct. Since elf need bitmap1/2, so the needed memory for
bufsize_cyclicneed be double. Hence free size should be 40% of free memory, however theneeded_sizewhichfree_sizeis compared with should beinfo->max_mapnr/ BITPERBYTE, but not 2 times of it.Because of this, in case of free memory in 2nd kernel is not too much, OOM will happen very often.
but I still don't think this bug causes OOM.
Even ifneeded_sizeis calculated as so much size wrongly,bufsize_cyclicwill not exceed 40% of free memory by the check below:
Quoted 7 lines
info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.
I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its needed_size is 3200K*2 == 6.4M, if currently free memory is only 15M left, the free_size will be 15M*0.4 which is 6M. So info->bufsize_cyclic is assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.
Quoted 27 lines
Thanks
Atsushi KumagaiQuoted 22 lines
Quoted 22 lines
Signed-off-by Baoquan He <[email protected]>--- makedumpfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)makedumpfile.c +1
-2@@ -9010,7 +9010,7 @@ calculate_cyclic_buffer_size(void) {*/ if (info->flag_elf_dumpfile) { free_size =get_free_memory_size () *0.4 ; - needed_size = (info->max_mapnr *2 ) /BITPERBYTE ; + needed_size = info->max_mapnr /BITPERBYTE ; }else { free_size =get_free_memory_size () *0.8 ; needed_size = info->max_mapnr /BITPERBYTE ; --1.8.5.3
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 11 lines · Baoquan He
Quoted 10 lines · Atsushi Kumagai
Hello Baoquan,
Quoted 4 lines
Hi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Great, thanks for your help!
However, I still have questions.
Quoted 6 lines
Quoted 3 lines
First, what is the difference between yours and mine?
http://lists.infradead.org/pipermail/kexec/2014-April/011535.html
Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.
must<--
Quoted 16 lines
Quoted 16 lines
My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,Quoted 1 line · Baoquan He
but I still don't think this bug causes OOM.
Even ifneeded_sizeis calculated as so much size wrongly,bufsize_cyclicwill not exceed 40% of free memory by the check below:info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.
Quoted 9 lines
Quoted 2 lines
I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.
BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines
Hello Baoquan,
Quoted 4 lines
Hi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Great, thanks for your help!
However, I still have questions.Quoted 6 lines
Quoted 3 lines
First, what is the difference between yours and mine?
http://lists.infradead.org/pipermail/kexec/2014-April/011535.html
Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.
must<--Quoted 16 lines
Quoted 16 lines
My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,Quoted 1 line · Baoquan He
but I still don't think this bug causes OOM.
Even ifneeded_sizeis calculated as so much size wrongly,bufsize_cyclicwill not exceed 40% of free memory by the check below:info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.
Quoted 9 lines
Quoted 2 lines
I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.
Quoted 2 lines
BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.
What I think is:
1. Use a constant value as safe limit to calculate bufsize_cyclic instead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.
2. If free memory is smaller than the value, makedumpfile gives up to work early.
This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.
Any comments are welcome.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Quoted 7 lines
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (base_value + linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.
Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of base_value is needed. So a constant value as you said may not be needed.
Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.
Quoted 17 lines
I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.
What I think is:
1. Use a constant value as safe limit to calculate
bufsize_cyclicinstead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.2. If free memory is smaller than the value, makedumpfile gives up to work early.
This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.
Quoted 5 lines
Any comments are welcome.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 25 lines · On Fri, 18 Apr 2014 22:29:12 +0800 "[email protected]" <[email protected]> wrote:
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Quoted 7 lines
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (
base_value+ linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.
Quoted 6 lines
Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of
base_valueis needed. So a constant value as you said may not be needed.Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.
I believe these 80% come from the default value of vm.dirty_ratio, which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.
I figured out that the actual requirements also depend on the target device driver, filesystem (e.g. NFS is considerably more memory-hungry than ext3) and (not the least) number of online CPUs.
The calculation is quite complex, and that's probably why nobody has done it properly yet.
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Atsushi KumagaiQuoted 48 lines · Atsushi Kumagai
Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines
Hello Baoquan,
Quoted 4 lines
Hi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Great, thanks for your help!
However, I still have questions.Quoted 6 lines
Quoted 3 lines
First, what is the difference between yours and mine?
http://lists.infradead.org/pipermail/kexec/2014-April/011535.html
Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.
must<--Quoted 16 lines
Quoted 16 lines
My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,Quoted 1 line · Baoquan He
but I still don't think this bug causes OOM.
Even ifneeded_sizeis calculated as so much size wrongly,bufsize_cyclicwill not exceed 40% of free memory by the check below:info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.
Quoted 9 lines
Quoted 2 lines
I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.Quoted 2 lines
BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Quoted 7 lines
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?
Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.
Quoted 4 lines
I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.
Quoted 8 lines
What I think is:
1. Use a constant value as safe limit to calculate
bufsize_cyclicinstead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.2. If free memory is smaller than the value, makedumpfile gives up to work early.
What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.
Quoted 4 lines
This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.
I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.
Secondly, I think using absolute values is not a good idea. It will be very hard to keep track and udpate that value.
At max we can warn saying makedumpfile needs X MB of memory and only Y MB of free memory is available.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to [email protected]Quoted 25 lines · On Fri, Apr 18, 2014 at 10:29:12PM +0800, [email protected] wrote:
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Quoted 7 lines
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (
base_value+ linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.
Quoted 3 lines
Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of
base_valueis needed. So a constant value as you said may not be needed.
That logic is old and we probably should get rid of at some point. We don't want makedumpfile's memory usage to go up because system has more physical RAM. That's why cyclic mode was introduced.
Quoted 3 lines
Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.
I had come up with this 80% number randomly. So you think that's the problem?
I am still scratching my head that why 30MB is not sufficient for makdumpfile.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Petr TesarikQuoted 28 lines · Petr Tesarik
Quoted 25 lines · On Fri, 18 Apr 2014 22:29:12 +0800 "[email protected]" <[email protected]> wrote:
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Quoted 7 lines
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (
base_value+ linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.
Quoted 8 lines
Quoted 6 lines
Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of
base_valueis needed. So a constant value as you said may not be needed.Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.
I believe these 80% come from the default value of vm.dirty_ratio,
Actually I had suggested this 80% number when --cyclic feature was implemented. And I did not base it on dirty_ratio. Just a random suggestion.
Quoted 5 lines
which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.
We can start playig with reducing dirty_raio too and see how does it go.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 4 lines · On Mon, 21 Apr 2014 11:19:14 -0400 Vivek Goyal <[email protected]> wrote:
Quoted 2 lines · Petr Tesarik
On Fri, 18 Apr 2014 22:29:12 +0800
"[email protected]" <[email protected]> wrote:[...]
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.
I believe these 80% come from the default value of vm.dirty_ratio,
Actually I had suggested this 80% number when --cyclic feature was implemented. And I did not base it on
dirty_ratio. Just a random suggestion.
Hm. OK, you seem to have very good suggestions, even if they are random. ;-)
Quoted 7 lines
Quoted 5 lines
which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.
We can start playig with reducing
dirty_raiotoo and see how does it go.
Or use direct I/O... Is it still broken?
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 25 lines · Petr Tesarik
Quoted 4 lines · On Mon, 21 Apr 2014 11:19:14 -0400 Vivek Goyal <[email protected]> wrote:
Quoted 2 lines · Petr Tesarik
On Fri, 18 Apr 2014 22:29:12 +0800
"[email protected]" <[email protected]> wrote:[...]
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.
I believe these 80% come from the default value of vm.dirty_ratio,
Actually I had suggested this 80% number when --cyclic feature was implemented. And I did not base it on
dirty_ratio. Just a random suggestion.Hm. OK, you seem to have very good suggestions, even if they are random. ;-)
Quoted 6 lines
Quoted 5 lines
which is 20%. In other words, the kernel won't block further writes until 20% of available RAM is used up by dirty cache. But if you fill up all free memory with dirty pages and then touch another (though allocated) page, the kernel will go into direct reclaim, and if nothing can be written out ATM, it will invoke the OOM Killer.
Quoted 3 lines
Quoted 1 line
We can start playig with reducing
dirty_raiotoo and see how does it go.Or use direct I/O... Is it still broken?
May be if that turns out to be an issue. Or just reduce dirty_ratio to say 5 instead of default 20. Different things can be tried once we know what's the root cause of the issue.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Vivek GoyalQuoted 52 lines · Atsushi Kumagai
Quoted 51 lines
Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines
Hello Baoquan,
Quoted 4 lines
Hi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Great, thanks for your help!
However, I still have questions.Quoted 6 lines
Quoted 3 lines
First, what is the difference between yours and mine?
http://lists.infradead.org/pipermail/kexec/2014-April/011535.html
Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.
must<--Quoted 16 lines
Quoted 16 lines
My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,Quoted 1 line · Baoquan He
but I still don't think this bug causes OOM.
Even ifneeded_sizeis calculated as so much size wrongly,bufsize_cyclicwill not exceed 40% of free memory by the check below:info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.
Quoted 9 lines
Quoted 2 lines
I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.Quoted 2 lines
BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Thanks for your description, I understand that situation and the nature of the problem.
Quoted 7 lines
Quoted 4 lines
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?
Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)
pattern | bitmap size | remains
----------------------------------------------+---------------+-------------
A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MB
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
Quoted 2 lines
Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.
According to the table above and Baoquan's report, 23.6MB was enough but 17.2 MB was short for the other works, it sounds too much requirement also to me.
Quoted 9 lines
Quoted 9 lines
I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.
What I think is:
1. Use a constant value as safe limit to calculate
bufsize_cyclicinstead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.
Quoted 5 lines
Quoted 2 lines
2. If free memory is smaller than the value, makedumpfile gives up to work early.
What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.
Oh, you got a point.
Quoted 7 lines
Quoted 4 lines
This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.
I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.
I agree. I'm trying to get the peak size of memory footprint except cyclic_buffer with memory cgroup (memory.max_usage_in_bytes), but I don't make it yet.
Quoted 5 lines
Secondly, I think using absolute values is not a good idea. It will be very hard to keep track and udpate that value.
At max we can warn saying makedumpfile needs X MB of memory and only Y MB of free memory is available.
Yes, I don't want to choose that way, and to maintain the constant value just to inform possibility of OOM sounds worthless.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Vivek GoyalQuoted 19 lines
Quoted 11 lines
Well, we have done some experiments to try to get the statistical memory range which kdump really need. Then a final reservation will be calculated automatically as (
base_value+ linear growth of total memory). If one machine has 200GB memory, its reservation will grow too. Since except of the bitmap cost, other memory cost is almost fixed.Per this scheme things should be go well, if memory always goes to the edge of OOM, an adjust of
base_valueis needed. So a constant value as you said may not be needed.That logic is old and we probably should get rid of at some point. We don't want makedumpfile's memory usage to go up because system has more physical RAM. That's why cyclic mode was introduced.
Quoted 1 line
Quoted 5 lines
Quoted 2 lines
Instead, I am wondering how the 80% comes from, and why 20% of free memory must be safe.
I had come up with this 80% number randomly. So you think that's the problem?
As Astushi said, when memory amount goes up, the bitmap also goes up. The cyclic should solve this problem, but seems it doesn't. In the bug we found, it always get the whole bitmap size to cyclic_bugsize. And that is because 80% number.
We are try to add debug code to see what is going on during makedumpfile running.
Quoted 11 lines
I am still scratching my head that why 30MB is not sufficient for makdumpfile.
Thanks
Vivek_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Atsushi KumagaiQuoted 25 lines
Quoted 10 lines
Quoted 7 lines
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?
Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MB
Quoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
Thanks Atsushi for this detailed table.
In fact in cyclic mode, the cyclic bugsize is the dynamic number which need be grieved, not the remaining memory which always got to be decreased. Since user set cyclic mode, they would like to accept the fact that dumping could cost a longer time. But we give them a OOM, that's not acceptable.
So I think we need think about how the cyclic_bufsize can be calculated better. Maybe dynamically adjust the 80% number?
Thanks
Baoquan
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Atsushi KumagaiQuoted 57 lines · Atsushi Kumagai
Quoted 57 lines · Atsushi Kumagai
Quoted 55 lines
Quoted 45 lines · Baoquan He
Quoted 11 lines · Atsushi Kumagai
Quoted 10 lines
Hello Baoquan,
Quoted 4 lines
Hi Atsushi,
I have got the test machine where bug reported and did a test. The changed code can make elf dump successful.
Great, thanks for your help!
However, I still have questions.Quoted 6 lines
Quoted 3 lines
First, what is the difference between yours and mine?
http://lists.infradead.org/pipermail/kexec/2014-April/011535.html
Yeah, you are right, it's the same on changing the code bug. I mush haven't read your patch carefully.
must<--Quoted 16 lines
Quoted 16 lines
My patch includes renaming some values, but the purpose looks the same as yours.
Further, you described as below,Quoted 1 line · Baoquan He
but I still don't think this bug causes OOM.
Even ifneeded_sizeis calculated as so much size wrongly,bufsize_cyclicwill not exceed 40% of free memory by the check below:info->bufsize_cyclic = (free_size <= needed_size) ? free_size : needed_size;So it looks that bitmap1(40%) and bitmap2(40%) will fit in 80% of free memory in any case.
Quoted 9 lines
Quoted 2 lines
I may misunderstand something since your patch has an effect on this issue in practice, could you correct me?
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.Quoted 2 lines
BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Quoted 6 lines
Quoted 2 lines
Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?
Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:
What I meant was memory allocation is of the order O(1) and not O(N).
Quoted 11 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MB
Quoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
Quoted 30 lines
Quoted 3 lines
Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.
According to the table above and Baoquan's report, 23.6MB was enough but 17.2 MB was short for the other works, it sounds too much requirement also to me.
Quoted 9 lines
Quoted 9 lines
I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.
What I think is:
1. Use a constant value as safe limit to calculate
bufsize_cyclicinstead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.Quoted 5 lines
Quoted 2 lines
2. If free memory is smaller than the value, makedumpfile gives up to work early.
What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.
Oh, you got a point.
Quoted 5 lines
Quoted 4 lines
This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.
Quoted 6 lines
Quoted 2 lines
I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.
I agree. I'm trying to get the peak size of memory footprint except
cyclic_bufferwith memory cgroup (memory.max_usage_in_bytes), but I don't make it yet.
That sounds like a good idea.
I am wondering if we are doing anything which requires kernel to allocate memory and that leads to OOM.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 20 lines · Vivek Goyal
Quoted 11 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MBQuoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Quoted 5 lines
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.
Thanks
Baoquan
Quoted 2 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 28 lines · Baoquan He
Quoted 20 lines · Vivek Goyal
Quoted 11 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MBQuoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Quoted 5 lines
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
Quoted 4 lines
How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.
Oh, this is what Astushi suggested earlier in his comments.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to [email protected]Quoted 28 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 20 lines · Vivek Goyal
Quoted 11 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MBQuoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Quoted 5 lines
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
Quoted 4 lines
How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.
I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.
Tricky bit here is to figure out how much memory does makedumpfile need.
A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Vivek GoyalQuoted 87 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines
It definitely will cause OOM. On my test machine, it has 100G memory. So per old code, its
needed_sizeis 3200K*2 == 6.4M, if currently free memory is only 15M left, thefree_sizewill be 15M*0.4 which is 6M. Soinfo->bufsize_cyclicis assigned to be 6M. and only 3M is left for other use, e.g page cache, dynamic allocation. OOM will happen.BTW, in our case, there's about 30M free memory when we started saving dump. It should be caused by my coarse estimation above.
Thanks for your description, I understand that situation and the nature of the problem.
That is, the assumption that 20% of free memory is enough for makedumpfile can be broken if free memory is too small. If your machine has 200GB memory, OOM will happen even after fix the too allocation bug.
Quoted 6 lines
Quoted 2 lines
Why? In cyclic mode, shouldn't makedumpfile's memory usage be fixed and should not be dependent on amount of RAM present in the system?
Strictly speaking, it's not fixed but just restricted by the safe limit(80% of free memory) like below:
What I meant was memory allocation is of the order O(1) and not O(N).
Quoted 11 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MBQuoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
Quoted 33 lines
Quoted 3 lines
Also, so even 30MB is not sufficient to run makedumpfile. That looks like a lot of free memory to me.
According to the table above and Baoquan's report, 23.6MB was enough but 17.2 MB was short for the other works, it sounds too much requirement also to me.
Quoted 9 lines
Quoted 9 lines
I don't think this is a problem, it's natural that a lack of memory causes OOM. However, there is a thing we can do for improvement.
What I think is:
1. Use a constant value as safe limit to calculate
bufsize_cyclicinstead of 80% of free memory. This value must be enough for makedumpfile's work except bitmap.Quoted 5 lines
Quoted 2 lines
2. If free memory is smaller than the value, makedumpfile gives up to work early.
What do we gain by makedumpfile giving up. System will reboot. System will reboot anyway after OOM.
Oh, you got a point.
Quoted 7 lines
Quoted 4 lines
This change may reduce the possibility of lack of memory, but the required memory size will be changing every version, so maintaining it sounds tough to me.
I think we need to dive deeper to figure out why 30MB of free memory is not sufficient. To me something looks wrong here.
Quoted 8 lines
Quoted 3 lines
I agree. I'm trying to get the peak size of memory footprint except
cyclic_bufferwith memory cgroup (memory.max_usage_in_bytes), but I don't make it yet.That sounds like a good idea.
I am wondering if we are doing anything which requires kernel to allocate memory and that leads to OOM.
max_usage_in_bytes seems to include page cache, so I used simpler way like the patch at the bottom of this mail instead.
Here is the result:
parameter | result
dump_lv | buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------
d31 | 1 | on | 5872
Ed31 | 1 | on | 5804
d31 | 1 | off | 1784
Ed31 | 1 | off | 1720
d31 | 1000 | on | 8884 (*1)
Ed31 | 1000 | on | 7812
d31 | 1000 | off | 4788 (*1)
Ed31 | 1000 | off | 3720*1. 2nd bitmap is allocated twice by the bug found by Arthur Zou.
According to this result and the estimation below produced from the design of makedumpfile,
base size + (buffer size * 2) + (for mmap size) = VmHWM
the base size may be about 2 MB, so about 6MB (base + mmap) will be the deadline. If 20% of the available memory is less than 6MB, OOM will happen.
As for Baoquan's case, remained 17.2MB sounds enough even if makedumpfile consumes 6MB as for the other works.
So I don't know why OOM happened yet, but the current safety limit looks still reasonable to me at least.
By the way, I noticed it's better to remove 4MB(for mmap) from the available memory before calculate the bitmap buffer size. I'll do this anyway.
Thanks
Atsushi Kumagai
makedumpfile.c +10 -0
@@ -37,6 +37,14 @@ struct DumpInfo *info = NULL;char filename_stdout[] =FILENAME_STDOUT ; +void +print_VmHWM (void ) +{ +char cmd[64 ]; +sprintf (cmd,"grep VmHWM /proc/%d/status" ,getpid ()); +system (cmd); +} +static void first_cycle (mdf_pfn_t start,mdf_pfn_t max,struct cycle *cycle) { cycle->start_pfn =round (start, info->pfn_cyclic);
@@ -9357,5 +9365,7 @@ out: }free_elf_info (); +print_VmHWM (); +return retcd; }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Vivek GoyalQuoted 34 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 28 lines · Vivek Goyal
Quoted 21 lines
Quoted 12 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MBQuoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Quoted 5 lines
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
Quoted 4 lines
How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.
Quoted 6 lines
I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.
Tricky bit here is to figure out how much memory does makedumpfile need.
Did you said using such value is bad idea since it's hard to update it? If we got the needed memory size, it would be changing every version. At least I think this may be an ideal way but not practical.
Quoted 2 lines
A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.
That's just specific tuning for you and Baoquan.
Now, I think this case is just lack of free memory caused by inappropriate parameter setting for your environment. You should increase crashkernel= to get enough free memory, 166M may be too small for your environment.
By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 39 lines · Atsushi Kumagai
Quoted 39 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 28 lines · Vivek Goyal
Quoted 21 lines
Quoted 12 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MBQuoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Quoted 5 lines
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
Quoted 4 lines
How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.
I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.
Quoted 5 lines
Quoted 1 line
Tricky bit here is to figure out how much memory does makedumpfile need.
Did you said using such value is bad idea since it's hard to update it? If we got the needed memory size, it would be changing every version. At least I think this may be an ideal way but not practical.
Yep, I am not too convinced about fixing makedumpfile memory usage at a particular value.
Quoted 10 lines
Quoted 3 lines
A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.
That's just specific tuning for you and Baoquan.
Now, I think this case is just lack of free memory caused by inappropriate parameter setting for your environment. You should increase crashkernel= to get enough free memory, 166M may be too small for your environment.
I don't think it is bad tuning from our side. makedumpfile has 30MB free memory when it was launched and still OOM happened.
30MB should be more than enough to save dump.
Quoted 3 lines
By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.
Sure, talk to you more about this once you are back.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Atsushi KumagaiQuoted 35 lines
Quoted 2 lines
I am wondering if we are doing anything which requires kernel to allocate memory and that leads to OOM.
max_usage_in_bytesseems to include page cache, so I used simpler way like the patch at the bottom of this mail instead.Here is the result:
parameter | result
dump_lv| buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------d31 | 1 | on | 5872 Ed31 | 1 | on | 5804 d31 | 1 | off | 1784 Ed31 | 1 | off | 1720 d31 | 1000 | on | 8884 (*1) Ed31 | 1000 | on | 7812 d31 | 1000 | off | 4788 (*1) Ed31 | 1000 | off | 3720*1. 2nd bitmap is allocated twice by the bug found by Arthur Zou.
According to this result and the estimation below produced from the design of makedumpfile,
base size + (buffer size * 2) + (for mmap size) = VmHWM
the base size may be about 2 MB, so about 6MB (base + mmap) will be the deadline. If 20% of the available memory is less than 6MB, OOM will happen.
As for Baoquan's case, remained 17.2MB sounds enough even if makedumpfile consumes 6MB as for the other works.
So I don't know why OOM happened yet, but the current safety limit looks still reasonable to me at least.
Quoted 3 lines
By the way, I noticed it's better to remove 4MB(for mmap) from the available memory before calculate the bitmap buffer size. I'll do this anyway.
Sorry, I did my test on a 1st kernel environment.
The dump file was on a disk and I used mmap() for it, so __do_fault() occurred and the VmHWM (MAX(MM_FILEPAGES + MM_ANONPAGES)) increased by 4MB.
Here is the result on a 2nd kernel environment:
parameter | result
dump_lv | buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------
d31 | 1 | on | 776
Ed31 | 1 | on | 712
d31 | 1 | off | 704
Ed31 | 1 | off | 708
d31 | 1000 | on | 1776
Ed31 | 1000 | on | 2716
d31 | 1000 | off | 1660
Ed31 | 1000 | off | 2556According to this, I think there is no need to take care of the memory usage of mmap() because the actual memory usage is small enough.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Vivek GoyalQuoted 60 lines · Atsushi Kumagai
Quoted 39 lines
Quoted 39 lines · On Thu, Apr 24, 2014 at 07:50:41AM +0800, [email protected] wrote:
Quoted 28 lines · Vivek Goyal
Quoted 21 lines
Quoted 12 lines
- bitmap size: used for 1st and 2nd bitmaps
- remains: can be used for the other works of makedumpfile (e.g. I/O buffer)pattern | bitmap size | remains
----------------------------------------------+---------------+-------------A. 100G memory with the too allocation bug | 12.8 MB | 17.2 MB B. 100G memory with fixed makedumpfile | 6.4 MB | 23.6 MB C. 200G memory with fixed makedumpfile | 12.8 MB | 17.2 MB D. 300G memory with fixed makedumpfile | 19.2 MB | 10.8 MB E. 400G memory with fixed makedumpfile | 24.0 MB | 6.0 MB F. 500G memory with fixed makedumpfile | 24.0 MB | 6.0 MBQuoted 4 lines
...
Baoquan got OOM in A pattern and didn't get it in B, so C must also fail due to OOM. This is just what I wanted to say.
ok, So here bitmap size is growing because we have not hit the 80% of available memory limit yet. But it gets limited at 24MB once we hit 80% limit. I think that's fine. That's what I was looking for.
Quoted 5 lines
Now key question will remain is that is using 80% of free memory by bitmaps too much. Are other things happening in system which consume memory and because memory is not available OOM hits. If that's the case we probably need to lower the amount of memory allocated to bit maps. Say 70% or 60% or may be 50%. But this should be data driven.
Quoted 4 lines
How about add anoter limit, say left memory safety limit, e.g 20M. If the remaining memory which is 20% of free memory is bigger than 20M, 80% can be taken to calculate the bitmap size. If smaller than 20M, we just take (total memory - safety limit) for bitmap size.
I think doing another internal limit for makedumpfile usage sounds fine. So say, if makedumpfile needs 5MB of memory for purposes other than bitmap, then remove 5MB from total memory and then take 80% of remaining memory to calculate bitmap size. I think that should be reasonable.
Quoted 5 lines
Quoted 1 line
Tricky bit here is to figure out how much memory does makedumpfile need.
Did you said using such value is bad idea since it's hard to update it? If we got the needed memory size, it would be changing every version. At least I think this may be an ideal way but not practical.
Yep, I am not too convinced about fixing makedumpfile memory usage at a particular value.
Quoted 10 lines
Quoted 3 lines
A simpler solution will be to just reserve 60% of total memory for bitmaps and leave rest for makedumpfile and kernel and other components.
That's just specific tuning for you and Baoquan.
Now, I think this case is just lack of free memory caused by inappropriate parameter setting for your environment. You should increase crashkernel= to get enough free memory, 166M may be too small for your environment.
Quoted 4 lines
I don't think it is bad tuning from our side. makedumpfile has 30MB free memory when it was launched and still OOM happened.
30MB should be more than enough to save dump.
There were more than 10 MB remain memory in your case (A pattern), and probably they were used as page cache mostly, so it sounds OOM couldn't happen since such pages are reclaimable.
I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:
/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8000 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [100.0 %] |
The dumpfile is saved to /mnt/tmp/dumpfile.E.
makedumpfile Completed.
VmHWM: 16456 kB
/ #
As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).
Of course, OOM happened when the memory usage exceeded the limit:
/ # free
total used free shared buffers
Mem: 37544 21608 15936 0 368
Swap: 0 0 0
Total: 37544 21608 15936
/ # /mnt/usr/sbin/makedumpfile_static -E --cyclic-buffer=8192 /proc/vmcore /mnt/tmp/dumpfile.E
Copying data : [ 0.0 %] /Out of memory: Kill process 1389 (makedumpfile_st) score 428 or sacrifice child
Killed process 1389 (makedumpfile_st) total-vm:22196kB, anon-rss:16524kB, file-rss:8kB KILL
/ #
I think we should investigate why OOM happened in your environment, otherwise we can't decide a safety limit of a user process's memory usage.
Thanks
Atsushi Kumagai
Quoted 6 lines
Quoted 3 lines
By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.
Sure, talk to you more about this once you are back.
Quoted 7 lines
Thanks
Vivek_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
On Fri, May 09, 2014 at 05:36:13AM +0000, Atsushi Kumagai wrote:
[..]
Quoted 14 lines
I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ #/mnt/usr/sbin/makedumpfile_static-E --cyclic-buffer=8000/proc/vmcore/mnt/tmp/dumpfile.E
Copying data : [100.0 %] |The dumpfile is saved to
/mnt/tmp/dumpfile.E.
Quoted 8 lines
makedumpfile Completed.
VmHWM: 16456 kB
/ #As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).
Of course, OOM happened when the memory usage exceeded the limit:
Hi Atsushi,
I think this is the key point. How did makedumpfile exceed the limit. So if you don't specify --cyclic-buffer=X, then makedumpfile will allocate 80% of available memory. That would be roughly 16MB of cyclic buffer (instead of 8MB).
In your test case looks like makeudmpfile used 16MB of memory. Out of which 8MB must have been used for bitmaps and rest 8MB for other purposes.
So clearly our calculation of using 80% of available memory for bitmaps is not right. Rest of the 20% memory might not be enough for fulfilling the needs of makeudmpfile.
We don't have a good way to determine how much *non-bitmap* memory allocation makedumpfile requires. I am wondering how about making some assumtions and modify the formula for bitmap memory.
- Assume makedumpfile requires 16MB of free memory for various purposes.
- Subtract 16MB from available memory and take 80% of remaining memory to calculate the bitmap memory size.
- Keep 4MB as minimum for bitmap buffer size.
One more person has reported makedumpfile OOM (even without -E option), on a machine. There also 30MB of memory is free. I think it has the same issue that 24MB must have been allocated in bitmaps and remainig 6MB is not sufficient to satisfy makedumpfile's needs.
What do you think?
Also, does anybody know what's the impact of bitmap buffer size on dump speed. Does dump speed change significantly with bitmap buffer size?
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Atsushi KumagaiQuoted 16 lines · Atsushi Kumagai
There were more than 10 MB remain memory in your case (A pattern), and probably they were used as page cache mostly, so it sounds OOM couldn't happen since such pages are reclaimable.
I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ #/mnt/usr/sbin/makedumpfile_static-E --cyclic-buffer=8000/proc/vmcore/mnt/tmp/dumpfile.E
Copying data : [100.0 %] |
Quoted 4 lines
The dumpfile is saved to
/mnt/tmp/dumpfile.E.makedumpfile Completed.
VmHWM: 16456 kB
Hi Atsushi,
What is the dump target in your test? The OOM bugs happened if and only if it nfs target. Other dump target including ssh/local fs/ didn't happen ever though those cyclic buffer size code bugs existed.
We added some debug code and made test, the result shows the OOM happened when the left memory is only about 2M. But when we drop page caches every 1000 times of write, it's OK to complete the dump. And the OOM happened when page cache need allocate page for writing.
So could you adjust your test to nfs dump with elf format or lzo compression? I think nfs dump have heavy page cache affect which is different with others.
I will take machines with 100G memory and 10G memory to test separately, will paste the result with our configuration.
Thanks
Baoquan
Quoted 19 lines
/ #
As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).
Of course, OOM happened when the memory usage exceeded the limit:
/ # free
total used free shared buffers
Mem: 37544 21608 15936 0 368
Swap: 0 0 0
Total: 37544 21608 15936
/ #/mnt/usr/sbin/makedumpfile_static-E --cyclic-buffer=8192/proc/vmcore/mnt/tmp/dumpfile.E
Copying data : [ 0.0 %] /Out of memory: Kill process 1389 (makedumpfile_st) score 428 or sacrifice child
Killed process 1389 (makedumpfile_st) total-vm:22196kB, anon-rss:16524kB, file-rss:8kB KILL
/ #
Quoted 11 lines
I think we should investigate why OOM happened in your environment, otherwise we can't decide a safety limit of a user process's memory usage.
Thanks
Atsushi KumagaiQuoted 4 lines
Quoted 3 lines
By the way, I'm going on holiday for 8 days, I can't reply during that period. Thanks in advance.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Vivek GoyalQuoted 24 lines · Vivek Goyal
On Fri, May 09, 2014 at 05:36:13AM +0000, Atsushi Kumagai wrote:
[..]
Quoted 21 lines
I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ #/mnt/usr/sbin/makedumpfile_static-E --cyclic-buffer=8000/proc/vmcore/mnt/tmp/dumpfile.E
Copying data : [100.0 %] |The dumpfile is saved to
/mnt/tmp/dumpfile.E.makedumpfile Completed.
VmHWM: 16456 kB
/ #As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).
Quoted 8 lines
Quoted 1 line
Of course, OOM happened when the memory usage exceeded the limit:
Hi Atsushi,
I think this is the key point. How did makedumpfile exceed the limit. So if you don't specify --cyclic-buffer=X, then makedumpfile will allocate 80% of available memory. That would be roughly 16MB of cyclic buffer (instead of 8MB).
Hi,
Here I think Vivek is right. In Astushi's test, when --cyclic-buffer=8000 and elf dump, the info->bufsize_cyclic is only 4M, because it need be divided by 2 for elf case.
I took tests on my local machine with 16G memory, the result suprised me. In my tests, even 70% is not safe. I didn't specify --cyclic-buffer=X like Astushi did, just change code like below, I think it's more accurate.
@@ -9049,8 +9055,9 @@ calculate_cyclic_buffer_size(void) {
* should be 40% of free memory to keep the size of cyclic
* buffer
* within 80% of free memory.
*/
if (info->flag_elf_dumpfile) {
- limit_size = get_free_memory_size() * 0.4;
+ limit_size = get_free_memory_size() * 0.30;
} else {
limit_size = get_free_memory_size() * 0.8;
}
@@ -9060,7 +9067,8 @@ calculate_cyclic_buffer_size(void) {
if (info->num_dumpfile > 1)
bitmap_size /= info->num_dumpfile;- info->bufsize_cyclic = MIN(limit_size, bitmap_size);
+ info->bufsize_cyclic = limit_size;
return TRUE;
}
All dump is elf format, and target is nfs and local separately. And the crashkernel=168M
Then in nfs dump, when bufsize_cyclic is 80% of memory, OOM happened 3 times in all 5 times. when bufsize_cyclic is 70% of memory, OOM happened 2 times in all 7 times. I paste the output message in the following (aaaa) section, it's 80% of memory for bufsize_cyclic.
Then I thought 70% is not safe, maybe local fs dump is better. So I took local fs dump, when bufsize_cyclic is 80%, OOM happened 3 times in all 5 times of test. When bufsize_cyclic is 70%, 2 of 5. when 60%, finally, it's safe now. I paste the output message in the following (bbbb) section, it's 80% of memory for bufsize_cyclic.
I can't get a conclusion, just give the test result for your reference.
(aaaa)
pre-pivot:/# cat /etc/kdump.conf
nfs 10.66.16.116:/var/crash
path /var/crashcore_collector makedumpfile -E --message-level 1 -d 31 default shell
pre-pivot:/#
pre-pivot:/# exit kdump: dump target is 10.66.16.116:/var/crash kdump: saving to /sysroot/mnt//var/crash/10.66.16.106-2014.05.15-09:08:03/ kdump: saving vmcore-dmesg.txt kdump: saving vmcore-dmesg.txt complete kdump: saving vmcore
calculate_cyclic_buffer_size, free memory 34795520
Buffer size for the cyclic mode: 13918208
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,before write_elf_headerfree, memory 6819840
Excluding unnecessary pages : [100.0 %] \
writeout_dumpfile,before write_elf_pages_cyclic, memory 6791168
In calculate_cyclic_buffer_size, free memory 6791168
Copying data : [ 1.1 %] /
In calculate_cyclic_buffer_size, free memory 2412544
Excluding unnecessary pages : [100.0 %] |
In calculate_cyclic_buffer_size, free memory 2691072
Excluding unnecessary pages : [100.0 %] \
In calculate_cyclic_buffer_size, free memory 2756608
Copying data : [ 2.8 %] |
In calculate_cyclic_buffer_size, free memory 2363392
Copying data : [ 17.2 %] \[ 56.567080] usb
2-1.1: USB disconnect, device number 3
[ 58.025437] usb 2-1.1: new low-speed USB device number 5 using ehci-pci [ 58.125797] usb 2-1.1: New USB device found, idVendor=03f0, idProduct=0b4a [ 58.132907] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 58.140431] usb 2-1.1: Product: USB Optical Mouse [ 58.145345] usb 2-1.1: Manufacturer: Logitech [ 58.153006] input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1:1.0/input/input4 [ 58.164942] hid-generic 0003:03F0:0B4A.0003: input,hidraw0: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:1d.0-1.1/inp0 [ 58.646175] makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0 [ 58.654581] makedumpfile cpuset=/ mems_allowed=0 [ 58.659411] CPU: 0 PID: 517 Comm: makedumpfile Not tainted 3.10.0-114.el7.x86_64 #1 [ 58.667275] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.02 03/09/2012 [ 58.676177] ffff880032db4440 000000003ed297af ffff88002e4b3880 ffffffff815ec0ca [ 58.683882] ffff88002e4b3910 ffffffff815e773d ffffffff810b6918 ffff880033aca0c0 [ 58.691584] ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81102e03 [ 58.699293] Call Trace: [ 58.701959] [<ffffffff815ec0ca>] dump_stack+0x19/0x1b [ 58.707309] [<ffffffff815e773d>] dump_header+0x8e/0x214 [ 58.712827] [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0 [ 58.718529] [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0 [ 58.724310] [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0 [ 58.730339] [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30 [ 58.736895] [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0 [ 58.742665] [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0 [ 58.749222] [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170 [ 58.755440] [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0 [ 58.761479] [<ffffffff81142566>] grab_cache_page_write_begin+0x76/0xd0 [ 58.768304] [<ffffffffa03f4767>] nfs_write_begin+0x77/0x210 [nfs] [ 58.774694] [<ffffffff8114158e>] generic_file_buffered_write+0x11e/0x290 [ 58.781693] [<ffffffff81050f00>] ? rbt_memtype_copy_nth_element+0xa0/0xa0 [ 58.788779] [<ffffffff811436e5>] __generic_file_aio_write+0x1d5/0x3e0 [ 58.795512] [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0 [ 58.801903] [<ffffffffa03f3c1b>] nfs_file_write+0xbb/0x1d0 [nfs] [ 58.808206] [<ffffffff811af21d>] do_sync_write+0x8d/0xd0 [ 58.813796] [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0 [ 58.819141] [<ffffffff811af4a4>] ? generic_file_llseek+0x24/0x30 [ 58.825434] [<ffffffff811b0408>] SyS_write+0x58/0xb0 [ 58.830676] [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b [ 58.836874] Mem-Info: [ 58.839349] Node 0 DMA per-cpu: [ 58.842714] CPU 0: hi: 0, btch: 1 usd: 0 [ 58.847704] Node 0 DMA32 per-cpu: [ 58.851244] CPU 0: hi: 42, btch: 7 usd: 0 [ 58.856235] active_anon:11294 inactive_anon:925 isolated_anon:0 [ 58.856235] active_file:1032 inactive_file:1653 isolated_file:32 [ 58.856235] unevictable:11030 dirty:0 writeback:1800 unstable:0 [ 58.856235] free:473 slab_reclaimable:2202 slab_unreclaimable:3849 [ 58.856235] mapped:1245 shmem:1103 pagetables:283 bounce:0 [ 58.856235] free_cma:0 [ 58.891859] Node 0 DMA free:492kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:8kB unevictabs [ 58.931901] lowmem_reserve[]: 0 122 122 122 [ 58.936393] Node 0 DMA32 free:1400kB min:1412kB low:1764kB high:2116kB active_anon:45176kB inactive_anon:3700kB active_file:4124kB inacts [ 58.981849] lowmem_reserve[]: 0 0 0 0 [ 58.985815] Node 0 DMA: 3*4kB (UM) 2*8kB (U) 1*16kB (M) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4B [ 58.999026] Node 0 DMA32: 38*4kB (EM) 78*8kB (UEMR) 11*16kB (UER) 0*32kB 3*64kB (R) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096B [ 59.012831] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 59.021469] 14852 total pagecache pages [ 59.025522] 0 pages in swap cache [ 59.029048] Swap cache stats: add 0, delete 0, find 0/0 [ 59.034491] Free swap = 0kB [ 59.037583] Total swap = 0kB [ 59.041703] 90211 pages RAM [ 59.044712] 53920 pages reserved [ 59.048157] 267612 pages shared [ 59.051514] 31588 pages non-shared [ 59.055133] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 59.063173] [ 64] 0 64 13020 570 24 0 0 systemd-journal [ 59.072172] [ 170] 32 170 9977 282 23 0 0 rpcbind [ 59.080474] [ 174] 0 174 11120 416 25 0 0 rpc.statd [ 59.088960] [ 179] 0 179 6366 169 16 0 0 rpc.idmapd [ 59.097551] [ 201] 0 201 8729 425 22 0 -1000 systemd-udevd [ 59.106402] [ 324] 0 324 25589 3179 47 0 0 dhclient [ 59.114815] [ 383] 0 383 3101 553 16 0 0 dracut-pre-pivo [ 59.123832] [ 517] 0 517 13333 7103 42 0 0 makedumpfile [ 59.132594] [ 518] 0 518 8728 272 21 0 0 systemd-udevd [ 59.141442] [ 519] 0 519 8728 214 21 0 0 systemd-udevd [ 59.150287] Out of memory: Kill process 517 (makedumpfile) score 190 or sacrifice child [ 59.158527] Killed process 517 (makedumpfile) total-vm:53332kB,
anon-rss:27524kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"
(bbbb)
kdump: dump target is /dev/sda2
kdump: saving t[ 13.492083] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
o /sysroot//var/crash/127.0.0.1-2014.05.15-11:24:51/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore
calculate_cyclic_buffer_size, free memory 77033472
Buffer size for the cyclic mode: 30813388
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,before write_elf_headerfree, memory 15233024
Excluding unnecessary pages : [100.0 %] -
writeout_dumpfile,before write_elf_pages_cyclic, memory 15204352
In calculate_cyclic_buffer_size, free memory 15204352
Excluding unnecessary pages : [100.0 %] /
In calculate_cyclic_buffer_size, free memory 2932736
Excluding unnecessary pages : [100.0 %] |
In calculate_cyclic_buffer_size, free memory 2387968
Excluding unnecessary pages : [100.0 %] \
In calculate_cyclic_buffer_size, free memory 2555904
Excluding unnecessary pages : [100.0 %] -
In calculate_cyclic_buffer_size, free memory 2838528
Copying data : [ 3.8 %] |[ 15.406789]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0
[ 15.415222] makedumpfile cpuset=/ mems_allowed=0 [ 15.420191] CPU: 0 PID: 287 Comm: makedumpfile Not tainted 3.10.0-114.el7.x86_64 #1 [ 15.428092] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.02 03/09/2012 [ 15.437280] ffff88002f8196c0 0000000043db25d9 ffff88002ff2b7e0 ffffffff815ec0ca [ 15.445105] ffff88002ff2b870 ffffffff815e773d ffffffff810b6918 ffff880030018bb0 [ 15.452935] ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81102e03 [ 15.460693] Call Trace: [ 15.463375] [<ffffffff815ec0ca>] dump_stack+0x19/0x1b [ 15.468743] [<ffffffff815e773d>] dump_header+0x8e/0x214 [ 15.474288] [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0 [ 15.479999] [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0 [ 15.485790] [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0 [ 15.491848] [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30 [ 15.498430] [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0 [ 15.504222] [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0 [ 15.510793] [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170 [ 15.517020] [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0 [ 15.523061] [<ffffffff81142566>] grab_cache_page_write_begin+0x76/0xd0 [ 15.529898] [<ffffffffa02ab133>] ext4_da_write_begin+0xa3/0x330 [ext4] [ 15.536728] [<ffffffff8114158e>] generic_file_buffered_write+0x11e/0x290 [ 15.543724] [<ffffffff811436e5>] __generic_file_aio_write+0x1d5/0x3e0 [ 15.550462] [<ffffffff81050f00>] ? rbt_memtype_copy_nth_element+0xa0/0xa0 [ 15.557551] [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0 [ 15.563952] [<ffffffffa02a1189>] ext4_file_write+0xa9/0x450 [ext4] [ 15.570434] [<ffffffff811797fc>] ? free_vmap_area_noflush+0x7c/0x90 [ 15.577009] [<ffffffff811af21d>] do_sync_write+0x8d/0xd0 [ 15.582616] [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0 [ 15.587967] [<ffffffff811b0408>] SyS_write+0x58/0xb0 [ 15.593223] [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b [ 15.599437] Mem-Info: [ 15.601912] Node 0 DMA per-cpu: [ 15.605283] CPU 0: hi: 0, btch: 1 usd: 0 [ 15.610279] Node 0 DMA32 per-cpu: [ 15.613825] CPU 0: hi: 42, btch: 7 usd: 29 [ 15.618827] active_anon:16072 inactive_anon:924 isolated_anon:0 [ 15.618827] active_file:2075 inactive_file:2232 isolated_file:0 [ 15.618827] unevictable:6464 dirty:0 writeback:3 unstable:0 [ 15.618827] free:511 slab_reclaimable:2077 slab_unreclaimable:3528 [ 15.618827] mapped:1055 shmem:1088 pagetables:146 bounce:0 [ 15.618827] free_cma:0 [ 15.654098] Node 0 DMA free:508kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictabs [ 15.693926] lowmem_reserve[]: 0 129 129 129 [ 15.698429] Node 0 DMA32 free:1536kB min:1452kB low:1812kB high:2176kB active_anon:64288kB inactive_anon:3696kB active_file:8300kB inacto [ 15.742999] lowmem_reserve[]: 0 0 0 0 [ 15.746981] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B [ 15.760499] Node 0 DMA32: 6*4kB (M) 7*8kB (R) 3*16kB (R) 2*32kB (M) 1*64kB (M) 0*128kB 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB [ 15.774255] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 15.782928] 11860 total pagecache pages [ 15.786986] 0 pages in swap cache [ 15.790524] Swap cache stats: add 0, delete 0, find 0/0 [ 15.795976] Free swap = 0kB [ 15.799080] Total swap = 0kB [ 15.803169] 90211 pages RAM [ 15.806188] 53919 pages reserved [ 15.809635] 6131 pages shared [ 15.812820] 29843 pages non-shared [ 15.816437] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 15.824495] [ 64] 0 64 13020 559 23 0 0 systemd-journal [ 15.833506] [ 119] 0 119 8864 549 23 0 -1000 systemd-udevd [ 15.842347] [ 204] 0 204 3071 531 15 0 0 dracut-pre-pivo [ 15.851353] [ 287] 0 287 21581 15351 58 0 0 makedumpfile [ 15.860103] Out of memory: Kill process 287 (makedumpfile) score 411 or sacrifice child [ 15.868333] Killed process 287 (makedumpfile) total-vm:86324kB,
anon-rss:60516kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/999
Generating "/run/initramfs/rdsosreport.txt"
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Quoted 34 lines · Vivek Goyal
Quoted 24 lines · Atsushi Kumagai
[..]
Quoted 21 lines
I tried to reproduce OOM in my environment.
Unfortunately, I couldn't get a chance to use a large memory machine, so I controlled the bitmap buffer size with --cyclic-buffer like below:/ # free
total used free shared buffers
Mem: 37544 19796 17748 0 56
Swap: 0 0 0
Total: 37544 19796 17748
/ #/mnt/usr/sbin/makedumpfile_static-E --cyclic-buffer=8000/proc/vmcore/mnt/tmp/dumpfile.E
Copying data : [100.0 %] |The dumpfile is saved to
/mnt/tmp/dumpfile.E.makedumpfile Completed.
VmHWM: 16456 kB
/ #As above, OOM didn't happen even when makedumpfile consumed most of the available memory (the remains were only 1MB).
Quoted 8 lines
Quoted 1 line
Of course, OOM happened when the memory usage exceeded the limit:
Hi Atsushi,
I think this is the key point. How did makedumpfile exceed the limit. So if you don't specify --cyclic-buffer=X, then makedumpfile will allocate 80% of available memory. That would be roughly 16MB of cyclic buffer (instead of 8MB).
Quoted 5 lines
Hi,
Here I think Vivek is right. In Astushi's test, when --cyclic-buffer=8000 and elf dump, the
info->bufsize_cyclicis only 4M, because it need be divided by 2 for elf case.
No, 16MB was used for bitmaps also in my case.
--cyclic-buffer option means specifying each bitmap size, so allocated bitmap size is the double of --cyclic-buffer.
(But now, this behavior is only for ELF case, just the specified size is allocated in kdump case. Yes, it's confusing...)
--cyclic-buffer buffer_size
Specify the buffer size in kilo bytes for analysis in the cyclic mode. Actually, the double of buffer_size kilo bytes will be allocated in memory. In the cyclic mode, the number of cycles is represented as:
So only 456KB was the requirement for other purposes, 20% of available memory (3.5MB) was enough, thus the current assumption is safe even when available memory is only 17MB like my test.
According to my test:
http://lists.infradead.org/pipermail/kexec/2014-May/011784.html
Quoted 5 lines
Here is the result on a 2nd kernel environment:
parameter | result
dump_lv| buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------
Quoted 8 lines
d31 | 1 | on | 776 // about 700
Ed31 | 1 | on | 712
d31 | 1 | off | 704
Ed31 | 1 | off | 708
d31 | 1000 | on | 1776 // about 700 + 1000
Ed31 | 1000 | on | 2716 // about 700 + 1000 * 2
d31 | 1000 | off | 1660
Ed31 | 1000 | off | 2556The requirement memory size for other purposes seems about 700KB when the dump level is 31, it's so small that we can ignore it.
The size will increase by KB order based on the system memory size in practice (this assumption comes from my code investigation), but 6MB (20% of 30MB) still sounds much enough as safety limit.
This is why I was wondering why OOM happened on your machine.
Now that I heard that OOM happened only on nfs, I guess just nfs requires lots of memory. (I did my test only on local fs.)
If we can get the required size, we could reflect it in the safety limit size.
I haven't checked your test result below yet, so I'll mention it later.
BTW, I prepared a temporal branch "oom" to investigate this issue, let's use this after this:
http://sourceforge.net/p/makedumpfile/code/ci/oom/tree/
Thanks
Atsushi Kumagai
Quoted 309 lines
I took tests on my local machine with 16G memory, the result suprised me. In my tests, even 70% is not safe. I didn't specify --cyclic-buffer=X like Astushi did, just change code like below, I think it's more accurate.
(quoted diff) +2
-2@@ -9049,8 +9055,9 @@ calculate_cyclic_buffer_size(void) { * should be 40% of free memory to keep the size of cyclic * buffer * within 80% of free memory. */ if (info->flag_elf_dumpfile) { - limit_size = get_free_memory_size() * 0.4; + limit_size = get_free_memory_size() * 0.30; } else { limit_size = get_free_memory_size() * 0.8; }@@ -9060,7 +9067,8 @@ calculate_cyclic_buffer_size(void) { if (info->num_dumpfile > 1) bitmap_size /= info->num_dumpfile; - info->bufsize_cyclic = MIN(limit_size, bitmap_size); + info->bufsize_cyclic = limit_size; return TRUE; }All dump is elf format, and target is nfs and local separately. And the crashkernel=168M
Then in nfs dump, when
bufsize_cyclicis 80% of memory, OOM happened 3 times in all 5 times. whenbufsize_cyclicis 70% of memory, OOM happened 2 times in all 7 times. I paste the output message in the following (aaaa) section, it's 80% of memory forbufsize_cyclic.Then I thought 70% is not safe, maybe local fs dump is better. So I took local fs dump, when
bufsize_cyclicis 80%, OOM happened 3 times in all 5 times of test. Whenbufsize_cyclicis 70%, 2 of 5. when 60%, finally, it's safe now. I paste the output message in the following (bbbb) section, it's 80% of memory forbufsize_cyclic.I can't get a conclusion, just give the test result for your reference.
(aaaa)
pre-pivot:/# cat /etc/kdump.confnfs 10.66.16.116:/var/crash
path/var/crashcore_collectormakedumpfile -E --message-level 1 -d 31 default shell
pre-pivot:/#pre-pivot:/# exit kdump: dump target is 10.66.16.116:/var/crash kdump: saving to /sysroot/mnt//var/crash/10.66.16.106-2014.05.15-09:08:03/ kdump: saving vmcore-dmesg.txt kdump: saving vmcore-dmesg.txt complete kdump: saving vmcore
calculate_cyclic_buffer_size, free memory 34795520Buffer size for the cyclic mode: 13918208
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,beforewrite_elf_headerfree, memory 6819840
Excluding unnecessary pages : [100.0 %] \
writeout_dumpfile,beforewrite_elf_pages_cyclic, memory 6791168In
calculate_cyclic_buffer_size, free memory 6791168
Copying data : [ 1.1 %] /
Incalculate_cyclic_buffer_size, free memory 2412544
Excluding unnecessary pages : [100.0 %] |
Incalculate_cyclic_buffer_size, free memory 2691072
Excluding unnecessary pages : [100.0 %] \
Incalculate_cyclic_buffer_size, free memory 2756608
Copying data : [ 2.8 %] |
Incalculate_cyclic_buffer_size, free memory 2363392
Copying data : [ 17.2 %] \[ 56.567080] usb
2-1.1: USB disconnect, device number 3[ 58.025437] usb 2-1.1: new low-speed USB device number 5 using ehci-pci [ 58.125797] usb 2-1.1: New USB device found, idVendor=03f0, idProduct=0b4a [ 58.132907] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 58.140431] usb 2-1.1: Product: USB Optical Mouse [ 58.145345] usb 2-1.1: Manufacturer: Logitech [ 58.153006] input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1:1.0/input/input4 [ 58.164942] hid-generic 0003:03F0:0B4A.0003: input,hidraw0: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:1d.0-1.1/inp0 [ 58.646175] makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0 [ 58.654581] makedumpfile cpuset=/ mems_allowed=0 [ 58.659411] CPU: 0 PID: 517 Comm: makedumpfile Not tainted 3.10.0-114.el7.x86_64 #1 [ 58.667275] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.02 03/09/2012 [ 58.676177] ffff880032db4440 000000003ed297af ffff88002e4b3880 ffffffff815ec0ca [ 58.683882] ffff88002e4b3910 ffffffff815e773d ffffffff810b6918 ffff880033aca0c0 [ 58.691584] ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81102e03 [ 58.699293] Call Trace: [ 58.701959] [<ffffffff815ec0ca>] dump_stack+0x19/0x1b [ 58.707309] [<ffffffff815e773d>] dump_header+0x8e/0x214 [ 58.712827] [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0 [ 58.718529] [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0 [ 58.724310] [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0 [ 58.730339] [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30 [ 58.736895] [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0 [ 58.742665] [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0 [ 58.749222] [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170 [ 58.755440] [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0 [ 58.761479] [<ffffffff81142566>] grab_cache_page_write_begin+0x76/0xd0 [ 58.768304] [<ffffffffa03f4767>] nfs_write_begin+0x77/0x210 [nfs] [ 58.774694] [<ffffffff8114158e>] generic_file_buffered_write+0x11e/0x290 [ 58.781693] [<ffffffff81050f00>] ? rbt_memtype_copy_nth_element+0xa0/0xa0 [ 58.788779] [<ffffffff811436e5>] __generic_file_aio_write+0x1d5/0x3e0 [ 58.795512] [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0 [ 58.801903] [<ffffffffa03f3c1b>] nfs_file_write+0xbb/0x1d0 [nfs] [ 58.808206] [<ffffffff811af21d>] do_sync_write+0x8d/0xd0 [ 58.813796] [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0 [ 58.819141] [<ffffffff811af4a4>] ? generic_file_llseek+0x24/0x30 [ 58.825434] [<ffffffff811b0408>] SyS_write+0x58/0xb0 [ 58.830676] [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b [ 58.836874] Mem-Info: [ 58.839349] Node 0 DMA per-cpu: [ 58.842714] CPU 0: hi: 0, btch: 1 usd: 0 [ 58.847704] Node 0 DMA32 per-cpu: [ 58.851244] CPU 0: hi: 42, btch: 7 usd: 0 [ 58.856235] active_anon:11294 inactive_anon:925 isolated_anon:0 [ 58.856235] active_file:1032 inactive_file:1653 isolated_file:32 [ 58.856235] unevictable:11030 dirty:0 writeback:1800 unstable:0 [ 58.856235] free:473 slab_reclaimable:2202 slab_unreclaimable:3849 [ 58.856235] mapped:1245 shmem:1103 pagetables:283 bounce:0 [ 58.856235] free_cma:0 [ 58.891859] Node 0 DMA free:492kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:8kB unevictabs [ 58.931901] lowmem_reserve[]: 0 122 122 122 [ 58.936393] Node 0 DMA32 free:1400kB min:1412kB low:1764kB high:2116kB active_anon:45176kB inactive_anon:3700kB active_file:4124kB inacts [ 58.981849] lowmem_reserve[]: 0 0 0 0 [ 58.985815] Node 0 DMA: 3*4kB (UM) 2*8kB (U) 1*16kB (M) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4B [ 58.999026] Node 0 DMA32: 38*4kB (EM) 78*8kB (UEMR) 11*16kB (UER) 0*32kB 3*64kB (R) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096B [ 59.012831] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 59.021469] 14852 total pagecache pages [ 59.025522] 0 pages in swap cache [ 59.029048] Swap cache stats: add 0, delete 0, find 0/0 [ 59.034491] Free swap = 0kB [ 59.037583] Total swap = 0kB [ 59.041703] 90211 pages RAM [ 59.044712] 53920 pages reserved [ 59.048157] 267612 pages shared [ 59.051514] 31588 pages non-shared [ 59.055133] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 59.063173] [ 64] 0 64 13020 570 24 0 0 systemd-journal [ 59.072172] [ 170] 32 170 9977 282 23 0 0 rpcbind [ 59.080474] [ 174] 0 174 11120 416 25 0 0 rpc.statd [ 59.088960] [ 179] 0 179 6366 169 16 0 0 rpc.idmapd [ 59.097551] [ 201] 0 201 8729 425 22 0 -1000 systemd-udevd [ 59.106402] [ 324] 0 324 25589 3179 47 0 0 dhclient [ 59.114815] [ 383] 0 383 3101 553 16 0 0 dracut-pre-pivo [ 59.123832] [ 517] 0 517 13333 7103 42 0 0 makedumpfile [ 59.132594] [ 518] 0 518 8728 272 21 0 0 systemd-udevd [ 59.141442] [ 519] 0 519 8728 214 21 0 0 systemd-udevd [ 59.150287] Out of memory: Kill process 517 (makedumpfile) score 190 or sacrifice child [ 59.158527] Killed process 517 (makedumpfile) total-vm:53332kB,anon-rss:27524kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"(bbbb)
kdump: dump target is/dev/sda2
kdump: saving t[ 13.492083] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
o /sysroot//var/crash/127.0.0.1-2014.05.15-11:24:51/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore
calculate_cyclic_buffer_size, free memory 77033472Buffer size for the cyclic mode: 30813388
Excluding unnecessary pages : [100.0 %] |
writeout_dumpfile,beforewrite_elf_headerfree, memory 15233024
Excluding unnecessary pages : [100.0 %] -
writeout_dumpfile,beforewrite_elf_pages_cyclic, memory 15204352In
calculate_cyclic_buffer_size, free memory 15204352
Excluding unnecessary pages : [100.0 %] /
Incalculate_cyclic_buffer_size, free memory 2932736
Excluding unnecessary pages : [100.0 %] |
Incalculate_cyclic_buffer_size, free memory 2387968
Excluding unnecessary pages : [100.0 %] \
Incalculate_cyclic_buffer_size, free memory 2555904
Excluding unnecessary pages : [100.0 %] -
Incalculate_cyclic_buffer_size, free memory 2838528
Copying data : [ 3.8 %] |[ 15.406789]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0[ 15.415222] makedumpfile cpuset=/ mems_allowed=0 [ 15.420191] CPU: 0 PID: 287 Comm: makedumpfile Not tainted 3.10.0-114.el7.x86_64 #1 [ 15.428092] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.02 03/09/2012 [ 15.437280] ffff88002f8196c0 0000000043db25d9 ffff88002ff2b7e0 ffffffff815ec0ca [ 15.445105] ffff88002ff2b870 ffffffff815e773d ffffffff810b6918 ffff880030018bb0 [ 15.452935] ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81102e03 [ 15.460693] Call Trace: [ 15.463375] [<ffffffff815ec0ca>] dump_stack+0x19/0x1b [ 15.468743] [<ffffffff815e773d>] dump_header+0x8e/0x214 [ 15.474288] [<ffffffff810b6918>] ? ktime_get_ts+0x48/0xe0 [ 15.479999] [<ffffffff81102e03>] ? delayacct_end+0x33/0xb0 [ 15.485790] [<ffffffff8114516e>] oom_kill_process+0x24e/0x3b0 [ 15.491848] [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30 [ 15.498430] [<ffffffff81145996>] out_of_memory+0x4b6/0x4f0 [ 15.504222] [<ffffffff8114b55b>] __alloc_pages_nodemask+0xa7b/0xac0 [ 15.510793] [<ffffffff811885f9>] alloc_pages_current+0xa9/0x170 [ 15.517020] [<ffffffff81141957>] __page_cache_alloc+0x87/0xb0 [ 15.523061] [<ffffffff81142566>] grab_cache_page_write_begin+0x76/0xd0 [ 15.529898] [<ffffffffa02ab133>] ext4_da_write_begin+0xa3/0x330 [ext4] [ 15.536728] [<ffffffff8114158e>] generic_file_buffered_write+0x11e/0x290 [ 15.543724] [<ffffffff811436e5>] __generic_file_aio_write+0x1d5/0x3e0 [ 15.550462] [<ffffffff81050f00>] ? rbt_memtype_copy_nth_element+0xa0/0xa0 [ 15.557551] [<ffffffff8114394d>] generic_file_aio_write+0x5d/0xc0 [ 15.563952] [<ffffffffa02a1189>] ext4_file_write+0xa9/0x450 [ext4] [ 15.570434] [<ffffffff811797fc>] ? free_vmap_area_noflush+0x7c/0x90 [ 15.577009] [<ffffffff811af21d>] do_sync_write+0x8d/0xd0 [ 15.582616] [<ffffffff811af9bd>] vfs_write+0xbd/0x1e0 [ 15.587967] [<ffffffff811b0408>] SyS_write+0x58/0xb0 [ 15.593223] [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b [ 15.599437] Mem-Info: [ 15.601912] Node 0 DMA per-cpu: [ 15.605283] CPU 0: hi: 0, btch: 1 usd: 0 [ 15.610279] Node 0 DMA32 per-cpu: [ 15.613825] CPU 0: hi: 42, btch: 7 usd: 29 [ 15.618827] active_anon:16072 inactive_anon:924 isolated_anon:0 [ 15.618827] active_file:2075 inactive_file:2232 isolated_file:0 [ 15.618827] unevictable:6464 dirty:0 writeback:3 unstable:0 [ 15.618827] free:511 slab_reclaimable:2077 slab_unreclaimable:3528 [ 15.618827] mapped:1055 shmem:1088 pagetables:146 bounce:0 [ 15.618827] free_cma:0 [ 15.654098] Node 0 DMA free:508kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictabs [ 15.693926] lowmem_reserve[]: 0 129 129 129 [ 15.698429] Node 0 DMA32 free:1536kB min:1452kB low:1812kB high:2176kB active_anon:64288kB inactive_anon:3696kB active_file:8300kB inacto [ 15.742999] lowmem_reserve[]: 0 0 0 0 [ 15.746981] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B [ 15.760499] Node 0 DMA32: 6*4kB (M) 7*8kB (R) 3*16kB (R) 2*32kB (M) 1*64kB (M) 0*128kB 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB [ 15.774255] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 15.782928] 11860 total pagecache pages [ 15.786986] 0 pages in swap cache [ 15.790524] Swap cache stats: add 0, delete 0, find 0/0 [ 15.795976] Free swap = 0kB [ 15.799080] Total swap = 0kB [ 15.803169] 90211 pages RAM [ 15.806188] 53919 pages reserved [ 15.809635] 6131 pages shared [ 15.812820] 29843 pages non-shared [ 15.816437] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 15.824495] [ 64] 0 64 13020 559 23 0 0 systemd-journal [ 15.833506] [ 119] 0 119 8864 549 23 0 -1000 systemd-udevd [ 15.842347] [ 204] 0 204 3071 531 15 0 0 dracut-pre-pivo [ 15.851353] [ 287] 0 287 21581 15351 58 0 0 makedumpfile [ 15.860103] Out of memory: Kill process 287 (makedumpfile) score 411 or sacrifice child [ 15.868333] Killed process 287 (makedumpfile) total-vm:86324kB,
Quoted 3 lines
anon-rss:60516kB, file-rss:888kB
//lib/dracut/hooks/pre-pivot/999
Generating "/run/initramfs/rdsosreport.txt"
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
Hi Astushi,
Quoted 11 lines
No, 16MB was used for bitmaps also in my case.
--cyclic-buffer option means specifying each bitmap size, so allocated bitmap size is the double of --cyclic-buffer.
(But now, this behavior is only for ELF case, just the specified size is allocated in kdump case. Yes, it's confusing...)--cyclic-buffer
buffer_size
Specify the buffer size in kilo bytes for analysis in the cyclic mode. Actually, the double ofbuffer_sizekilo bytes will be allocated in memory. In the cyclic mode, the number of cycles is represented as:
Yeah, I was wrong about this. If specified --cyclic-buffer doesn't exceed the memory limit, it is equal to info->bufsize_cyclic. But for elf dump, it need be double. E.g if free memory is 30M, --cyclic-buffer is 16M, it will absolutely fail. But the code check here should be done separately. Since it's meaningless for elf dump if 2 x --cyclic-buffer exceeds the free memory.
if (info->bufsize_cyclic > free_memory) {
.......
{I think code here need be changed.
Quoted 34 lines
So only 456KB was the requirement for other purposes, 20% of available memory (3.5MB) was enough, thus the current assumption is safe even when available memory is only 17MB like my test.
According to my test:
http://lists.infradead.org/pipermail/kexec/2014-May/011784.html
Quoted 5 lines
Here is the result on a 2nd kernel environment:
parameter | result
dump_lv| buffer[KiB] | mmap (=4MiB) | VmHWM [KiB]
--------+-------------+---------------+------------------Quoted 8 lines
d31 | 1 | on | 776 // about 700 Ed31 | 1 | on | 712 d31 | 1 | off | 704 Ed31 | 1 | off | 708 d31 | 1000 | on | 1776 // about 700 + 1000 Ed31 | 1000 | on | 2716 // about 700 + 1000 * 2 d31 | 1000 | off | 1660 Ed31 | 1000 | off | 2556The requirement memory size for other purposes seems about 700KB when the dump level is 31, it's so small that we can ignore it.
The size will increase by KB order based on the system memory size in practice (this assumption comes from my code investigation), but 6MB (20% of 30MB) still sounds much enough as safety limit.
This is why I was wondering why OOM happened on your machine.Now that I heard that OOM happened only on nfs, I guess just nfs requires lots of memory. (I did my test only on local fs.)
If we can get the required size, we could reflect it in the safety limit size.
Quoted 6 lines
I haven't checked your test result below yet, so I'll mention it later.
BTW, I prepared a temporal branch "oom" to investigate this issue, let's use this after this:
About OOM, I don't know why the difference between yours and mine is so big. My test machine has 16G memory, and the reserved crashkernel=161M. The OOM happened in a very high frequency. You can check the last section of kdump failed log, OOM failed when use 80% for info->bufsize_buffer though left free memory is 13M.
I changed the code like below.
---------------------------------------------------
makedumpfile.c +1 -0
@@ -3157,6 +3157,7 @@ out:return FALSE;if (info->flag_cyclic) { + printf("\ncalculate_cyclic_buffer_size,
get_free_memory_size: %llu\n", get_free_memory_size());
if (info->bufsize_cyclic == 0) {
if (!calculate_cyclic_buffer_size())
return FALSE;@@ -3204,6 +3205,7 @@ out:
DEBUG_MSG("\n");
DEBUG_MSG("Buffer size for the cyclic mode: %ld\n",info->bufsize_cyclic);
+ printf("\n Buffer size for the cyclic mode: %ld\n",
info->bufsize_cyclic);
}
if (!is_xen_memory() && !cache_init())
@@ -9060,7 +9062,7 @@ calculate_cyclic_buffer_size(void) {
if (info->num_dumpfile > 1)
bitmap_size /= info->num_dumpfile;- info->bufsize_cyclic = MIN(limit_size, bitmap_size);
+ info->bufsize_cyclic = limit_size;
-------------------------------------------------
bhe# cat /etc/kdump.conf
path /var/crashcore_collector makedumpfile -E --message-level 1 -d 31
------------------------------------------
kdump: dump target is /dev/sda2
kdump: saving [ 9.595153] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
to /sysroot//var/crash/127.0.0.1-2014.05.19-18:50:18/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore
calculate_cyclic_buffer_size, get_free_memory_size: 68857856
Buffer size for the cyclic mode: 27543142
Copying data : [ 15.9 %] -[ 14.955468]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0
[ 14.963876] makedumpfile cpuset=/ mems_allowed=0 [ 14.968723] CPU: 0 PID: 286 Comm: makedumpfile Not tainted 3.10.0-123.el7.x86_64 #1 [ 14.976606] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.02 03/09/2012 [ 14.985567] ffff88002fedc440 00000000f650c592 ffff88002fcb57d0 ffffffff815e19ba [ 14.993291] ffff88002fcb5860 ffffffff815dd02d ffffffff810b68f8 ffff8800359dc0c0 [ 15.001013] ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81102e03 [ 15.008733] Call Trace: [ 15.011413] [<ffffffff815e19ba>] dump_stack+0x19/0x1b [ 15.016778] [<ffffffff815dd02d>] dump_header+0x8e/0x214 [ 15.022321] [<ffffffff810b68f8>] ? ktime_get_ts+0x48/0xe0 [ 15.028036] [<ffffffff81102e03>] ? proc_do_uts_string+0xe3/0x130 [ 15.034383] [<ffffffff8114520e>] oom_kill_process+0x24e/0x3b0 [ 15.040446] [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30 [ 15.047068] [<ffffffff81145a36>] out_of_memory+0x4b6/0x4f0 [ 15.052864] [<ffffffff8114b579>] __alloc_pages_nodemask+0xa09/0xb10 [ 15.059482] [<ffffffff81188779>] alloc_pages_current+0xa9/0x170 [ 15.065711] [<ffffffff811419f7>] __page_cache_alloc+0x87/0xb0 [ 15.071804] [<ffffffff81142606>] grab_cache_page_write_begin+0x76/0xd0 [ 15.078646] [<ffffffffa02aa133>] ext4_da_write_begin+0xa3/0x330 [ext4] [ 15.085495] [<ffffffff8114162e>] generic_file_buffered_write+0x11e/0x290 [ 15.092504] [<ffffffff81143785>] __generic_file_aio_write+0x1d5/0x3e0 [ 15.099294] [<ffffffff81050f00>] ? rbt_memtype_copy_nth_element+0xa0/0xa0 [ 15.106385] [<ffffffff811439ed>] generic_file_aio_write+0x5d/0xc0 [ 15.112841] [<ffffffffa02a0189>] ext4_file_write+0xa9/0x450 [ext4] [ 15.119321] [<ffffffff8117997c>] ? free_vmap_area_noflush+0x7c/0x90 [ 15.125884] [<ffffffff811af36d>] do_sync_write+0x8d/0xd0 [ 15.131492] [<ffffffff811afb0d>] vfs_write+0xbd/0x1e0 [ 15.136839] [<ffffffff811b0558>] SyS_write+0x58/0xb0 [ 15.142091] [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b [ 15.148293] Mem-Info: [ 15.150770] Node 0 DMA per-cpu: [ 15.154138] CPU 0: hi: 0, btch: 1 usd: 0 [ 15.159133] Node 0 DMA32 per-cpu: [ 15.162741] CPU 0: hi: 42, btch: 7 usd: 12 [ 15.167732] active_anon:14395 inactive_anon:1034 isolated_anon:0 [ 15.167732] active_file:2406 inactive_file:2533 isolated_file:0 [ 15.167732] unevictable:7137 dirty:2 writeback:1511 unstable:0 [ 15.167732] free:488 slab_reclaimable:2371 slab_unreclaimable:3533 [ 15.167732] mapped:1110 shmem:1065 pagetables:166 bounce:0 [ 15.167732] free_cma:0 [ 15.203076] Node 0 DMA free:508kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictabs [ 15.242882] lowmem_reserve[]: 0 128 128 128 [ 15.247447] Node 0 DMA32 free:1444kB min:1444kB low:1804kB high:2164kB active_anon:57580kB inactive_anon:4136kB active_file:9624kB inacts [ 15.292683] lowmem_reserve[]: 0 0 0 0 [ 15.296761] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B [ 15.310372] Node 0 DMA32: 78*4kB (UEM) 52*8kB (UEM) 17*16kB (UM) 12*32kB (UM) 2*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*40B [ 15.324412] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 15.333088] 13144 total pagecache pages [ 15.337161] 0 pages in swap cache [ 15.340708] Swap cache stats: add 0, delete 0, find 0/0 [ 15.346165] Free swap = 0kB [ 15.349280] Total swap = 0kB [ 15.353385] 90211 pages RAM [ 15.356420] 53902 pages reserved [ 15.359880] 6980 pages shared [ 15.363088] 29182 pages non-shared [ 15.366719] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 15.374788] [ 85] 0 85 13020 553 24 0 0 systemd-journal [ 15.383818] [ 134] 0 134 8860 547 22 0 -1000 systemd-udevd [ 15.392664] [ 146] 0 146 5551 245 23 0 0 plymouthd [ 15.401167] [ 230] 0 230 3106 537 16 0 0 dracut-pre-pivo [ 15.410181] [ 286] 0 286 19985 13756 55 0 0 makedumpfile [ 15.418942] Out of memory: Kill process 286 (makedumpfile) score 368 or sacrifice child [ 15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,
anon-rss:54132kB, file-rss:892kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"
Quoted 5 lines
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
On Mon, May 19, 2014 at 07:15:38PM +0800, [email protected] wrote:
[..]
Quoted 14 lines
-------------------------------------------------
bhe# cat/etc/kdump.conf
path/var/crashcore_collectormakedumpfile -E --message-level 1 -d 31------------------------------------------
kdump: dump target is/dev/sda2
kdump: saving [ 9.595153] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
to /sysroot//var/crash/127.0.0.1-2014.05.19-18:50:18/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore
Quoted 3 lines
calculate_cyclic_buffer_size,get_free_memory_size: 68857856Buffer size for the cyclic mode: 27543142
Bao,
So 68857856 is 65MB. So we have around 65MB free when makedumpfile started.
27543142 is 26MB. So we reserved 26MB for bitmaps or we reserved 52MB for bitmaps?
Looking at the backtrace, larry pointed out few things.
- makedumpfile has already allocated around 52MB of anonymous memory. I guess this primarily comes from bitmaps and looks like we are reserving 52MB in bitmaps and not 26MB. I think this could be consistent with current 80% logic as 80% of 65MB is around 52MB.
[ 15.427173] Killed process 286 (makedumpfile) total-vm:79940kB, anon-rss:54132kB, file-rss:892kB
- So we are left with 65-52 = 13MB of total memory for kernel as well as makedumpfile.
- We have around 1500 pages in page cache which are in writeback stage. That means around 6MB of pages are dirty and being written back to disk. That means makedumpfile might not require lot of memory but kernel does require free memory in dirty/writeback pages when dump file is being written.
[ 15.167732] unevictable:7137 dirty:2 writeback:1511 unstable:0
- Larry mentioend that there are around 5000 pages (20MB of memory) sitting in file pages in page cache which ideally should be reclaimable. It is not clear why that memory is not being reclaimed fast enough.
[ 15.167732] active_file:2406 inactive_file:2533 isolated_file:0
So to me bottom line is that once the write out starts, kernel needs memory for holding dirty and writeback pages in cache too. So we probably are being too aggresive in allocating 80% of free memory for bitmaps. May be we should drop it down to 50-60% of free memory for bitmaps.
Thanks
Vivek
Quoted 91 lines
Copying data : [ 15.9 %] -[ 14.955468]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0[ 14.963876] makedumpfile cpuset=/ mems_allowed=0 [ 14.968723] CPU: 0 PID: 286 Comm: makedumpfile Not tainted 3.10.0-123.el7.x86_64 #1 [ 14.976606] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.02 03/09/2012 [ 14.985567] ffff88002fedc440 00000000f650c592 ffff88002fcb57d0 ffffffff815e19ba [ 14.993291] ffff88002fcb5860 ffffffff815dd02d ffffffff810b68f8 ffff8800359dc0c0 [ 15.001013] ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81102e03 [ 15.008733] Call Trace: [ 15.011413] [<ffffffff815e19ba>] dump_stack+0x19/0x1b [ 15.016778] [<ffffffff815dd02d>] dump_header+0x8e/0x214 [ 15.022321] [<ffffffff810b68f8>] ? ktime_get_ts+0x48/0xe0 [ 15.028036] [<ffffffff81102e03>] ? proc_do_uts_string+0xe3/0x130 [ 15.034383] [<ffffffff8114520e>] oom_kill_process+0x24e/0x3b0 [ 15.040446] [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30 [ 15.047068] [<ffffffff81145a36>] out_of_memory+0x4b6/0x4f0 [ 15.052864] [<ffffffff8114b579>] __alloc_pages_nodemask+0xa09/0xb10 [ 15.059482] [<ffffffff81188779>] alloc_pages_current+0xa9/0x170 [ 15.065711] [<ffffffff811419f7>] __page_cache_alloc+0x87/0xb0 [ 15.071804] [<ffffffff81142606>] grab_cache_page_write_begin+0x76/0xd0 [ 15.078646] [<ffffffffa02aa133>] ext4_da_write_begin+0xa3/0x330 [ext4] [ 15.085495] [<ffffffff8114162e>] generic_file_buffered_write+0x11e/0x290 [ 15.092504] [<ffffffff81143785>] __generic_file_aio_write+0x1d5/0x3e0 [ 15.099294] [<ffffffff81050f00>] ? rbt_memtype_copy_nth_element+0xa0/0xa0 [ 15.106385] [<ffffffff811439ed>] generic_file_aio_write+0x5d/0xc0 [ 15.112841] [<ffffffffa02a0189>] ext4_file_write+0xa9/0x450 [ext4] [ 15.119321] [<ffffffff8117997c>] ? free_vmap_area_noflush+0x7c/0x90 [ 15.125884] [<ffffffff811af36d>] do_sync_write+0x8d/0xd0 [ 15.131492] [<ffffffff811afb0d>] vfs_write+0xbd/0x1e0 [ 15.136839] [<ffffffff811b0558>] SyS_write+0x58/0xb0 [ 15.142091] [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b [ 15.148293] Mem-Info: [ 15.150770] Node 0 DMA per-cpu: [ 15.154138] CPU 0: hi: 0, btch: 1 usd: 0 [ 15.159133] Node 0 DMA32 per-cpu: [ 15.162741] CPU 0: hi: 42, btch: 7 usd: 12 [ 15.167732] active_anon:14395 inactive_anon:1034 isolated_anon:0 [ 15.167732] active_file:2406 inactive_file:2533 isolated_file:0 [ 15.167732] unevictable:7137 dirty:2 writeback:1511 unstable:0 [ 15.167732] free:488 slab_reclaimable:2371 slab_unreclaimable:3533 [ 15.167732] mapped:1110 shmem:1065 pagetables:166 bounce:0 [ 15.167732] free_cma:0 [ 15.203076] Node 0 DMA free:508kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictabs [ 15.242882] lowmem_reserve[]: 0 128 128 128 [ 15.247447] Node 0 DMA32 free:1444kB min:1444kB low:1804kB high:2164kB active_anon:57580kB inactive_anon:4136kB active_file:9624kB inacts [ 15.292683] lowmem_reserve[]: 0 0 0 0 [ 15.296761] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B [ 15.310372] Node 0 DMA32: 78*4kB (UEM) 52*8kB (UEM) 17*16kB (UM) 12*32kB (UM) 2*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*40B [ 15.324412] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 15.333088] 13144 total pagecache pages [ 15.337161] 0 pages in swap cache [ 15.340708] Swap cache stats: add 0, delete 0, find 0/0 [ 15.346165] Free swap = 0kB [ 15.349280] Total swap = 0kB [ 15.353385] 90211 pages RAM [ 15.356420] 53902 pages reserved [ 15.359880] 6980 pages shared [ 15.363088] 29182 pages non-shared [ 15.366719] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 15.374788] [ 85] 0 85 13020 553 24 0 0 systemd-journal [ 15.383818] [ 134] 0 134 8860 547 22 0 -1000 systemd-udevd [ 15.392664] [ 146] 0 146 5551 245 23 0 0 plymouthd [ 15.401167] [ 230] 0 230 3106 537 16 0 0 dracut-pre-pivo [ 15.410181] [ 286] 0 286 19985 13756 55 0 0 makedumpfile [ 15.418942] Out of memory: Kill process 286 (makedumpfile) score 368 or sacrifice child [ 15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,
Quoted 9 lines
anon-rss:54132kB, file-rss:892kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"Quoted 5 lines
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to [email protected]Hello Bao,
Quoted 22 lines
Hi Astushi,
Quoted 11 lines
No, 16MB was used for bitmaps also in my case.
--cyclic-buffer option means specifying each bitmap size, so allocated bitmap size is the double of --cyclic-buffer.
(But now, this behavior is only for ELF case, just the specified size is allocated in kdump case. Yes, it's confusing...)--cyclic-buffer
buffer_size
Specify the buffer size in kilo bytes for analysis in the cyclic mode. Actually, the double ofbuffer_sizekilo bytes will be allocated in memory. In the cyclic mode, the number of cycles is represented as:Yeah, I was wrong about this. If specified --cyclic-buffer doesn't exceed the memory limit, it is equal to
info->bufsize_cyclic. But for elf dump, it need be double. E.g if free memory is 30M, --cyclic-buffer is 16M, it will absolutely fail. But the code check here should be done separately. Since it's meaningless for elf dump if 2 x --cyclic-buffer exceeds the free memory.
Quoted 5 lines
if (info->bufsize_cyclic > free_memory) { ....... {I think code here need be changed.
Exactly, this is a bug.
I can fix it simply by modification to the check above, but I think it's better to change the ELF path to get rid of the 1st bitmap buffer since it's unused. Then the required buffer size will be just <--cyclic-buffer> in both cases, so I'll post the patch to do it later.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
In reply to Vivek GoyalQuoted 21 lines · On Mon, May 19, 2014 at 07:15:38PM +0800, [email protected] wrote:
[..]
Quoted 14 lines
-------------------------------------------------
bhe# cat/etc/kdump.conf
path/var/crashcore_collectormakedumpfile -E --message-level 1 -d 31------------------------------------------
kdump: dump target is/dev/sda2
kdump: saving [ 9.595153] EXT4-fs (sda2): re-mounted. Opts:
data=ordered
to /sysroot//var/crash/127.0.0.1-2014.05.19-18:50:18/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcoreQuoted 3 lines
calculate_cyclic_buffer_size,get_free_memory_size: 68857856Buffer size for the cyclic mode: 27543142
Quoted 7 lines
Bao,
So 68857856 is 65MB. So we have around 65MB free when makedumpfile started.
27543142 is 26MB. So we reserved 26MB for bitmaps or we reserved 52MB for bitmaps?
52MB is correct, so the larry's view below looks right.
Quoted 25 lines
Looking at the backtrace, larry pointed out few things.
- makedumpfile has already allocated around 52MB of anonymous memory. I guess this primarily comes from bitmaps and looks like we are reserving 52MB in bitmaps and not 26MB. I think this could be consistent with current 80% logic as 80% of 65MB is around 52MB.
[ 15.427173] Killed process 286 (makedumpfile) total-vm:79940kB, anon-rss:54132kB, file-rss:892kB- So we are left with 65-52 = 13MB of total memory for kernel as well as makedumpfile.
- We have around 1500 pages in page cache which are in writeback stage. That means around 6MB of pages are dirty and being written back to disk. That means makedumpfile might not require lot of memory but kernel does require free memory in dirty/writeback pages when dump file is being written.
[ 15.167732] unevictable:7137 dirty:2 writeback:1511 unstable:0- Larry mentioend that there are around 5000 pages (20MB of memory) sitting in file pages in page cache which ideally should be reclaimable. It is not clear why that memory is not being reclaimed fast enough.
Quoted 6 lines
[ 15.167732] active_file:2406 inactive_file:2533 isolated_file:0So to me bottom line is that once the write out starts, kernel needs memory for holding dirty and writeback pages in cache too. So we probably are being too aggresive in allocating 80% of free memory for bitmaps. May be we should drop it down to 50-60% of free memory for bitmaps.
I don't disagree to changing the 80% limit but I prefer to remove such a percentage threshold because it's dependent on the environment. Actually, I think it makes this problem more complex.
Now, thanks to page_is_buddy(), the performance degradation caused by multi-cycle processing looks very small according to the benchmark on 2TB memory:
https://lkml.org/lkml/2013/3/26/914
This result means we don't need to make an effort to allocate the bitmap buffer as large as possible. So how about just setting a small fixed value like 5MB as a safety limit?
It may be safer, and it will be easier to estimate the total memory usage of makedumpfile, so I think it's better way if the most users especially large machine users accept it.
Thanks
Atsushi Kumagai
Quoted 97 lines
Quoted 97 lines
Copying data : [ 15.9 %] -[ 14.955468]
makedumpfile invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0[ 14.963876] makedumpfile cpuset=/ mems_allowed=0 [ 14.968723] CPU: 0 PID: 286 Comm: makedumpfile Not tainted 3.10.0-123.el7.x86_64 #1 [ 14.976606] Hardware name: Hewlett-Packard HP Z420 Workstation/1589, BIOS J61 v01.02 03/09/2012 [ 14.985567] ffff88002fedc440 00000000f650c592 ffff88002fcb57d0 ffffffff815e19ba [ 14.993291] ffff88002fcb5860 ffffffff815dd02d ffffffff810b68f8 ffff8800359dc0c0 [ 15.001013] ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81102e03 [ 15.008733] Call Trace: [ 15.011413] [<ffffffff815e19ba>] dump_stack+0x19/0x1b [ 15.016778] [<ffffffff815dd02d>] dump_header+0x8e/0x214 [ 15.022321] [<ffffffff810b68f8>] ? ktime_get_ts+0x48/0xe0 [ 15.028036] [<ffffffff81102e03>] ? proc_do_uts_string+0xe3/0x130 [ 15.034383] [<ffffffff8114520e>] oom_kill_process+0x24e/0x3b0 [ 15.040446] [<ffffffff8106af3e>] ? has_capability_noaudit+0x1e/0x30 [ 15.047068] [<ffffffff81145a36>] out_of_memory+0x4b6/0x4f0 [ 15.052864] [<ffffffff8114b579>] __alloc_pages_nodemask+0xa09/0xb10 [ 15.059482] [<ffffffff81188779>] alloc_pages_current+0xa9/0x170 [ 15.065711] [<ffffffff811419f7>] __page_cache_alloc+0x87/0xb0 [ 15.071804] [<ffffffff81142606>] grab_cache_page_write_begin+0x76/0xd0 [ 15.078646] [<ffffffffa02aa133>] ext4_da_write_begin+0xa3/0x330 [ext4] [ 15.085495] [<ffffffff8114162e>] generic_file_buffered_write+0x11e/0x290 [ 15.092504] [<ffffffff81143785>] __generic_file_aio_write+0x1d5/0x3e0 [ 15.099294] [<ffffffff81050f00>] ? rbt_memtype_copy_nth_element+0xa0/0xa0 [ 15.106385] [<ffffffff811439ed>] generic_file_aio_write+0x5d/0xc0 [ 15.112841] [<ffffffffa02a0189>] ext4_file_write+0xa9/0x450 [ext4] [ 15.119321] [<ffffffff8117997c>] ? free_vmap_area_noflush+0x7c/0x90 [ 15.125884] [<ffffffff811af36d>] do_sync_write+0x8d/0xd0 [ 15.131492] [<ffffffff811afb0d>] vfs_write+0xbd/0x1e0 [ 15.136839] [<ffffffff811b0558>] SyS_write+0x58/0xb0 [ 15.142091] [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b [ 15.148293] Mem-Info: [ 15.150770] Node 0 DMA per-cpu: [ 15.154138] CPU 0: hi: 0, btch: 1 usd: 0 [ 15.159133] Node 0 DMA32 per-cpu: [ 15.162741] CPU 0: hi: 42, btch: 7 usd: 12 [ 15.167732] active_anon:14395 inactive_anon:1034 isolated_anon:0 [ 15.167732] active_file:2406 inactive_file:2533 isolated_file:0 [ 15.167732] unevictable:7137 dirty:2 writeback:1511 unstable:0 [ 15.167732] free:488 slab_reclaimable:2371 slab_unreclaimable:3533 [ 15.167732] mapped:1110 shmem:1065 pagetables:166 bounce:0 [ 15.167732] free_cma:0 [ 15.203076] Node 0 DMA free:508kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictabs [ 15.242882] lowmem_reserve[]: 0 128 128 128 [ 15.247447] Node 0 DMA32 free:1444kB min:1444kB low:1804kB high:2164kB active_anon:57580kB inactive_anon:4136kB active_file:9624kB inacts [ 15.292683] lowmem_reserve[]: 0 0 0 0 [ 15.296761] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB B [ 15.310372] Node 0 DMA32: 78*4kB (UEM) 52*8kB (UEM) 17*16kB (UM) 12*32kB (UM) 2*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*40B [ 15.324412] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 15.333088] 13144 total pagecache pages [ 15.337161] 0 pages in swap cache [ 15.340708] Swap cache stats: add 0, delete 0, find 0/0 [ 15.346165] Free swap = 0kB [ 15.349280] Total swap = 0kB [ 15.353385] 90211 pages RAM [ 15.356420] 53902 pages reserved [ 15.359880] 6980 pages shared [ 15.363088] 29182 pages non-shared [ 15.366719] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 15.374788] [ 85] 0 85 13020 553 24 0 0 systemd-journal [ 15.383818] [ 134] 0 134 8860 547 22 0 -1000 systemd-udevd [ 15.392664] [ 146] 0 146 5551 245 23 0 0 plymouthd [ 15.401167] [ 230] 0 230 3106 537 16 0 0 dracut-pre-pivo [ 15.410181] [ 286] 0 286 19985 13756 55 0 0 makedumpfile [ 15.418942] Out of memory: Kill process 286 (makedumpfile) score 368 or sacrifice child [ 15.427173] Killed process 286 (makedumpfile) total-vm:79940kB,anon-rss:54132kB, file-rss:892kB
//lib/dracut/hooks/pre-pivot/9999-kdump.sh: line
Generating "/run/initramfs/rdsosreport.txt"Quoted 2 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH] makedumpfile: change the wrong code to calculate bufsize_cyclic for elf dump
On Tue, May 27, 2014 at 05:34:05AM +0000, Atsushi Kumagai wrote:
[..]
Quoted 13 lines
Quoted 4 lines
So to me bottom line is that once the write out starts, kernel needs memory for holding dirty and writeback pages in cache too. So we probably are being too aggresive in allocating 80% of free memory for bitmaps. May be we should drop it down to 50-60% of free memory for bitmaps.
I don't disagree to changing the 80% limit but I prefer to remove such a percentage threshold because it's dependent on the environment. Actually, I think it makes this problem more complex.
Now, thanks to
page_is_buddy(), the performance degradation caused by multi-cycle processing looks very small according to the benchmark on 2TB memory:
Quoted 8 lines
https://lkml.org/lkml/2013/3/26/914
This result means we don't need to make an effort to allocate the bitmap buffer as large as possible. So how about just setting a small fixed value like 5MB as a safety limit?
It may be safer, and it will be easier to estimate the total memory usage of makedumpfile, so I think it's better way if the most users especially large machine users accept it.
Hi Atsushi,
If increasing buffer size does not have any significant increase in dump time, then it is reasonable to have fixed buffer size for bitmaps. (instead of trying to maximize bitmap size).
We can probably go for 4MB as bitmap size (instead of 5MB).
Also can we modify the logic a bit so that we automatically shrhink the size of bitmap if sufficient memory is not available. Say assume that 60% of available memory can be used for bitmaps. If it is less then 4MB then we drop the buffer size and that hopefully still makes makedumpfile successful (instead of being OOMed).
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec