makedumpfile: Use max_pfn from mem_map array
Subsystems (2)
PATCH 2/2 makedumpfile: Use max_pfn from mem_map array
There are dump mechansims like s390 stand-alond dump or KVM virsh dump that write the physical memory of a machine and are not aware of the dumped operating system. For those dump mechanisms it can happen that for the Linux kernel of the dumped system the "mem=" kernel parameter has been specified. In this case max_mapnr that makedumpfile gets from the ELF header can be bigger than the maximum page frame number used by the dumped Linux kernel.
With this patch makedumpfile gets the maximum page frame number from the mem_map array and adjusts info->max_mapnr if this value is smaller than the value calculated from the ELF header.
--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
makedumpfile.c +13 -1
@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;switch (get_mem_type ()) {case SPARSEMEM :
@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) { +if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr =MIN (info->max_mapnr, max_pfn);return ret; }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
From: Michael Holzheu <[email protected]>
Subject: [PATCH 2/2] makedumpfile: Use max_pfn from mem_map array
Date: Tue, 25 Mar 2014 17:14:20 +0100
Quoted 20 lines
There are dump mechansims like s390 stand-alond dump or KVM virsh dump that write the physical memory of a machine and are not aware of the dumped operating system. For those dump mechanisms it can happen that for the Linux kernel of the dumped system the "mem=" kernel parameter has been specified. In this case
max_mapnrthat makedumpfile gets from the ELF header can be bigger than the maximum page frame number used by the dumped Linux kernel.With this patch makedumpfile gets the maximum page frame number from the
mem_maparray and adjustsinfo->max_mapnrif this value is smaller than the value calculated from the ELF header.Signed-off-by Michael Holzheu <[email protected]>--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
makedumpfile.c get_mem_map_without_mm(void) · line 2833
Quoted 6 lines
makedumpfile.c +2
-1@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;
Please define max_pfn as unsigned long long.
And for i,
makedumpfile.c get_mem_map(void) · line 2870
Quoted 13 lines
makedumpfile.c +6
-0switch (get_mem_type ()) {case SPARSEMEM :@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) {
info->num_mem_map is defined as unsigned int. I guess some warning about comparison with different signedness occurs.
makedumpfile.c get_mem_map(void) · line 2878
Quoted 9 lines
makedumpfile.c +5
-0+if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr =MIN (info->max_mapnr, max_pfn);return ret; }
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 3 lines · On Wed, 26 Mar 2014 10:55:07 +0100 (a/T) HATAYAMA Daisuke…
From: Michael Holzheu <[email protected]>
Subject: [PATCH 2/2] makedumpfile: Usemax_pfnfrommem_maparray
Date: Tue, 25 Mar 2014 17:14:20 +0100
[snip]
Quoted 9 lines
Quoted 9 lines
With this patch makedumpfile gets the maximum page frame number from the
mem_maparray and adjustsinfo->max_mapnrif this value is smaller than the value calculated from the ELF header.Signed-off-by Michael Holzheu <[email protected]>--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
Quoted 11 lines
makedumpfile.c get_mem_map_without_mm(void) · line 2833
Quoted 9 lines
makedumpfile.c +2
-1@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;Please define
max_pfnas unsigned long long.
Ok done.
Quoted 4 lines
And for i,
Quoted 1 line
Quoted 15 lines
makedumpfile.c get_mem_map(void) · line 2870
Quoted 12 lines
makedumpfile.c +6
-0switch (get_mem_type ()) {case SPARSEMEM :@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) {
info->num_mem_mapis defined as unsigned int. I guess some warning about comparison with different signedness occurs.
Ah ok...
With the default CFLAGS for makedumpfile-1.5.5 tarball I do not get any warning. When I add "-W" to CFLAGS, I get lots of warnings including the one you mentioned.
Here the fixed patch:
---
[PATCH 2/2] makedumpfile: Use max_pfn from mem_map array
There are dump mechansims like s390 stand-alond dump or KVM virsh dump that write the physical memory of a machine and are not aware of the dumped operating system. For those dump mechanisms it can happen that for the Linux kernel of the dumped system the "mem=" kernel parameter has been specified. In this case max_mapnr that makedumpfile gets from the ELF header can be bigger than the maximum page frame number used by the dumped Linux kernel.
With this patch makedumpfile gets the maximum page frame number from the mem_map array and adjusts info->max_mapnr if this value is smaller than the value calculated from the ELF header.
--- makedumpfile.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
makedumpfile.c +13 -0
@@ -2829,6 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { +unsigned long long max_pfn =0 ; +unsigned int i;int ret;switch (get_mem_type ()) {
@@ -2861,6 +2863,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) { +if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr =MIN (info->max_mapnr, max_pfn);return ret; }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Michael,
Quoted 52 lines
Quoted 3 lines · On Wed, 26 Mar 2014 10:55:07 +0100 (a/T) HATAYAMA Daisuke…
From: Michael Holzheu <[email protected]>
Subject: [PATCH 2/2] makedumpfile: Usemax_pfnfrommem_maparray
Date: Tue, 25 Mar 2014 17:14:20 +0100[snip]
Quoted 9 lines
Quoted 9 lines
With this patch makedumpfile gets the maximum page frame number from the
mem_maparray and adjustsinfo->max_mapnrif this value is smaller than the value calculated from the ELF header.Signed-off-by Michael Holzheu <[email protected]>--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)Quoted 11 lines
makedumpfile.c get_mem_map_without_mm(void) · line 2833
Quoted 9 lines
makedumpfile.c +2
-1@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;Please define
max_pfnas unsigned long long.Ok done.
Quoted 4 lines
And for i,
Quoted 1 line
Quoted 15 lines
makedumpfile.c get_mem_map(void) · line 2870
Quoted 12 lines
makedumpfile.c +6
-0switch (get_mem_type ()) {case SPARSEMEM :@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) {
info->num_mem_mapis defined as unsigned int. I guess some warning about comparison with different signedness occurs.
Quoted 7 lines
Ah ok...
With the default CFLAGS for makedumpfile-1.5.5 tarball I do not get any warning. When I add "-W" to CFLAGS, I get lots of warnings including the one you mentioned.
Here the fixed patch:
Thanks, I'll merge the fixed version into v1.5.6.
Atsushi Kumagai
Quoted 51 lines
---
[PATCH 2/2] makedumpfile: Usemax_pfnfrommem_maparrayThere are dump mechansims like s390 stand-alond dump or KVM virsh dump that write the physical memory of a machine and are not aware of the dumped operating system. For those dump mechanisms it can happen that for the Linux kernel of the dumped system the "mem=" kernel parameter has been specified. In this case
max_mapnrthat makedumpfile gets from the ELF header can be bigger than the maximum page frame number used by the dumped Linux kernel.With this patch makedumpfile gets the maximum page frame number from the
mem_maparray and adjustsinfo->max_mapnrif this value is smaller than the value calculated from the ELF header.Signed-off-by Michael Holzheu <[email protected]>---
makedumpfile.c| 13 +++++++++++++1 file changed, 13 insertions(+)makedumpfile.c +0
-0@@ -2829,6 +2829,8 @@ get_mem_map_without_mm(void)int
get_mem_map(void)
{makedumpfile.c +2
-0+unsigned long long max_pfn =0 ; +unsigned int i;int ret; switch (get_mem_type()) {makedumpfile.c +11
-0@@ -2861,6 +2863,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) { +if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr =MIN (info->max_mapnr, max_pfn);return ret; }
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 56 lines · On Thu, 27 Mar 2014 05:19:06 +0000 Atsushi Kumagai…
Hello Michael,
Quoted 54 lines
Quoted 3 lines · On Wed, 26 Mar 2014 10:55:07 +0100 (a/T) HATAYAMA Daisuke…
From: Michael Holzheu <[email protected]>
Subject: [PATCH 2/2] makedumpfile: Usemax_pfnfrommem_maparray
Date: Tue, 25 Mar 2014 17:14:20 +0100[snip]
Quoted 9 lines
Quoted 9 lines
With this patch makedumpfile gets the maximum page frame number from the
mem_maparray and adjustsinfo->max_mapnrif this value is smaller than the value calculated from the ELF header.Signed-off-by Michael Holzheu <[email protected]>--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)Quoted 11 lines
makedumpfile.c get_mem_map_without_mm(void) · line 2833
Quoted 9 lines
makedumpfile.c +2
-1@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;Please define
max_pfnas unsigned long long.Ok done.
Quoted 4 lines
And for i,
Quoted 1 line
Quoted 15 lines
makedumpfile.c get_mem_map(void) · line 2870
Quoted 12 lines
makedumpfile.c +6
-0switch (get_mem_type ()) {case SPARSEMEM :@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) {
info->num_mem_mapis defined as unsigned int. I guess some warning about comparison with different signedness occurs.Ah ok...
Quoted 7 lines
Quoted 5 lines
With the default CFLAGS for makedumpfile-1.5.5 tarball I do not get any warning. When I add "-W" to CFLAGS, I get lots of warnings including the one you mentioned.
Here the fixed patch:
Thanks, I'll merge the fixed version into v1.5.6.
Great!
Thanks for your support!
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 63 lines · On Thu, 27 Mar 2014 14:54:41 +0100 Michael Holzheu…
Quoted 60 lines · On Thu, 27 Mar 2014 05:19:06 +0000 Atsushi Kumagai…
Hello Michael,
Quoted 58 lines
Quoted 3 lines · On Wed, 26 Mar 2014 10:55:07 +0100 (a/T) HATAYAMA Daisuke…
From: Michael Holzheu <[email protected]>
Subject: [PATCH 2/2] makedumpfile: Usemax_pfnfrommem_maparray
Date: Tue, 25 Mar 2014 17:14:20 +0100[snip]
Quoted 9 lines
Quoted 9 lines
With this patch makedumpfile gets the maximum page frame number from the
mem_maparray and adjustsinfo->max_mapnrif this value is smaller than the value calculated from the ELF header.Signed-off-by Michael Holzheu <[email protected]>--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)Quoted 11 lines
makedumpfile.c get_mem_map_without_mm(void) · line 2833
Quoted 9 lines
makedumpfile.c +2
-1@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;Please define
max_pfnas unsigned long long.Ok done.
Quoted 4 lines
And for i,
Quoted 1 line
Quoted 15 lines
makedumpfile.c get_mem_map(void) · line 2870
Quoted 12 lines
makedumpfile.c +6
-0switch (get_mem_type ()) {case SPARSEMEM :@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) {
info->num_mem_mapis defined as unsigned int. I guess some warning about comparison with different signedness occurs.Ah ok...
With the default CFLAGS for makedumpfile-1.5.5 tarball I do not get any warning. When I add "-W" to CFLAGS, I get lots of warnings including the one you mentioned.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Here the fixed patch:
Thanks, I'll merge the fixed version into v1.5.6.
Great!
I'm sorry to spoil the party, but this patch broke Xen dumps for me. I'm getting an long series of these messages:
set_bitmap: Can't read the bitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read the bitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read the bitmap(/tmp/kdump_bitmap91pbsO). Invalid argument
...
In fact, it most likely broke all non-cyclic dumps.
That's because the bitmap length is calculated in prepare_bitmap_buffer using info->max_mapnr, but create_1st_bitmap() still loops over all PT_LOAD segments, calling set_bit_on_1st_bitmap() for each PFN. The offset may easily fall beyond the bitmap size.
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 80 lines · On Fri, 28 Mar 2014 12:00:47 +0100 Petr Tesarik <[email protected]> wrote:
Quoted 63 lines · On Thu, 27 Mar 2014 14:54:41 +0100 Michael Holzheu…
Quoted 60 lines · On Thu, 27 Mar 2014 05:19:06 +0000 Atsushi Kumagai…
Hello Michael,
Quoted 58 lines
Quoted 3 lines · On Wed, 26 Mar 2014 10:55:07 +0100 (a/T) HATAYAMA Daisuke…
From: Michael Holzheu <[email protected]>
Subject: [PATCH 2/2] makedumpfile: Usemax_pfnfrommem_maparray
Date: Tue, 25 Mar 2014 17:14:20 +0100[snip]
Quoted 9 lines
Quoted 9 lines
With this patch makedumpfile gets the maximum page frame number from the
mem_maparray and adjustsinfo->max_mapnrif this value is smaller than the value calculated from the ELF header.Signed-off-by Michael Holzheu <[email protected]>--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)Quoted 11 lines
makedumpfile.c get_mem_map_without_mm(void) · line 2833
Quoted 9 lines
makedumpfile.c +2
-1@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;Please define
max_pfnas unsigned long long.Ok done.
Quoted 4 lines
And for i,
Quoted 1 line
Quoted 15 lines
makedumpfile.c get_mem_map(void) · line 2870
Quoted 12 lines
makedumpfile.c +6
-0switch (get_mem_type ()) {case SPARSEMEM :@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) {
info->num_mem_mapis defined as unsigned int. I guess some warning about comparison with different signedness occurs.Ah ok...
With the default CFLAGS for makedumpfile-1.5.5 tarball I do not get any warning. When I add "-W" to CFLAGS, I get lots of warnings including the one you mentioned.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Here the fixed patch:
Thanks, I'll merge the fixed version into v1.5.6.
Great!
I'm sorry to spoil the party, but this patch broke Xen dumps for me. I'm getting an long series of these messages:
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argument
...
Quoted 6 lines
In fact, it most likely broke all non-cyclic dumps.
That's because the bitmap length is calculated in
prepare_bitmap_bufferusinginfo->max_mapnr, butcreate_1st_bitmap()still loops over allPT_LOADsegments, callingset_bit_on_1st_bitmap()for each PFN. The offset may easily fall beyond the bitmap size.
Hello Petr,
I can reproduce your issue with the --non-cyclic option and I will look into this.
Thanks for testing this!
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 2 lines · On Fri, 28 Mar 2014 12:00:47 +0100 Petr Tesarik <[email protected]> wrote:
On Thu, 27 Mar 2014 14:54:41 +0100
Michael Holzheu <[email protected]> wrote:
[snip]
Quoted 14 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Here the fixed patch:
Thanks, I'll merge the fixed version into v1.5.6.
Great!
I'm sorry to spoil the party, but this patch broke Xen dumps for me. I'm getting an long series of these messages:
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argument
...
Quoted 6 lines
In fact, it most likely broke all non-cyclic dumps.
That's because the bitmap length is calculated in
prepare_bitmap_bufferusinginfo->max_mapnr, butcreate_1st_bitmap()still loops over allPT_LOADsegments, callingset_bit_on_1st_bitmap()for each PFN. The offset may easily fall beyond the bitmap size.
What about the following patch. It works for me when I specify the "--non-cyclic" option.
Michael
---
[PATCH] makedumpfile: Fix bitmap create for adjusted info->max_mapnr
If info->max_mapnr has been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
set_bitmap: Can't read the bitmap(/tmp/kdump_bitmap96s9V8). Success
Fix this and consider "info->max_mapnr" in the create bitmap functions.
--- makedumpfile.c | 9 +++++++++ 1 file changed, 9 insertions(+)
makedumpfile.c +9 -0
@@ -4402,6 +4402,9 @@ create_1st_bitmap(void) pfn_start =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); +if (pfn_start > info->max_mapnr) +continue ; + pfn_end =MIN (phys_end, info->max_mapnr);for (pfn = pfn_start; pfn < pfn_end; pfn++) {set_bit_on_1st_bitmap (pfn);
@@ -7511,6 +7514,9 @@ exclude_xen3_user_domain(void) pfn =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); size = pfn_end - pfn; +if (pfn > info->max_mapnr) +continue ; + pfn_end =MIN (phys_end, info->max_mapnr);for (j =0 ; pfn < pfn_end; pfn++, j++) { print_progress(PROGRESS_XEN_DOMAIN, j + (size * i),
@@ -7575,6 +7581,9 @@ exclude_xen4_user_domain(void) pfn =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); size = pfn_end - pfn; +if (pfn > info->max_mapnr) +continue ; + pfn_end =MIN (phys_end, info->max_mapnr);for (j =0 ; pfn < pfn_end; pfn++, j++) { print_progress(PROGRESS_XEN_DOMAIN, j + (size * i),
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 3 lines · On Fri, 28 Mar 2014 17:46:22 +0100 Michael Holzheu…
On Fri, 28 Mar 2014 12:00:47 +0100
Petr Tesarik <[email protected]> wrote:
[snip]
Quoted 23 lines
Quoted 4 lines
That's because the bitmap length is calculated in
prepare_bitmap_bufferusinginfo->max_mapnr, butcreate_1st_bitmap()still loops over allPT_LOADsegments, callingset_bit_on_1st_bitmap()for each PFN. The offset may easily fall beyond the bitmap size.What about the following patch. It works for me when I specify the "--non-cyclic" option.
Michael
---
[PATCH] makedumpfile: Fix bitmap create for adjustedinfo->max_mapnrIf
info->max_mapnrhas been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmap96s9V8). SuccessFix this and consider "
info->max_mapnr" in the create bitmap functions.Signed-off-by Michael Holzheu <[email protected]>---
[snip]
Review exclude_xen4_user_domain(void) · line 7586
Quoted 7 lines
(quoted diff) +3
-0@@ -7575,6 +7581,9 @@ exclude_xen4_user_domain(void) pfn = paddr_to_pfn(phys_start); pfn_end = paddr_to_pfn(phys_end); size = pfn_end - pfn; + if (pfn > info->max_mapnr) + continue; + pfn_end = MIN(phys_end, info->max_mapnr);
Hmm, probably time for weekend. Of course this should be:
pfn_end = MIN(pfn_end, info->max_mapnr);
Here the updated patch:
---
[PATCH] makedumpfile: Fix bitmap create for adjusted info->max_mapnr
If info->max_mapnr has been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
set_bitmap: Can't read the bitmap(/tmp/kdump_bitmap96s9V8). Success
Fix this and consider "info->max_mapnr" in the create bitmap functions.
--- makedumpfile.c | 9 +++++++++ 1 file changed, 9 insertions(+)
makedumpfile.c +9 -0
@@ -4402,6 +4402,9 @@ create_1st_bitmap(void) pfn_start =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); +if (pfn_start > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (pfn = pfn_start; pfn < pfn_end; pfn++) {set_bit_on_1st_bitmap (pfn);
@@ -7511,6 +7514,9 @@ exclude_xen3_user_domain(void) pfn =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); size = pfn_end - pfn; +if (pfn > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (j =0 ; pfn < pfn_end; pfn++, j++) { print_progress(PROGRESS_XEN_DOMAIN, j + (size * i),
@@ -7575,6 +7581,9 @@ exclude_xen4_user_domain(void) pfn =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); size = pfn_end - pfn; +if (pfn > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (j =0 ; pfn < pfn_end; pfn++, j++) { print_progress(PROGRESS_XEN_DOMAIN, j + (size * i),
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
[snip]
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
That's because the bitmap length is calculated in
prepare_bitmap_bufferusinginfo->max_mapnr, butcreate_1st_bitmap()still loops over allPT_LOADsegments, callingset_bit_on_1st_bitmap()for each PFN. The offset may easily fall beyond the bitmap size.What about the following patch. It works for me when I specify the "--non-cyclic" option.
Michael
---
[PATCH] makedumpfile: Fix bitmap create for adjustedinfo->max_mapnr
Quoted 6 lines
Quoted 6 lines
If
info->max_mapnrhas been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmap96s9V8). Success
^^^^^^^^
This looks confusing, is it an actual message ? I suppose it must be "Invalid argument" like Petr's log.
Quoted 18 lines
Quoted 5 lines
Fix this and consider "
info->max_mapnr" in the create bitmap functions.Signed-off-by Michael Holzheu <[email protected]>---
[snip]
makedumpfile.c exclude_xen4_user_domain(void) · line 7586
Quoted 7 lines
makedumpfile.c +3
-0@@ -7575,6 +7581,9 @@ exclude_xen4_user_domain(void) pfn =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); size = pfn_end - pfn; +if (pfn > info->max_mapnr) +continue ; + pfn_end =MIN (phys_end, info->max_mapnr);Hmm, probably time for weekend. Of course this should be:
Quoted 3 lines
pfn_end=MIN(pfn_end, info->max_mapnr);Here the updated patch:
I found another issue of truncating max_mapnr for Xen.
The bitmap manages MFN(machine frame number) for Xen
while __exclude_unnecessary_pages() treats PFN(guest-physical frame number). __exclude_unnecessary_pages() expects that all bits of PFNs
are mapped in the bitmap even if it was reduced by truncated max_mapnr. However, PtoM mapping isn't linear(probably...), there is no guarantee that a set of continuous PFNs is mapped
in a set of continuous MFNs.
So the actual I/O offset can exceed the bitmap size when the bitmap size is reduced.
In the first place, we shouldn't truncate max_mapnr based on dom0's mem_section since there are some domU's memories on Xen dumps. Now, I think a better way for Xen is just leaving max_mapnr as it is.
Do you agree with my view ?
Thanks
Atsushi Kumagai
Quoted 51 lines
---
[PATCH] makedumpfile: Fix bitmap create for adjustedinfo->max_mapnrIf
info->max_mapnrhas been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmap96s9V8). SuccessFix this and consider "
info->max_mapnr" in the create bitmap functions.Signed-off-by Michael Holzheu <[email protected]>---
makedumpfile.c| 9 +++++++++1 file changed, 9 insertions(+)makedumpfile.c +9
-0@@ -4402,6 +4402,9 @@ create_1st_bitmap(void) pfn_start =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); +if (pfn_start > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (pfn = pfn_start; pfn < pfn_end; pfn++) {set_bit_on_1st_bitmap (pfn);@@ -7511,6 +7514,9 @@ exclude_xen3_user_domain(void) pfn =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); size = pfn_end - pfn; +if (pfn > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (j =0 ; pfn < pfn_end; pfn++, j++) { print_progress(PROGRESS_XEN_DOMAIN, j + (size * i),@@ -7575,6 +7581,9 @@ exclude_xen4_user_domain(void) pfn =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); size = pfn_end - pfn; +if (pfn > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (j =0 ; pfn < pfn_end; pfn++, j++) { print_progress(PROGRESS_XEN_DOMAIN, j + (size * i),
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines · On Mon, 31 Mar 2014 09:48:05 +0000 Atsushi Kumagai…
[...]
In the first place, we shouldn't truncate
max_mapnrbased on dom0'smem_sectionsince there are some domU's memories on Xen dumps. Now, I think a better way for Xen is just leavingmax_mapnras it is.Do you agree with my view ?
Definitely! For Xen, info->max_mapnr gives the maximum machine PFN (ie. it corresponds to total memory installed in the machine).
The data in mem_section describes Dom0 kernel memory map (and gets initialized from info->dom0_mapnr). It may be substantially smaller than info->max_mapnr...
A "clean" solution would be to change info->max_mapnr so that it always gives the maximum physical PFN, but that doesn't fly very well in practice, because memory bitmaps and other stuff must still be sized according to the number of machine PFNs, so I would have to add special cases all around the place...
OTOH, Michal's patch is still neded to fix non-Xen non-cyclic dumps.
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
[...]
Quoted 6 lines
Quoted 6 lines
In the first place, we shouldn't truncate
max_mapnrbased on dom0'smem_sectionsince there are some domU's memories on Xen dumps. Now, I think a better way for Xen is just leavingmax_mapnras it is.
Quoted 8 lines
Quoted 1 line
Do you agree with my view ?
Definitely! For Xen,
info->max_mapnrgives the maximum machine PFN (ie. it corresponds to total memory installed in the machine).The data in
mem_sectiondescribes Dom0 kernel memory map (and gets initialized frominfo->dom0_mapnr). It may be substantially smaller thaninfo->max_mapnr...
Thanks for your confirming.
Quoted 5 lines
A "clean" solution would be to change
info->max_mapnrso that it always gives the maximum physical PFN, but that doesn't fly very well in practice, because memory bitmaps and other stuff must still be sized according to the number of machine PFNs, so I would have to add special cases all around the place...
I don't know how to capture a dump on Xen well, so do you have any idea how to produce the difference between actual memory regions and the ELF header like the s390 case ?
If it isn't possible, we don't need to change info->max_mapnr since the value calculated from the ELF header must be correct.
Quoted 1 line
OTOH, Michal's patch is still neded to fix non-Xen non-cyclic dumps.
Yes, the fix for create_1st_bitmap() is still necessary.
Michael, could you fix your patch ? We need to add the conditional check for Xen like below:
+ if (!is_xen_memory()) {
+ for (i = 0; i < info->num_mem_map; i++) {
+ if (info->mem_map_data[i].mem_map == NOT_MEMMAP_ADDR)
+ continue;
+ max_pfn = MAX(max_pfn, info->mem_map_data[i].pfn_end);
+ }
+ info->max_mapnr = MIN(info->max_mapnr, max_pfn);
+ }
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 24 lines · On Tue, 1 Apr 2014 05:06:33 +0000 Atsushi Kumagai…
[...]
Quoted 6 lines
Quoted 6 lines
In the first place, we shouldn't truncate
max_mapnrbased on dom0'smem_sectionsince there are some domU's memories on Xen dumps. Now, I think a better way for Xen is just leavingmax_mapnras it is.Quoted 8 lines
Quoted 1 line
Do you agree with my view ?
Definitely! For Xen,
info->max_mapnrgives the maximum machine PFN (ie. it corresponds to total memory installed in the machine).The data in
mem_sectiondescribes Dom0 kernel memory map (and gets initialized frominfo->dom0_mapnr). It may be substantially smaller thaninfo->max_mapnr...Thanks for your confirming.
Quoted 5 lines
A "clean" solution would be to change
info->max_mapnrso that it always gives the maximum physical PFN, but that doesn't fly very well in practice, because memory bitmaps and other stuff must still be sized according to the number of machine PFNs, so I would have to add special cases all around the place...
Quoted 3 lines
I don't know how to capture a dump on Xen well, so do you have any idea how to produce the difference between actual memory regions and the ELF header like the s390 case ?
I don't quite see what would be the Xen equivalent. Like I said in a previous mail, memory regions under Xen are sized by Dom0's max_pfn, so it makes no sense to set this value back from the memory regions.
Quoted 2 lines
If it isn't possible, we don't need to change
info->max_mapnrsince the value calculated from the ELF header must be correct.
Ah, if we're talking about the ELF headers, then the extents are determined by kexec-tools using information passed on by the Xen hypervisor on boot. If the available memory is reduced using Xen's mem= or availmem= command line parameter, then these headers are correct. AFAIK there is no mechanism to change the amount of RAM used by the hypervisor at run-time.
In short, I agree that the adjustment should be simply skipped for Xen, exactly as you proposed in your patch:
+ if (!is_xen_memory()) {
+ for (i = 0; i < info->num_mem_map; i++) {
+ if (info->mem_map_data[i].mem_map == NOT_MEMMAP_ADDR)
+ continue;
+ max_pfn = MAX(max_pfn, info->mem_map_data[i].pfn_end);
+ }
+ info->max_mapnr = MIN(info->max_mapnr, max_pfn);
+ }
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
On Tue, 1 Apr 2014 05:06:33 +0000
Atsushi Kumagai <[email protected]> wrote:
[...]
Quoted 7 lines
Quoted 1 line
OTOH, Michal's patch is still neded to fix non-Xen non-cyclic dumps.
Yes, the fix for
create_1st_bitmap()is still necessary.Michael, could you fix your patch ? We need to add the conditional check for Xen like below:
Review
Quoted 8 lines
(quoted diff) +8
-0+ if (!is_xen_memory()) { + for (i = 0; i < info->num_mem_map; i++) { + if (info->mem_map_data[i].mem_map == NOT_MEMMAP_ADDR) + continue; + max_pfn = MAX(max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr = MIN(info->max_mapnr, max_pfn); + }
Hello Atsushi and Petr,
Based on the discussion I removed the checks in exclude_xen3_user_domain() and exclude_xen4_user_domain() and added the is_xen_memory() check int get_mem_map().
Here the updated patch:
---
[PATCH] makedumpfile: Fix bitmap create for adjusted info->max_mapnr
If info->max_mapnr has been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
set_bitmap: Can't read the bitmap(/tmp/kdump_bitmapBsKAUe). Invalid argument
Fix this and consider "info->max_mapnr" in the create_1st_bitmap() function.
In addition to this, do not adjust max_mapnr for Xen dumps. For Xen info->max_mapnr gives the maximum machine PFN and the data in mem_section describes the Dom0 kernel memory map that gets initialized from info->dom0_mapnr. It may be substantially smaller than info->max_mapnr.
--- makedumpfile.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
makedumpfile.c +10 -5
@@ -2868,12 +2868,14 @@ get_mem_map(void)* than is dumped. For example when "mem=" has been used for the * dumped system. */ -for (i =0 ; i < info->num_mem_map; i++) { -if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) -continue ; - max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); +if (!is_xen_memory ()) { +for (i =0 ; i < info->num_mem_map; i++) { +if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr =MIN (info->max_mapnr, max_pfn); } - info->max_mapnr =MIN (info->max_mapnr, max_pfn);return ret; }
@@ -4402,6 +4404,9 @@ create_1st_bitmap(void) pfn_start =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); +if (pfn_start > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (pfn = pfn_start; pfn < pfn_end; pfn++) {set_bit_on_1st_bitmap (pfn);
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 23 lines
On Tue, 1 Apr 2014 05:06:33 +0000
Atsushi Kumagai <[email protected]> wrote:[...]
Quoted 7 lines
Quoted 1 line
OTOH, Michal's patch is still neded to fix non-Xen non-cyclic dumps.
Yes, the fix for
create_1st_bitmap()is still necessary.Michael, could you fix your patch ? We need to add the conditional check for Xen like below:
makedumpfile.c get_mem_map(void)
Quoted 8 lines
makedumpfile.c +8
-0+if (!is_xen_memory ()) { +for (i =0 ; i < info->num_mem_map; i++) { +if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr =MIN (info->max_mapnr, max_pfn); + }Hello Atsushi and Petr,
Quoted 5 lines
Based on the discussion I removed the checks in
exclude_xen3_user_domain()andexclude_xen4_user_domain()and added theis_xen_memory()check intget_mem_map().Here the updated patch:
Good, I'll merge this into v1.5.6.
Thanks for your work, Michael and Petr !
Atsushi Kumagai
Quoted 46 lines
---
[PATCH] makedumpfile: Fix bitmap create for adjustedinfo->max_mapnrIf
info->max_mapnrhas been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmapBsKAUe). Invalid argumentFix this and consider "
info->max_mapnr" in thecreate_1st_bitmap()function.In addition to this, do not adjust
max_mapnrfor Xen dumps. For Xeninfo->max_mapnrgives the maximum machine PFN and the data inmem_sectiondescribes the Dom0 kernel memory map that gets initialized frominfo->dom0_mapnr. It may be substantially smaller thaninfo->max_mapnr.Signed-off-by Michael Holzheu <[email protected]>---
makedumpfile.c| 15 ++++++++++-----1 file changed, 10 insertions(+), 5 deletions(-)makedumpfile.c +7
-5@@ -2868,12 +2868,14 @@ get_mem_map(void)* than is dumped. For example when "mem=" has been used for the * dumped system. */ -for (i =0 ; i < info->num_mem_map; i++) { -if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) -continue ; - max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); +if (!is_xen_memory ()) { +for (i =0 ; i < info->num_mem_map; i++) { +if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr =MIN (info->max_mapnr, max_pfn); } - info->max_mapnr =MIN (info->max_mapnr, max_pfn);return ret; }
makedumpfile.c create_1st_bitmap(void) · line 4412
Quoted 9 lines
makedumpfile.c +3
-0@@ -4402,6 +4404,9 @@ create_1st_bitmap(void) pfn_start =paddr_to_pfn (phys_start); pfn_end =paddr_to_pfn (phys_end); +if (pfn_start > info->max_mapnr) +continue ; + pfn_end =MIN (pfn_end, info->max_mapnr);for (pfn = pfn_start; pfn < pfn_end; pfn++) {set_bit_on_1st_bitmap (pfn);
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 19 lines · On Mon, 31 Mar 2014 09:48:05 +0000 Atsushi Kumagai…
[snip]
Quoted 17 lines
Quoted 17 lines
Quoted 4 lines
That's because the bitmap length is calculated in
prepare_bitmap_bufferusinginfo->max_mapnr, butcreate_1st_bitmap()still loops over allPT_LOADsegments, callingset_bit_on_1st_bitmap()for each PFN. The offset may easily fall beyond the bitmap size.What about the following patch. It works for me when I specify the "--non-cyclic" option.
Michael
---
[PATCH] makedumpfile: Fix bitmap create for adjustedinfo->max_mapnrIf
info->max_mapnrhas been adjusted, for example because the dumped system has specified the "mem=" kernel parameter, makedumpfile writes the following error messages for Xen dumps or when the "--non-cyclic" option has been specified:
Quoted 4 lines
Quoted 1 line
Quoted 1 line
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmap96s9V8). Success^^^^^^^^This looks confusing, is it an actual message ? I suppose it must be "Invalid argument" like Petr's log.
Right, I get "Invalid argument".
No idea from where I pasted "Success" here...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Fix this and consider "
info->max_mapnr" in the create bitmap functions.Signed-off-by Michael Holzheu <[email protected]>---
[snip]
Quoted 12 lines
I found another issue of truncating
max_mapnrfor Xen.The bitmap manages MFN(machine frame number) for Xen
while__exclude_unnecessary_pages()treats PFN(guest-physical frame number).__exclude_unnecessary_pages()expects that all bits of PFNs
are mapped in the bitmap even if it was reduced by truncatedmax_mapnr. However, PtoM mapping isn't linear(probably...), there is no guarantee that a set of continuous PFNs is mapped
in a set of continuous MFNs.
So the actual I/O offset can exceed the bitmap size when the bitmap size is reduced.
Quoted 6 lines
In the first place, we shouldn't truncate
max_mapnrbased on dom0'smem_sectionsince there are some domU's memories on Xen dumps. Now, I think a better way for Xen is just leavingmax_mapnras it is.Do you agree with my view ?
I don't know the Xen details so I would leave it to Petr to answer this question.
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 24 lines · On Fri, 28 Mar 2014 17:46:22 +0100 Michael Holzheu…
Quoted 2 lines · On Fri, 28 Mar 2014 12:00:47 +0100 Petr Tesarik <[email protected]> wrote:
On Thu, 27 Mar 2014 14:54:41 +0100
Michael Holzheu <[email protected]> wrote:[snip]
Quoted 16 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Here the fixed patch:
Thanks, I'll merge the fixed version into v1.5.6.
Great!
I'm sorry to spoil the party, but this patch broke Xen dumps for me. I'm getting an long series of these messages:
set_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argumentset_bitmap: Can't read thebitmap(/tmp/kdump_bitmap91pbsO). Invalid argument
...In fact, it most likely broke all non-cyclic dumps.
Quoted 7 lines
Quoted 4 lines
That's because the bitmap length is calculated in
prepare_bitmap_bufferusinginfo->max_mapnr, butcreate_1st_bitmap()still loops over allPT_LOADsegments, callingset_bit_on_1st_bitmap()for each PFN. The offset may easily fall beyond the bitmap size.What about the following patch. It works for me when I specify the "--non-cyclic" option.
I'm still getting a bunch of these:
set_bitmap: Can't read the bitmap(/tmp/kdump_bitmap7lSPrl). Invalid argument
This time they come in like this:
create_2nd_bitmap
-> exclude_free_page
-> _exclude_free_page
-> reset_bitmap_of_free_pages
-> clear_bit_on_2nd_bitmap_for_kernel
(here, physical PFN is translated to machine PFN)
-> clear_bit_on_2nd_bitmapThe resulting machine PFN is beyond the bitmap extents.
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec