makedumpfile: get_max_mapnr() from ELF header problem
Subsystems (2)
Hello Atsushi,
On s390 we have the following little problem:
We use hypervisor or stand-alone dump tools to create Linux system dumps. These tools do not know the kernel parameter line and dump the full physical memory.
We use makedumpfile to filter those dumps.
If a Linux system has specified the "mem=" parameter, the dump tools still dump the whole phypsical memory.
Unfortunately in "get_max_mapnr()" makedumpfile uses the ELF header to get the maxmimum page frame number. Since this is not the correct value in our case makedumpfile fails to filter the dump.
We get the following error on s390 with makedumpfile version 1.5.3:
makedumpfile -c -d 31 vmcore dump.kdump
cyclic buffer size has been changed: 22156083 => 22156032
Excluding unnecessary pages : [ 21 %] vtop_s390x: Address too big for the number of page table levels.
readmem: Can't convert a virtual address(8000180104670) to physical address.
readmem: type_addr: 0, addr:8000180104670, size:32768 __exclude_unnecessary_pages: Can't read the buffer of struct page.
Excluding unnecessary pages : [ 23 %] vtop_s390x: Address too big for the number of page table levels. readmem: Can't convert a
virtual address(8000180104670) to physical address. readmem: type_addr:
0, addr:8000180104670, size:327681.5.5 __exclude_unnecessary_pages:
Can't read the buffer of struct page.
Since version 1.5.4 makedumpfile seems to loop in __exclude_unnecessary_pages().
We thought about several ways to fix this problem but have not found a good solution up to now.
Do you have an idea how we could fix that?
Best Regards,
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Michael,
Quoted 4 lines
Hello Atsushi,
On s390 we have the following little problem:
Quoted 8 lines
We use hypervisor or stand-alone dump tools to create Linux system dumps. These tools do not know the kernel parameter line and dump the full physical memory.
We use makedumpfile to filter those dumps.
If a Linux system has specified the "mem=" parameter, the dump tools still dump the whole phypsical memory.
I guess this is a problem of the tools, it sounds that the tools ignore the actual memory map and just make wrong ELF headers.
How do the tools decide the range of System RAM to create ELF headers ?
At least, if the tools respect the actual memory map like /proc/vmcore, it can create correct ELF headers and makedumpfile will work normally.
Quoted 3 lines
Unfortunately in "
get_max_mapnr()" makedumpfile uses the ELF header to get the maxmimum page frame number. Since this is not the correct value in our case makedumpfile fails to filter the dump.
makedumpfile depends on the ELF file format, you know. I think you should fix the tools to create correct ELF files.
Thanks
Atsushi Kumagai
Quoted 16 lines
We get the following error on s390 with makedumpfile version 1.5.3:
makedumpfile -c -d 31 vmcore dump.kdump
cyclic buffer size has been changed: 22156083 => 22156032
Excluding unnecessary pages : [ 21 %]vtop_s390x: Address too big for the number of page table levels.
readmem: Can't convert a virtual address(8000180104670) to physical address.
readmem:type_addr: 0, addr:8000180104670, size:32768__exclude_unnecessary_pages: Can't read the buffer of struct page.
Excluding unnecessary pages : [ 23 %]vtop_s390x: Address too big for the number of page table levels. readmem:
Can't convert a
virtual address(8000180104670) to physical address. readmem:type_addr:
0, addr:8000180104670, size:327681.5.5__exclude_unnecessary_pages:
Can't read the buffer of struct page.Since version 1.5.4 makedumpfile seems to loop in
__exclude_unnecessary_pages().
Quoted 7 lines
We thought about several ways to fix this problem but have not found a good solution up to now.
Do you have an idea how we could fix that?
Best Regards,
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 10 lines · On Mon, 3 Mar 2014 03:11:23 +0000 Atsushi Kumagai…
Hello Michael,
Quoted 8 lines
Hello Atsushi,
On s390 we have the following little problem:
We use hypervisor or stand-alone dump tools to create Linux system dumps. These tools do not know the kernel parameter line and dump the full physical memory.
Quoted 8 lines
Quoted 4 lines
We use makedumpfile to filter those dumps.
If a Linux system has specified the "mem=" parameter, the dump tools still dump the whole phypsical memory.
I guess this is a problem of the tools, it sounds that the tools ignore the actual memory map and just make wrong ELF headers.
How do the tools decide the range of System RAM to create ELF headers ?
The tools do a physical memory detection and that defines the range of memory to be dumped and also defines the memory chunks for the ELF header.
And I think we are not the only ones that have this problem. For example, the KVM virsh dump probably also has that problem.
Quoted 3 lines
At least, if the tools respect the actual memory map like
/proc/vmcore, it can create correct ELF headers and makedumpfile will work normally.
As I said, the tools do not know the Linux memory map. They only know the physical available memory.
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 18 lines
Quoted 15 lines · On Mon, 3 Mar 2014 03:11:23 +0000 Atsushi Kumagai…
Hello Michael,
Quoted 12 lines
Hello Atsushi,
On s390 we have the following little problem:
We use hypervisor or stand-alone dump tools to create Linux system dumps. These tools do not know the kernel parameter line and dump the full physical memory.
We use makedumpfile to filter those dumps.
If a Linux system has specified the "mem=" parameter, the dump tools still dump the whole phypsical memory.
Quoted 7 lines
Quoted 3 lines
I guess this is a problem of the tools, it sounds that the tools ignore the actual memory map and just make wrong ELF headers.
How do the tools decide the range of System RAM to create ELF headers ?The tools do a physical memory detection and that defines the range of memory to be dumped and also defines the memory chunks for the ELF header.
makedumpfile is designed for kdump, this means it relies on dependable ELF headers. If we support such an incorrect ELF header, makedumpfile has to get the actual memory map from vmcore (but I have no ideas how to do it now) and re-calculate all PT_LOAD regions with it. It sounds too much work for irregular case, I don't plan to take care of it now.
Quoted 2 lines
And I think we are not the only ones that have this problem. For example, the KVM virsh dump probably also has that problem.
virsh dump seems to have the same issue as you said, but I suppose qemu developers don't worry about that because they are developing an original way to dump guest's memory in kdump-compressed format as "dump-guest-memory" command. It seems that they know such case is out of the scope of makedumpfile.
Thanks
Atsushi Kumagai
Quoted 6 lines
Quoted 2 lines
At least, if the tools respect the actual memory map like
/proc/vmcore, it can create correct ELF headers and makedumpfile will work normally.As I said, the tools do not know the Linux memory map. They only know the physical available memory.
Quoted 7 lines
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 11 lines · On Tue, 11 Mar 2014 06:22:41 +0000 Atsushi Kumagai…
Quoted 5 lines
On Mon, 3 Mar 2014 03:11:23 +0000
Atsushi Kumagai <[email protected]> wrote:
The tools do a physical memory detection and that defines the range of memory to be dumped and also defines the memory chunks for the ELF header.makedumpfile is designed for kdump, this means it relies on dependable ELF headers. If we support such an incorrect ELF header, makedumpfile has to get the actual memory map from vmcore (but I have no ideas how to do it now) and re-calculate all
PT_LOADregions with it. It sounds too much work for irregular case, I don't plan to take care of it now.
Ok, fair.
Quoted 7 lines
Quoted 2 lines
And I think we are not the only ones that have this problem. For example, the KVM virsh dump probably also has that problem.
virsh dump seems to have the same issue as you said, but I suppose qemu developers don't worry about that because they are developing an original way to dump guest's memory in kdump-compressed format as "dump-guest-memory" command. It seems that they know such case is out of the scope of makedumpfile.
Even if they create a kdump-compressed format dump, they (probably) do not filter while dumping. Therefore for large dumps post-processing with makedumpfile could still make sense, e.g. for transfering the dumps.
Because qemu is not aware of kernel parameters this will also fail when "mem=" has been used.
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 32 lines · Michael Holzheu
Hello Atsushi,
On s390 we have the following little problem:
We use hypervisor or stand-alone dump tools to create Linux system dumps. These tools do not know the kernel parameter line and dump the full physical memory.
We use makedumpfile to filter those dumps.
If a Linux system has specified the "mem=" parameter, the dump tools still dump the whole phypsical memory.
Unfortunately in "
get_max_mapnr()" makedumpfile uses the ELF header to get the maxmimum page frame number. Since this is not the correct value in our case makedumpfile fails to filter the dump.We get the following error on s390 with makedumpfile version 1.5.3:
makedumpfile -c -d 31 vmcore dump.kdump
cyclic buffer size has been changed: 22156083 => 22156032
Excluding unnecessary pages : [ 21 %]vtop_s390x: Address too big for the number of page table levels.
readmem: Can't convert a virtual address(8000180104670) to physical address.
readmem:type_addr: 0, addr:8000180104670, size:32768__exclude_unnecessary_pages: Can't read the buffer of struct page.
Excluding unnecessary pages : [ 23 %]vtop_s390x: Address too big for the number of page table levels. readmem: Can't convert a
virtual address(8000180104670) to physical address. readmem:type_addr:
0, addr:8000180104670, size:327681.5.5__exclude_unnecessary_pages:
Can't read the buffer of struct page.Since version 1.5.4 makedumpfile seems to loop in
__exclude_unnecessary_pages().
Quoted 8 lines
We thought about several ways to fix this problem but have not found a good solution up to now.
Do you have an idea how we could fix that?
Best Regards,
Michael
At least on x86, makedumpfile appears to work well for dumps generated by sadump and virsh dump. In particular, virsh dump --memory-only generates dump in ELF, whose PT_LOAD entries are generated from RAM list managed by qemu, not managed by kernel.
Looking into source code a little, max_mapnr is used only for calculating a size of two bitmaps. I guess there's any s390-specific issue.
Signature · 8 lines
Thanks. HATAYAMA, Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 43 lines
Quoted 19 lines · Michael Holzheu
Hello Atsushi,
On s390 we have the following little problem:
We use hypervisor or stand-alone dump tools to create Linux system dumps. These tools do not know the kernel parameter line and dump the full physical memory.
We use makedumpfile to filter those dumps.
If a Linux system has specified the "mem=" parameter, the dump tools still dump the whole phypsical memory.
Unfortunately in "
get_max_mapnr()" makedumpfile uses the ELF header to get the maxmimum page frame number. Since this is not the correct value in our case makedumpfile fails to filter the dump.We get the following error on s390 with makedumpfile version 1.5.3:
Quoted 7 lines
makedumpfile -c -d 31 vmcore dump.kdump
cyclic buffer size has been changed: 22156083 => 22156032
Excluding unnecessary pages : [ 21 %]vtop_s390x: Address too big for the number of page table levels.
readmem: Can't convert a virtual address(8000180104670) to physical address.
readmem:type_addr: 0, addr:8000180104670, size:32768__exclude_unnecessary_pages: Can't read the buffer of struct page.
Excluding unnecessary pages : [ 23 %]vtop_s390x: Address too big for the number of page table levels. readmem:Can't convert a
Quoted 6 lines
virtual address(8000180104670) to physical address. readmem:
type_addr:
0, addr:8000180104670, size:327681.5.5__exclude_unnecessary_pages:
Can't read the buffer of struct page.Since version 1.5.4 makedumpfile seems to loop in
__exclude_unnecessary_pages().Quoted 8 lines
We thought about several ways to fix this problem but have not found a good solution up to now.
Do you have an idea how we could fix that?
Best Regards,
Michael
Quoted 6 lines
At least on x86, makedumpfile appears to work well for dumps generated by sadump and virsh dump. In particular, virsh dump --memory-only generates dump in ELF, whose
PT_LOADentries are generated from RAM list managed by qemu, not managed by kernel.Looking into source code a little,
max_mapnris used only for calculating a size of two bitmaps. I guess there's any s390-specific issue.
On second thought, Michael's log looks strange.
Quoted 4 lines
Quoted 4 lines
Excluding unnecessary pages : [ 21 %]
vtop_s390x: Address too big for the number of page table levels.
readmem: Can't convert a virtual address(8000180104670) to physical address.
readmem:type_addr: 0, addr:8000180104670, size:32768__exclude_unnecessary_pages: Can't read the buffer of struct page.
This message was shown during translating the virtual address of mem_map to the physical address:
__exclude_unnecessary_pages():
if (!readmem(VADDR, mem_map,
page_cache + (index_pg * SIZE(page)),
SIZE(page) * pfn_mm)) {
ERRMSG("Can't read the buffer of struct page.\n");
return FALSE;
}However, this should succeed even if mem= was specified because the corresponding page table must exist in the memory image since it was used by kernel actually. The address translation logic may have an issue.
Thanks
Atsushi Kumagai
Quoted 9 lines
--
Thanks.
HATAYAMA, Daisuke_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 2 lines · On Wed, 12 Mar 2014 06:01:47 +0000 Atsushi Kumagai…
Quoted 2 lines
Quoted 1 line · Michael Holzheu
Hello Atsushi,
[snip]
Quoted 22 lines
Quoted 2 lines
Looking into source code a little,
max_mapnris used only for calculating a size of two bitmaps. I guess there's any s390-specific issue.On second thought, Michael's log looks strange.
Quoted 4 lines
Quoted 4 lines
Excluding unnecessary pages : [ 21 %]
vtop_s390x: Address too big for the number of page table levels.
readmem: Can't convert a virtual address(8000180104670) to physical address.
readmem:type_addr: 0, addr:8000180104670, size:32768__exclude_unnecessary_pages: Can't read the buffer of struct page.This message was shown during translating the virtual address of
mem_mapto the physical address:
__exclude_unnecessary_pages():if (!readmem(VADDR, mem_map, page_cache + (index_pg * SIZE(page)), SIZE(page) * pfn_mm)) { ERRMSG("Can't read the buffer of struct page.\n"); return FALSE; }
Quoted 3 lines
However, this should succeed even if mem= was specified because the corresponding page table must exist in the memory image since it was used by kernel actually. The address translation logic may have an issue.
To be honest I don't really understand what happens when the error occurs.
My test is a 1 TiB dump of a Linux system that has set mem=1G.
With makedumpfile 1.5.3 I see the following stack backtrace:
(gdb) bt
#0 vtop_s390x (vaddr=2251803034918936) at ./arch/s390x.c:236 #1 0x000000008001de44 in vaddr_to_paddr_s390x (vaddr=2251803034918936)
at ./arch/s390x.c:300
#2 0x000000008001fb50 in readmem (type_addr=0, addr=2251803034918936,
bufptr=0x3ffffff6cf0, size=32768) at makedumpfile.c:349
#3 0x0000000080034cf2 in __exclude_unnecessary_pages (
mem_map=2251803034918936, pfn_start=16777216, pfn_end=16842752)
at makedumpfile.c:4189
#4 0x0000000080035716 in exclude_unnecessary_pages_cyclic ()
at makedumpfile.c:4349
#5 0x00000000800358e4 in update_cyclic_region (pfn=0) at makedumpfile.c:4380
#6 0x00000000800384e0 in get_num_dumpable_cyclic () at makedumpfile.c:5060
#7 0x0000000080036850 in create_dump_bitmap () at makedumpfile.c:4585
#8 0x00000000800429c8 in create_dumpfile () at makedumpfile.c:7533
#9 0x00000000800490fc in main (argc=5, argv=0x3fffffff3d8)
at makedumpfile.c:8651Looks like makdumpfile wants to read a virtual address 2251803034918936 (hex 0x80000C0002018) which can't be resolved by the three level kernel page table (max is 4 TiB here).
In the __exclude_unnecessary_pages() function the variables have the following values:
(gdb) print pfn_end
$1 = 16842752
(gdb) print pfn
$2 = 16777216
(gdb) print pfn_start
$3 = 16777216
(gdb) print mem_map
$4 = 2251803034918936
I would appreciate any hints!
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 38 lines
My test is a 1 TiB dump of a Linux system that has set mem=1G.
With makedumpfile 1.5.3 I see the following stack backtrace:
(gdb) bt
#0vtop_s390x(vaddr=2251803034918936) at./arch/s390x.c:236#10x000000008001de44invaddr_to_paddr_s390x(vaddr=2251803034918936)
at./arch/s390x.c:300
#20x000000008001fb50in readmem (type_addr=0, addr=2251803034918936,
bufptr=0x3ffffff6cf0, size=32768) at makedumpfile.c:349
#30x0000000080034cf2in__exclude_unnecessary_pages(
mem_map=2251803034918936, pfn_start=16777216, pfn_end=16842752) at makedumpfile.c:4189
#40x0000000080035716inexclude_unnecessary_pages_cyclic()
at makedumpfile.c:4349#5 0x00000000800358e4 in update_cyclic_region (pfn=0) at makedumpfile.c:4380 #6 0x00000000800384e0 in get_num_dumpable_cyclic () at makedumpfile.c:5060 #7 0x0000000080036850 in create_dump_bitmap () at makedumpfile.c:4585 #8 0x00000000800429c8 in create_dumpfile () at makedumpfile.c:7533 #9 0x00000000800490fc in main (argc=5, argv=0x3fffffff3d8)at makedumpfile.c:8651
Looks like makdumpfile wants to read a virtual address 2251803034918936 (hex
0x80000C0002018) which can't be resolved by the three level kernel page table (max is 4 TiB here).In the
__exclude_unnecessary_pages()function the variables have the following values:(gdb) print
pfn_end
$1 = 16842752
(gdb) print pfn
$2 = 16777216
(gdb) printpfn_start
$3 = 16777216
(gdb) printmem_map
$4 = 2251803034918936
Quoted 1 line
I would appreciate any hints!
What were these values when you didn't specify mem=1G ? The mem_map information can be shown with -D option like below:
$ makedumpfile -D -cd31 vmcore dumpfile
... Memory type : SPARSEMEM
mem_map (0)
mem_map : f6d7f000
pfn_start : 0
pfn_end : 20000
mem_map (1)
mem_map : f697f000
pfn_start : 20000
pfn_end : 40000
...Could you show me both log of mem=1G and log of no mem= option ? The difference may help us.
BTW, is the command below the actual command line you used ?
makedumpfile -c -d 31 vmcore dump.kdump
If yes, the dumpfile must have VMOCREINFO since you didn't specify neither -x nor -i option, i.e. the s390 dump tool generates a PT_NOTE entry as VMCOREINFO, really ?
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Atsushi,
I debugged my problem a bit further and tried to implement a function that gets the maximum page frame number from the Linux kernel memory management structures.
I am no memory management expert, so the following patch probably is not complete, but at least for my setup it worked.
Michael
--- makedumpfile.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)
makedumpfile.c +45 -0
@@ -2029,6 +2029,48 @@ pgdat4:return SYMBOL (contig_page_data); } +int +get_max_pfn (void ) +{ +unsigned long pgdat, node_start_pfn, node_spanned_pages, max_pfn =0 ; +int num_nodes, node; + +if ((node =next_online_node (0 )) <0 ) { +ERRMSG ("Can't get next online node.\n" ); +return FALSE; + } +if (!(pgdat =next_online_pgdat (node))) { +ERRMSG ("Can't get pgdat list.\n" ); +return FALSE; + } +for (num_nodes =1 ; num_nodes <= vt.numnodes; num_nodes++) { +if (!readmem (VADDR , pgdat +OFFSET (pglist_data.node_start_pfn), + &node_start_pfn,sizeof node_start_pfn)) { +ERRMSG ("Can't get node_start_pfn.\n" ); +return FALSE; + } +if (!readmem (VADDR , + pgdat +OFFSET (pglist_data.node_spanned_pages), + &node_spanned_pages,sizeof node_spanned_pages)) { +ERRMSG ("Can't get node_spanned_pages.\n" ); +return FALSE; + } + max_pfn =MAX (max_pfn, (node_start_pfn + node_spanned_pages)); +if (num_nodes < vt.numnodes) { +if ((node =next_online_node (node +1 )) <0 ) { +ERRMSG ("Can't get next online node.\n" ); +return FALSE; + }else if (!(pgdat =next_online_pgdat (node))) { +ERRMSG ("Can't determine pgdat list (node %d).\n" , + node); +return FALSE; + } + } + } + info->max_mapnr = max_pfn; +return TRUE; +} +void dump_mem_map (unsigned long long pfn_start,unsigned long long pfn_end,unsigned long mem_map,int num_mm)
@@ -2853,6 +2908,9 @@ out:if (!get_numnodes ())return FALSE; +if (!get_max_pfn ()) +return FALSE; +if (!get_mem_map ())return FALSE;
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines
Hello Atsushi,
I debugged my problem a bit further and tried to implement a function that gets the maximum page frame number from the Linux kernel memory management structures.
I am no memory management expert, so the following patch probably is not complete, but at least for my setup it worked.
The patch looks good for your case, but I don't think it's a proper approach for this problem.
Now, I think this is a problem of get_mm_sparsemem() in makedumpfile. To say in more detail, the problem is "wrong calculating the address of unused mem_map".
Looking at the log you sent, some addresses of mem_map corresponding to unused pages look invalid like below:
mem_map (256)
mem_map : 80000c0002018 pfn_start : 1000000 pfn_end : 1010000 mem_map (257) mem_map : 800001840400000 pfn_start : 1010000 pfn_end : 1020000 ... mem_map (544) mem_map : a82400012f14fffc pfn_start : 2200000 pfn_end : 2210000
...(and more)
However, makedumpfile should calculate such unused mem_map addresses as 0(NOT_MEMMAP_ADDR). Actually it works as expected at least in my environment(x86_64):
...mem_map (16)
mem_map : 0 pfn_start : 80000 pfn_end : 88000 mem_map (17) mem_map : 0 pfn_start : 88000 pfn_end : 90000
...
makedumpfile get the address from mem_section.section_mem_map, it will be initialized with zero:
[CONFIG_SPARSEMEM_EXTREAM]
paging_init()
sparse_memory_present_with_active_regions()
memory_present()
sparse_index_init()
sparse_index_alloc() // allocate mem_section with kzalloc()
makedumpfile assumes the value of unused mem_section will remain as 0, but I suspect this assumption may be broken in your environment.
Moreover, if it's true, the problem will happen when memmap= parameter is specified even if max_mapnr is correct.
This is because the unused pages created by memmap= will be placed below max_mapnr and its mem_map will be calculated wrongly by get_mm_sparsemem().
I'll continue to investigate to find better solution for this problem, any comments are helpful.
Thanks
Atsushi Kumagai
Quoted 68 lines
Michael
---makedumpfile.c| 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++1 file changed, 58 insertions(+)makedumpfile.c +42
-0@@ -2029,6 +2029,48 @@ pgdat4:return SYMBOL (contig_page_data); } +int +get_max_pfn (void ) +{ +unsigned long pgdat, node_start_pfn, node_spanned_pages, max_pfn =0 ; +int num_nodes, node; + +if ((node =next_online_node (0 )) <0 ) { +ERRMSG ("Can't get next online node.\n" ); +return FALSE; + } +if (!(pgdat =next_online_pgdat (node))) { +ERRMSG ("Can't get pgdat list.\n" ); +return FALSE; + } +for (num_nodes =1 ; num_nodes <= vt.numnodes; num_nodes++) { +if (!readmem (VADDR , pgdat +OFFSET (pglist_data.node_start_pfn), + &node_start_pfn,sizeof node_start_pfn)) { +ERRMSG ("Can't get node_start_pfn.\n" ); +return FALSE; + } +if (!readmem (VADDR , + pgdat +OFFSET (pglist_data.node_spanned_pages), + &node_spanned_pages,sizeof node_spanned_pages)) { +ERRMSG ("Can't get node_spanned_pages.\n" ); +return FALSE; + } + max_pfn =MAX (max_pfn, (node_start_pfn + node_spanned_pages)); +if (num_nodes < vt.numnodes) { +if ((node =next_online_node (node +1 )) <0 ) { +ERRMSG ("Can't get next online node.\n" ); +return FALSE; + }else if (!(pgdat =next_online_pgdat (node))) { +ERRMSG ("Can't determine pgdat list (node %d).\n" , + node); +return FALSE; + } + } + } + info->max_mapnr = max_pfn; +return TRUE; +} +void
dump_mem_map(unsigned long longpfn_start,
unsigned long longpfn_end, unsigned longmem_map, intnum_mm)makedumpfile.c +3
-0@@ -2853,6 +2908,9 @@ out:if (!get_numnodes ())return FALSE; +if (!get_max_pfn ()) +return FALSE; +if (!get_mem_map ())return FALSE;
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 60 lines · On Wed, 19 Mar 2014 07:14:25 +0000 Atsushi Kumagai…
Quoted 8 lines
Hello Atsushi,
I debugged my problem a bit further and tried to implement a function that gets the maximum page frame number from the Linux kernel memory management structures.
I am no memory management expert, so the following patch probably is not complete, but at least for my setup it worked.
The patch looks good for your case, but I don't think it's a proper approach for this problem.
Now, I think this is a problem of
get_mm_sparsemem()in makedumpfile. To say in more detail, the problem is "wrong calculating the address of unusedmem_map".Looking at the log you sent, some addresses of
mem_mapcorresponding to unused pages look invalid like below:
mem_map(256)mem_map : 80000c0002018 pfn_start : 1000000 pfn_end : 1010000 mem_map (257) mem_map : 800001840400000 pfn_start : 1010000 pfn_end : 1020000 ... mem_map (544) mem_map : a82400012f14fffc pfn_start : 2200000 pfn_end : 2210000...(and more)
However, makedumpfile should calculate such unused
mem_mapaddresses as 0(NOT_MEMMAP_ADDR). Actually it works as expected at least in myenvironment(x86_64):...
mem_map(16)mem_map : 0 pfn_start : 80000 pfn_end : 88000 mem_map (17) mem_map : 0 pfn_start : 88000 pfn_end : 90000...
makedumpfile get the address from mem_section.section_mem_map, it will be initialized with zero:
[
CONFIG_SPARSEMEM_EXTREAM]
paging_init()
sparse_memory_present_with_active_regions()
memory_present()
sparse_index_init()
sparse_index_alloc()// allocatemem_sectionwithkzalloc()
Quoted 3 lines
makedumpfile assumes the value of unused
mem_sectionwill remain as 0, but I suspect this assumption may be broken in your environment.
Hello Atshushi,
I noticed that my last patch was not complete. It only checked the mem_section[] array for zero entries. But as you noticed, we also have to check the section array that we get from the mem_section entries.
So I updated the patch.
Michael
--- makedumpfile.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
makedumpfile.c +17 -5
@@ -2690,11 +2690,14 @@ nr_to_section(unsigned long nr, unsigned {unsigned long addr; -if (is_sparsemem_extreme ()) +if (is_sparsemem_extreme ()) { +if (mem_sec[SECTION_NR_TO_ROOT (nr)] ==0 ) +return NOT_KV_ADDR ; addr = mem_sec[SECTION_NR_TO_ROOT (nr)] + (nr &SECTION_ROOT_MASK ()) *SIZE (mem_section); -else + }else { addr =SYMBOL (mem_section) + (nr *SIZE (mem_section)); + }if (!is_kvaddr (addr))return NOT_KV_ADDR ;
@@ -2778,10 +2781,19 @@ get_mm_sparsemem(void) }for (section_nr =0 ; section_nr < num_section; section_nr++) { section =nr_to_section (section_nr, mem_sec); - mem_map =section_mem_map_addr (section); - mem_map =sparse_decode_mem_map (mem_map, section_nr); -if (!is_kvaddr (mem_map)) +if (section ==NOT_KV_ADDR ) { mem_map =NOT_MEMMAP_ADDR ; + }else { + mem_map =section_mem_map_addr (section); +if (mem_map ==0 ) { + mem_map =NOT_MEMMAP_ADDR ; + }else { + mem_map =sparse_decode_mem_map (mem_map, + section_nr); +if (!is_kvaddr (mem_map)) + mem_map =NOT_MEMMAP_ADDR ; + } + } pfn_start = section_nr *PAGES_PER_SECTION (); pfn_end = pfn_start +PAGES_PER_SECTION ();if (info->max_mapnr < pfn_end)
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 11 lines · On Wed, 19 Mar 2014 07:14:25 +0000 Atsushi Kumagai…
Quoted 8 lines
Hello Atsushi,
I debugged my problem a bit further and tried to implement a function that gets the maximum page frame number from the Linux kernel memory management structures.
I am no memory management expert, so the following patch probably is not complete, but at least for my setup it worked.
The patch looks good for your case, but I don't think it's a proper approach for this problem.
Hello Atsushi,
If you don't like that solution, what about using the mem_map_data[] array of makedumpfile to adjust "max_mapnr"?
The patch below also works fine for my dump.
Michael
--- makedumpfile.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
makedumpfile.c +13 -1
@@ -2829,7 +2829,8 @@ get_mem_map_without_mm(void)int get_mem_map (void ) { -int ret; +unsigned long max_pfn =0 ; +int ret, i;switch (get_mem_type ()) {case SPARSEMEM :
@@ -2861,6 +2862,17 @@ get_mem_map(void) ret = FALSE;break ; } +/* +* Adjust "max_mapnr" for the case that Linux uses less memory +* than is dumped. For example when "mem=" has been used for the +* dumped system. +*/ +for (i =0 ; i < info->num_mem_map; i++) { +if (info->mem_map_data[i].mem_map ==NOT_MEMMAP_ADDR ) +continue ; + max_pfn =MAX (max_pfn, info->mem_map_data[i].pfn_end); + } + info->max_mapnr = max_pfn;return ret; }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
A message that is not in the archive (sent off-list or not yet received): 20140319180903.2c6e2b72@holzheu
Quoted 54 lines
Quoted 48 lines
Now, I think this is a problem of
get_mm_sparsemem()in makedumpfile. To say in more detail, the problem is "wrong calculating the address of unusedmem_map".Looking at the log you sent, some addresses of
mem_mapcorresponding to unused pages look invalid like below:
mem_map(256)mem_map : 80000c0002018 pfn_start : 1000000 pfn_end : 1010000 mem_map (257) mem_map : 800001840400000 pfn_start : 1010000 pfn_end : 1020000 ... mem_map (544) mem_map : a82400012f14fffc pfn_start : 2200000 pfn_end : 2210000...(and more)
However, makedumpfile should calculate such unused
mem_mapaddresses as 0(NOT_MEMMAP_ADDR). Actually it works as expected at least in myenvironment(x86_64):...
mem_map(16)mem_map : 0 pfn_start : 80000 pfn_end : 88000 mem_map (17) mem_map : 0 pfn_start : 88000 pfn_end : 90000...
makedumpfile get the address from mem_section.section_mem_map, it will be initialized with zero:
[
CONFIG_SPARSEMEM_EXTREAM]
paging_init()
sparse_memory_present_with_active_regions()
memory_present()
sparse_index_init()
sparse_index_alloc()// allocatemem_sectionwithkzalloc()Quoted 2 lines
makedumpfile assumes the value of unused
mem_sectionwill remain as 0, but I suspect this assumption may be broken in your environment.No, I think your assumption is true also for my environment. For my dump the "
mem_section" array is zero except for the first entry.
Quoted 8 lines
crash> print/x
mem_section
$1 = {0x2fe6f800,0x0,0x0,0x0,0x0,0x0,0x0,0x0, ...}But it looks like
get_mm_sparsemem()does not check for zero. Thenr_to_section()function just returns an invalid address (something between 0 and 4096) for section in case we get zero from the "mem_section" entry. This is address is then used for calculating "mem_map":
In other architectures, the check by is_kaddr() avoids to read invalid address, but it doesn't do anything in the case of s390 due to the its memory management mechanism:
s390x: Fix KVBASE to correct value for s390x architecture.
http://lists.infradead.org/pipermail/kexec/2011-March/004930.html
Finally I've understood the cause of this issue completely, thanks for your report.
Quoted 10 lines
mem_map=section_mem_map_addr(section);mem_map=sparse_decode_mem_map(mem_map, section_nr);With the patch below I could use makedumpfile (1.5.3) successfully on the 1TB dump with mem=1G. I attached the -D output that is created by makedumpfile with the patch.
But compared to my first patch it takes much longer and the resulting dump is bigger (version 1.5.3):
Quoted 8 lines
| Dump time | Dump size
-------------+-------------+-----------First patch | 10 sec | 124 MB Second patch | 87 minutes | 6348 MBNo idea why the dump is bigger with the second patch. I think the time is consumed in
write_kdump_pages_cyclic()by checking for zero pages for the whole range:
I suppose this difference was resolved with the v2 of the second patch, right?
Quoted 21 lines
5970 for (pfn =
start_pfn; pfn <end_pfn; pfn++) {
(gdb) n
5972 if ((num_dumped% per) == 0)
(gdb) n
5978 if (!is_dumpable_cyclic(info->partial_bitmap2, pfn))
(gdb) n
5981 num_dumped++;
(gdb) n
5983 if (!read_pfn(pfn, buf))
(gdb) n
5989 if ((info->dump_level&DL_EXCLUDE_ZERO)
(gdb) n
5990 &&is_zero_page(buf, info->page_size)) {
(gdb) n
5991 if (!write_cache(cd_header,pd_zero,sizeof(page_desc_t)))
(gdb) n
5993 pfn_zero++;
(gdb) n
5994 continue;
Quoted 5 lines
(gdb) print
end_pfn
$3 = 268435456So the first patch would be better for my scenario. What in particular are your concerns with that patch?
I think the v2 second patch is a reasonable patch to fix the bug of get_mm_sparsemem().
Additionally, the latest patch you posted to adjust max_mapnr (which using mem_map_data[]) is acceptable instead of the first patch.
So could you re-post the two as a formal patch set?
I mean patch descriptions and your signature are needed.
Thanks
Atsushi Kumagai
Quoted 34 lines
Michael
The following patch adds the zero check for "
mem_section" entries
---makedumpfile.c| 17 ++++++++++++-----1 file changed, 12 insertions(+), 5 deletions(-)makedumpfile.c +6
-5@@ -2402,11 +2402,14 @@ nr_to_section(unsigned long nr, unsigned {unsigned long addr; -if (is_sparsemem_extreme ()) +if (is_sparsemem_extreme ()) { +if (mem_sec[SECTION_NR_TO_ROOT (nr)] ==0 ) +return NOT_KV_ADDR ; addr = mem_sec[SECTION_NR_TO_ROOT (nr)] + (nr &SECTION_ROOT_MASK ()) *SIZE (mem_section); -else + }else { addr =SYMBOL (mem_section) + (nr *SIZE (mem_section)); + }if (!is_kvaddr (addr))return NOT_KV_ADDR ;@@ -2490,10 +2493,14 @@ get_mm_sparsemem(void) }for (section_nr =0 ; section_nr < num_section; section_nr++) { section =nr_to_section (section_nr, mem_sec); - mem_map =section_mem_map_addr (section); - mem_map =sparse_decode_mem_map (mem_map, section_nr); -if (!is_kvaddr (mem_map)) +if (section ==NOT_KV_ADDR ) {
makedumpfile.c get_mm_sparsemem(void) · line 2506
Quoted 10 lines
makedumpfile.c +6
-0mem_map =NOT_MEMMAP_ADDR ; + }else { + mem_map =section_mem_map_addr (section); + mem_map =sparse_decode_mem_map (mem_map, section_nr); +if (!is_kvaddr (mem_map)) + mem_map =NOT_MEMMAP_ADDR ; + } pfn_start = section_nr *PAGES_PER_SECTION (); pfn_end = pfn_start +PAGES_PER_SECTION ();if (info->max_mapnr < pfn_end)
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
On Tue, 25 Mar 2014 01:14:21 +0000
Atsushi Kumagai <[email protected]> wrote:
[snip]
Quoted 6 lines
Quoted 5 lines
But it looks like
get_mm_sparsemem()does not check for zero. Thenr_to_section()function just returns an invalid address (something between 0 and 4096) for section in case we get zero from the "mem_section" entry. This is address is then used for calculating "mem_map":
Quoted 6 lines
In other architectures, the check by
is_kaddr()avoids to read invalid address, but it doesn't do anything in the case of s390 due to the its memory management mechanism:s390x: Fix KVBASE to correct value for s390x architecture.
http://lists.infradead.org/pipermail/kexec/2011-March/004930.html
Right, for s390 the zero page is valid.
Quoted 18 lines
Finally I've understood the cause of this issue completely, thanks for your report.
Quoted 15 lines
mem_map=section_mem_map_addr(section);mem_map=sparse_decode_mem_map(mem_map, section_nr);With the patch below I could use makedumpfile (1.5.3) successfully on the 1TB dump with mem=1G. I attached the -D output that is created by makedumpfile with the patch.
But compared to my first patch it takes much longer and the resulting dump is bigger (version 1.5.3):
| Dump time | Dump size
-------------+-------------+-----------First patch | 10 sec | 124 MB Second patch | 87 minutes | 6348 MB
Quoted 6 lines
Quoted 3 lines
No idea why the dump is bigger with the second patch. I think the time is consumed in
write_kdump_pages_cyclic()by checking for zero pages for the whole range:I suppose this difference was resolved with the v2 of the second patch, right?
Right, with the last patch the dump time and size were ok.
[snip]
Quoted 10 lines
Quoted 2 lines
So the first patch would be better for my scenario. What in particular are your concerns with that patch?
I think the v2 second patch is a reasonable patch to fix the bug of
get_mm_sparsemem().
Additionally, the latest patch you posted to adjustmax_mapnr(which using mem_map_data[]) is acceptable instead of the first patch.
So could you re-post the two as a formal patch set?
I mean patch descriptions and your signature are needed.
Ok great! I will resend the patches.
Michael
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec