makedumpfile: readpage_elf: handle 0-pages not stored in the ELF file
Subsystems (2)
PATCH 1/1 makedumpfile: readpage_elf: handle 0-pages not stored in the ELF file
Handle pages filled with zeros that are not stored in the ELF file directly, but have addresses between p_filesz and p_memsz of a segment.
This allows makedumpfile to dump-dmesg from a previously makedumpfile-ed vmcore where all 0-pages were excluded (dump_level & 1) for example.
--- elf_info.c | 22 ++++++++++++++++++++++ elf_info.h | 1 + makedumpfile.c | 16 ++++++++++++++++ 3 files changed, 39 insertions(+)
elf_info.c +22 -0
@@ -42,6 +42,7 @@ struct pt_load_segment {unsigned long long phys_end;unsigned long long virt_start;unsigned long long virt_end; +size_t mem_size; };static int nr_cpus;/* number of cpu */
@@ -166,6 +167,7 @@ dump_Elf_load(Elf64_Phdr *prog, int num_load) pls->virt_start = prog->p_vaddr; pls->virt_end = pls->virt_start + prog->p_filesz; pls->file_offset = prog->p_offset; + pls->mem_size = prog->p_memsz;DEBUG_MSG ("LOAD (%d)\n" , num_load);DEBUG_MSG (" phys_start : %llx\n" , pls->phys_start);
@@ -584,6 +586,26 @@ offset_to_pt_load_end(off_t offset) }/* +* Return true if a page is in the zone between p_filesz and p_memsz of a +* segment, which is a page filled with zero and not stored in the ELF file. +*/ +int +paddr_is_null (unsigned long long paddr) +{ +int i; +struct pt_load_segment *pls; + +for (i =0 ; i < num_pt_loads; i++) { + pls = &pt_loads[i]; +if ((paddr >= pls->phys_start) + && (paddr + info->page_size + <= pls->phys_start + pls->mem_size)) +return TRUE; + } +return FALSE; +} + +/* * Judge whether the page is fractional or not. */ int
elf_info.h +1 -0
@@ -36,6 +36,7 @@ unsigned long long page_head_to_phys_start(unsigned long long head_paddr);unsigned long long page_head_to_phys_end (unsigned long long head_paddr);off_t offset_to_pt_load_start (off_t offset);off_t offset_to_pt_load_end (off_t offset); +int paddr_is_null (unsigned long long paddr);unsigned long long vaddr_to_paddr_general (unsigned long long vaddr);off_t vaddr_to_offset_slow (int fd,char *filename,unsigned long long vaddr);unsigned long long get_max_paddr (void );
makedumpfile.c +16 -0
@@ -654,6 +654,14 @@ readpage_elf(unsigned long long paddr, void *bufptr) phys_end = paddr + info->page_size;/* +* Check the 0-filled pages that are not in the ELF file. +*/ +if (!offset1 && !offset2 &&paddr_is_null (paddr)) { +memset (bufptr,0 , info->page_size); +return TRUE; + } + +/* * Check the case phys_start isn't aligned by page size like below: * * phys_start
@@ -728,6 +736,14 @@ readpage_elf_parallel(int fd_memory, unsigned long long paddr, void *bufptr) phys_end = paddr + info->page_size;/* +* Check the 0-filled pages that are not in the ELF file. +*/ +if (!offset1 && !offset2 &&paddr_is_null (paddr)) { +memset (bufptr,0 , info->page_size); +return TRUE; + } + +/* * Check the case phys_start isn't aligned by page size like below: * * phys_start
Signature · 10 lines
2.7.0 -- Ivan "Colona" Delalande Arista Networks _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Hello Ivan,
Quoted 2 lines
Handle pages filled with zeros that are not stored in the ELF file directly, but have addresses between
p_fileszandp_memszof a segment.
This fix looks reasonable, thanks for your work.
Quoted 2 lines
This allows makedumpfile to dump-dmesg from a previously makedumpfile-ed vmcore where all 0-pages were excluded (
dump_level& 1) for example.
I have some comments below:
Quoted 28 lines
Signed-off-by Ivan Delalande <[email protected]>---
elf_info.c | 22 ++++++++++++++++++++++ elf_info.h | 1 + makedumpfile.c | 16 ++++++++++++++++ 3 files changed, 39 insertions(+)elf_info.c +1
-0@@ -42,6 +42,7 @@ struct pt_load_segment {unsigned long long phys_end;unsigned long long virt_start;unsigned long long virt_end; +size_t mem_size; };static int
nr_cpus; /* number of cpu */elf_info.c +1
-0@@ -166,6 +167,7 @@ dump_Elf_load(Elf64_Phdr *prog, int num_load) pls->virt_start = prog->p_vaddr; pls->virt_end = pls->virt_start + prog->p_filesz; pls->file_offset = prog->p_offset; + pls->mem_size = prog->p_memsz;DEBUG_MSG ("LOAD (%d)\n" , num_load);DEBUG_MSG (" phys_start : %llx\n" , pls->phys_start);
elf_info.c offset_to_pt_load_end(off_t offset) · line 591
Quoted 6 lines
elf_info.c +3
-0@@ -584,6 +586,26 @@ offset_to_pt_load_end(off_t offset) }/* +* Return true if a page is in the zone between p_filesz and p_memsz of a +* segment, which is a page filled with zero and not stored in the ELF file. +*/
This function checks just "if a page fits within a PT_LOAD", but this comment and the function name don't match with it.
Quoted 5 lines
elf_info.c +5
-0+int +paddr_is_null (unsigned long long paddr) +{ +int i; +struct pt_load_segment *pls;
elf_info.c offset_to_pt_load_end(off_t offset) · line 606
Quoted 10 lines
elf_info.c +10
-0+ +for (i =0 ; i < num_pt_loads; i++) { + pls = &pt_loads[i]; +if ((paddr >= pls->phys_start) + && (paddr + info->page_size + <= pls->phys_start + pls->mem_size)) +return TRUE; + } +return FALSE; +}
If you keep the function name, the conditional expression should be:
if ((paddr >= pls->phys_end) && (paddr + info->page_size <= pls->phys_start + pls->mem_size)) return TRUE;
or,
Quoted 22 lines
elf_info.c +2
-0+ +/* * Judge whether the page is fractional or not. */ int
elf_info.h +1
-0@@ -36,6 +36,7 @@ unsigned long long page_head_to_phys_start(unsigned long long head_paddr);unsigned long long page_head_to_phys_end (unsigned long long head_paddr);off_t offset_to_pt_load_start (off_t offset);off_t offset_to_pt_load_end (off_t offset); +int paddr_is_null (unsigned long long paddr);unsigned long long vaddr_to_paddr_general (unsigned long long vaddr);off_t vaddr_to_offset_slow (int fd,char *filename,unsigned long long vaddr);unsigned long long get_max_paddr (void );makedumpfile.c +0
-0
makedumpfile.c readpage_elf(unsigned long long paddr, void *bufptr) · line 662
Quoted 9 lines
makedumpfile.c +6
-0@@ -654,6 +654,14 @@ readpage_elf(unsigned long long paddr, void *bufptr) phys_end = paddr + info->page_size;/* +* Check the 0-filled pages that are not in the ELF file. +*/ +if (!offset1 && !offset2 &&paddr_is_null (paddr)) { +memset (bufptr,0 , info->page_size); +return TRUE; + }
Since this combination of three conditions is exactly *paddr_is_null()*, the current paddr_is_null() should be renamed so that the name reflects its contents.
Thanks,
Atsushi Kumagai
Quoted 23 lines
makedumpfile.c +10
-1+ +/* * Check the case phys_start isn't aligned by page size like below: * * phys_start @@ -728,6 +736,14 @@ readpage_elf_parallel(int fd_memory, unsigned long long paddr, void *bufptr) phys_end = paddr + info->page_size;/* +* Check the 0-filled pages that are not in the ELF file. +*/ +if (!offset1 && !offset2 &&paddr_is_null (paddr)) { +memset (bufptr,0 , info->page_size); +return TRUE; + } + +/* * Check the case phys_start isn't aligned by page size like below: * * phys_start -- 2.7.0
Quoted 3 lines
--
Ivan "Colona" Delalande
Arista Networks
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Resent to the list.
The list does not like attachments, it seems.
Petr T
Quoted 24 lines · On Wed, 27 Jan 2016 08:58:21 +0100 Petr Tesarik <[email protected]> wrote:
Hello Atsushi-san,
Quoted 11 lines · On Wed, 27 Jan 2016 02:21:57 +0000 Atsushi Kumagai <[email protected]>…
Hello Ivan,
Quoted 2 lines
Handle pages filled with zeros that are not stored in the ELF file directly, but have addresses between
p_fileszandp_memszof a segment.This fix looks reasonable, thanks for your work.
Quoted 2 lines
This allows makedumpfile to dump-dmesg from a previously makedumpfile-ed vmcore where all 0-pages were excluded (
dump_level& 1) for example.I have some comments below:
Actually, I have already hit another issue with filtered ELF files. When refiltering an ELF file that had already been filtered, the memory size vs. file size can cause wrong
max_mapnrcalculation for a compressed file. In turn, if some pages were removed at the end of physical memory, those pages are not marked as filtered in the output file, but instead the total RAM appears to be smaller by that amount.
Quoted 4 lines
I don't think my patch addresses the same issue that Ivan found, but it definitely touches the same area. Please have a look.
Petr Tesarik
From: Petr Tesarik <[email protected]>
Date: Thu Aug 20 10:43:22 2015 +0200
Subject: Keep segment memory size when re-filtering ELF dumps
Originally, makedumpfile was designed to read from /proc/vmcore, where each segment's p_memsz is equal to its p_filesz. However, makedumpfile can also be used to re-filter an already filtered ELF dump file, where memory size may be larger than file size. In that case the memory size should be used as the size of the segment. This affects:
1. max_mapnr
This value is computed as the highest phys_end. If the last segment was filtered, max_mapnr may be too small, and the crash utility will refuse to read that memory (even with --zero_excluded).
2. p_memsz field in ELF dumps
The resulting ELF segment p_memsz will be capped to original file's p_filesz, ignoring the original p_memsz.
3. memory holes in KDUMP dumps
Pages excluded in the original ELF dump will be appear as memory holes in the resulting KDUMP file's first bitmap.
4. vtop translation
Virtual addresses that were filtered out in the original ELF file cannot be translated to physical addresses using the generic vtop translation.
My fix uses p_memsz to set physical and virtual extents of ELF segments, because this is the actual size. However, file size is important when accessing page data, so it must be stored separately and checked where appropriate. Last but not least, pages that were filtered in the original dump file must also be filtered in the destination file, i.e. pages between p_filesz and p_memsz must have their corresponding bit cleared in the 2nd bitmap.
--- elf_info.c | 37 ++++++++++++++++++++++++++++++------- elf_info.h | 4 ++++ makedumpfile.c | 26 ++++++++++++++++++++++++++ 3 files changed, 60 insertions(+), 7 deletions(-)
elf_info.c +30 -7
@@ -38,6 +38,7 @@struct pt_load_segment {off_t file_offset; +off_t file_size;unsigned long long phys_start;unsigned long long phys_end;unsigned long long virt_start;
@@ -162,10 +163,11 @@ dump_Elf_load(Elf64_Phdr *prog, int num_ pls = &pt_loads[num_load]; pls->phys_start = prog->p_paddr; - pls->phys_end = pls->phys_start + prog->p_filesz; + pls->phys_end = pls->phys_start + prog->p_memsz; pls->virt_start = prog->p_vaddr; - pls->virt_end = pls->virt_start + prog->p_filesz; + pls->virt_end = pls->virt_start + prog->p_memsz; pls->file_offset = prog->p_offset; + pls->file_size = prog->p_filesz;DEBUG_MSG ("LOAD (%d)\n" , num_load);DEBUG_MSG (" phys_start : %llx\n" , pls->phys_start);
@@ -462,7 +464,7 @@ paddr_to_offset(unsigned long long paddrfor (i = offset =0 ; i < num_pt_loads; i++) { pls = &pt_loads[i];if ((paddr >= pls->phys_start) - && (paddr < pls->phys_end)) { + && (paddr < pls->phys_start + pls->file_size)) { offset = (off_t)(paddr - pls->phys_start) + pls->file_offset;break ;
@@ -480,16 +482,14 @@ paddr_to_offset2(unsigned long long padd {int i;off_t offset; -unsigned long long len;struct pt_load_segment *pls;for (i = offset =0 ; i < num_pt_loads; i++) { pls = &pt_loads[i]; - len = pls->phys_end - pls->phys_start;if ((paddr >= pls->phys_start) - && (paddr < pls->phys_end) + && (paddr < pls->phys_start + pls->file_size) && (hint >= pls->file_offset) - && (hint < pls->file_offset + len)) { + && (hint < pls->file_offset + pls->file_size)) { offset = (off_t)(paddr - pls->phys_start) + pls->file_offset;break ;
@@ -1095,6 +1095,29 @@ get_pt_load(int idx,return TRUE; } + +int +get_pt_extents (int idx, +unsigned long long *phys_start, +unsigned long long *phys_end, +off_t *file_size) +{ +struct pt_load_segment *pls; + +if (num_pt_loads <= idx) +return FALSE; + + pls = &pt_loads[idx]; + +if (phys_start) + *phys_start = pls->phys_start; +if (phys_end) + *phys_end = pls->phys_end; +if (file_size) + *file_size = pls->file_size; + +return TRUE; +}unsigned int get_num_pt_loads (void )
elf_info.h +4 -0
@@ -61,6 +61,10 @@ int get_pt_load(int idx,unsigned long long *phys_end, unsigned long long *virt_start, unsigned long long *virt_end); +int get_pt_extents (int idx, +unsigned long long *phys_start, +unsigned long long *phys_end, +off_t *file_size);unsigned int get_num_pt_loads (void );void set_nr_cpus (int num);
makedumpfile.c +26 -0
@@ -4395,6 +4395,26 @@ read_start_flat_header(void)return TRUE; } +static void +exclude_nodata_pages (void ) +{ +int i; +unsigned long long phys_start, phys_end; +off_t file_size; + + i =0 ; +while (get_pt_extents (i, &phys_start, &phys_end, &file_size)) { +unsigned long long pfn, pfn_end; + + pfn =paddr_to_pfn (phys_start + file_size); + pfn_end =paddr_to_pfn (phys_end); +while (pfn <= pfn_end) +clear_bit_on_2nd_bitmap (pfn); + ++pfn; + ++i; + } +} +int read_flat_data_header (struct makedumpfile_data_header *fdh) {
@@ -6087,6 +6107,12 @@ create_2nd_bitmap(struct cycle *cycle) }/* +* If re-filtering ELF dump, exclude pages that were already +* excluded in the original file. +*/ +exclude_nodata_pages (); + +/* * Exclude cache pages, cache private pages, user data pages, * and hwpoison pages. */
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 1/1 makedumpfile: readpage_elf: handle 0-pages not stored in the ELF file
In reply to Atsushi KumagaiHandle pages filled with zero that are not stored in the ELF file directly, but have addresses between p_filesz and p_memsz of a segment.
This allows makedumpfile to dump-dmesg from a previously makedumpfile-ed vmcore where all 0-page were excluded (dump_level & 1) for example.
--- elf_info.c | 22 ++++++++++++++++++++++ elf_info.h | 1 + makedumpfile.c | 17 +++++++++++++++++ 3 files changed, 40 insertions(+)
elf_info.c +22 -0
@@ -42,6 +42,7 @@ struct pt_load_segment {unsigned long long phys_end;unsigned long long virt_start;unsigned long long virt_end; +size_t mem_size; };static int nr_cpus;/* number of cpu */
@@ -166,6 +167,7 @@ dump_Elf_load(Elf64_Phdr *prog, int num_load) pls->virt_start = prog->p_vaddr; pls->virt_end = pls->virt_start + prog->p_filesz; pls->file_offset = prog->p_offset; + pls->mem_size = prog->p_memsz;DEBUG_MSG ("LOAD (%d)\n" , num_load);DEBUG_MSG (" phys_start : %llx\n" , pls->phys_start);
@@ -584,6 +586,26 @@ offset_to_pt_load_end(off_t offset) }/* +* Return true if a page is in the zone between p_filesz and p_memsz of a +* segment, which is a page filled with zero and not stored in the ELF file. +*/ +int +paddr_is_null (unsigned long long paddr) +{ +int i; +struct pt_load_segment *pls; + +for (i =0 ; i < num_pt_loads; i++) { + pls = &pt_loads[i]; +if ((paddr >= pls->phys_end) + && (paddr + info->page_size + <= pls->phys_start + pls->mem_size)) +return TRUE; + } +return FALSE; +} + +/* * Judge whether the page is fractional or not. */ int
elf_info.h +1 -0
@@ -36,6 +36,7 @@ unsigned long long page_head_to_phys_start(unsigned long long head_paddr);unsigned long long page_head_to_phys_end (unsigned long long head_paddr);off_t offset_to_pt_load_start (off_t offset);off_t offset_to_pt_load_end (off_t offset); +int paddr_is_null (unsigned long long paddr);unsigned long long vaddr_to_paddr_general (unsigned long long vaddr);off_t vaddr_to_offset_slow (int fd,char *filename,unsigned long long vaddr);unsigned long long get_max_paddr (void );
makedumpfile.c +17 -0
@@ -652,11 +652,20 @@ readpage_elf(unsigned long long paddr, void *bufptr)size_t size1, size2;unsigned long long phys_start, phys_end, frac_head =0 ; +/* +* Check the 0-filled pages that are not in the ELF file. +*/ +if (paddr_is_null (paddr)) { +memset (bufptr,0 , info->page_size); +return TRUE; + } + offset1 =paddr_to_offset (paddr); offset2 =paddr_to_offset (paddr + info->page_size); phys_start = paddr; phys_end = paddr + info->page_size; +/* * Check the case phys_start isn't aligned by page size like below: *
@@ -726,6 +735,14 @@ readpage_elf_parallel(int fd_memory, unsigned long long paddr, void *bufptr)size_t size1, size2;unsigned long long phys_start, phys_end, frac_head =0 ; +/* +* Check the 0-filled pages that are not in the ELF file. +*/ +if (paddr_is_null (paddr)) { +memset (bufptr,0 , info->page_size); +return TRUE; + } + offset1 =paddr_to_offset (paddr); offset2 =paddr_to_offset (paddr + info->page_size); phys_start = paddr;
Signature · 11 lines
2.7.0 -- Ivan "Colona" Delalande Arista Networks _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Hello Petr,
Quoted 37 lines
handle 0-pages not stored in the ELF file
Resent to the list.
The list does not like attachments, it seems.
Petr T
Quoted 26 lines · On Wed, 27 Jan 2016 08:58:21 +0100 Petr Tesarik <[email protected]> wrote:
Hello Atsushi-san,
Quoted 11 lines · On Wed, 27 Jan 2016 02:21:57 +0000 Atsushi Kumagai <[email protected]>…
Hello Ivan,
Quoted 2 lines
Handle pages filled with zeros that are not stored in the ELF file directly, but have addresses between
p_fileszandp_memszof a segment.This fix looks reasonable, thanks for your work.
Quoted 2 lines
This allows makedumpfile to dump-dmesg from a previously makedumpfile-ed vmcore where all 0-pages were excluded (
dump_level& 1) for example.I have some comments below:
Actually, I have already hit another issue with filtered ELF files. When refiltering an ELF file that had already been filtered, the memory size vs. file size can cause wrong
max_mapnrcalculation for a compressed file. In turn, if some pages were removed at the end of physical memory, those pages are not marked as filtered in the output file, but instead the total RAM appears to be smaller by that amount.I don't think my patch addresses the same issue that Ivan found, but it definitely touches the same area. Please have a look.
Good catch, thanks for your report.
Quoted 11 lines
Quoted 1 line
Petr Tesarik
From: Petr Tesarik <[email protected]>
Date: Thu Aug 20 10:43:22 2015 +0200
Subject: Keep segment memory size when re-filtering ELF dumpsOriginally, makedumpfile was designed to read from
/proc/vmcore, where each segment'sp_memszis equal to itsp_filesz. However, makedumpfile can also be used to re-filter an already filtered ELF dump file, where memory size may be larger than file size. In that case the memory size should be used as the size of the segment. This affects:
Does this problem occur only if makedumpfile has done filtering ? According to the man 5 elf, even the original ELF file can have "unstored zero pages".
PT_LOAD The array element specifies a loadable segment, described by p_filesz and p_memsz.
The bytes from the file are mapped to the beginning of the memory segment. If the segment's memory size p_memsz is larger than the file size p_filesz, the "extra" bytes are defined to hold the value 0 and to follow the segment's initialized area.
If unstored pages will be made only by makedumpfile, what I said Below has no meaning.
Quoted 4 lines
1.
max_mapnr
This value is computed as the highestphys_end. If the last segment was filtered,max_mapnrmay be too small, and the crash utility will refuse to read that memory (even with --zero_excluded).
Yes, should be computed based on p_memsz.
Quoted 3 lines
2.
p_memszfield in ELF dumps
The resulting ELF segmentp_memszwill be capped to original file'sp_filesz, ignoring the originalp_memsz.
Yes, should be fixed.
Quoted 3 lines
3. memory holes in KDUMP dumps
Pages excluded in the original ELF dump will be appear as memory holes in the resulting KDUMP file's first bitmap.
a. If an unstored page is a just zero page, it is neither on a memory hole
nor a filtered page.
b. If an unstored page is the result of makedumpfile filtering, it should be
handled as a filtered page.
However, I think it's impossible to distinguish whether former or latter after filtering.
Quoted 4 lines
4. vtop translation
Virtual addresses that were filtered out in the original ELF file cannot be translated to physical addresses using the generic vtop translation.
Exactly, should refer to p_memsz for it as you did.
Quoted 7 lines
My fix uses
p_memszto set physical and virtual extents of ELF segments, because this is the actual size. However, file size is important when accessing page data, so it must be stored separately and checked where appropriate. Last but not least, pages that were filtered in the original dump file must also be filtered in the destination file, i.e. pages betweenp_fileszandp_memszmust have their corresponding bit cleared in the 2nd bitmap.
As I said above, I suspect not all of unstored pages are filtered pages, I'm not sure exclude_nodata_pages() does right things.
As Ivan's patch does, I guess reading them as zero pages fits ELF's format specification.
Thanks,
Atsushi Kumagai
Quoted 146 lines
Signed-off-by Petr Tesarik <[email protected]>---
elf_info.c | 37 ++++++++++++++++++++++++++++++------- elf_info.h | 4 ++++ makedumpfile.c | 26 ++++++++++++++++++++++++++ 3 files changed, 60 insertions(+), 7 deletions(-)elf_info.c +30
-7@@ -38,6 +38,7 @@struct pt_load_segment {off_t file_offset; +off_t file_size;unsigned long long phys_start;unsigned long long phys_end;unsigned long long virt_start;@@ -162,10 +163,11 @@ dump_Elf_load(Elf64_Phdr *prog, int num_ pls = &pt_loads[num_load]; pls->phys_start = prog->p_paddr; - pls->phys_end = pls->phys_start + prog->p_filesz; + pls->phys_end = pls->phys_start + prog->p_memsz; pls->virt_start = prog->p_vaddr; - pls->virt_end = pls->virt_start + prog->p_filesz; + pls->virt_end = pls->virt_start + prog->p_memsz; pls->file_offset = prog->p_offset; + pls->file_size = prog->p_filesz;DEBUG_MSG ("LOAD (%d)\n" , num_load);DEBUG_MSG (" phys_start : %llx\n" , pls->phys_start);@@ -462,7 +464,7 @@ paddr_to_offset(unsigned long long paddrfor (i = offset =0 ; i < num_pt_loads; i++) { pls = &pt_loads[i];if ((paddr >= pls->phys_start) - && (paddr < pls->phys_end)) { + && (paddr < pls->phys_start + pls->file_size)) { offset = (off_t)(paddr - pls->phys_start) + pls->file_offset;break ;@@ -480,16 +482,14 @@ paddr_to_offset2(unsigned long long padd {int i;off_t offset; -unsigned long long len;struct pt_load_segment *pls;for (i = offset =0 ; i < num_pt_loads; i++) { pls = &pt_loads[i]; - len = pls->phys_end - pls->phys_start;if ((paddr >= pls->phys_start) - && (paddr < pls->phys_end) + && (paddr < pls->phys_start + pls->file_size) && (hint >= pls->file_offset) - && (hint < pls->file_offset + len)) { + && (hint < pls->file_offset + pls->file_size)) { offset = (off_t)(paddr - pls->phys_start) + pls->file_offset;break ;@@ -1095,6 +1095,29 @@ get_pt_load(int idx,return TRUE; } + +int +get_pt_extents (int idx, +unsigned long long *phys_start, +unsigned long long *phys_end, +off_t *file_size) +{ +struct pt_load_segment *pls; + +if (num_pt_loads <= idx) +return FALSE; + + pls = &pt_loads[idx]; + +if (phys_start) + *phys_start = pls->phys_start; +if (phys_end) + *phys_end = pls->phys_end; +if (file_size) + *file_size = pls->file_size; + +return TRUE; +}unsigned int
get_num_pt_loads(void)elf_info.h +4
-0@@ -61,6 +61,10 @@ int get_pt_load(int idx,unsigned long long *phys_end, unsigned long long *virt_start, unsigned long long *virt_end); +int get_pt_extents (int idx, +unsigned long long *phys_start, +unsigned long long *phys_end, +off_t *file_size);unsigned int get_num_pt_loads (void );void set_nr_cpus (int num);makedumpfile.c +20
-0@@ -4395,6 +4395,26 @@ read_start_flat_header(void)return TRUE; } +static void +exclude_nodata_pages (void ) +{ +int i; +unsigned long long phys_start, phys_end; +off_t file_size; + + i =0 ; +while (get_pt_extents (i, &phys_start, &phys_end, &file_size)) { +unsigned long long pfn, pfn_end; + + pfn =paddr_to_pfn (phys_start + file_size); + pfn_end =paddr_to_pfn (phys_end); +while (pfn <= pfn_end) +clear_bit_on_2nd_bitmap (pfn); + ++pfn; + ++i; + } +} +int
read_flat_data_header(struct makedumpfile_data_header *fdh){makedumpfile.c +6
-0@@ -6087,6 +6107,12 @@ create_2nd_bitmap(struct cycle *cycle) }/* +* If re-filtering ELF dump, exclude pages that were already +* excluded in the original file. +*/ +exclude_nodata_pages (); + +/* * Exclude cache pages, cache private pages, user data pages, * and hwpoison pages. */
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Atsushi,
Quoted 8 lines · On Mon, 1 Feb 2016 06:48:13 +0000 Atsushi Kumagai <[email protected]>…
[...]
Quoted 5 lines
Originally, makedumpfile was designed to read from
/proc/vmcore, where each segment'sp_memszis equal to itsp_filesz. However, makedumpfile can also be used to re-filter an already filtered ELF dump file, where memory size may be larger than file size. In that case the memory size should be used as the size of the segment. This affects:Does this problem occur only if makedumpfile has done filtering ?
Indeed, I have only seen it in a previously filtered dump file, but see below.
Quoted 4 lines
According to the man 5 elf, even the original ELF file can have "unstored zero pages".
[...]
I'm aware of that.
Quoted 2 lines
If unstored pages will be made only by makedumpfile, what I said Below has no meaning.
So, I had a look at the kernel code for /proc/vmcore, and it turns out that the p_memsz and p_filesz fields for PT_LOAD segments are not changed at all. This means they are prepared by:
a. kexec(8) or
b. kexec_file_load(2) in the old kernel, or
c. elfcorehdr_alloc() in the new kernel (s390).
For option a, kexec/crashdump-elf.c says:
phdr->p_filesz = phdr->p_memsz = elf_info->kern_size;
and:
phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;
For option b, arch/x86/kernel/crash.c says:
phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;
and:
phdr->p_filesz = phdr->p_memsz = _end - _text;
For option c, arch/s390/kernel/crash_dump.c says:
phdr->p_filesz = end - start; phdr->p_memsz = end - start;
To sum it up, both fields (p_filesz and p_memsz) may originate outside the dump kernel (except s390), hence they cannot be fully trusted. Theoretically, you can write a custom tool which creates ELF segments with p_memsz greater than p_filesz and pass it to the secondary kernel. However, I'm not sure how it should be interpreted in a kernel dump
file: The "file" is in fact physical RAM, so such a segment would in effect forcibly replace existing RAM content with zeros.
OTOH another tool may post-process /proc/vmcore, translating zero-filled pages to more segments with a smaller p_filesz (just like makedumpfile does). If makedumpfile is supposed to interpret the output of such a (hypothetical) tool correctly, then yes, you must follow the ELF specification and treat the pages as zero.
Quoted 5 lines
[...]
Quoted 3 lines
3. memory holes in KDUMP dumps
Pages excluded in the original ELF dump will be appear as memory holes in the resulting KDUMP file's first bitmap.
Quoted 7 lines
a. If an unstored page is a just zero page, it is neither on a memory hole
nor a filtered page.
b. If an unstored page is the result of makedumpfile filtering, it should be
handled as a filtered page.However, I think it's impossible to distinguish whether former or latter after filtering.
Correct.
A clean solution would be to store this information in the filtered ELF file, e.g. with a ELF note or with an OS-specific program header flag.
Quoted 5 lines
[...]
As I said above, I suspect not all of unstored pages are filtered pages, I'm not sureexclude_nodata_pages()does right things.
As Ivan's patch does, I guess reading them as zero pages fits ELF's format specification.
That's right. It follows the ELF specification.
It may replace filtered out pages with zero-filled pages when converting an already-filtered ELF file to the compressed format; but they may be filtered again if bit 0 is set in the dump level, so Ivan's approach is cleaner, in fact. If you ever find a way to mark filtered out pages in an ELF file, then this behaviour can be improved later.
Anyway, are you going to take the patch by Ivan, or my patch (after I remove exclude_nodata_pages)?
Regards,
Petr Tesarik
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 56 lines
Quoted 7 lines
Quoted 5 lines
Originally, makedumpfile was designed to read from
/proc/vmcore, where each segment'sp_memszis equal to itsp_filesz. However, makedumpfile can also be used to re-filter an already filtered ELF dump file, where memory size may be larger than file size. In that case the memory size should be used as the size of the segment. This affects:Does this problem occur only if makedumpfile has done filtering ?
Indeed, I have only seen it in a previously filtered dump file, but see below.
Quoted 4 lines
According to the man 5 elf, even the original ELF file can have "unstored zero pages".
[...]
I'm aware of that.
Quoted 2 lines
If unstored pages will be made only by makedumpfile, what I said Below has no meaning.
So, I had a look at the kernel code for
/proc/vmcore, and it turns out that thep_memszandp_fileszfields forPT_LOADsegments are not changed at all. This means they are prepared by:a. kexec(8) or
b.kexec_file_load(2)in the old kernel, or
c.elfcorehdr_alloc()in the new kernel (s390).For option a,
kexec/crashdump-elf.csays:phdr->p_filesz = phdr->p_memsz = elf_info->kern_size;and:
phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;For option b,
arch/x86/kernel/crash.csays:phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;and:
phdr->p_filesz = phdr->p_memsz = _end - _text;For option c,
arch/s390/kernel/crash_dump.csays:phdr->p_filesz = end - start; phdr->p_memsz = end - start;To sum it up, both fields (
p_fileszandp_memsz) may originate outside the dump kernel (except s390), hence they cannot be fully trusted. Theoretically, you can write a custom tool which creates ELF segments withp_memszgreater thanp_fileszand pass it to the secondary kernel. However, I'm not sure how it should be interpreted in a kernel dump
file: The "file" is in fact physical RAM, so such a segment would in effect forcibly replace existing RAM content with zeros.
Quoted 5 lines
OTOH another tool may post-process
/proc/vmcore, translating zero-filled pages to more segments with a smallerp_filesz(just like makedumpfile does). If makedumpfile is supposed to interpret the output of such a (hypothetical) tool correctly, then yes, you must follow the ELF specification and treat the pages as zero.
Great, that's helpful investigation.
I'm not going to care about such irregular case, but I reconsidered the policy.
Quoted 10 lines
Quoted 10 lines
[...]
Quoted 3 lines
3. memory holes in KDUMP dumps
Pages excluded in the original ELF dump will be appear as memory holes in the resulting KDUMP file's first bitmap.a. If an unstored page is a just zero page, it is neither on a memory hole
nor a filtered page.
b. If an unstored page is the result of makedumpfile filtering, it should be
handled as a filtered page.
Quoted 7 lines
Quoted 2 lines
However, I think it's impossible to distinguish whether former or latter after filtering.
Correct.
A clean solution would be to store this information in the filtered ELF file, e.g. with a ELF note or with an OS-specific program header flag.
You are right, but I think it's too much fix.
I've noticed that we don't need to restore an excluded page in makedumpfile, I don't think we need to distinguish the two cases.
Quoted 7 lines
Quoted 5 lines
[...]
As I said above, I suspect not all of unstored pages are filtered pages, I'm not sureexclude_nodata_pages()does right things.
As Ivan's patch does, I guess reading them as zero pages fits ELF's format specification.That's right. It follows the ELF specification.
Since crash treats excluded pages as zero-filled pages, I thought it's better way also for makedumpfile, but
Quoted 5 lines
It may replace filtered out pages with zero-filled pages when converting an already-filtered ELF file to the compressed format; but they may be filtered again if bit 0 is set in the dump level, so Ivan's approach is cleaner, in fact. If you ever find a way to mark filtered out pages in an ELF file, then this behaviour can be improved later.
in the case of makedumpfile, reproducing zero pages consumes actual disk space, it sounds silly.
Anyhow we can't restore the excluded pages, there is no point compensating them with zero pages in writing process. Now I think the unstored pages should be written as they are to keep the file size. "read as zero pages" is necessary, but "written as (actual) zero pages" isn't.
Quoted 2 lines
Anyway, are you going to take the patch by Ivan, or my patch (after I remove
exclude_nodata_pages)?
Ivan's patch is necessary to follow the ELF specification, but also your patch (with exclude_nodata_pages) should be merged.
The two patches have different approach to expand struct pt_load_segment, hence could you manage both work ?
Ivan, your v2 patch has no problems, but could I leave this work to Petr since the two patches touch the same area ?
Thanks,
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 10 lines · Atsushi Kumagai
Quoted 2 lines
Anyway, are you going to take the patch by Ivan, or my patch (after I remove
exclude_nodata_pages)?Ivan's patch is necessary to follow the ELF specification, but also your patch (with
exclude_nodata_pages) should be merged.
The two patches have different approach to expand structpt_load_segment, hence could you manage both work ?Ivan, your v2 patch has no problems, but could I leave this work to Petr since the two patches touch the same area ?
Sure, this sounds reasonable.
Signature · 7 lines
Ivan "Colona" Delalande Arista Networks _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 9 lines · On Tue, 2 Feb 2016 08:00:44 +0100 Ivan Delalande <[email protected]> wrote:
Quoted 8 lines · Atsushi Kumagai
Quoted 2 lines
Anyway, are you going to take the patch by Ivan, or my patch (after I remove
exclude_nodata_pages)?Ivan's patch is necessary to follow the ELF specification, but also your patch (with
exclude_nodata_pages) should be merged.
The two patches have different approach to expand structpt_load_segment, hence could you manage both work ?
Quoted 4 lines
Quoted 2 lines
Ivan, your v2 patch has no problems, but could I leave this work to Petr since the two patches touch the same area ?
Sure, this sounds reasonable.
Hello,
just a short status update. I haven't forgotten this patch, but I've had some troubles reproducing the issue. To make sure I'm not on the wrong track, what happened when you tried to dump dmesg on the affected dump file without the patch?
For me, I crafted a dump file which causes a segfault. Did makedumpfile also terminate on SIGSEGV? Did it report an error? Or did it just produce wrong output?
Thanks,
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Kexec on ARM?
All,
After compiling the exec tools from my ubuntu machine to run on an ARMv7 target,
1) I loaded the kernel using : kexec -f uImage --append="$ ( cat /proc/cmdline )"
2) Afterwards, I ran: kexec -e
Note: uImage was successfully loaded using u-boot.
From my serial output, I get the following messages:
Bye!
Uncompressing Linux... done, booting the kernel. And it just hangs there.
Occasionaly, it goes further and actually prints:
Linux version xxxxxx
Date: XXX
CPU:XXX
Machine:XXX
Early serial console at MMIO32 0Xxxxxxxxxx
Does anyone have any idea on how to help fix this.
I am also constrained to use kexect-tools 2.0.2
Thanks,
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi,
Quoted 15 lines · Petr Tesarik
Quoted 9 lines · On Tue, 2 Feb 2016 08:00:44 +0100 Ivan Delalande <[email protected]> wrote:
Quoted 8 lines · Atsushi Kumagai
Quoted 2 lines
Anyway, are you going to take the patch by Ivan, or my patch (after I remove
exclude_nodata_pages)?Ivan's patch is necessary to follow the ELF specification, but also your patch (with
exclude_nodata_pages) should be merged.
The two patches have different approach to expand structpt_load_segment, hence could you manage both work ?Quoted 4 lines
Quoted 2 lines
Ivan, your v2 patch has no problems, but could I leave this work to Petr since the two patches touch the same area ?
Sure, this sounds reasonable.
Quoted 8 lines
just a short status update. I haven't forgotten this patch, but I've had some troubles reproducing the issue. To make sure I'm not on the wrong track, what happened when you tried to dump dmesg on the affected dump file without the patch?
For me, I crafted a dump file which causes a segfault. Did makedumpfile also terminate on SIGSEGV? Did it report an error? Or did it just produce wrong output?
Yeah, just a segfault, without any other message or output. It was coming from readpage_elf, at the call to memset in the if (!offset1) block, as it thinks it is handling one of this weird overlapping segments from ia64 and computes bad offsets.
Thank you,
Signature · 7 lines
Ivan "Colona" Delalande Arista Networks _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 9 lines · On Wed, 10 Feb 2016 04:21:18 +0100 Ivan Delalande <[email protected]> wrote:
Hi,
On Tue, Feb 09, 2016 at 09:31:50AM +0100, Petr Tesarik wrote:
[...]Quoted 5 lines
just a short status update. I haven't forgotten this patch, but I've had some troubles reproducing the issue. To make sure I'm not on the wrong track, what happened when you tried to dump dmesg on the affected dump file without the patch?
Quoted 8 lines
Quoted 3 lines
For me, I crafted a dump file which causes a segfault. Did makedumpfile also terminate on SIGSEGV? Did it report an error? Or did it just produce wrong output?
Yeah, just a segfault, without any other message or output. It was coming from
readpage_elf, at the call to memset in theif (!offset1)block, as it thinks it is handling one of this weird overlapping segments from ia64 and computes bad offsets.
Good. Then I was looking at the same bug, but fixed it differently. Anyway, thank you very much for the original fix! It set me in the right direction.
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Kexec on ARM?
In reply to Petr TesarikAll,
After compiling the exec tools from my ubuntu machine to run on an ARMv7 target,
1) I loaded the kernel using : kexec -f uImage --append="$ ( cat /proc/cmdline )"
2) Afterwards, I ran: kexec -e
Note: uImage was successfully loaded using u-boot.
From my serial output, I get the following messages:
Bye!
Uncompressing Linux... done, booting the kernel. And it just hangs there.
Occasionaly, it goes further and actually prints:
Linux version xxxxxx
Date: XXX
CPU:XXX
Machine:XXX
Early serial console at MMIO32 0Xxxxxxxxxx
Does anyone have any idea on how to help fix this.
I am also constrained to use kexect-tools 2.0.2
Thanks,
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec