Support free page filtering looking up mem_map array
Subsystems (2)
v2 cover Support free page filtering looking up mem_map array
This patch set implements filtering free pages looking up mem_map array instead of free lists. This is compatible for cyclic mode because looking up mem_map can be divided into cycles. On the other hand, dividing free pages contaiend in free lists is difficult since they are not sorted in physical-address order.
* Changes
v1 => v2:
- If debuginfo is not available enough, switch logic to freelist one. On v1, free page filtering was disabled in this case.
- Add hard-coded values in wider kernel versions.
- If some free pages possibly fail to be filtered, try to correct cyclic buffer size appropreately.
- Correct the comment explaining the cyclic buffer overrun, which was broken on v1.
RFC => v1:
- Logic is automatically selected at runtime according to the current mode. In cyclic mode, mem_map array logic is used. In non-cyclic mode, free list logic is used.
- The RFC version is:
http://lists.infradead.org/pipermail/kexec/2012-June/006441.html
* TODO
Add the following values in VMCOREINFO on the upstream kernel. These are used in the mem_map logic.
- OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
- NUMBER(PG_buddy)
- NUMBER(PG_slab)
- NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)
* Test
I tested this patch set on the following kernel versions.
- 3.4
- 3.1
- 2.6.38
- 2.6.32
- 2.6.18
On the test, I manually specified VMCOREINFO while extending it with the following values according to the kernel versions.
- 3.1, 3.4
NUMBER(PG_slab)=7
SIZE(pageflags)=4
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
- 2.6.38
SIZE(pageflags)=4
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-2
- 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4
- 2.6.18
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
---
HATAYAMA Daisuke (10):
Warn cyclic buffer overrun and correct it if possible Add page_is_buddy for old kernels
Add page_is_buddy for PG_buddy
Add page_is_buddy for recent kernels
Exclude free pages by looking up mem_map array
Add hardcoded page flag values
Add debuginfo-related processing for VMCOREINFO/VMLINUX Add new parameters to various tables
Add debuginfo interface for enum type size
Move page flags setup for old kernels after debuginfo initialization
dwarf_info.c | 18 ++++ dwarf_info.h | 1 makedumpfile.c | 226 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- makedumpfile.h | 35 ++++++++- 4 files changed, 269 insertions(+), 11 deletions(-)
Signature · 7 lines
Thanks. HATAYAMA, Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 1/10 Move page flags setup for old kernels after debuginfo initialization
Page flags setup needs to be done after debuginfo initialization. We use hard coded values only when debugging information doesn't provide corresponding values.
---
makedumpfile.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
makedumpfile.c +3 -2
@@ -2723,8 +2723,6 @@ initial(void) debug_info = TRUE; } -if (!get_value_for_old_linux ()) -return FALSE; out:if (!info->page_size) {/*
@@ -2789,6 +2787,9 @@ out:if (is_xen_memory () && !get_dom0_mapnr ())return FALSE; +if (!get_value_for_old_linux ()) +return FALSE; +return TRUE; }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 2/10 Add debuginfo interface for enum type size
In reply to HATAYAMA DaisukeThis is need to determine if a specified enumeration type is present in a given debug information.
In this patch set, I'll use this to check if enum pageflags is present.
The newly introduced interface is a simple extension from the existing one for enumeration value.
---
dwarf_info.c | 18 +++++++++++++++++- dwarf_info.h | 1 + makedumpfile.h | 13 +++++++++++-- 3 files changed, 29 insertions(+), 3 deletions(-)
dwarf_info.c +17 -1
@@ -607,7 +607,7 @@ search_structure(Dwarf_Die *die, int *found)static void search_number (Dwarf_Die *die,int *found) { -int tag; +int tag, bytesize;Dwarf_Word const_value;Dwarf_Attribute attr;Dwarf_Die child, *walker;
@@ -618,6 +618,22 @@ search_number(Dwarf_Die *die, int *found)if (tag != DW_TAG_enumeration_type)continue ; +if (dwarf_info.cmd ==DWARF_INFO_GET_ENUMERATION_TYPE_SIZE ) { + name =dwarf_diename (die); + +if (!name ||strcmp (name, dwarf_info.struct_name)) +continue ; + +if ((bytesize =dwarf_bytesize (die)) <=0 ) +continue ; + + *found = TRUE; + + dwarf_info.struct_size = bytesize; + +return ; + } +if (dwarf_child (die, &child) !=0 )continue ;
dwarf_info.h +1 -0
@@ -46,6 +46,7 @@ enum {DWARF_INFO_CHECK_SYMBOL_ARRAY_TYPE ,DWARF_INFO_GET_SYMBOL_TYPE ,DWARF_INFO_GET_MEMBER_TYPE , + DWARF_INFO_GET_ENUMERATION_TYPE_SIZE, };char *get_dwarf_module_name (void );
makedumpfile.h +11 -2
@@ -256,14 +256,23 @@ do { \#define ARRAY_LENGTH (X ) (array_table.X)#define SIZE_INIT (X ,Y ) \ do { \ - if ((SIZE(X) = get_structure_size(Y, 0)) == FAILED_DWARFINFO) \ + if ((SIZE(X) = get_structure_size(Y, DWARF_INFO_GET_STRUCT_SIZE)) \ + == FAILED_DWARFINFO) \ return FALSE; \ } while (0)#define TYPEDEF_SIZE_INIT (X ,Y ) \ do { \ - if ((SIZE(X) = get_structure_size(Y, 1)) == FAILED_DWARFINFO) \ + if ((SIZE(X) = get_structure_size(Y, DWARF_INFO_GET_TYPEDEF_SIZE)) \ + == FAILED_DWARFINFO) \ return FALSE; \ } while (0) +#define ENUM_TYPE_SIZE_INIT (X ,Y ) \ +do { \ + if ((SIZE(X) = get_structure_size(Y, \ + DWARF_INFO_GET_ENUMERATION_TYPE_SIZE)) \ + == FAILED_DWARFINFO) \ + return FALSE; \ +} while (0)#define OFFSET_INIT (X ,Y ,Z ) \ do { \ if ((OFFSET(X) = get_member_offset(Y, Z, DWARF_INFO_GET_MEMBER_OFFSET)) \
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 3/10 Add new parameters to various tables
In reply to HATAYAMA DaisukeAdd new parameters required for mem_map-based freepages filtering, to various tables.
---
makedumpfile.h | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
makedumpfile.h +8 -0
@@ -1163,6 +1163,8 @@ struct size_table {long cpumask_t;long kexec_segment;long elf64_hdr; + +long pageflags; };struct offset_table {
@@ -1171,6 +1173,8 @@ struct offset_table {long _count;long mapping;long lru; +long _mapcount; +long private; } page;struct mem_section {long section_mem_map;
@@ -1332,6 +1336,10 @@ struct number_table {long PG_lru;long PG_private;long PG_swapcache; +long PG_buddy; +long PG_slab; + +long PAGE_BUDDY_MAPCOUNT_VALUE ; };struct srcfile_table {
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 4/10 Add debuginfo-related processing for VMCOREINFO/VMLINUX
In reply to HATAYAMA Daisuke---
makedumpfile.c | 21 ++++++++++++++++++++- 1 files changed, 20 insertions(+), 1 deletions(-)
makedumpfile.c +20 -1
@@ -914,6 +914,8 @@ get_structure_info(void)OFFSET_INIT (page.flags,"page" ,"flags" );OFFSET_INIT (page._count,"page" ,"_count" );OFFSET_INIT (page.mapping,"page" ,"mapping" ); +OFFSET_INIT (page._mapcount,"page" ,"_mapcount" ); +OFFSET_INIT (page.private,"page" ,"private" );/* * Some vmlinux(s) don't have debugging information about
@@ -1005,6 +1007,10 @@ get_structure_info(void)ENUM_NUMBER_INIT (PG_lru,"PG_lru" );ENUM_NUMBER_INIT (PG_private,"PG_private" );ENUM_NUMBER_INIT (PG_swapcache,"PG_swapcache" ); +ENUM_NUMBER_INIT (PG_buddy,"PG_buddy" ); +ENUM_NUMBER_INIT (PG_slab,"PG_slab" ); + +ENUM_TYPE_SIZE_INIT (pageflags,"pageflags" );TYPEDEF_SIZE_INIT (nodemask_t,"nodemask_t" );
@@ -1355,6 +1361,7 @@ write_vmcoreinfo_data(void)WRITE_STRUCTURE_SIZE ("list_head" , list_head);WRITE_STRUCTURE_SIZE ("node_memblk_s" , node_memblk_s);WRITE_STRUCTURE_SIZE ("nodemask_t" , nodemask_t); +WRITE_STRUCTURE_SIZE ("pageflags" , pageflags);/* * write the member offset of 1st kernel
@@ -1363,6 +1370,8 @@ write_vmcoreinfo_data(void)WRITE_MEMBER_OFFSET ("page._count" , page._count);WRITE_MEMBER_OFFSET ("page.mapping" , page.mapping);WRITE_MEMBER_OFFSET ("page.lru" , page.lru); +WRITE_MEMBER_OFFSET ("page._mapcount" , page._mapcount); +WRITE_MEMBER_OFFSET ("page.private" , page.private);WRITE_MEMBER_OFFSET ("mem_section.section_mem_map" , mem_section.section_mem_map);WRITE_MEMBER_OFFSET ("pglist_data.node_zones" , pglist_data.node_zones);
@@ -1407,6 +1416,10 @@ write_vmcoreinfo_data(void)WRITE_NUMBER ("PG_lru" , PG_lru);WRITE_NUMBER ("PG_private" , PG_private);WRITE_NUMBER ("PG_swapcache" , PG_swapcache); +WRITE_NUMBER ("PG_buddy" , PG_buddy); +WRITE_NUMBER ("PG_slab" , PG_slab); + +WRITE_NUMBER ("PAGE_BUDDY_MAPCOUNT_VALUE" ,PAGE_BUDDY_MAPCOUNT_VALUE );/* * write the source file of 1st kernel
@@ -1654,11 +1667,14 @@ read_vmcoreinfo(void)READ_STRUCTURE_SIZE ("list_head" , list_head);READ_STRUCTURE_SIZE ("node_memblk_s" , node_memblk_s);READ_STRUCTURE_SIZE ("nodemask_t" , nodemask_t); +READ_STRUCTURE_SIZE ("pageflags" , pageflags);READ_MEMBER_OFFSET ("page.flags" , page.flags);READ_MEMBER_OFFSET ("page._count" , page._count);READ_MEMBER_OFFSET ("page.mapping" , page.mapping);READ_MEMBER_OFFSET ("page.lru" , page.lru); +READ_MEMBER_OFFSET ("page._mapcount" , page._mapcount); +READ_MEMBER_OFFSET ("page.private" , page.private);READ_MEMBER_OFFSET ("mem_section.section_mem_map" , mem_section.section_mem_map);READ_MEMBER_OFFSET ("pglist_data.node_zones" , pglist_data.node_zones);
@@ -1695,9 +1711,12 @@ read_vmcoreinfo(void)READ_NUMBER ("PG_lru" , PG_lru);READ_NUMBER ("PG_private" , PG_private);READ_NUMBER ("PG_swapcache" , PG_swapcache); - +READ_NUMBER ("PG_slab" , PG_slab); +READ_NUMBER ("PG_buddy" , PG_buddy);READ_SRCFILE ("pud_t" , pud_t); +READ_NUMBER ("PAGE_BUDDY_MAPCOUNT_VALUE" ,PAGE_BUDDY_MAPCOUNT_VALUE ); +return TRUE; }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 5/10 Add hardcoded page flag values
In reply to HATAYAMA DaisukeAlthough these values should basically be exported from VMCOREINFO, we cannot modify old kernels. Luckily, these values had not so frequently been changed that it's relatively easy to determine appropreate values for a given kernel version without any symbol or type information.
On the other hand, the mem_map logic also needs the values for some members of page structure. But it much depends on kernel versions. We aovid to hard code the values.
---
makedumpfile.c | 32 ++++++++++++++++++++++++++++++++ makedumpfile.h | 9 +++++++++ 2 files changed, 41 insertions(+), 0 deletions(-)
makedumpfile.c +32 -0
@@ -1210,6 +1210,38 @@ get_value_for_old_linux(void)NUMBER (PG_private) = PG_private_ORIGINAL;if (NUMBER (PG_swapcache) ==NOT_FOUND_NUMBER )NUMBER (PG_swapcache) = PG_swapcache_ORIGINAL; +if (NUMBER (PG_slab) ==NOT_FOUND_NUMBER ) +NUMBER (PG_slab) = PG_slab_ORIGINAL; +/* +* The values from here are for free page filtering based on +* mem_map array. These are minimum effort to cover old +* kernels. +* +* The logic also needs offset values for some members of page +* structure. But it much depends on kernel versions. We avoid +* to hard code the values. +*/ +if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER ) { +if (info->kernel_version >=KERNEL_VERSION (2 ,6 ,17 ) + && info->kernel_version <=KERNEL_VERSION (2 ,6 ,26 )) +NUMBER (PG_buddy) = PG_buddy_v2_6_17_to_v2_6_26; +if (info->kernel_version >=KERNEL_VERSION (2 ,6 ,27 ) + && info->kernel_version <=KERNEL_VERSION (2 ,6 ,37 )) +NUMBER (PG_buddy) = PG_buddy_v2_6_27_to_v2_6_37; + } +if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) ==NOT_FOUND_NUMBER ) { +if (info->kernel_version ==KERNEL_VERSION (2 ,6 ,38 )) +NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) = + PAGE_BUDDY_MAPCOUNT_VALUE_v2_6_38; +if (info->kernel_version >=KERNEL_VERSION (2 ,6 ,39 )) +NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) = + PAGE_BUDDY_MAPCOUNT_VALUE_v2_6_39_to_latest_version; + } +if (SIZE (pageflags) ==NOT_FOUND_STRUCTURE ) { +if (info->kernel_version >=KERNEL_VERSION (2 ,6 ,27 )) +SIZE (pageflags) = + PAGE_FLAGS_SIZE_v2_6_27_to_latest_version; + }return TRUE; }
makedumpfile.h +9 -0
@@ -71,9 +71,18 @@ int get_mem_type(void);* The following values are for linux-2.6.25 or former. */ #define PG_lru_ORIGINAL (5) +#define PG_slab_ORIGINAL (7)#define PG_private_ORIGINAL (11)/* Has something at ->private */ #define PG_swapcache_ORIGINAL (15)/* Swap page: swp_entry_t in private */ +#define PG_buddy_v2_6_17_to_v2_6_26 (19) +#define PG_buddy_v2_6_27_to_v2_6_37 (18) + +#define PAGE_BUDDY_MAPCOUNT_VALUE_v2_6_38 (-2) +#define PAGE_BUDDY_MAPCOUNT_VALUE_v2_6_39_to_latest_version (-128) + +#define PAGE_FLAGS_SIZE_v2_6_27_to_latest_version (4) +#define PAGE_MAPPING_ANON (1)#define LSEEKED_BITMAP (1)
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 6/10 Exclude free pages by looking up mem_map array
In reply to HATAYAMA DaisukeAdd free page filtering logic in __exclude_unnecessary_pages.
Unlike other filtering levels, the number of free pages indicated by buddy page is multiple. So I put this logic in the first position, in front of page cache, to avoid filtering the pages that has already been filtered as free pages.
Basically, this mem_map array logic is in cyclic mode only. The exceptional case is that debug information necessary for mem_map logic is not available enough. Then, it is switched to freelist logic.
In non cyclic mode, existing freelist logic is used.
Newly introduced page_is_buddy handler abstracts condition of buddy page that varies depending on kernel versions. On the kernel versions supported by makedumpfile, there are three kinds of buddy conditions. Later patches will introduce them in order.
If failing to choose a correct page_is_buddy handler due to absence of debug information, we switch the logic to freelist logic.
---
makedumpfile.c | 45 ++++++++++++++++++++++++++++++++++++++++----- makedumpfile.h | 5 +++++ 2 files changed, 45 insertions(+), 5 deletions(-)
makedumpfile.c +40 -5
@@ -58,6 +58,8 @@ do { \ *ptr_long_table = value; \ }while (0 ) +static void setup_page_is_buddy(void); +void initialize_tables (void ) {
@@ -2841,6 +2843,9 @@ out:if (!get_value_for_old_linux ())return FALSE; +if (info->flag_cyclic && (info->dump_level &DL_EXCLUDE_FREE )) +setup_page_is_buddy (); +return TRUE; }
@@ -3663,6 +3668,18 @@ exclude_free_page(void)return TRUE; } +static void +setup_page_is_buddy (void ) +{ +if (OFFSET (page.private) ==NOT_FOUND_STRUCTURE ) + goto out; + +out: +if (!info->page_is_buddy) +DEBUG_MSG ("Can't select page_is_buddy handler; " +"follow free lists instead of mem_map array.\n" ); +} +/* * If using a dumpfile in kdump-compressed format as a source file * instead of /proc/vmcore, 1st-bitmap of a new dumpfile must be
@@ -3873,8 +3890,8 @@ __exclude_unnecessary_pages(unsigned long mem_map,unsigned long long pfn_read_start, pfn_read_end, index_pg;unsigned char page_cache[SIZE (page) *PGMM_CACHED ];unsigned char *pcache; -unsigned int _count; -unsigned long flags, mapping; +unsigned int _count, _mapcount =0 ; +unsigned long flags, mapping, private =0 ;/* * Refresh the buffer of struct page, when changing mem_map.
@@ -3928,11 +3945,28 @@ __exclude_unnecessary_pages(unsigned long mem_map, flags =ULONG (pcache +OFFSET (page.flags)); _count =UINT (pcache +OFFSET (page._count)); mapping =ULONG (pcache +OFFSET (page.mapping)); +if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) + _mapcount =UINT (pcache +OFFSET (page._mapcount)); +if (OFFSET (page.private) !=NOT_FOUND_STRUCTURE ) + private =ULONG (pcache +OFFSET (page.private));/* +* Exclude the free page managed by a buddy +*/ +if ((info->dump_level &DL_EXCLUDE_FREE ) + && info->flag_cyclic + && info->page_is_buddy + && info->page_is_buddy (flags, _mapcount, private, _count)) { +int i; + +for (i =0 ; i < (1 << private); ++i) +clear_bit_on_2nd_bitmap_for_kernel (pfn + i); + pfn_free += i; + } +/* * Exclude the cache page without the private page. */ -if ((info->dump_level &DL_EXCLUDE_CACHE ) +else if ((info->dump_level &DL_EXCLUDE_CACHE ) && (isLRU (flags) ||isSwapCache (flags)) && !isPrivate (flags) && !isAnon (mapping)) {if (clear_bit_on_2nd_bitmap_for_kernel (pfn))
@@ -4013,7 +4047,7 @@ exclude_unnecessary_pages_cyclic(void)*/ copy_bitmap_cyclic (); -if (info->dump_level &DL_EXCLUDE_FREE ) +if ((info->dump_level &DL_EXCLUDE_FREE ) && !info->page_is_buddy)if (!exclude_free_page ())return FALSE;
@@ -4022,7 +4056,8 @@ exclude_unnecessary_pages_cyclic(void)*/ if (info->dump_level &DL_EXCLUDE_CACHE || info->dump_level &DL_EXCLUDE_CACHE_PRI || - info->dump_level &DL_EXCLUDE_USER_DATA ) { + info->dump_level &DL_EXCLUDE_USER_DATA || + ((info->dump_level &DL_EXCLUDE_FREE ) && info->page_is_buddy)) {gettimeofday (&tv_start,NULL );
makedumpfile.h +5 -0
@@ -1037,6 +1037,11 @@ struct DumpInfo {*/ int flag_sadump_diskset;enum sadump_format_type flag_sadump;/* sadump format type */ +/* +* for filtering free pages managed by buddy system: +*/ +int (*page_is_buddy)(unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count); };extern struct DumpInfo *info;
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 7/10 Add page_is_buddy for recent kernels
In reply to HATAYAMA DaisukeOn kernels from v2.6.38 and later kernels, buddy page is marked by _mapcount == PAGE_BUDDY_MAPCOUNT_VALUE, which varies once as follows:
kernel version | PAGE_BUDDY_MAPCOUNT_VALUE
------------------+--------------------------
v2.6.38 | -2
v2.6.39 and later | -128
One more notice is that _mapcount shares its memory with other fields for SLAB/SLUB when PG_slab is set. Before looking up _mapcount value, we need to check if PG_slab is set.
Since this page_is_buddy needs _mapcount, we use freelist logic if _mapcount is not available.
Recall that PG_slab has been 7 since v2.6.15. No need to check it.
---
makedumpfile.c | 21 +++++++++++++++++++++ 1 files changed, 21 insertions(+), 0 deletions(-)
makedumpfile.c +21 -0
@@ -3668,12 +3668,33 @@ exclude_free_page(void)return TRUE; } +/* +* For v2.6.38 and later kernel versions. +*/ +static int +page_is_buddy_v3 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if (flags & (1UL <<NUMBER (PG_slab))) +return FALSE; + +if (_mapcount == (int )NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE )) +return TRUE; + +return FALSE; +} +static void setup_page_is_buddy (void ) {if (OFFSET (page.private) ==NOT_FOUND_STRUCTURE ) goto out; +if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) { +if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) + info->page_is_buddy = page_is_buddy_v3; + } + out:if (!info->page_is_buddy) DEBUG_MSG("Can't select page_is_buddy handler; "
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 8/10 Add page_is_buddy for PG_buddy
In reply to HATAYAMA DaisukeOn kernels from v2.6.18 to v2.6.37, buddy page is marked by the PG_buddy flag.
kernel version | PG_buddy
------------------ +---------------------------------
v2.6.17 to v2.6.26 | 19
v2.6.27 to v2.6.37 | 19 if CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwise
We don't need to care about CONFIG_PAGEFLAGS_EXTEND because the architectures specifying this as y are um and xtensa only. They are not included in the supported architectures of makedumpfile.
---
makedumpfile.c | 24 ++++++++++++++++++++---- 1 files changed, 20 insertions(+), 4 deletions(-)
makedumpfile.c +20 -4
@@ -3669,6 +3669,19 @@ exclude_free_page(void) }/* +* For the kernel versions from v2.6.17 to v2.6.37. +*/ +static int +page_is_buddy_v2 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if (flags & (1UL <<NUMBER (PG_buddy))) +return TRUE; + +return FALSE; +} + +/* * For v2.6.38 and later kernel versions. */ static int
@@ -3690,10 +3703,13 @@ setup_page_is_buddy(void)if (OFFSET (page.private) ==NOT_FOUND_STRUCTURE ) goto out; -if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) { -if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) - info->page_is_buddy = page_is_buddy_v3; - } +if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER ) { +if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) { +if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) + info->page_is_buddy = page_is_buddy_v3; + } + }else + info->page_is_buddy = page_is_buddy_v2; out:if (!info->page_is_buddy)
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 08/10] Add page_is_buddy for PG_buddy
Hello HATAYAMA-san,
Quoted 5 lines · On Fri, 16 Nov 2012 14:02:13 +0900 HATAYAMA Daisuke…
On kernels from v2.6.18 to v2.6.37, buddy page is marked by the
PG_buddyflag.kernel version |
PG_buddy
------------------ +---------------------------------
Quoted 7 lines
v2.6.17 to v2.6.26 | 19v2.6.27 to v2.6.37 | 19 if
CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwiseWe don't need to care about
CONFIG_PAGEFLAGS_EXTENDbecause the architectures specifying this as y are um and xtensa only. They are not included in the supported architectures of makedumpfile.
Sorry for too late point out.
I did regression test for v1.5.1 on x86_64 machine in the last weekend and found the issue that page_is_buddy_v2() can't exclude free pages correctly for kernel 2.6.30 to 2.6.37.
I checked the configuration of these kernels, I made sure that CONFIG_PAGEFLAGS_EXTENDED was set as y, so PG_buddy=18 is invalid for them.
According to this fact, PG_buddy is variable for the supported architectures of makedumpfile, too.
So, how did you confirm that CONFIG_PAGEFLAGS_EXTENDED is only related with um and xtensa ?
And why did you test with PG_buddy=19 for 2.6.32 when you posted this patch set?
Quoted 6 lines
- 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4
Thanks
Atsushi Kumagai
Quoted 50 lines
Signed-off-by HATAYAMA Daisuke <[email protected]>---
makedumpfile.c | 24 ++++++++++++++++++++---- 1 files changed, 20 insertions(+), 4 deletions(-)makedumpfile.c +20
-4@@ -3669,6 +3669,19 @@ exclude_free_page(void) }/* +* For the kernel versions from v2.6.17 to v2.6.37. +*/ +static int +page_is_buddy_v2 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if (flags & (1UL <<NUMBER (PG_buddy))) +return TRUE; + +return FALSE; +} + +/* * For v2.6.38 and later kernel versions. */ static int@@ -3690,10 +3703,13 @@ setup_page_is_buddy(void)if (OFFSET (page.private) ==NOT_FOUND_STRUCTURE ) goto out; -if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) { -if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) - info->page_is_buddy = page_is_buddy_v3; - } +if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER ) { +if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) { +if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) + info->page_is_buddy = page_is_buddy_v3; + } + }else + info->page_is_buddy = page_is_buddy_v2; out:if (!info->page_is_buddy)
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 08/10] Add page_is_buddy for PG_buddy
Quoted 13 lines · On Tue, 27 Nov 2012 15:00:48 +0900 Atsushi Kumagai…
Hello HATAYAMA-san,
Quoted 8 lines · On Fri, 16 Nov 2012 14:02:13 +0900 HATAYAMA Daisuke…
On kernels from v2.6.18 to v2.6.37, buddy page is marked by the
PG_buddyflag.kernel version |
PG_buddy
------------------ +---------------------------------v2.6.17 to v2.6.26 | 19v2.6.27 to v2.6.37 | 19 if
CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwise
Additionally, PG_private_2 was introduced in v2.6.30 with the commit below:
commit 266cf658efcf6ac33541a46740f74f50c79d2b6b
Author: David Howells <[email protected]>
Date: Fri Apr 3 16:42:36 2009 +0100
FS-Cache: Recruit a page flags for cache management
Recruit a page flag to aid in cache management. The following extra flag is defined:
(1) PG_fscache (PG_private_2)
Therefore, PG_buddy varies as below:
kernel version | PG_buddy
------------------ +---------------------------------
v2.6.27 to v2.6.29 | 18 if CONFIG_PAGEFLAGS_EXTEND=y
| 17 otherwise
v2.6.30 to v2.6.37 | 19 if CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwiseThanks
Atsushi Kumagai
Quoted 76 lines
Quoted 3 lines
We don't need to care about
CONFIG_PAGEFLAGS_EXTENDbecause the architectures specifying this as y are um and xtensa only. They are not included in the supported architectures of makedumpfile.Sorry for too late point out.
I did regression test for v1.5.1 on
x86_64machine in the last weekend and found the issue thatpage_is_buddy_v2()can't exclude free pages correctly for kernel 2.6.30 to 2.6.37.
I checked the configuration of these kernels, I made sure thatCONFIG_PAGEFLAGS_EXTENDEDwas set as y, so PG_buddy=18 is invalid for them.According to this fact,
PG_buddyis variable for the supported architectures of makedumpfile, too.
So, how did you confirm thatCONFIG_PAGEFLAGS_EXTENDEDis only related with um and xtensa ?And why did you test with PG_buddy=19 for 2.6.32 when you posted this patch set?
Quoted 6 lines
- 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4Thanks
Atsushi KumagaiQuoted 46 lines
Signed-off-by HATAYAMA Daisuke <[email protected]>---
makedumpfile.c | 24 ++++++++++++++++++++---- 1 files changed, 20 insertions(+), 4 deletions(-)makedumpfile.c +20
-4@@ -3669,6 +3669,19 @@ exclude_free_page(void) }/* +* For the kernel versions from v2.6.17 to v2.6.37. +*/ +static int +page_is_buddy_v2 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if (flags & (1UL <<NUMBER (PG_buddy))) +return TRUE; + +return FALSE; +} + +/* * For v2.6.38 and later kernel versions. */ static int@@ -3690,10 +3703,13 @@ setup_page_is_buddy(void)if (OFFSET (page.private) ==NOT_FOUND_STRUCTURE ) goto out; -if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) { -if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) - info->page_is_buddy = page_is_buddy_v3; - } +if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER ) { +if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) { +if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) + info->page_is_buddy = page_is_buddy_v3; + } + }else + info->page_is_buddy = page_is_buddy_v2;
Quoted 8 lines
Quoted 8 lines
makedumpfile.c +0
-0out:if (!info->page_is_buddy)_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RE: [PATCH v2 08/10] Add page_is_buddy for PG_buddy
In reply to Atsushi KumagaiQuoted 31 lines
From: [email protected] [mailto:[email protected]] On Behalf Of Atsushi Kumagai
Sent: Tuesday, November 27, 2012 3:01 PM
Subject: Re: [PATCH v2 08/10] Addpage_is_buddyforPG_buddyHello HATAYAMA-san,
Quoted 5 lines · On Fri, 16 Nov 2012 14:02:13 +0900 HATAYAMA Daisuke…
On kernels from v2.6.18 to v2.6.37, buddy page is marked by the
PG_buddyflag.kernel version |
PG_buddy
------------------ +---------------------------------Quoted 7 lines
v2.6.17 to v2.6.26 | 19v2.6.27 to v2.6.37 | 19 if
CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwiseWe don't need to care about
CONFIG_PAGEFLAGS_EXTENDbecause the architectures specifying this as y are um and xtensa only. They are not included in the supported architectures of makedumpfile.Sorry for too late point out.
I did regression test for v1.5.1 on
x86_64machine in the last weekend and found the issue thatpage_is_buddy_v2()can't exclude free pages correctly for kernel 2.6.30 to 2.6.37.
I checked the configuration of these kernels, I made sure thatCONFIG_PAGEFLAGS_EXTENDEDwas set as y, so PG_buddy=18 is invalid for them.
Quoted 5 lines
According to this fact,
PG_buddyis variable for the supported architectures of makedumpfile, too.
So, how did you confirm thatCONFIG_PAGEFLAGS_EXTENDEDis only related with um and xtensa ?
I guess I used git grep on different kernel source tree.
Quoted 10 lines
And why did you test with PG_buddy=19 for 2.6.32 when you posted this patch set?
Quoted 6 lines
- 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4
Sorry, I was not aware of this.
Anyway, the current hard-coded values doesn't work at all since before talking about PG_buddy, we cannot get values of page structure's members, such as offset value of private member, so non-cyclic is always used instead.
How about dropping hard-coded values?
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 08/10] Add page_is_buddy for PG_buddy
Hello HATAYAMA-san,
Quoted 50 lines · Daisuke
Quoted 31 lines
From: [email protected] [mailto:[email protected]] On Behalf Of Atsushi Kumagai
Sent: Tuesday, November 27, 2012 3:01 PM
Subject: Re: [PATCH v2 08/10] Addpage_is_buddyforPG_buddyHello HATAYAMA-san,
Quoted 5 lines · On Fri, 16 Nov 2012 14:02:13 +0900 HATAYAMA Daisuke…
On kernels from v2.6.18 to v2.6.37, buddy page is marked by the
PG_buddyflag.kernel version |
PG_buddy
------------------ +---------------------------------Quoted 7 lines
v2.6.17 to v2.6.26 | 19v2.6.27 to v2.6.37 | 19 if
CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwiseWe don't need to care about
CONFIG_PAGEFLAGS_EXTENDbecause the architectures specifying this as y are um and xtensa only. They are not included in the supported architectures of makedumpfile.Sorry for too late point out.
I did regression test for v1.5.1 on
x86_64machine in the last weekend and found the issue thatpage_is_buddy_v2()can't exclude free pages correctly for kernel 2.6.30 to 2.6.37.
I checked the configuration of these kernels, I made sure thatCONFIG_PAGEFLAGS_EXTENDEDwas set as y, so PG_buddy=18 is invalid for them.Quoted 5 lines
According to this fact,
PG_buddyis variable for the supported architectures of makedumpfile, too.
So, how did you confirm thatCONFIG_PAGEFLAGS_EXTENDEDis only related with um and xtensa ?I guess I used git grep on different kernel source tree.
Quoted 10 lines
And why did you test with PG_buddy=19 for 2.6.32 when you posted this patch set?
Quoted 6 lines
- 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4
Quoted 1 line
Sorry, I was not aware of this.
OK, I see.
Quoted 5 lines
Anyway, the current hard-coded values doesn't work at all since before talking about
PG_buddy, we cannot get values of page structure's members, such as offset value of private member, so non-cyclic is always used instead.How about dropping hard-coded values?
It seems better to drop hard-coded values.
vmlinux (or vmcoreinfo) is necessary to get values of page structure's members and we can get NUMBER(PG_buddy) from vmlinux as well. So, I think the hard-coding for PG_buddy brings few benefits.
I misunderstood that enumeration values can't be gotten from vmlinux, so I thought the hard-coding is reasonable.
(I seem to just faced the gcc bug below.)
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41065
Anyway, to sum it up:
1. If vmlinux or vmcoreinfo is prepared and running on cyclic mode,
page_is_buddy_v2() works correctly for v2.6.18 to v2.6.37.
2. Otherwise, free list logic is used.
I think it's good policy for current kernels.
Thanks
Atsushi Kumagai
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 9/10 Add page_is_buddy for old kernels
In reply to HATAYAMA DaisukeOn kernels from v2.6.15 to v2.6.17, buddy page is marked by the condition that PG_private flag is set and _count == 0.
Since this page_is_buddy needs _count, we use freelist logic if _count is not available.
Unfortunately, I have yet to test this logic on these kernel versions simply because I've been failing to boot them on my box.
Note that on these kernels, free list can be corrupted due to the bug that the above two conditions are not checked atomically. The reason why PG_buddy was introduced is a fix for this bug.
---
makedumpfile.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-)
makedumpfile.c +17 -0
@@ -3669,6 +3669,20 @@ exclude_free_page(void) }/* +* For the kernel versions from v2.6.15 to v2.6.17. +*/ +static int +page_is_buddy_v1 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if ((flags & (1UL <<NUMBER (PG_private))) + && _count ==0 ) +return TRUE; + +return FALSE; +} + +/* * For the kernel versions from v2.6.17 to v2.6.37. */ static int
@@ -3707,6 +3721,9 @@ setup_page_is_buddy(void)if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) {if (OFFSET (page._mapcount) !=NOT_FOUND_STRUCTURE ) info->page_is_buddy = page_is_buddy_v3; + }else if (SIZE (pageflags) ==NOT_FOUND_STRUCTURE ) { +if (OFFSET (page._count) !=NOT_FOUND_STRUCTURE ) + info->page_is_buddy = page_is_buddy_v1; } }else info->page_is_buddy = page_is_buddy_v2;
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v2 10/10 Warn cyclic buffer overrun and correct it if possible
In reply to HATAYAMA DaisukeClearling bits on cyclic buffer can overrun the cyclic buffer according to some combination of MAX_ORDER and cyclic buffer size.
The cyclic buffer size is corrected if possible.
---
makedumpfile.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 70 insertions(+), 1 deletions(-)
makedumpfile.c +70 -1
@@ -58,6 +58,7 @@ do { \ *ptr_long_table = value; \ }while (0 ) +static void check_cyclic_buffer_overrun(void);static void setup_page_is_buddy (void );void
@@ -2832,6 +2833,9 @@ out: !sadump_generate_elf_note_from_dumpfile())return FALSE; +if (info->flag_cyclic && info->dump_level &DL_EXCLUDE_FREE ) +check_cyclic_buffer_overrun (); + }else {if (!get_mem_map_without_mm ())return FALSE;
@@ -3669,6 +3673,61 @@ exclude_free_page(void) }/* +* Let C be a cyclic buffer size and B a bitmap size used for +* representing maximum block size managed by buddy allocator. +* +* For some combinations of C and B, clearing operation can overrun +* the cyclic buffer. Let's consider three cases. +* +* - If C == B, this is trivially safe. +* +* - If B > C, overrun can easily happen. +* +* - In case of C > B, if C mod B != 0, then there exist n > m > 0, +* B > b > 0 such that n x C = m x B + b. This means that clearing +* operation overruns cyclic buffer (B - b)-bytes in the +* combination of n-th cycle and m-th block. +* +* Note that C mod B != 0 iff (m x C) mod B != 0 for some m. +* +* If C == B, C mod B == 0 always holds. Again, if B > C, C mod B != 0 +* always holds. Hence, it's always sufficient to check the condition +* C mod B != 0 in order to determine whether overrun can happen or +* not. +* +* The bitmap size used for maximum block size B is calculated from +* MAX_ORDER as: +* +* B := DIVIDE_UP((1 << (MAX_ORDER - 1)), BITS_PER_BYTE) +* +* Normally, MAX_ORDER is 11 at default. This is configurable through +* CONFIG_FORCE_MAX_ZONEORDER. +*/ +static void +check_cyclic_buffer_overrun (void ) +{ +int max_order =ARRAY_LENGTH (zone.free_area); +int max_order_nr_pages =1 << (max_order -1 ); +unsigned long max_block_size =roundup (max_order_nr_pages,BITPERBYTE ); + +if (info->bufsize_cyclic % +roundup (max_order_nr_pages,BITPERBYTE )) { +unsigned long bufsize; + +if (max_block_size > info->bufsize_cyclic) { +MSG ("WARNING: some free pages are not filtered.\n" ); +return ; + } + + bufsize = info->bufsize_cyclic; + info->bufsize_cyclic =round (bufsize, max_block_size); + +MSG ("cyclic buffer size has been changed: %lu => %lu\n" , + bufsize, info->bufsize_cyclic); + } +} + +/* * For the kernel versions from v2.6.15 to v2.6.17. */ static int
@@ -4013,8 +4072,18 @@ __exclude_unnecessary_pages(unsigned long mem_map, && info->page_is_buddy(flags, _mapcount, private, _count)) {int i; -for (i =0 ; i < (1 << private); ++i) +for (i =0 ; i < (1 << private); ++i) { +/* +* According to combination of +* MAX_ORDER and size of cyclic +* buffer, this clearing bit operation +* can overrun the cyclic buffer. +* +* See check_cyclic_buffer_overrun() +* for the detail. +*/ clear_bit_on_2nd_bitmap_for_kernel (pfn + i); + } pfn_free += i; }/*
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 10/10] Warn cyclic buffer overrun and correct it if possible
Hello HATAYAMA-san,
Quoted 6 lines · HATAYAMA Daisuke
Clearling bits on cyclic buffer can overrun the cyclic buffer according to some combination of
MAX_ORDERand cyclic buffer size.The cyclic buffer size is corrected if possible.
Signed-off-by HATAYAMA Daisuke <[email protected]>
I know it's so late, I found that updating pfn_cyclic is missing. It can cause memory corruption.
From: Atsushi Kumagai <[email protected]>
Date: Wed, 11 Sep 2013 14:50:10 +0900
Subject: [PATCH] Update pfn_cyclic when the cyclic buffer size is
corrected.
When the clearing bit operation for excluding free pages can overrun the cyclic buffer, the buffer size is changed with check_cyclic_buffer_overrun().
Then pfn_cyclic should be recalculated.
--- makedumpfile.c | 1 + 1 file changed, 1 insertion(+)
makedumpfile.c +1 -0
@@ -4091,6 +4091,7 @@ check_cyclic_buffer_overrun(void) bufsize = info->bufsize_cyclic; info->bufsize_cyclic =round (bufsize, max_block_size); + info->pfn_cyclic = info->bufsize_cyclic *BITPERBYTE ;MSG ("cyclic buffer size has been changed: %lu => %lu\n" , bufsize, info->bufsize_cyclic);
Signature · 7 lines
1.8.0.2 I'll merge this patch into v1.5.5. Thanks Atsushi Kumagai
Quoted 99 lines
---
makedumpfile.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 70 insertions(+), 1 deletions(-)makedumpfile.c +64
-1@@ -58,6 +58,7 @@ do { \ *ptr_long_table = value; \ }while (0 ) +static void check_cyclic_buffer_overrun(void);static void setup_page_is_buddy (void );void @@ -2832,6 +2833,9 @@ out: !sadump_generate_elf_note_from_dumpfile())return FALSE; +if (info->flag_cyclic && info->dump_level &DL_EXCLUDE_FREE ) +check_cyclic_buffer_overrun (); + }else {if (!get_mem_map_without_mm ())return FALSE;@@ -3669,6 +3673,61 @@ exclude_free_page(void) }/* +* Let C be a cyclic buffer size and B a bitmap size used for +* representing maximum block size managed by buddy allocator. +* +* For some combinations of C and B, clearing operation can overrun +* the cyclic buffer. Let's consider three cases. +* +* - If C == B, this is trivially safe. +* +* - If B > C, overrun can easily happen. +* +* - In case of C > B, if C mod B != 0, then there exist n > m > 0, +* B > b > 0 such that n x C = m x B + b. This means that clearing +* operation overruns cyclic buffer (B - b)-bytes in the +* combination of n-th cycle and m-th block. +* +* Note that C mod B != 0 iff (m x C) mod B != 0 for some m. +* +* If C == B, C mod B == 0 always holds. Again, if B > C, C mod B != 0 +* always holds. Hence, it's always sufficient to check the condition +* C mod B != 0 in order to determine whether overrun can happen or +* not. +* +* The bitmap size used for maximum block size B is calculated from +* MAX_ORDER as: +* +* B := DIVIDE_UP((1 << (MAX_ORDER - 1)), BITS_PER_BYTE) +* +* Normally, MAX_ORDER is 11 at default. This is configurable through +* CONFIG_FORCE_MAX_ZONEORDER. +*/ +static void +check_cyclic_buffer_overrun (void ) +{ +int max_order =ARRAY_LENGTH (zone.free_area); +int max_order_nr_pages =1 << (max_order -1 ); +unsigned long max_block_size =roundup (max_order_nr_pages,BITPERBYTE ); + +if (info->bufsize_cyclic % +roundup (max_order_nr_pages,BITPERBYTE )) { +unsigned long bufsize; + +if (max_block_size > info->bufsize_cyclic) { +MSG ("WARNING: some free pages are not filtered.\n" ); +return ; + } + + bufsize = info->bufsize_cyclic; + info->bufsize_cyclic =round (bufsize, max_block_size); + +MSG ("cyclic buffer size has been changed: %lu => %lu\n" , + bufsize, info->bufsize_cyclic); + } +} + +/* * For the kernel versions from v2.6.15 to v2.6.17. */ static int@@ -4013,8 +4072,18 @@ __exclude_unnecessary_pages(unsigned long mem_map, && info->page_is_buddy(flags, _mapcount, private, _count)) {int i; -for (i =0 ; i < (1 << private); ++i) +for (i =0 ; i < (1 << private); ++i) { +/* +* According to combination of +* MAX_ORDER and size of cyclic +* buffer, this clearing bit operation
makedumpfile.c __exclude_unnecessary_pages(unsigned long mem_map, · line 4089
Quoted 10 lines
makedumpfile.c +6
-0+* can overrun the cyclic buffer. +* +* See check_cyclic_buffer_overrun() +* for the detail. +*/ clear_bit_on_2nd_bitmap_for_kernel (pfn + i); + } pfn_free += i; }/*
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
HATAYAMA Daisuke
Reviewed-byRe: [PATCH v2 10/10] Warn cyclic buffer overrun and correct it if possible
Quoted 6 lines · Atsushi Kumagai
Hello HATAYAMA-san,
Quoted 3 lines · HATAYAMA Daisuke
Clearling bits on cyclic buffer can overrun the cyclic buffer according to some combination of
MAX_ORDERand cyclic buffer size.
Quoted 7 lines
Quoted 3 lines
The cyclic buffer size is corrected if possible.
Signed-off-by HATAYAMA Daisuke <[email protected]>I know it's so late, I found that updating
pfn_cyclicis missing. It can cause memory corruption.
Hello Kumagai-san,
It might be even better to introduce some kind of helper function that sets up these cyclic-mode-related parameters and then to use it in initial() and check_cyclic_buffer_overrun().
Signature · 8 lines
Thanks. HATAYAMA, Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 10/10] Warn cyclic buffer overrun and correct it if possible
Quoted 15 lines · HATAYAMA Daisuke
Quoted 6 lines · Atsushi Kumagai
Hello HATAYAMA-san,
Quoted 3 lines · HATAYAMA Daisuke
Clearling bits on cyclic buffer can overrun the cyclic buffer according to some combination of
MAX_ORDERand cyclic buffer size.Quoted 7 lines
Quoted 3 lines
The cyclic buffer size is corrected if possible.
Signed-off-by HATAYAMA Daisuke <[email protected]>I know it's so late, I found that updating
pfn_cyclicis missing. It can cause memory corruption.
Quoted 8 lines
Hello Kumagai-san,
Reviewed-by HATAYAMA Daisuke <[email protected]>It might be even better to introduce some kind of helper function that sets up these cyclic-mode-related parameters and then to use it in
initial()andcheck_cyclic_buffer_overrun().
Hello Kumaga-san,
I found one more bug. Could you review it?
From c98375b9af6c19dff88823166eaf13674b4a47ec Mon Sep 17 00:00:00 2001
From: HATAYAMA Daisuke <[email protected]>
Date: Thu, 12 Sep 2013 10:35:17 +0900
Subject: [PATCH] Use divideup() to calculate maximum required bitmap size
Currently, check_cyclic_buffer_overrun() wrongly calculates maximum bitmap size required to represent maximum block size managed by buddy allocator with roundup(). Then, max_block_size is BITPERBYTE-time larger than its correct size. As a result, although the bug never affect free-page filtering since roundup(max_order_nr_pages, BITPERBYTE) is a multiple of divideup(max_order_nr_pages, BITPERBYTE), the following sanity check, (max_block_size > info->bufsize_cyclic), and recalculation of info->bufsize_cyclic becomes BITPERBYTE-time conservative and inefficient.
--- makedumpfile.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
makedumpfile.c +3 -3
@@ -4078,10 +4078,10 @@ check_cyclic_buffer_overrun(void) {int max_order =ARRAY_LENGTH (zone.free_area);int max_order_nr_pages =1 << (max_order -1 ); -unsigned long max_block_size =roundup (max_order_nr_pages,BITPERBYTE ); +unsigned long max_block_size =divideup (max_order_nr_pages, +BITPERBYTE ); -if (info->bufsize_cyclic % -roundup (max_order_nr_pages,BITPERBYTE )) { +if (info->bufsize_cyclic % max_block_size) {unsigned long bufsize;if (max_block_size > info->bufsize_cyclic) {
Signature · 13 lines
1.8.3.1 -- Thanks. HATAYAMA, Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 10/10] Warn cyclic buffer overrun and correct it if possible
Hello HATAYAMA-san,
Quoted 20 lines · HATAYAMA Daisuke
Quoted 19 lines · HATAYAMA Daisuke
Quoted 6 lines · Atsushi Kumagai
Hello HATAYAMA-san,
Quoted 3 lines · HATAYAMA Daisuke
Clearling bits on cyclic buffer can overrun the cyclic buffer according to some combination of
MAX_ORDERand cyclic buffer size.Quoted 7 lines
Quoted 3 lines
The cyclic buffer size is corrected if possible.
Signed-off-by HATAYAMA Daisuke <[email protected]>I know it's so late, I found that updating
pfn_cyclicis missing. It can cause memory corruption.Hello Kumagai-san,
Reviewed-by HATAYAMA Daisuke <[email protected]>
Quoted 8 lines
Quoted 4 lines
It might be even better to introduce some kind of helper function that sets up these cyclic-mode-related parameters and then to use it in
initial()andcheck_cyclic_buffer_overrun().Hello Kumaga-san,
I found one more bug. Could you review it?
Thanks, acked and pushed to devel branch.
Atsushi Kumagai
Quoted 29 lines
From
c98375b9af6c19dff88823166eaf13674b4a47ecMon Sep 17 00:00:00 2001
From: HATAYAMA Daisuke <[email protected]>
Date: Thu, 12 Sep 2013 10:35:17 +0900
Subject: [PATCH] Usedivideup()to calculate maximum required bitmap sizeCurrently,
check_cyclic_buffer_overrun()wrongly calculates maximum bitmap size required to represent maximum block size managed by buddy allocator withroundup(). Then,max_block_sizeis BITPERBYTE-time larger than its correct size. As a result, although the bug never affect free-page filtering sinceroundup(max_order_nr_pages, BITPERBYTE)is a multiple ofdivideup(max_order_nr_pages, BITPERBYTE), the following sanity check, (max_block_size>info->bufsize_cyclic), and recalculation ofinfo->bufsize_cyclicbecomes BITPERBYTE-time conservative and inefficient.Signed-off-by HATAYAMA Daisuke <[email protected]>--- makedumpfile.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)makedumpfile.c +0
-0@@ -4078,10 +4078,10 @@ check_cyclic_buffer_overrun(void) {int max_order =ARRAY_LENGTH (zone.free_area);int max_order_nr_pages =1 << (max_order -1 );
makedumpfile.c check_cyclic_buffer_overrun(void) · line 4087
Quoted 10 lines
makedumpfile.c +3
-3-unsigned long max_block_size =roundup (max_order_nr_pages,BITPERBYTE ); +unsigned long max_block_size =divideup (max_order_nr_pages, +BITPERBYTE ); -if (info->bufsize_cyclic % -roundup (max_order_nr_pages,BITPERBYTE )) { +if (info->bufsize_cyclic % max_block_size) {unsigned long bufsize;if (max_block_size > info->bufsize_cyclic) {
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello HATAYAMA-san,
Quoted 5 lines · On Fri, 16 Nov 2012 14:01:30 +0900 HATAYAMA Daisuke…
This patch set implements filtering free pages looking up
mem_maparray instead of free lists. This is compatible for cyclic mode because looking upmem_mapcan be divided into cycles. On the other hand, dividing free pages contaiend in free lists is difficult since they are not sorted in physical-address order.
Thank you very much for your hard work.
I will release v1.5.1-rc with this patch set soon.
Thanks
Atsushi Kumagai
Quoted 86 lines
* Changes
v1 => v2:
- If debuginfo is not available enough, switch logic to freelist one. On v1, free page filtering was disabled in this case.
- Add hard-coded values in wider kernel versions.
- If some free pages possibly fail to be filtered, try to correct cyclic buffer size appropreately.
- Correct the comment explaining the cyclic buffer overrun, which was broken on v1.RFC => v1:
- Logic is automatically selected at runtime according to the current mode. In cyclic mode,mem_maparray logic is used. In non-cyclic mode, free list logic is used.
- The RFC version is:http://lists.infradead.org/pipermail/kexec/2012-June/006441.html* TODO
Add the following values in VMCOREINFO on the upstream kernel. These are used in the
mem_maplogic.-
OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
-NUMBER(PG_buddy)
-NUMBER(PG_slab)
-NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)* Test
I tested this patch set on the following kernel versions.
- 3.4
- 3.1
- 2.6.38
- 2.6.32
- 2.6.18On the test, I manually specified VMCOREINFO while extending it with the following values according to the kernel versions.
- 3.1, 3.4
NUMBER(PG_slab)=7
SIZE(pageflags)=4
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128- 2.6.38
SIZE(pageflags)=4
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-2- 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4- 2.6.18
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16---
HATAYAMA Daisuke (10):
Warn cyclic buffer overrun and correct it if possible Addpage_is_buddyfor old kernels
Addpage_is_buddyforPG_buddy
Addpage_is_buddyfor recent kernels
Exclude free pages by looking upmem_maparray
Add hardcoded page flag values
Add debuginfo-related processing for VMCOREINFO/VMLINUX Add new parameters to various tables
Add debuginfo interface for enum type size
Move page flags setup for old kernels after debuginfo initialization
Quoted 10 lines
dwarf_info.c | 18 ++++ dwarf_info.h | 1 makedumpfile.c | 226 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- makedumpfile.h | 35 ++++++++- 4 files changed, 269 insertions(+), 11 deletions(-)Signature · 2 lines
Thanks. HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec