Support free page filtering looking up mem_map array
Subsystems (2)
cover Support free page filtering looking up mem_map array
Sorry for late posting. I made RFC patch set for free page filtering looking up mem_map array. Unlike exiting method looking up free page list, this is done in constant space.
I intend this patch set to be merged with Kumagai-san's cyclic patch set, so I mark these with RFC. See TODO below. Also, I have yet to test the logic for old kernels from v2.6.15 to v2.6.17.
This new free page filtering needs the following values.
- OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
- NUMBER(PG_buddy)
- NUMBER(PG_slab)
- NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)
Unfortunately, OFFSET(_mapcount) and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.
To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.
TODO
1. Add new values in VMCOREINFO on the upstream kernel.
2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.
3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.
---
HATAYAMA Daisuke (10):
Add page_is_buddy for old kernels
Add page_is_buddy for PG_buddy
Add page_is_buddy for recent kernels
Add excldue free pages by looking up mem_map array
Add command-line processing for free page filtering looking up mem_map array Add page flag values as hardcoded values
Add debuginfo-related processing for VMCOREINFO/VMLINUX
Add new parameters for various tables
Add debuginfo interface for enum type size
Move page flags setup for old kernels after debuginfo initialization
dwarf_info.c | 29 +++++++++++---- dwarf_info.h | 1 + makedumpfile.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- makedumpfile.h | 34 ++++++++++++++++- 4 files changed, 158 insertions(+), 17 deletions(-)
Signature · 7 lines
Thanks. HATAYAMA, Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 1/10 Move page flags setup for old kernels after debuginfo initialization
The hard coded values need to be used only if the corresponding values are not specified by debuginfo. So it should be done after debuginfo initialization.
---
makedumpfile.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
makedumpfile.c +3 -2
@@ -2643,8 +2643,6 @@ initial(void) debug_info = TRUE; } -if (!get_value_for_old_linux ()) -return FALSE; out:if (!info->page_size) {/*
@@ -2706,6 +2704,9 @@ out:return FALSE; } +if (!get_value_for_old_linux ()) +return FALSE; +return TRUE; }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 2/10 Add debuginfo interface for enum type size
In reply to HATAYAMA DaisukeThis is needed in this patch set to determine whether or not a certain enumeration type exists from a given debuginfo. The interface is a simple extension from the existing one for enumeration value.
---
dwarf_info.c | 29 +++++++++++++++++++++-------- dwarf_info.h | 1 + makedumpfile.h | 13 +++++++++++-- 3 files changed, 33 insertions(+), 10 deletions(-)
dwarf_info.c +21 -8
@@ -75,7 +75,8 @@ is_search_structure(int cmd)static int is_search_number (int cmd) { -if (cmd ==DWARF_INFO_GET_ENUM_NUMBER ) +if ((cmd ==DWARF_INFO_GET_ENUM_NUMBER ) + || (cmd ==DWARF_INFO_GET_ENUMERATION_TYPE_SIZE ))return TRUE;else return FALSE;
@@ -647,7 +648,7 @@ search_structure(Dwarf_Die *die, int *found)static void search_number (Dwarf_Die *die,int *found) { -int tag; +int tag, bytesize;Dwarf_Word const_value;Dwarf_Attribute attr;Dwarf_Die child, *walker;
@@ -658,6 +659,22 @@ search_number(Dwarf_Die *die, int *found)if (tag != DW_TAG_enumeration_type)continue ; +if (dwarf_info.cmd ==DWARF_INFO_GET_ENUMERATION_TYPE_SIZE ) { + name =dwarf_diename (die); + +if (!name ||strcmp (name, dwarf_info.struct_name)) +continue ; + +if ((bytesize =dwarf_bytesize (die)) <=0 ) +continue ; + + *found = TRUE; + + dwarf_info.struct_size = bytesize; + +return ; + } +if (dwarf_child (die, &child) !=0 )continue ;
@@ -1026,13 +1043,9 @@ out:* Get the size of structure. */ long -get_structure_size (char *structname,int flag_typedef) +get_structure_size (char *structname,int cmd) { -if (flag_typedef) - dwarf_info.cmd =DWARF_INFO_GET_TYPEDEF_SIZE ; -else - dwarf_info.cmd =DWARF_INFO_GET_STRUCT_SIZE ; - + dwarf_info.cmd = cmd; dwarf_info.struct_name = structname; dwarf_info.struct_size =NOT_FOUND_STRUCTURE ;
dwarf_info.h +1 -0
@@ -47,6 +47,7 @@ enum {DWARF_INFO_CHECK_SYMBOL_ARRAY_TYPE ,DWARF_INFO_GET_SYMBOL_TYPE ,DWARF_INFO_GET_MEMBER_TYPE , + DWARF_INFO_GET_ENUMERATION_TYPE_SIZE, };char *get_dwarf_module_name (void );
makedumpfile.h +11 -2
@@ -225,14 +225,23 @@ do { \#define ARRAY_LENGTH (X ) (array_table.X)#define SIZE_INIT (X ,Y ) \ do { \ - if ((SIZE(X) = get_structure_size(Y, 0)) == FAILED_DWARFINFO) \ + if ((SIZE(X) = get_structure_size(Y, DWARF_INFO_GET_STRUCT_SIZE)) \ + == FAILED_DWARFINFO) \ return FALSE; \ } while (0)#define TYPEDEF_SIZE_INIT (X ,Y ) \ do { \ - if ((SIZE(X) = get_structure_size(Y, 1)) == FAILED_DWARFINFO) \ + if ((SIZE(X) = get_structure_size(Y, DWARF_INFO_GET_TYPEDEF_SIZE)) \ + == FAILED_DWARFINFO) \ return FALSE; \ } while (0) +#define ENUM_TYPE_SIZE_INIT (X ,Y ) \ +do { \ + if ((SIZE(X) = get_structure_size(Y, \ + DWARF_INFO_GET_ENUMERATION_TYPE_SIZE)) \ + == FAILED_DWARFINFO) \ + return FALSE; \ + } while (0)#define OFFSET_INIT (X ,Y ,Z ) \ do { \ if ((OFFSET(X) = get_member_offset(Y, Z, DWARF_INFO_GET_MEMBER_OFFSET)) \
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 3/10 Add new parameters for various tables
In reply to HATAYAMA Daisuke---
makedumpfile.h | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
makedumpfile.h +8 -0
@@ -1073,6 +1073,8 @@ struct size_table {long cpumask_t;long kexec_segment;long elf64_hdr; + +long pageflags; };struct offset_table {
@@ -1081,6 +1083,8 @@ struct offset_table {long _count;long mapping;long lru; +long _mapcount; +long private; } page;struct mem_section {long section_mem_map;
@@ -1242,6 +1246,10 @@ struct number_table {long PG_lru;long PG_private;long PG_swapcache; +long PG_buddy; +long PG_slab; + +long PAGE_BUDDY_MAPCOUNT_VALUE ; };struct srcfile_table {
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 4/10 Add debuginfo-related processing for VMCOREINFO/VMLINUX
In reply to HATAYAMA Daisuke---
makedumpfile.c | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-)
makedumpfile.c +20 -0
@@ -876,6 +876,8 @@ get_structure_info(void)OFFSET_INIT (page._count,"page" ,"_count" );OFFSET_INIT (page.mapping,"page" ,"mapping" ); +OFFSET_INIT (page._mapcount,"page" ,"_mapcount" ); +OFFSET_INIT (page.private,"page" ,"private" );/* * On linux-2.6.16 or later, page.mapping is defined
@@ -974,6 +976,10 @@ get_structure_info(void)ENUM_NUMBER_INIT (PG_lru,"PG_lru" );ENUM_NUMBER_INIT (PG_private,"PG_private" );ENUM_NUMBER_INIT (PG_swapcache,"PG_swapcache" ); +ENUM_NUMBER_INIT (PG_buddy,"PG_buddy" ); +ENUM_NUMBER_INIT (PG_slab,"PG_slab" ); + +ENUM_TYPE_SIZE_INIT (pageflags,"pageflags" );TYPEDEF_SIZE_INIT (nodemask_t,"nodemask_t" );
@@ -1324,6 +1330,7 @@ write_vmcoreinfo_data(void)WRITE_STRUCTURE_SIZE ("list_head" , list_head);WRITE_STRUCTURE_SIZE ("node_memblk_s" , node_memblk_s);WRITE_STRUCTURE_SIZE ("nodemask_t" , nodemask_t); +WRITE_STRUCTURE_SIZE ("pageflags" , pageflags);/* * write the member offset of 1st kernel
@@ -1332,6 +1339,8 @@ write_vmcoreinfo_data(void)WRITE_MEMBER_OFFSET ("page._count" , page._count);WRITE_MEMBER_OFFSET ("page.mapping" , page.mapping);WRITE_MEMBER_OFFSET ("page.lru" , page.lru); +WRITE_MEMBER_OFFSET ("page._mapcount" , page._mapcount); +WRITE_MEMBER_OFFSET ("page.private" , page.private);WRITE_MEMBER_OFFSET ("mem_section.section_mem_map" , mem_section.section_mem_map);WRITE_MEMBER_OFFSET ("pglist_data.node_zones" , pglist_data.node_zones);
@@ -1376,6 +1385,10 @@ write_vmcoreinfo_data(void)WRITE_NUMBER ("PG_lru" , PG_lru);WRITE_NUMBER ("PG_private" , PG_private);WRITE_NUMBER ("PG_swapcache" , PG_swapcache); +WRITE_NUMBER ("PG_buddy" , PG_buddy); +WRITE_NUMBER ("PG_slab" , PG_slab); + +WRITE_NUMBER ("PAGE_BUDDY_MAPCOUNT_VALUE" ,PAGE_BUDDY_MAPCOUNT_VALUE );/* * write the source file of 1st kernel
@@ -1623,11 +1636,14 @@ read_vmcoreinfo(void)READ_STRUCTURE_SIZE ("list_head" , list_head);READ_STRUCTURE_SIZE ("node_memblk_s" , node_memblk_s);READ_STRUCTURE_SIZE ("nodemask_t" , nodemask_t); +READ_STRUCTURE_SIZE ("pageflags" , pageflags);READ_MEMBER_OFFSET ("page.flags" , page.flags);READ_MEMBER_OFFSET ("page._count" , page._count);READ_MEMBER_OFFSET ("page.mapping" , page.mapping);READ_MEMBER_OFFSET ("page.lru" , page.lru); +READ_MEMBER_OFFSET ("page._mapcount" , page._mapcount); +READ_MEMBER_OFFSET ("page.private" , page.private);READ_MEMBER_OFFSET ("mem_section.section_mem_map" , mem_section.section_mem_map);READ_MEMBER_OFFSET ("pglist_data.node_zones" , pglist_data.node_zones);
@@ -1664,6 +1680,10 @@ read_vmcoreinfo(void)READ_NUMBER ("PG_lru" , PG_lru);READ_NUMBER ("PG_private" , PG_private);READ_NUMBER ("PG_swapcache" , PG_swapcache); +READ_NUMBER ("PG_slab" , PG_slab); +READ_NUMBER ("PG_buddy" , PG_buddy); + +READ_NUMBER ("PAGE_BUDDY_MAPCOUNT_VALUE" ,PAGE_BUDDY_MAPCOUNT_VALUE );READ_SRCFILE ("pud_t" , pud_t);
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 5/10 Add page flag values as hardcoded values
In reply to HATAYAMA DaisukeThese should normally be exported as VMCOREINFO from kernel, but we cannot modify old kernels and still they are easy to be determined becasue of a small kernel version dependency with no symbol and type information.
- PG_slab has had the same value 7 since v2.6.15, and
- PG_buddy has been defined as macro value from v2.6.17 to v2.6.26
On other versions, pageflags enumeration type is introduced and at the version, PG_buddy's value depends on CONFIG_PAGEFLAGS_EXTEND; luckily, this can be determined by looking at error_states array, for example, but I don't implement it in this patch.
---
makedumpfile.c | 6 ++++++ makedumpfile.h | 3 +++ 2 files changed, 9 insertions(+), 0 deletions(-)
makedumpfile.c +6 -0
@@ -1179,6 +1179,12 @@ get_value_for_old_linux(void)NUMBER (PG_private) = PG_private_ORIGINAL;if (NUMBER (PG_swapcache) ==NOT_FOUND_NUMBER )NUMBER (PG_swapcache) = PG_swapcache_ORIGINAL; +if (NUMBER (PG_slab) ==NOT_FOUND_NUMBER ) +NUMBER (PG_slab) = PG_slab_ORIGINAL; +if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER + && info->kernel_version >=KERNEL_VERSION (2 ,6 ,17 ) + && info->kernel_version <=KERNEL_VERSION (2 ,6 ,26 )) +NUMBER (PG_buddy) = PG_buddy_v2_6_17_to_v2_6_26;return TRUE; }
makedumpfile.h +3 -0
@@ -68,9 +68,12 @@ int get_mem_type(void);* The following values are for linux-2.6.25 or former. */ #define PG_lru_ORIGINAL (5) +#define PG_slab_ORIGINAL (7)#define PG_private_ORIGINAL (11)/* Has something at ->private */ #define PG_swapcache_ORIGINAL (15)/* Swap page: swp_entry_t in private */ +#define PG_buddy_v2_6_17_to_v2_6_26 (19) +#define PAGE_MAPPING_ANON (1)#define LSEEKED_BITMAP (1)
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 6/10 Add command-line processing for free page filtering looking up mem_map array
In reply to HATAYAMA DaisukeI choose dump level 32 for free page filtering looking up mem_map array. But I don't mean this is the final version now, I mean experimental use for RFC, only. I prefer the setting that this logic looking up mem_map array is automatically used if dump level 16 and --cyclic option are specified at the same time.
---
makedumpfile.c | 3 ++- makedumpfile.h | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-)
makedumpfile.c +2 -1
@@ -3853,7 +3853,8 @@ create_2nd_bitmap(void)*/ if (info->dump_level &DL_EXCLUDE_CACHE || info->dump_level &DL_EXCLUDE_CACHE_PRI || - info->dump_level &DL_EXCLUDE_USER_DATA ) { + info->dump_level &DL_EXCLUDE_USER_DATA || + info->dump_level &DL_EXCLUDE_FREE_CONST ) {if (!exclude_unnecessary_pages ()) {ERRMSG ("Can't exclude unnecessary pages.\n" );return FALSE;
makedumpfile.h +3 -1
@@ -146,7 +146,7 @@ isAnon(unsigned long mapping)* Dump Level */ #define MIN_DUMP_LEVEL (0) -#define MAX_DUMP_LEVEL (31) +#define MAX_DUMP_LEVEL (63)#define NUM_ARRAY_DUMP_LEVEL (MAX_DUMP_LEVEL + 1)/* enough to allocate all the dump_level */ #define DL_EXCLUDE_ZERO (0x001)/* Exclude Pages filled with Zeros */
@@ -156,6 +156,8 @@ isAnon(unsigned long mapping)with Private Pages */ #define DL_EXCLUDE_USER_DATA (0x008)/* Exclude UserProcessData Pages */ #define DL_EXCLUDE_FREE (0x010)/* Exclude Free Pages */ +#define DL_EXCLUDE_FREE_CONST (0x020)/* Exclude Free Pages looking +up mem_map */ /*
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 7/10 Add excldue free pages by looking up mem_map array
In reply to HATAYAMA DaisukeAdd free page filtering logic in the processing looking up mem_map array. page_is_buddy handler is newly introduced, which abstracts condition of determining buddy page that varies depending on kernel versions. On makedumpfile suppoting range of kernel versions, there are three kinds of page_is_buddy, which are introduced in later patches. Also, _mapcount and private fields of page struct are newly introduced, which are used in the page_is_buddy handlers.
---
makedumpfile.c | 21 ++++++++++++++++++--- makedumpfile.h | 6 ++++++ 2 files changed, 24 insertions(+), 3 deletions(-)
makedumpfile.c +18 -3
@@ -3689,8 +3689,8 @@ __exclude_unnecessary_pages(unsigned long mem_map,unsigned long long pfn_read_start, pfn_read_end, index_pg;unsigned char page_cache[SIZE (page) *PGMM_CACHED ];unsigned char *pcache; -unsigned int _count; -unsigned long flags, mapping; +unsigned int _count, _mapcount; +unsigned long flags, mapping, private;/* * Refresh the buffer of struct page, when changing mem_map.
@@ -3738,11 +3738,26 @@ __exclude_unnecessary_pages(unsigned long mem_map, flags =ULONG (pcache +OFFSET (page.flags)); _count =UINT (pcache +OFFSET (page._count)); mapping =ULONG (pcache +OFFSET (page.mapping)); + _mapcount =UINT (pcache +OFFSET (page._mapcount)); + private =ULONG (pcache +OFFSET (page.private)); +/* +* Exclude the free page managed by a buddy. +*/ +if ((info->dump_level &DL_EXCLUDE_FREE_CONST ) + && info->page_is_buddy + && info->page_is_buddy (flags, _mapcount, private, + _count)) { +int i; + +for (i =0 ; i < (1 <<private); ++i) +clear_bit_on_2nd_bitmap_for_kernel (pfn + i); + pfn_free += i; + }/* * Exclude the cache page without the private page. */ -if ((info->dump_level &DL_EXCLUDE_CACHE ) +else if ((info->dump_level &DL_EXCLUDE_CACHE ) && (isLRU (flags) ||isSwapCache (flags)) && !isPrivate (flags) && !isAnon (mapping)) {clear_bit_on_2nd_bitmap_for_kernel (pfn);
makedumpfile.h +6 -0
@@ -943,6 +943,12 @@ struct DumpInfo {*/ int flag_sadump_diskset;enum sadump_format_type flag_sadump;/* sadump format type */ + +/* +* for filtering free pages managed by buddy system: +*/ +int (*page_is_buddy)(unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count); };extern struct DumpInfo *info;
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 8/10 Add page_is_buddy for recent kernels
In reply to HATAYAMA DaisukeOn kernels from v2.6.38 and later kernels, buddy page is marked by _mapcount == PAGE_BUDDY_MAPCOUNT_VALUE, which varies once as follows:
kernel version | PAGE_BUDDY_MAPCOUNT_VALUE
------------------+--------------------------
v2.6.38 | -2
v2.6.39 and later | -128
One more notice is that _mapcount shares its memory with other fields for SLAB/SLUB when PG_slab is set. We need to check if PG_slab is set or not before looking up _mapcount value.
---
makedumpfile.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)
makedumpfile.c +30 -0
@@ -55,6 +55,8 @@ do { \ *ptr_long_table = value; \ }while (0 ) +static void setup_page_is_buddy(void); +void initialize_tables (void ) {
@@ -2733,6 +2735,9 @@ out:if (!get_value_for_old_linux ())return FALSE; +if (info->dump_level &DL_EXCLUDE_FREE_CONST ) +setup_page_is_buddy (); +return TRUE; }
@@ -3512,6 +3517,31 @@ exclude_free_page(void)return TRUE; } +static int +page_is_buddy_v3 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if (flags & (1UL <<NUMBER (PG_slab))) +return FALSE; + +if (_mapcount == (int )NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE )) +return TRUE; + +return FALSE; +} + +static void +setup_page_is_buddy (void ) +{ +if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER + &&SIZE (pageflags) !=NOT_FOUND_STRUCTURE + &&NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) + info->page_is_buddy = page_is_buddy_v3; + +MSG ("Can't select page_is_buddy handler; " +"filtering free pages is disabled.\n" ); +} +/* * If using a dumpfile in kdump-compressed format as a source file * instead of /proc/vmcore, 1st-bitmap of a new dumpfile must be
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 9/10 Add page_is_buddy for PG_buddy
In reply to HATAYAMA DaisukeOn kernels from v2.6.18 to v2.6.37, buddy page is marked by the PG_buddy flag.
kernel version | PG_buddy
------------------ +---------------------------------
v2.6.17 to v2.6.26 | 19
v2.6.27 to v2.6.37 | 19 if CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwise
Note for hard coding: it's possible to determine whether CONFIG_PAGEFLAGS_EXTEND=yes or not by looking at error_status array defined in mm/memory-failure.c, for example.
---
makedumpfile.c | 27 ++++++++++++++++++++------- 1 files changed, 20 insertions(+), 7 deletions(-)
makedumpfile.c +20 -7
@@ -3518,6 +3518,16 @@ exclude_free_page(void) }static int +page_is_buddy_v2 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if (flags & (1UL <<NUMBER (PG_buddy))) +return TRUE; + +return FALSE; +} + +static int page_is_buddy_v3 (unsigned long flags,unsigned int _mapcount,unsigned long private,unsigned int _count) {
@@ -3533,13 +3543,16 @@ page_is_buddy_v3(unsigned long flags, unsigned int _mapcount,static void setup_page_is_buddy (void ) { -if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER - &&SIZE (pageflags) !=NOT_FOUND_STRUCTURE - &&NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) - info->page_is_buddy = page_is_buddy_v3; - -MSG ("Can't select page_is_buddy handler; " -"filtering free pages is disabled.\n" ); +if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER ) { +if (SIZE (pageflags) !=NOT_FOUND_STRUCTURE + &&NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) + info->page_is_buddy = page_is_buddy_v3; +else { +MSG ("Can't select page_is_buddy handler; " +"filtering free pages is disabled.\n" ); + } + }else + info->page_is_buddy = page_is_buddy_v2; }/*
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH 10/10 Add page_is_buddy for old kernels
In reply to HATAYAMA DaisukeOn kernels from v2.6.15 to v2.6.17 buddy page is marked by the condition that PG_private flag is set and _count == 0.
Unfortunately, I have yet to test this logic on these kernel versions simply because I've been failing to boot them on my box.
Note that on these kernels, free list can be corrupted due to the bug that the above two conditions are not checked atomically. The reason why PG_buddy was introduced is a fix for this bug. Thus, the bug can also affect the logic based on mem_map array, which we cannot avoid definitely.
---
makedumpfile.c | 17 +++++++++++++++-- 1 files changed, 15 insertions(+), 2 deletions(-)
makedumpfile.c +15 -2
@@ -3518,6 +3518,18 @@ exclude_free_page(void) }static int +page_is_buddy_v1 (unsigned long flags,unsigned int _mapcount, +unsigned long private,unsigned int _count) +{ +if ((flags & (1UL <<NUMBER (PG_private))) + && _count ==0 + && private <=ARRAY_LENGTH (zone.free_area)) +return TRUE; + +return FALSE; +} + +static int page_is_buddy_v2 (unsigned long flags,unsigned int _mapcount,unsigned long private,unsigned int _count) {
@@ -3544,8 +3556,9 @@ static voidsetup_page_is_buddy (void) {if (NUMBER (PG_buddy) ==NOT_FOUND_NUMBER ) { -if (SIZE (pageflags) !=NOT_FOUND_STRUCTURE - &&NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) +if (SIZE (pageflags) ==NOT_FOUND_STRUCTURE ) + info->page_is_buddy = page_is_buddy_v1; +else if (NUMBER (PAGE_BUDDY_MAPCOUNT_VALUE ) !=NOT_FOUND_NUMBER ) info->page_is_buddy = page_is_buddy_v3;else { MSG("Can't select page_is_buddy handler; "
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
From: HATAYAMA Daisuke <[email protected]>
Subject: [RFC PATCH 00/10] Support free page filtering looking up mem_map array
Date: Fri, 29 Jun 2012 02:37:57 +0900
Quoted 9 lines
This new free page filtering needs the following values.
-
OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
-NUMBER(PG_buddy)
-NUMBER(PG_slab)
-NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)
Quoted 8 lines
Unfortunately,
OFFSET(_mapcount)and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.
Concretely, I added the following values in VMCOREINFO generated with -g option or using strings command directly to vmcore.
* 3.1.0-7.fc16.x86_64
NUMBER(PG_slab)=7
SIZE(pageflags)=4
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
* 2.6.38
SIZE(pageflags)=4
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-2
* 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4
* 2.6.18
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello HATAYAMA-san,
Quoted 34 lines · On Fri, 29 Jun 2012 02:37:57 +0900 HATAYAMA Daisuke…
Sorry for late posting. I made RFC patch set for free page filtering looking up
mem_maparray. Unlike exiting method looking up free page list, this is done in constant space.I intend this patch set to be merged with Kumagai-san's cyclic patch set, so I mark these with RFC. See TODO below. Also, I have yet to test the logic for old kernels from v2.6.15 to v2.6.17.
This new free page filtering needs the following values.
-
OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
-NUMBER(PG_buddy)
-NUMBER(PG_slab)
-NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)Unfortunately,
OFFSET(_mapcount)and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.
TODO
1. Add new values in VMCOREINFO on the upstream kernel.
2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.
Quoted 4 lines
3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.
Thank you always for your work.
I will review your patches and measure executing time with v2 patch of cyclic processing. If your patches are effective, then I will consider TODO above.
Please wait for a while.
Thanks
Atsushi Kumagai
Quoted 15 lines
---
HATAYAMA Daisuke (10):
Addpage_is_buddyfor old kernels
Addpage_is_buddyforPG_buddy
Addpage_is_buddyfor recent kernels
Add excldue free pages by looking upmem_maparray
Add command-line processing for free page filtering looking upmem_maparray Add page flag values as hardcoded values
Add debuginfo-related processing for VMCOREINFO/VMLINUX
Add new parameters for various tables
Add debuginfo interface for enum type size
Move page flags setup for old kernels after debuginfo initialization
Quoted 10 lines
dwarf_info.c | 29 +++++++++++---- dwarf_info.h | 1 + makedumpfile.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- makedumpfile.h | 34 ++++++++++++++++- 4 files changed, 158 insertions(+), 17 deletions(-)Signature · 2 lines
Thanks. HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello,
Quoted 44 lines · On Fri, 29 Jun 2012 15:23:37 +0900 Atsushi Kumagai…
Hello HATAYAMA-san,
Quoted 34 lines · On Fri, 29 Jun 2012 02:37:57 +0900 HATAYAMA Daisuke…
Sorry for late posting. I made RFC patch set for free page filtering looking up
mem_maparray. Unlike exiting method looking up free page list, this is done in constant space.I intend this patch set to be merged with Kumagai-san's cyclic patch set, so I mark these with RFC. See TODO below. Also, I have yet to test the logic for old kernels from v2.6.15 to v2.6.17.
This new free page filtering needs the following values.
-
OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
-NUMBER(PG_buddy)
-NUMBER(PG_slab)
-NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)Unfortunately,
OFFSET(_mapcount)and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.
TODO
1. Add new values in VMCOREINFO on the upstream kernel.
2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.
Quoted 4 lines
3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.
Quoted 6 lines
Thank you always for your work.
I will review your patches and measure executing time with v2 patch of cyclic processing. If your patches are effective, then I will consider TODO above.
Please wait for a while.
I did performance measuring with v2 patches of cyclic processing and HATAYAMA-san's patches. And I fixed v2 patches to reduce wasteful process, please see the end of this mail.
How to measure:
- The source data is a vmcore saved on the disk, the size is 5,099,292,912 bytes.
- makedumpfile writes dumpfile to the same disk as the source data.
- I measured the execution time with time(1) and adopted the average of 5 times.
Test Cases:
- _mapcount:
This logic is implemented by HATAYAMA-san.
This logic looks up members of page structure instead of free_list to filter out free pages.
- free_list
v2 patches choose this logic.
This logic looks up whole free_list to filter out free pages every cycle.
- upstream (v1.4.4):
This logic is NOT CYCLIC, uses temporary bitmap file as usual.
Example:
- _mapcount:
$ time makedumpfile --cyclic -d32 -i vmcoreinfo vmcore dumpfile.d32
- free_list:
$ time makedumpfile --cyclic -d16 -i vmcoreinfo vmcore dumpfile.d16
- upstream:
$ time makedumpfile -d16 -i vmcoreinfo vmcore dumpfile.d16
Result:
a) exclude only free pages
BUFSIZE_CYCLIC | | execution time [sec]
[byte] | num of cycle | _mapcount (-d32) | free_list (-d16) | upstream (-d16)
------------------+----------------+------------------+------------------+-----------------
1024 | 152 | 20.5204 | 28.8028 | -
1024 * 10 | 16 | 14.7460 | 18.7904 | -
1024 * 100 | 2 | 14.3962 | 17.9356 | -
1024 * 200 | 1 | 14.3166 | 17.8762 | 17.7928b) exclude all unnecessary pages
BUFSIZE_CYCLIC | | execution time [sec]
[byte] | num of cycle | _mapcount (-d47) | free_list (-d31) | upstream (-d31)
------------------+----------------+------------------+------------------+-----------------
1024 | 152 | 11.5086 | 27.2906 | -
1024 * 10 | 16 | 6.0740 | 10.9998 | -
1024 * 100 | 2 | 5.7928 | 9.1534 | -
1024 * 200 | 1 | 5.6378 | 8.9924 | 5.0516I expected that the difference of execution time increases based on number of cycle, because I think that repeating scanning free_list is high cost.
And according to result, it seems right.
_mapcount logic can be expected good performance in almost case when --cyclic is specified. So, I think that making effort to resolve TODO is worth for us.
However, I think more consideration is needed to decide whether to choose _mapcount logic or not when --cyclic isn't specified.
Quoted 3 lines
TODO
1. Add new values in VMCOREINFO on the upstream kernel.
I will send this request to the upstream kernel.
Quoted 4 lines
2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.
I think 2) is reasonable from the result.
Quoted 4 lines
3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.
I don't want to choose 2), I think it's inefficient.
I want to require that users prepare VMCOREINFO file with -g option from vmlinux, if cyclic processing is needed.
Do you have any comments ?
Thanks
Atsushi Kumagai
makedumpfile.c +19 -8
@@ -3824,6 +3824,9 @@ __exclude_unnecessary_pages(unsigned long mem_map,for (pfn = pfn_start; pfn < pfn_end; pfn++, mem_map +=SIZE (page)) { +if (info->flag_cyclic && !is_cyclic_region (pfn)) +continue ; +/* * Exclude the memory hole. */
@@ -3960,17 +3963,25 @@ exclude_unnecessary_pages_cyclic(void)if (!exclude_free_page ())return FALSE; -for (mm =0 ; mm < info->num_mem_map; mm++) { +/* +* Exclude cache pages, cache private pages, user data pages, and free pages. +*/ +if (info->dump_level &DL_EXCLUDE_CACHE || + info->dump_level &DL_EXCLUDE_CACHE_PRI || + info->dump_level &DL_EXCLUDE_USER_DATA || + info->dump_level &DL_EXCLUDE_FREE_CONST ) { +for (mm =0 ; mm < info->num_mem_map; mm++) { - mmd = &info->mem_map_data[mm]; + mmd = &info->mem_map_data[mm]; -if (mmd->mem_map ==NOT_MEMMAP_ADDR ) -continue ; +if (mmd->mem_map ==NOT_MEMMAP_ADDR ) +continue ; -if (mmd->pfn_end >= info->cyclic_start_pfn || mmd->pfn_start <= info->cyclic_end_pfn) { -if (!__exclude_unnecessary_pages (mmd->mem_map, - mmd->pfn_start, mmd->pfn_end)) -return FALSE; +if (mmd->pfn_end >= info->cyclic_start_pfn || mmd->pfn_start <= info->cyclic_end_pfn) { +if (!__exclude_unnecessary_pages (mmd->mem_map, + mmd->pfn_start, mmd->pfn_end)) +return FALSE; + } } }
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec