Skip to contentSkip to conversation

Support free page filtering looking up mem_map array

RFC PATCH In review HATAYAMA Daisuke 2 participants 14 messages Jun 28, 2012 – Jul 13, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Support free page filtering looking up mem_map array

Sorry for late posting. I made RFC patch set for free page filtering looking up mem_map array. Unlike exiting method looking up free page list, this is done in constant space.

I intend this patch set to be merged with Kumagai-san's cyclic patch set, so I mark these with RFC. See TODO below. Also, I have yet to test the logic for old kernels from v2.6.15 to v2.6.17.

This new free page filtering needs the following values.

- OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
- NUMBER(PG_buddy)
- NUMBER(PG_slab)
- NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)

Unfortunately, OFFSET(_mapcount) and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.

To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.

TODO

1. Add new values in VMCOREINFO on the upstream kernel.

2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.

3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.

---

HATAYAMA Daisuke (10):
Add page_is_buddy for old kernels
Add page_is_buddy for PG_buddy
Add page_is_buddy for recent kernels
Add excldue free pages by looking up mem_map array
Add command-line processing for free page filtering looking up mem_map array Add page flag values as hardcoded values
Add debuginfo-related processing for VMCOREINFO/VMLINUX
Add new parameters for various tables
Add debuginfo interface for enum type size
Move page flags setup for old kernels after debuginfo initialization

 dwarf_info.c   |   29 +++++++++++----
 dwarf_info.h   |    1 +
 makedumpfile.c |  111 +++++++++++++++++++++++++++++++++++++++++++++++++++++---
 makedumpfile.h |   34 ++++++++++++++++-
 4 files changed, 158 insertions(+), 17 deletions(-)
Signature · 7 lines
Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

RFC PATCH 1/10 Move page flags setup for old kernels after debuginfo initialization

The hard coded values need to be used only if the corresponding values are not specified by debuginfo. So it should be done after debuginfo initialization.

---

 makedumpfile.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
makedumpfile.c +3 -2
index d024e95..a6b3de7 100644
@@ -2643,8 +2643,6 @@ initial(void)
 		debug_info = TRUE;
 	}
 
-	if (!get_value_for_old_linux())
-		return FALSE;
 out:
 	if (!info->page_size) {
 		/*
@@ -2706,6 +2704,9 @@ out:
 			return FALSE;
 	}
 
+	if (!get_value_for_old_linux())
+		return FALSE;
+
 	return TRUE;
 }
 

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

RFC PATCH 2/10 Add debuginfo interface for enum type size

In reply to HATAYAMA Daisuke

This is needed in this patch set to determine whether or not a certain enumeration type exists from a given debuginfo. The interface is a simple extension from the existing one for enumeration value.

---

 dwarf_info.c   |   29 +++++++++++++++++++++--------
 dwarf_info.h   |    1 +
 makedumpfile.h |   13 +++++++++++--
 3 files changed, 33 insertions(+), 10 deletions(-)
dwarf_info.c +21 -8
index 1429858..98efc26 100644
@@ -75,7 +75,8 @@ is_search_structure(int cmd)
 static int
 is_search_number(int cmd)
 {
-	if (cmd == DWARF_INFO_GET_ENUM_NUMBER)
+	if ((cmd == DWARF_INFO_GET_ENUM_NUMBER)
+	    || (cmd == DWARF_INFO_GET_ENUMERATION_TYPE_SIZE))
 		return TRUE;
 	else
 		return FALSE;
@@ -647,7 +648,7 @@ search_structure(Dwarf_Die *die, int *found)
 static void
 search_number(Dwarf_Die *die, int *found)
 {
-	int tag;
+	int tag, bytesize;
 	Dwarf_Word const_value;
 	Dwarf_Attribute attr;
 	Dwarf_Die child, *walker;
@@ -658,6 +659,22 @@ search_number(Dwarf_Die *die, int *found)
 		if (tag != DW_TAG_enumeration_type)
 			continue;
 
+		if (dwarf_info.cmd == DWARF_INFO_GET_ENUMERATION_TYPE_SIZE) {
+			name = dwarf_diename(die);
+
+			if (!name || strcmp(name, dwarf_info.struct_name))
+				continue;
+
+			if ((bytesize = dwarf_bytesize(die)) <= 0)
+				continue;
+
+			*found = TRUE;
+
+			dwarf_info.struct_size = bytesize;
+
+			return;
+		}
+
 		if (dwarf_child(die, &child) != 0)
 			continue;
 
@@ -1026,13 +1043,9 @@ out:
  * Get the size of structure.
  */
 long
-get_structure_size(char *structname, int flag_typedef)
+get_structure_size(char *structname, int cmd)
 {
-	if (flag_typedef)
-		dwarf_info.cmd = DWARF_INFO_GET_TYPEDEF_SIZE;
-	else
-		dwarf_info.cmd = DWARF_INFO_GET_STRUCT_SIZE;
-
+	dwarf_info.cmd = cmd;
 	dwarf_info.struct_name = structname;
 	dwarf_info.struct_size = NOT_FOUND_STRUCTURE;
 
dwarf_info.h +1 -0
index 1e07484..b445738 100644
@@ -47,6 +47,7 @@ enum {
 	DWARF_INFO_CHECK_SYMBOL_ARRAY_TYPE,
 	DWARF_INFO_GET_SYMBOL_TYPE,
 	DWARF_INFO_GET_MEMBER_TYPE,
+	DWARF_INFO_GET_ENUMERATION_TYPE_SIZE,
 };
 
 char *get_dwarf_module_name(void);
makedumpfile.h +11 -2
index 6f5489d..95c0abc 100644
@@ -225,14 +225,23 @@ do { \
 #define ARRAY_LENGTH(X)		(array_table.X)
 #define SIZE_INIT(X, Y) \
 do { \
-	if ((SIZE(X) = get_structure_size(Y, 0)) == FAILED_DWARFINFO) \
+	if ((SIZE(X) = get_structure_size(Y, DWARF_INFO_GET_STRUCT_SIZE)) \
+	     == FAILED_DWARFINFO) \
 		return FALSE; \
 } while (0)
 #define TYPEDEF_SIZE_INIT(X, Y) \
 do { \
-	if ((SIZE(X) = get_structure_size(Y, 1)) == FAILED_DWARFINFO) \
+	if ((SIZE(X) = get_structure_size(Y, DWARF_INFO_GET_TYPEDEF_SIZE)) \
+	     == FAILED_DWARFINFO) \
 		return FALSE; \
 } while (0)
+#define ENUM_TYPE_SIZE_INIT(X, Y) \
+do { \
+	if ((SIZE(X) = get_structure_size(Y,	\
+		DWARF_INFO_GET_ENUMERATION_TYPE_SIZE))	\
+	     == FAILED_DWARFINFO)			\
+ 		return FALSE; \
+ } while (0)
 #define OFFSET_INIT(X, Y, Z) \
 do { \
 	if ((OFFSET(X) = get_member_offset(Y, Z, DWARF_INFO_GET_MEMBER_OFFSET)) \

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

RFC PATCH 3/10 Add new parameters for various tables

In reply to HATAYAMA Daisuke

---

 makedumpfile.h |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
makedumpfile.h +8 -0
index 95c0abc..2808871 100644
@@ -1073,6 +1073,8 @@ struct size_table {
 	long	cpumask_t;
 	long	kexec_segment;
 	long	elf64_hdr;
+
+	long	pageflags;
 };
 
 struct offset_table {
@@ -1081,6 +1083,8 @@ struct offset_table {
 		long	_count;
 		long	mapping;
 		long	lru;
+		long	_mapcount;
+		long	private;
 	} page;
 	struct mem_section {
 		long	section_mem_map;
@@ -1242,6 +1246,10 @@ struct number_table {
 	long	PG_lru;
 	long	PG_private;
 	long	PG_swapcache;
+	long	PG_buddy;
+	long	PG_slab;
+
+	long	PAGE_BUDDY_MAPCOUNT_VALUE;
 };
 
 struct srcfile_table {

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw

RFC PATCH 4/10 Add debuginfo-related processing for VMCOREINFO/VMLINUX

In reply to HATAYAMA Daisuke

---

 makedumpfile.c |   20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)
makedumpfile.c +20 -0
index a6b3de7..d8da608 100644
@@ -876,6 +876,8 @@ get_structure_info(void)
 	OFFSET_INIT(page._count, "page", "_count");
 
 	OFFSET_INIT(page.mapping, "page", "mapping");
+	OFFSET_INIT(page._mapcount, "page", "_mapcount");
+	OFFSET_INIT(page.private, "page", "private");
 
 	/*
 	 * On linux-2.6.16 or later, page.mapping is defined
@@ -974,6 +976,10 @@ get_structure_info(void)
 	ENUM_NUMBER_INIT(PG_lru, "PG_lru");
 	ENUM_NUMBER_INIT(PG_private, "PG_private");
 	ENUM_NUMBER_INIT(PG_swapcache, "PG_swapcache");
+	ENUM_NUMBER_INIT(PG_buddy, "PG_buddy");
+	ENUM_NUMBER_INIT(PG_slab, "PG_slab");
+
+	ENUM_TYPE_SIZE_INIT(pageflags, "pageflags");
 
 	TYPEDEF_SIZE_INIT(nodemask_t, "nodemask_t");
 
@@ -1324,6 +1330,7 @@ write_vmcoreinfo_data(void)
 	WRITE_STRUCTURE_SIZE("list_head", list_head);
 	WRITE_STRUCTURE_SIZE("node_memblk_s", node_memblk_s);
 	WRITE_STRUCTURE_SIZE("nodemask_t", nodemask_t);
+	WRITE_STRUCTURE_SIZE("pageflags", pageflags);
 
 	/*
 	 * write the member offset of 1st kernel
@@ -1332,6 +1339,8 @@ write_vmcoreinfo_data(void)
 	WRITE_MEMBER_OFFSET("page._count", page._count);
 	WRITE_MEMBER_OFFSET("page.mapping", page.mapping);
 	WRITE_MEMBER_OFFSET("page.lru", page.lru);
+	WRITE_MEMBER_OFFSET("page._mapcount", page._mapcount);
+	WRITE_MEMBER_OFFSET("page.private", page.private);
 	WRITE_MEMBER_OFFSET("mem_section.section_mem_map",
 	    mem_section.section_mem_map);
 	WRITE_MEMBER_OFFSET("pglist_data.node_zones", pglist_data.node_zones);
@@ -1376,6 +1385,10 @@ write_vmcoreinfo_data(void)
 	WRITE_NUMBER("PG_lru", PG_lru);
 	WRITE_NUMBER("PG_private", PG_private);
 	WRITE_NUMBER("PG_swapcache", PG_swapcache);
+	WRITE_NUMBER("PG_buddy", PG_buddy);
+	WRITE_NUMBER("PG_slab", PG_slab);
+
+	WRITE_NUMBER("PAGE_BUDDY_MAPCOUNT_VALUE", PAGE_BUDDY_MAPCOUNT_VALUE);
 
 	/*
 	 * write the source file of 1st kernel
@@ -1623,11 +1636,14 @@ read_vmcoreinfo(void)
 	READ_STRUCTURE_SIZE("list_head", list_head);
 	READ_STRUCTURE_SIZE("node_memblk_s", node_memblk_s);
 	READ_STRUCTURE_SIZE("nodemask_t", nodemask_t);
+	READ_STRUCTURE_SIZE("pageflags", pageflags);
 
 	READ_MEMBER_OFFSET("page.flags", page.flags);
 	READ_MEMBER_OFFSET("page._count", page._count);
 	READ_MEMBER_OFFSET("page.mapping", page.mapping);
 	READ_MEMBER_OFFSET("page.lru", page.lru);
+	READ_MEMBER_OFFSET("page._mapcount", page._mapcount);
+	READ_MEMBER_OFFSET("page.private", page.private);
 	READ_MEMBER_OFFSET("mem_section.section_mem_map",
 	    mem_section.section_mem_map);
 	READ_MEMBER_OFFSET("pglist_data.node_zones", pglist_data.node_zones);
@@ -1664,6 +1680,10 @@ read_vmcoreinfo(void)
 	READ_NUMBER("PG_lru", PG_lru);
 	READ_NUMBER("PG_private", PG_private);
 	READ_NUMBER("PG_swapcache", PG_swapcache);
+	READ_NUMBER("PG_slab", PG_slab);
+	READ_NUMBER("PG_buddy", PG_buddy);
+
+	READ_NUMBER("PAGE_BUDDY_MAPCOUNT_VALUE", PAGE_BUDDY_MAPCOUNT_VALUE);
 
 	READ_SRCFILE("pud_t", pud_t);
 

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw

RFC PATCH 5/10 Add page flag values as hardcoded values

In reply to HATAYAMA Daisuke

These should normally be exported as VMCOREINFO from kernel, but we cannot modify old kernels and still they are easy to be determined becasue of a small kernel version dependency with no symbol and type information.

- PG_slab has had the same value 7 since v2.6.15, and

- PG_buddy has been defined as macro value from v2.6.17 to v2.6.26

On other versions, pageflags enumeration type is introduced and at the version, PG_buddy's value depends on CONFIG_PAGEFLAGS_EXTEND; luckily, this can be determined by looking at error_states array, for example, but I don't implement it in this patch.

---

 makedumpfile.c |    6 ++++++
 makedumpfile.h |    3 +++
 2 files changed, 9 insertions(+), 0 deletions(-)
makedumpfile.c +6 -0
index d8da608..bed74df 100644
@@ -1179,6 +1179,12 @@ get_value_for_old_linux(void)
 		NUMBER(PG_private) = PG_private_ORIGINAL;
 	if (NUMBER(PG_swapcache) == NOT_FOUND_NUMBER)
 		NUMBER(PG_swapcache) = PG_swapcache_ORIGINAL;
+	if (NUMBER(PG_slab) == NOT_FOUND_NUMBER)
+		NUMBER(PG_slab) = PG_slab_ORIGINAL;
+	if (NUMBER(PG_buddy) == NOT_FOUND_NUMBER
+	    && info->kernel_version >= KERNEL_VERSION(2, 6, 17)
+	    && info->kernel_version <= KERNEL_VERSION(2, 6, 26))
+		NUMBER(PG_buddy) = PG_buddy_v2_6_17_to_v2_6_26;
 	return TRUE;
 }
 
makedumpfile.h +3 -0
index 2808871..3059a9e 100644
@@ -68,9 +68,12 @@ int get_mem_type(void);
  * The following values are for linux-2.6.25 or former.
  */
 #define PG_lru_ORIGINAL	 	(5)
+#define PG_slab_ORIGINAL	(7)
 #define PG_private_ORIGINAL	(11)	/* Has something at ->private */
 #define PG_swapcache_ORIGINAL	(15)	/* Swap page: swp_entry_t in private */
 
+#define PG_buddy_v2_6_17_to_v2_6_26	(19)
+
 #define PAGE_MAPPING_ANON	(1)
 
 #define LSEEKED_BITMAP	(1)

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

RFC PATCH 6/10 Add command-line processing for free page filtering looking up mem_map array

In reply to HATAYAMA Daisuke

I choose dump level 32 for free page filtering looking up mem_map array. But I don't mean this is the final version now, I mean experimental use for RFC, only. I prefer the setting that this logic looking up mem_map array is automatically used if dump level 16 and --cyclic option are specified at the same time.

---

 makedumpfile.c |    3 ++-
 makedumpfile.h |    4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)
makedumpfile.c +2 -1
index bed74df..d1eded0 100644
@@ -3853,7 +3853,8 @@ create_2nd_bitmap(void)
 	 */
 	if (info->dump_level & DL_EXCLUDE_CACHE ||
 	    info->dump_level & DL_EXCLUDE_CACHE_PRI ||
-	    info->dump_level & DL_EXCLUDE_USER_DATA) {
+	    info->dump_level & DL_EXCLUDE_USER_DATA ||
+	    info->dump_level & DL_EXCLUDE_FREE_CONST) {
 		if (!exclude_unnecessary_pages()) {
 			ERRMSG("Can't exclude unnecessary pages.\n");
 			return FALSE;
makedumpfile.h +3 -1
index 3059a9e..404f00e 100644
@@ -146,7 +146,7 @@ isAnon(unsigned long mapping)
  * Dump Level
  */
 #define MIN_DUMP_LEVEL		(0)
-#define MAX_DUMP_LEVEL		(31)
+#define MAX_DUMP_LEVEL		(63)
 #define NUM_ARRAY_DUMP_LEVEL	(MAX_DUMP_LEVEL + 1) /* enough to allocate
 							all the dump_level */
 #define DL_EXCLUDE_ZERO		(0x001) /* Exclude Pages filled with Zeros */
@@ -156,6 +156,8 @@ isAnon(unsigned long mapping)
 				           with Private Pages */
 #define DL_EXCLUDE_USER_DATA	(0x008) /* Exclude UserProcessData Pages */
 #define DL_EXCLUDE_FREE		(0x010)	/* Exclude Free Pages */
+#define DL_EXCLUDE_FREE_CONST	(0x020)	/* Exclude Free Pages looking
+					   up mem_map */
 
 
 /*

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw

RFC PATCH 7/10 Add excldue free pages by looking up mem_map array

In reply to HATAYAMA Daisuke

Add free page filtering logic in the processing looking up mem_map array. page_is_buddy handler is newly introduced, which abstracts condition of determining buddy page that varies depending on kernel versions. On makedumpfile suppoting range of kernel versions, there are three kinds of page_is_buddy, which are introduced in later patches. Also, _mapcount and private fields of page struct are newly introduced, which are used in the page_is_buddy handlers.

---

 makedumpfile.c |   21 ++++++++++++++++++---
 makedumpfile.h |    6 ++++++
 2 files changed, 24 insertions(+), 3 deletions(-)
makedumpfile.c +18 -3
index d1eded0..37c371e 100644
@@ -3689,8 +3689,8 @@ __exclude_unnecessary_pages(unsigned long mem_map,
 	unsigned long long pfn_read_start, pfn_read_end, index_pg;
 	unsigned char page_cache[SIZE(page) * PGMM_CACHED];
 	unsigned char *pcache;
-	unsigned int _count;
-	unsigned long flags, mapping;
+	unsigned int _count, _mapcount;
+	unsigned long flags, mapping, private;
 
 	/*
 	 * Refresh the buffer of struct page, when changing mem_map.
@@ -3738,11 +3738,26 @@ __exclude_unnecessary_pages(unsigned long mem_map,
 		flags   = ULONG(pcache + OFFSET(page.flags));
 		_count  = UINT(pcache + OFFSET(page._count));
 		mapping = ULONG(pcache + OFFSET(page.mapping));
+		_mapcount = UINT(pcache + OFFSET(page._mapcount));
+		private = ULONG(pcache + OFFSET(page.private));
 
+ 		/*
+		 * Exclude the free page managed by a buddy.
+		 */
+		if ((info->dump_level & DL_EXCLUDE_FREE_CONST)
+		    && info->page_is_buddy
+		    && info->page_is_buddy(flags, _mapcount, private,
+					   _count)) {
+			int i;
+
+			for (i = 0; i < (1<<private); ++i)
+				clear_bit_on_2nd_bitmap_for_kernel(pfn + i);
+			pfn_free += i;
+		}
 		/*
 		 * Exclude the cache page without the private page.
 		 */
-		if ((info->dump_level & DL_EXCLUDE_CACHE)
+		else if ((info->dump_level & DL_EXCLUDE_CACHE)
 		    && (isLRU(flags) || isSwapCache(flags))
 		    && !isPrivate(flags) && !isAnon(mapping)) {
 			clear_bit_on_2nd_bitmap_for_kernel(pfn);
makedumpfile.h +6 -0
index 404f00e..d867dfe 100644
@@ -943,6 +943,12 @@ struct DumpInfo {
 	 */
 	int flag_sadump_diskset;
 	enum sadump_format_type flag_sadump;         /* sadump format type */
+
+	/*
+	 * for filtering free pages managed by buddy system:
+	 */
+	int (*page_is_buddy)(unsigned long flags, unsigned int _mapcount,
+			     unsigned long private, unsigned int _count);
 };
 extern struct DumpInfo		*info;
 

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw

RFC PATCH 8/10 Add page_is_buddy for recent kernels

In reply to HATAYAMA Daisuke

On kernels from v2.6.38 and later kernels, buddy page is marked by _mapcount == PAGE_BUDDY_MAPCOUNT_VALUE, which varies once as follows:

kernel version | PAGE_BUDDY_MAPCOUNT_VALUE
------------------+--------------------------
v2.6.38 | -2
v2.6.39 and later | -128

One more notice is that _mapcount shares its memory with other fields for SLAB/SLUB when PG_slab is set. We need to check if PG_slab is set or not before looking up _mapcount value.

---

 makedumpfile.c |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)
makedumpfile.c +30 -0
index 37c371e..aea956f 100644
@@ -55,6 +55,8 @@ do { \
 		*ptr_long_table = value; \
 } while (0)
 
+static void setup_page_is_buddy(void);
+
 void
 initialize_tables(void)
 {
@@ -2733,6 +2735,9 @@ out:
 	if (!get_value_for_old_linux())
 		return FALSE;
 
+	if (info->dump_level & DL_EXCLUDE_FREE_CONST)
+		setup_page_is_buddy();
+
 	return TRUE;
 }
 
@@ -3512,6 +3517,31 @@ exclude_free_page(void)
 	return TRUE;
 }
 
+static int
+page_is_buddy_v3(unsigned long flags, unsigned int _mapcount,
+		 unsigned long private, unsigned int _count)
+{
+	if (flags & (1UL << NUMBER(PG_slab)))
+		return FALSE;
+
+	if (_mapcount == (int)NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE))
+		return TRUE;
+
+	return FALSE;
+}
+
+static void
+setup_page_is_buddy(void)
+{
+	if (NUMBER(PG_buddy) == NOT_FOUND_NUMBER
+	    && SIZE(pageflags) != NOT_FOUND_STRUCTURE
+	    && NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE) != NOT_FOUND_NUMBER)
+		info->page_is_buddy = page_is_buddy_v3;
+
+	MSG("Can't select page_is_buddy handler; "
+	    "filtering free pages is disabled.\n");
+}
+
 /*
  * If using a dumpfile in kdump-compressed format as a source file
  * instead of /proc/vmcore, 1st-bitmap of a new dumpfile must be

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw

RFC PATCH 9/10 Add page_is_buddy for PG_buddy

In reply to HATAYAMA Daisuke

On kernels from v2.6.18 to v2.6.37, buddy page is marked by the PG_buddy flag.

kernel version | PG_buddy
------------------ +---------------------------------

  v2.6.17 to v2.6.26 | 19

v2.6.27 to v2.6.37 | 19 if CONFIG_PAGEFLAGS_EXTEND=y
| 18 otherwise

Note for hard coding: it's possible to determine whether CONFIG_PAGEFLAGS_EXTEND=yes or not by looking at error_status array defined in mm/memory-failure.c, for example.

---

 makedumpfile.c |   27 ++++++++++++++++++++-------
 1 files changed, 20 insertions(+), 7 deletions(-)
makedumpfile.c +20 -7
index aea956f..675b47e 100644
@@ -3518,6 +3518,16 @@ exclude_free_page(void)
 }
 
 static int
+page_is_buddy_v2(unsigned long flags, unsigned int _mapcount,
+		 unsigned long private, unsigned int _count)
+{
+	if (flags & (1UL << NUMBER(PG_buddy)))
+		return TRUE;
+
+	return FALSE;
+}
+
+static int
 page_is_buddy_v3(unsigned long flags, unsigned int _mapcount,
 		 unsigned long private, unsigned int _count)
 {
@@ -3533,13 +3543,16 @@ page_is_buddy_v3(unsigned long flags, unsigned int _mapcount,
 static void
 setup_page_is_buddy(void)
 {
-	if (NUMBER(PG_buddy) == NOT_FOUND_NUMBER
-	    && SIZE(pageflags) != NOT_FOUND_STRUCTURE
-	    && NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE) != NOT_FOUND_NUMBER)
-		info->page_is_buddy = page_is_buddy_v3;
-
-	MSG("Can't select page_is_buddy handler; "
-	    "filtering free pages is disabled.\n");
+	if (NUMBER(PG_buddy) == NOT_FOUND_NUMBER) {
+		if (SIZE(pageflags) != NOT_FOUND_STRUCTURE
+		    && NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE) != NOT_FOUND_NUMBER)
+			info->page_is_buddy = page_is_buddy_v3;
+		else {
+			MSG("Can't select page_is_buddy handler; "
+			    "filtering free pages is disabled.\n");
+		}
+	} else
+		info->page_is_buddy = page_is_buddy_v2;
 }
 
 /*

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw

RFC PATCH 10/10 Add page_is_buddy for old kernels

In reply to HATAYAMA Daisuke

On kernels from v2.6.15 to v2.6.17 buddy page is marked by the condition that PG_private flag is set and _count == 0.

Unfortunately, I have yet to test this logic on these kernel versions simply because I've been failing to boot them on my box.

Note that on these kernels, free list can be corrupted due to the bug that the above two conditions are not checked atomically. The reason why PG_buddy was introduced is a fix for this bug. Thus, the bug can also affect the logic based on mem_map array, which we cannot avoid definitely.

---

 makedumpfile.c |   17 +++++++++++++++--
 1 files changed, 15 insertions(+), 2 deletions(-)
makedumpfile.c +15 -2
index 675b47e..b73cc64 100644
@@ -3518,6 +3518,18 @@ exclude_free_page(void)
 }
 
 static int
+page_is_buddy_v1(unsigned long flags, unsigned int _mapcount,
+		 unsigned long private, unsigned int _count)
+{
+	if ((flags & (1UL << NUMBER(PG_private)))
+	    && _count == 0
+	    && private <= ARRAY_LENGTH(zone.free_area))
+		return TRUE;
+
+	return FALSE;
+}
+
+static int
 page_is_buddy_v2(unsigned long flags, unsigned int _mapcount,
 		 unsigned long private, unsigned int _count)
 {
@@ -3544,8 +3556,9 @@ static void
 setup_page_is_buddy(void)
 {
 	if (NUMBER(PG_buddy) == NOT_FOUND_NUMBER) {
-		if (SIZE(pageflags) != NOT_FOUND_STRUCTURE
-		    && NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE) != NOT_FOUND_NUMBER)
+		if (SIZE(pageflags) == NOT_FOUND_STRUCTURE)
+			info->page_is_buddy = page_is_buddy_v1;
+		else if (NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE) != NOT_FOUND_NUMBER)
 			info->page_is_buddy = page_is_buddy_v3;
 		else {
 			MSG("Can't select page_is_buddy handler; "

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw
In reply to HATAYAMA Daisuke

From: HATAYAMA Daisuke <[email protected]>
Subject: [RFC PATCH 00/10] Support free page filtering looking up mem_map array
Date: Fri, 29 Jun 2012 02:37:57 +0900

Quoted 9 lines

This new free page filtering needs the following values.

- OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
- NUMBER(PG_buddy)
- NUMBER(PG_slab)
- NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)

Quoted 8 lines

Unfortunately, OFFSET(_mapcount) and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.

To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.

Concretely, I added the following values in VMCOREINFO generated with -g option or using strings command directly to vmcore.

* 3.1.0-7.fc16.x86_64

NUMBER(PG_slab)=7
SIZE(pageflags)=4
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128

* 2.6.38
SIZE(pageflags)=4
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-2

* 2.6.32
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16
SIZE(pageflags)=4

* 2.6.18
NUMBER(PG_slab)=7
NUMBER(PG_buddy)=19
OFFSET(page._mapcount)=12
OFFSET(page.private)=16

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw
In reply to HATAYAMA Daisuke

Hello HATAYAMA-san,

Quoted 34 lines · On Fri, 29 Jun 2012 02:37:57 +0900 HATAYAMA Daisuke…

Sorry for late posting. I made RFC patch set for free page filtering looking up mem_map array. Unlike exiting method looking up free page list, this is done in constant space.

I intend this patch set to be merged with Kumagai-san's cyclic patch set, so I mark these with RFC. See TODO below. Also, I have yet to test the logic for old kernels from v2.6.15 to v2.6.17.

This new free page filtering needs the following values.

- OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
- NUMBER(PG_buddy)
- NUMBER(PG_slab)
- NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)

Unfortunately, OFFSET(_mapcount) and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.

To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.

TODO

1. Add new values in VMCOREINFO on the upstream kernel.

2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.

Quoted 4 lines

3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.

Thank you always for your work.

I will review your patches and measure executing time with v2 patch of cyclic processing. If your patches are effective, then I will consider TODO above.
Please wait for a while.

Thanks
Atsushi Kumagai

Quoted 15 lines

---

HATAYAMA Daisuke (10):
Add page_is_buddy for old kernels
Add page_is_buddy for PG_buddy
Add page_is_buddy for recent kernels
Add excldue free pages by looking up mem_map array
Add command-line processing for free page filtering looking up mem_map array Add page flag values as hardcoded values
Add debuginfo-related processing for VMCOREINFO/VMLINUX
Add new parameters for various tables
Add debuginfo interface for enum type size
Move page flags setup for old kernels after debuginfo initialization

Quoted 10 lines
 dwarf_info.c   |   29 +++++++++++----
 dwarf_info.h   |    1 +
 makedumpfile.c |  111 +++++++++++++++++++++++++++++++++++++++++++++++++++++---
 makedumpfile.h |   34 ++++++++++++++++-
 4 files changed, 158 insertions(+), 17 deletions(-)
Signature · 2 lines
Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#13 lore raw

Hello,

Quoted 44 lines · On Fri, 29 Jun 2012 15:23:37 +0900 Atsushi Kumagai…

Hello HATAYAMA-san,

Quoted 34 lines · On Fri, 29 Jun 2012 02:37:57 +0900 HATAYAMA Daisuke…

Sorry for late posting. I made RFC patch set for free page filtering looking up mem_map array. Unlike exiting method looking up free page list, this is done in constant space.

I intend this patch set to be merged with Kumagai-san's cyclic patch set, so I mark these with RFC. See TODO below. Also, I have yet to test the logic for old kernels from v2.6.15 to v2.6.17.

This new free page filtering needs the following values.

- OFFSET(page._mapcount)
- OFFSET(page.private)
- SIZE(pageflags)
- NUMBER(PG_buddy)
- NUMBER(PG_slab)
- NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)

Unfortunately, OFFSET(_mapcount) and OFFSET(private) fields of page structure cannot be obtained from VMLINUX using the exiting library in makedumpfile since two members are anonymous components of union types. We need a new interface for them.

To try to use this patch set, it's handy to pass manually editted VMCOREINFO file via -i option.

TODO

1. Add new values in VMCOREINFO on the upstream kernel.

2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.

Quoted 4 lines

3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.

Quoted 6 lines

Thank you always for your work.

I will review your patches and measure executing time with v2 patch of cyclic processing. If your patches are effective, then I will consider TODO above.
Please wait for a while.

I did performance measuring with v2 patches of cyclic processing and HATAYAMA-san's patches. And I fixed v2 patches to reduce wasteful process, please see the end of this mail.

How to measure:

- The source data is a vmcore saved on the disk, the size is 5,099,292,912 bytes.
- makedumpfile writes dumpfile to the same disk as the source data.
- I measured the execution time with time(1) and adopted the average of 5 times.

Test Cases:

- _mapcount:
This logic is implemented by HATAYAMA-san.
This logic looks up members of page structure instead of free_list to filter out free pages.

- free_list
v2 patches choose this logic.
This logic looks up whole free_list to filter out free pages every cycle.

- upstream (v1.4.4):
This logic is NOT CYCLIC, uses temporary bitmap file as usual.

Example:

- _mapcount:

    $ time makedumpfile --cyclic -d32  -i vmcoreinfo vmcore dumpfile.d32

- free_list:

    $ time makedumpfile --cyclic -d16  -i vmcoreinfo vmcore dumpfile.d16

- upstream:

    $ time makedumpfile -d16  -i vmcoreinfo vmcore dumpfile.d16

Result:

a) exclude only free pages

     BUFSIZE_CYCLIC  |                |                  execution time [sec]
         [byte]      |  num of cycle  | _mapcount (-d32) | free_list (-d16) | upstream (-d16)
   ------------------+----------------+------------------+------------------+-----------------
       1024          |      152       |      20.5204     |      28.8028     |        -
       1024 * 10     |       16       |      14.7460     |      18.7904     |        -
       1024 * 100    |        2       |      14.3962     |      17.9356     |        -
       1024 * 200    |        1       |      14.3166     |      17.8762     |     17.7928

b) exclude all unnecessary pages

     BUFSIZE_CYCLIC  |                |                  execution time [sec]
         [byte]      |  num of cycle  | _mapcount (-d47) | free_list (-d31) | upstream (-d31)
   ------------------+----------------+------------------+------------------+-----------------
       1024          |      152       |      11.5086     |      27.2906     |        -
       1024 * 10     |       16       |       6.0740     |      10.9998     |        -
       1024 * 100    |        2       |       5.7928     |       9.1534     |        -
       1024 * 200    |        1       |       5.6378     |       8.9924     |      5.0516

I expected that the difference of execution time increases based on number of cycle, because I think that repeating scanning free_list is high cost.
And according to result, it seems right.

_mapcount logic can be expected good performance in almost case when --cyclic is specified. So, I think that making effort to resolve TODO is worth for us.

However, I think more consideration is needed to decide whether to choose _mapcount logic or not when --cyclic isn't specified.

Quoted 3 lines

TODO

1. Add new values in VMCOREINFO on the upstream kernel.

I will send this request to the upstream kernel.

Quoted 4 lines

2. Decide when to use this logic instead of the existing free list logic. Option is 1) introduce new dump level or 2) use it automatically if --cyclic is specified. This patch chooses 1) only for RFC use.

I think 2) is reasonable from the result.

Quoted 4 lines

3. Consider how to deal with old kernels on which we cannot add the values in VMCOREINFO. Options is 1) to force users to use VMLINUX,
2) to cover them full hard coding or 3) to give up support on full range of kernel versions ever.

I don't want to choose 2), I think it's inefficient.
I want to require that users prepare VMCOREINFO file with -g option from vmlinux, if cyclic processing is needed.

Do you have any comments ?

Thanks
Atsushi Kumagai

makedumpfile.c +19 -8
index 0e4660f..981d72a 100644
@@ -3824,6 +3824,9 @@ __exclude_unnecessary_pages(unsigned long mem_map,
 
 	for (pfn = pfn_start; pfn < pfn_end; pfn++, mem_map += SIZE(page)) {
 
+		if (info->flag_cyclic && !is_cyclic_region(pfn))
+			continue;
+
 		/*
 		 * Exclude the memory hole.
 		 */
@@ -3960,17 +3963,25 @@ exclude_unnecessary_pages_cyclic(void)
 		if (!exclude_free_page())
 			return FALSE;
 
-	for (mm = 0; mm < info->num_mem_map; mm++) {
+	/*
+	 * Exclude cache pages, cache private pages, user data pages, and free pages.
+	 */
+	if (info->dump_level & DL_EXCLUDE_CACHE ||
+	    info->dump_level & DL_EXCLUDE_CACHE_PRI ||
+	    info->dump_level & DL_EXCLUDE_USER_DATA ||
+	    info->dump_level & DL_EXCLUDE_FREE_CONST) {
+		for (mm = 0; mm < info->num_mem_map; mm++) {
 
-		mmd = &info->mem_map_data[mm];
+			mmd = &info->mem_map_data[mm];
 
-		if (mmd->mem_map == NOT_MEMMAP_ADDR)
-			continue;
+			if (mmd->mem_map == NOT_MEMMAP_ADDR)
+				continue;
 
-		if (mmd->pfn_end >= info->cyclic_start_pfn || mmd->pfn_start <= info->cyclic_end_pfn) {
-			if (!__exclude_unnecessary_pages(mmd->mem_map,
-							 mmd->pfn_start, mmd->pfn_end))
-				return FALSE;
+			if (mmd->pfn_end >= info->cyclic_start_pfn || mmd->pfn_start <= info->cyclic_end_pfn) {
+				if (!__exclude_unnecessary_pages(mmd->mem_map,
+								 mmd->pfn_start, mmd->pfn_end))
+					return FALSE;
+			}
 		}
 	}
 

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec