Skip to contentSkip to conversation

btf/kallsyms based makedumpfile extension for mm page filtering

PATCH In review Tao Liu 2 participants 16 messages Jun 17 – Jun 23

Subsystems (3)
lore
Share & tools
lore

Tao Liu

author
#0 lore raw

v6 cover btf/kallsyms based makedumpfile extension for mm page filtering

A) This patchset will introduce the following features to makedumpfile:

1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.

B) The purpose of the features are:

1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:

a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.

But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.

2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:

a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.

makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.

With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.

3) Compile and run makedumpfile extensions:

  $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on EXTENSION=on
  $ make extensions
  $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
    /tmp/extension.out --extension amdgpu_filter.so
    Loaded extension: ./extensions/amdgpu_filter.so
    makedumpfile Completed.
        User time (seconds): 5.08
        System time (seconds): 0.84
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.95
        Maximum resident set size (kbytes): 17360
        ...

To contrast with eppic script of v2 [2]:

  $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
    /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c   
    makedumpfile Completed.
        User time (seconds): 8.23
        System time (seconds): 0.88
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
        Maximum resident set size (kbytes): 57128
        ...

-rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
-rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
-rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore

C) Discussion:

1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. rhel8.x is not supported, others are not tested.

D) Testing:

If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:

Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@

Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"

[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df

v6 -> v5:

1) Add erase_sample extension for data erasure sample.
2) Replace fprintf() to MSG()/ERRMSG() in kallsyms.h.
3) Use handle_cbs_len in extension_has_callback() for callback check.
4) Fix bug[5].
5) Add handle_cbs[i]->cb = NULL in init_extensions() to reset the callback if one extension's init() is skipped.
6) The patches are organized as follows:

---- <customization specific> ---
10. Add amdgpu mm pages filtering extension

--- <code should be merged> ---
9. Doc: Add --extension option to makedumpfile manual
8. Add erase_sample extension as an data erase reference
7. Add sample extension as an example reference
6. Add makedumpfile extensions support
5. Implement kernel module's btf resolving
4. Implement kernel module's kallsyms resolving
3. Implement kernel btf resolving
2. Implement kernel kallsyms resolving
1. Reserve sections for makedumpfile and extenions

Patch 10 is customization specific, merging depends on the strategy of maintenance.
Patch 1 ~ 9 are common code which should be merged with makedumpfile.

[5]: https://lore.kernel.org/kexec/[email protected]/

Link to v5: https://lore.kernel.org/kexec/[email protected]/
Link to v4: https://lore.kernel.org/kexec/[email protected]/
Link to v3: https://lore.kernel.org/kexec/[email protected]/
Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/

Tao Liu (10):
Reserve sections for makedumpfile and extenions Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel module's kallsyms resolving Implement kernel module's btf resolving
Add makedumpfile extensions support
Add sample extension as an example reference Add erase_sample extension as an data erase reference
Doc: Add --extension option to makedumpfile manual Add amdgpu mm pages filtering extension

 Makefile                   |  15 +-
 README                     |   6 +
 btf_info.c                 | 371 +++++++++++++++++++++++++
 btf_info.h                 |  77 ++++++
 extension.c                | 355 ++++++++++++++++++++++++
 extension.h                |  16 ++
 extensions/Makefile        |  13 +
 extensions/amdgpu_filter.c | 221 +++++++++++++++
 extensions/erase_sample.c  |  78 ++++++
 extensions/maple_tree.c    | 328 ++++++++++++++++++++++
 extensions/maple_tree.h    |   7 +
 extensions/sample.c        |  69 +++++
 kallsyms.c                 | 550 +++++++++++++++++++++++++++++++++++++
 kallsyms.h                 |  86 ++++++
 makedumpfile.8.in          |  11 +-
 makedumpfile.c             |  44 ++-
 makedumpfile.h             |  12 +
 makedumpfile.ld            |  16 ++
 18 files changed, 2266 insertions(+), 9 deletions(-)
 create mode 100644 btf_info.c
 create mode 100644 btf_info.h
 create mode 100644 extension.c
 create mode 100644 extension.h
 create mode 100644 extensions/Makefile
 create mode 100644 extensions/amdgpu_filter.c
 create mode 100644 extensions/erase_sample.c
 create mode 100644 extensions/maple_tree.c
 create mode 100644 extensions/maple_tree.h
 create mode 100644 extensions/sample.c
 create mode 100644 kallsyms.c
 create mode 100644 kallsyms.h
 create mode 100644 makedumpfile.ld
Signature · 1 line
2.54.0
author
#1 lore raw

PATCH v6 1/10 Reserve sections for makedumpfile and extenions

This patch makes preparation for btf/kallsyms support of makedumpfile and extensions. Any needed kernel symbols/types will be reserved within a special section, .init_ksyms for kallsyms symbols and .init_ktypes for kernel types. During makedumpfile kallsyms/btf initialization, those missing info will be resolved. A makedumpfile.ld script is introduced for the purpose.

---
 Makefile        |  2 +-
 makedumpfile.ld | 16 ++++++++++++++++
 2 files changed, 17 insertions(+), 1 deletion(-)
 create mode 100644 makedumpfile.ld
Makefile +1 -1
index 74acbe2..9482332 100644
@@ -113,7 +113,7 @@ $(OBJ_ARCH): $(SRC_ARCH)
 	$(CC) $(CFLAGS_ARCH) -c -o ./$@ $(VPATH)$(@:.o=.c)
 
 makedumpfile: $(SRC_BASE) $(OBJ_PART) $(OBJ_ARCH)
-	$(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -o $@ $< $(LIBS)
+	$(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -Wl,-T,makedumpfile.ld -o $@ $< $(LIBS)
 	@sed -e "s/@DATE@/$(DATE)/" \
 	     -e "s/@VERSION@/$(VERSION)/" \
 	     $(VPATH)makedumpfile.8.in > $(VPATH)makedumpfile.8
makedumpfile.ld new file +16 -0
new file mode 100644
index 0000000..474ad41
@@ -0,0 +1,16 @@
+SECTIONS
+{
+	.init_ksyms ALIGN(8) : {
+		__start_init_ksyms = .;
+		KEEP(*(.init_ksyms*))
+		__stop_init_ksyms = .;
+	}
+
+	.init_ktypes ALIGN(8) : {
+		__start_init_ktypes = .;
+		KEEP(*(.init_ktypes*))
+		__stop_init_ktypes = .;
+	}
+}
+INSERT AFTER .data;
+
Signature · 1 line
2.54.0
author
#2 lore raw

PATCH v6 2/10 Implement kernel kallsyms resolving

In reply to Tao Liu

This patch will parse kernel's kallsyms data. During the parsing process, the .init_ksyms sections of makedumpfile and the extensions will be iterated, so the kallsyms symbols which belongs to vmlinux can be resolved at this moment.

---
 Makefile       |   2 +-
 kallsyms.c     | 392 +++++++++++++++++++++++++++++++++++++++++++++++++
 kallsyms.h     |  83 +++++++++++
 makedumpfile.c |   3 +
 makedumpfile.h |  11 ++
 5 files changed, 490 insertions(+), 1 deletion(-)
 create mode 100644 kallsyms.c
 create mode 100644 kallsyms.h
Makefile +1 -1
index 9482332..3cd729f 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
kallsyms.c new file +392 -0

392 lines not on this page · raw message

kallsyms.h new file +83 -0

83 lines not on this page · raw message

makedumpfile.c +3 -0
index 80637a7..53a8812 100644
@@ -27,6 +27,7 @@
 #include <limits.h>
 #include <assert.h>
 #include <zlib.h>
+#include "kallsyms.h"
 
 struct symbol_table	symbol_table;
 struct size_table	size_table;
@@ -3114,6 +3115,8 @@ read_vmcoreinfo_from_vmcore(off_t offset, unsigned long size, int flag_xen_hv)
 		if (!read_vmcoreinfo())
 			goto out;
 	}
+	read_vmcoreinfo_kallsyms();
+
 	close_vmcoreinfo();
 
 	ret = TRUE;
makedumpfile.h +11 -0
index 66cbd29..19bb6a5 100644
@@ -259,6 +259,7 @@ static inline int string_exists(char *s) { return (s ? TRUE : FALSE); }
 #define UINT(ADDR)	*((unsigned int *)(ADDR))
 #define ULONG(ADDR)	*((unsigned long *)(ADDR))
 #define ULONGLONG(ADDR)	*((unsigned long long *)(ADDR))
+#define VOID_PTR(ADDR)	*((void **)(ADDR))
 
 
 /*
@@ -1919,6 +1920,16 @@ struct symbol_table {
 	 * symbols on sparc64 arch
 	 */
 	unsigned long long		vmemmap_table;
+
+	/*
+	 * kallsyms related
+	 */
+	unsigned long long		kallsyms_names;
+	unsigned long long		kallsyms_num_syms;
+	unsigned long long		kallsyms_token_table;
+	unsigned long long		kallsyms_token_index;
+	unsigned long long		kallsyms_offsets;
+	unsigned long long		kallsyms_relative_base;
 };
 
 struct size_table {
Signature · 1 line
2.54.0
#3 lore raw

Re: [PATCH v6][makedumpfile 02/10] Implement kernel kallsyms resolving

Hi Tao,

Quoted 36 lines · Tao Liu

This patch will parse kernel's kallsyms data. During the parsing process, the .init_ksyms sections of makedumpfile and the extensions will be iterated, so the kallsyms symbols which belongs to vmlinux can be resolved at this moment.

---
  Makefile       |   2 +-
  kallsyms.c     | 392 +++++++++++++++++++++++++++++++++++++++++++++++++
  kallsyms.h     |  83 +++++++++++
  makedumpfile.c |   3 +
  makedumpfile.h |  11 ++
  5 files changed, 490 insertions(+), 1 deletion(-)

create mode 100644 kallsyms.c
create mode 100644 kallsyms.h

Makefile +1 -1
index 9482332..3cd729f 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
  endif
 
  SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
  OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
  SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
  OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
kallsyms.c new file +0 -0
new file mode 100644
index 0000000..a55dcc6

kallsyms.c line 1

Quoted 1 line
kallsyms.c new file +1 -0
@@ -0,0 +1,392 @@
+#define _GNU_SOURCE

Just for confirmation, is this for what? I can build without this line.

$ git grep _GNU_SOURCE
kallsyms.c:#define _GNU_SOURCE

If unnecessary, I would not like to add something new for compatibility. is it ok to remove this when merging?

Thanks,
Kazu

author
#4 lore raw

Re: [PATCH v6][makedumpfile 02/10] Implement kernel kallsyms resolving

Hi Kazu,

Yes, it is needed for reallocarray() in kallsyms.c. For "man reallocarray" gives:

reallocarray():
Since glibc 2.29:
_DEFAULT_SOURCE
glibc 2.28 and earlier:
_GNU_SOURCE

For glibc <= 2.28, we need to use _GNU_SOURCE in order to use reallocarray(). E.g I can encounter the compiling warning in rhel8 once removed.

kallsyms.c: In function ‘add_to_arr’:
kallsyms.c:46:9: warning: implicit declaration of function ‘reallocarray’; did you mean ‘realloc’? [-Wimplicit-function-declaration]

   tmp = reallocarray(*arr, new_cap, sizeof(void *));
         ^~~~~~~~~~~~
         realloc
kallsyms.c:46:7: warning: assignment to ‘void *’ from ‘int’ makes
pointer from integer without a cast [-Wint-conversion]
   tmp = reallocarray(*arr, new_cap, sizeof(void *));
       ^

Thanks,
Tao Liu

Quoted 52 lines · HAGIO KAZUHITO(萩尾 一仁)

Hi Tao,

Quoted 36 lines · Tao Liu

This patch will parse kernel's kallsyms data. During the parsing process, the .init_ksyms sections of makedumpfile and the extensions will be iterated, so the kallsyms symbols which belongs to vmlinux can be resolved at this moment.

---
  Makefile       |   2 +-
  kallsyms.c     | 392 +++++++++++++++++++++++++++++++++++++++++++++++++
  kallsyms.h     |  83 +++++++++++
  makedumpfile.c |   3 +
  makedumpfile.h |  11 ++
  5 files changed, 490 insertions(+), 1 deletion(-)

create mode 100644 kallsyms.c
create mode 100644 kallsyms.h

Makefile +1 -1
index 9482332..3cd729f 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
  endif
 
  SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
  OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
  SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
  OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
kallsyms.c new file +0 -0
new file mode 100644
index 0000000..a55dcc6

kallsyms.c line 1

Quoted 1 line
kallsyms.c new file +1 -0
@@ -0,0 +1,392 @@
+#define _GNU_SOURCE

Just for confirmation, is this for what? I can build without this line.

$ git grep _GNU_SOURCE
kallsyms.c:#define _GNU_SOURCE

If unnecessary, I would not like to add something new for compatibility. is it ok to remove this when merging?

Thanks,
Kazu

#5 lore raw

Re: [PATCH v6][makedumpfile 02/10] Implement kernel kallsyms resolving

Quoted 11 lines · Tao Liu

Hi Kazu,

Yes, it is needed for reallocarray() in kallsyms.c. For "man reallocarray" gives:

reallocarray():
Since glibc 2.29:
_DEFAULT_SOURCE
glibc 2.28 and earlier:
_GNU_SOURCE

Quoted 3 lines

For glibc <= 2.28, we need to use _GNU_SOURCE in order to use reallocarray(). E.g I can encounter the compiling warning in rhel8 once removed.

Thanks for the info. so let's keep it.

Thanks,
Kazu

author
#6 lore raw

PATCH v6 3/10 Implement kernel btf resolving

In reply to Tao Liu

This patch will parse kernel's btf data using libbpf. The kernel's btf data is located between __start_BTF and __stop_BTF symbols which are resolved by kallsyms of the previous patch. Same as the previous one, the .init_ktypes section of makedumpfile and the extensions will be iterated, and any types which belongs to vmlinux can be resolved at this time.

Another primary function implemented in this patch, is recursively diving into anonymous struct/union when encountered any, to find a member by given its name.

---
 Makefile   |   2 +-
 btf_info.c | 238 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 btf_info.h |  75 +++++++++++++++++
 3 files changed, 314 insertions(+), 1 deletion(-)
 create mode 100644 btf_info.c
 create mode 100644 btf_info.h
Makefile +1 -1
index 3cd729f..1c51802 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
btf_info.c new file +238 -0

238 lines not on this page · raw message

btf_info.h new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#7 lore raw

PATCH v6 4/10 Implement kernel module's kallsyms resolving

In reply to Tao Liu

With kernel's kallsyms and btf ready, we can get any kernel types and symbol addresses. So we can iterate kernel modules' linked list, and parse each one of kernel module's structure to get its kallsyms data. At this time, kernel modules' kallsyms symbol defined within .init_ksyms section will be resolved.

---
 kallsyms.c | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 kallsyms.h |   3 +
 2 files changed, 161 insertions(+)
kallsyms.c +158 -0
index a55dcc6..0037600 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include "makedumpfile.h"
 #include "kallsyms.h"
+#include "btf_info.h"
 
 static uint32_t *kallsyms_offsets = NULL;
 static uint16_t *kallsyms_token_index = NULL;
@@ -381,6 +382,163 @@ out:
 	}
 	return ret;
 }
+
+INIT_MOD_SYM(vmlinux, modules);
+
+INIT_MOD_STRUCT_MEMBER(vmlinux, list_head, next);
+INIT_MOD_STRUCT_MEMBER(vmlinux, module, list);
+INIT_MOD_STRUCT_MEMBER(vmlinux, module, name);
+INIT_MOD_STRUCT_MEMBER(vmlinux, module, core_kallsyms);
+INIT_MOD_STRUCT_MEMBER(vmlinux, mod_kallsyms, symtab);
+INIT_MOD_STRUCT_MEMBER(vmlinux, mod_kallsyms, num_symtab);
+INIT_MOD_STRUCT_MEMBER(vmlinux, mod_kallsyms, strtab);
+INIT_MOD_STRUCT_MEMBER(vmlinux, elf64_sym, st_name);
+INIT_MOD_STRUCT_MEMBER(vmlinux, elf64_sym, st_value);
+
+#define MEMBER_OFF(S, M) \
+	GET_MOD_STRUCT_MEMBER_MOFF(vmlinux, S, M) / 8
+#define GET_KERN_STRUCT_MEMBER_MSIZE(S, M) \
+	GET_MOD_STRUCT_MEMBER_MSIZE(vmlinux, S, M)
+#define KERN_STRUCT_MEMBER_EXIST(S, M) \
+	MOD_STRUCT_MEMBER_EXIST(vmlinux, S, M)
+#define GET_KERN_STRUCT_MEMBER_SSIZE(S, M) \
+	GET_MOD_STRUCT_MEMBER_SSIZE(vmlinux, S, M)
+#define GET_KERN_SYM(SYM) GET_MOD_SYM(vmlinux, SYM)
+#define KERN_SYM_EXIST(SYM) MOD_SYM_EXIST(vmlinux, SYM)
+
+uint64_t next_list(uint64_t list)
+{
+	uint64_t next = 0;
+
+	if (!readmem(VADDR, list + MEMBER_OFF(list_head, next),
+		&next, GET_KERN_STRUCT_MEMBER_MSIZE(list_head, next))) {
+		ERRMSG("Can't get next list!\n");
+	}
+	return next;
+}
+
+bool init_module_kallsyms(void)
+{
+	uint64_t modules, list, value = 0, symtab = 0, strtab = 0;
+	uint32_t st_name = 0;
+	int num_symtab, i, j;
+	struct ksym_info **p;
+	char symname[512], ch;
+	char *modname = NULL;
+	bool ret = false;
+
+	modules = GET_KERN_SYM(modules);
+	if (!KERN_SYM_EXIST(modules)) {
+		/* Not a failure if no module enabled */
+		ret = true;
+		goto out;
+	}
+
+	if (!KERN_STRUCT_MEMBER_EXIST(list_head, next) ||
+	    !KERN_STRUCT_MEMBER_EXIST(module, list) ||
+	    !KERN_STRUCT_MEMBER_EXIST(module, name) ||
+	    !KERN_STRUCT_MEMBER_EXIST(module, core_kallsyms) ||
+	    !KERN_STRUCT_MEMBER_EXIST(mod_kallsyms, symtab) ||
+	    !KERN_STRUCT_MEMBER_EXIST(mod_kallsyms, num_symtab) ||
+	    !KERN_STRUCT_MEMBER_EXIST(mod_kallsyms, strtab) ||
+	    !KERN_STRUCT_MEMBER_EXIST(elf64_sym, st_name) ||
+	    !KERN_STRUCT_MEMBER_EXIST(elf64_sym, st_value)) {
+		/* Fail when module enabled but any required types not found */
+		ERRMSG("Missing required module syms/types!\n");
+		goto out;
+	}
+
+	modname = (char *)malloc(GET_KERN_STRUCT_MEMBER_MSIZE(module, name));
+	if (!modname)
+		goto no_mem;
+
+	for (list = next_list(modules); list != modules; list = next_list(list)) {
+		if (!readmem(VADDR, list - MEMBER_OFF(module, list) +
+				MEMBER_OFF(module, name),
+			modname, GET_KERN_STRUCT_MEMBER_MSIZE(module, name))) {
+			ERRMSG("Can't get module modname!\n");
+			goto out;
+		}
+		if (!check_ksyms_require_modname(modname, NULL))
+			continue;
+		if (!readmem(VADDR, list - MEMBER_OFF(module, list) +
+				MEMBER_OFF(module, core_kallsyms) +
+				MEMBER_OFF(mod_kallsyms, num_symtab),
+			&num_symtab, GET_KERN_STRUCT_MEMBER_MSIZE(mod_kallsyms, num_symtab))) {
+			ERRMSG("Can't get module num_symtab!\n");
+			goto out;
+		}
+		if (!readmem(VADDR, list - MEMBER_OFF(module, list) +
+				MEMBER_OFF(module, core_kallsyms) +
+				MEMBER_OFF(mod_kallsyms, symtab),
+			&symtab, GET_KERN_STRUCT_MEMBER_MSIZE(mod_kallsyms, symtab))) {
+			ERRMSG("Can't get module symtab!\n");
+			goto out;
+		}
+		if (!readmem(VADDR, list - MEMBER_OFF(module, list) +
+				MEMBER_OFF(module, core_kallsyms) +
+				MEMBER_OFF(mod_kallsyms, strtab),
+			&strtab, GET_KERN_STRUCT_MEMBER_MSIZE(mod_kallsyms, strtab))) {
+			ERRMSG("Can't get module strtab!\n");
+			goto out;
+		}
+		for (i = 0; i < num_symtab; i++) {
+			j = 0;
+			if (!readmem(VADDR, symtab + i * GET_KERN_STRUCT_MEMBER_SSIZE(elf64_sym, st_value) +
+					MEMBER_OFF(elf64_sym, st_value),
+				&value, GET_KERN_STRUCT_MEMBER_MSIZE(elf64_sym, st_value))) {
+				ERRMSG("Can't get module st_value!\n");
+				goto out;
+			}
+			if (!readmem(VADDR, symtab + i * GET_KERN_STRUCT_MEMBER_SSIZE(elf64_sym, st_name) +
+					MEMBER_OFF(elf64_sym, st_name),
+				&st_name, GET_KERN_STRUCT_MEMBER_MSIZE(elf64_sym, st_name))) {
+				ERRMSG("Can't get module st_name!\n");
+				goto out;
+			}
+			do {
+				if (!readmem(VADDR, strtab + st_name + j++, &ch, 1)) {
+					ERRMSG("Can't get module symname's next char!\n");
+					goto out;
+				}
+			} while (ch != '\0');
+			if (j == 1 || j > sizeof(symname))
+				/* Skip empty or too long string */
+				continue;
+			if (!readmem(VADDR, strtab + st_name, symname, j)) {
+				ERRMSG("Can't get module symname!\n");
+				goto out;
+			}
+
+			for (j = 0; j < sr_len; j++) {
+				for (p = (struct ksym_info **)(sr[j]->start);
+				     p < (struct ksym_info **)(sr[j]->stop);
+				     p++) {
+					if (!strcmp((*p)->modname, modname) &&
+					    !strcmp((*p)->symname, symname)) {
+						(*p)->value = value;
+						(*p)->index = i;
+					}
+				}
+			}
+		}
+	}
+	ret = true;
+	goto out;
+no_mem:
+	ERRMSG("Not enough memory!\n");
+out:
+	if (modname)
+		free(modname);
+	return ret;
+}
+
+void cleanup_kallsyms(void)
+{
+	cleanup_ksyms_section_range();
+	cleanup_ksyms_modname();
+}
+
 #else /* EXTENSION */
 
 bool read_vmcoreinfo_kallsyms(void)
kallsyms.h +3 -0
index 5d605ce..a69af15 100644
@@ -79,5 +79,8 @@ bool check_ksyms_require_modname(char *modname, int *total);
 bool register_ksym_section(char *start, char *stop);
 bool read_vmcoreinfo_kallsyms(void);
 bool init_kernel_kallsyms(void);
+uint64_t next_list(uint64_t list);
+bool init_module_kallsyms(void);
+void cleanup_kallsyms(void);
 #endif /* _KALLSYMS_H */
 
Signature · 1 line
2.54.0
author
#8 lore raw

PATCH v6 5/10 Implement kernel module's btf resolving

In reply to Tao Liu

Same as the previous patch, with kernel's kallsyms and btf ready, we can locate and iterate all kernel modules' btf data. So kernel modules' types specified within .init_ksyms section will be resolved.

---
 btf_info.c | 133 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 btf_info.h |   2 +
 2 files changed, 135 insertions(+)
btf_info.c +133 -0
index 8d907f6..fbb487b 100644
@@ -234,5 +234,138 @@ out:
 		free(buf);
 	return ret;
 }
+
+INIT_MOD_SYM(vmlinux, btf_modules);
+
+INIT_MOD_STRUCT_MEMBER(vmlinux, btf_module, list);
+INIT_MOD_STRUCT_MEMBER(vmlinux, btf_module, btf);
+INIT_MOD_STRUCT_MEMBER(vmlinux, btf_module, module);
+DECLARE_MOD_STRUCT_MEMBER(vmlinux, module, name);
+INIT_MOD_STRUCT_MEMBER(vmlinux, btf, data);
+INIT_MOD_STRUCT_MEMBER(vmlinux, btf, data_size);
+
+#define KERN_STRUCT_MEMBER_EXIST(S, M) MOD_STRUCT_MEMBER_EXIST(vmlinux, S, M)
+#define MEMBER_OFF(S, M) GET_MOD_STRUCT_MEMBER_MOFF(vmlinux, S, M) / 8
+#define GET_KERN_STRUCT_MEMBER_MSIZE(S, M) GET_MOD_STRUCT_MEMBER_MSIZE(vmlinux, S, M)
+#define GET_KERN_SYM(SYM) GET_MOD_SYM(vmlinux, SYM)
+
+bool init_module_btf(void)
+{
+	struct btf *btf_mod;
+	uint64_t btf_modules, list;
+	uint64_t btf = 0, data = 0, module = 0;
+	int data_size = 0;
+	bool ret = false;
+	char *btf_buf = NULL;
+	char *modname = NULL;
+	struct ktype_info **p;
+
+	btf_modules = GET_KERN_SYM(btf_modules);
+	if (!KERN_SYM_EXIST(btf_modules))
+		/* Maybe module is not enabled, this is not an error */
+		return true;
+
+	if (!KERN_STRUCT_MEMBER_EXIST(btf_module, list) ||
+	    !KERN_STRUCT_MEMBER_EXIST(btf_module, btf) ||
+	    !KERN_STRUCT_MEMBER_EXIST(btf_module, module) ||
+	    !KERN_STRUCT_MEMBER_EXIST(btf, data) ||
+	    !KERN_STRUCT_MEMBER_EXIST(btf, data_size)) {
+		/* Fail when module enabled but any required types not found */
+		ERRMSG("Missing required btf syms/types!\n");
+		goto out;
+	}
+
+	modname = (char *)malloc(GET_KERN_STRUCT_MEMBER_MSIZE(module, name));
+	if (!modname)
+		goto no_mem;
+
+	for (list = next_list(btf_modules); list != btf_modules; list = next_list(list)) {
+		if (!readmem(VADDR, list - MEMBER_OFF(btf_module, list) +
+				MEMBER_OFF(btf_module, btf),
+			&btf, GET_KERN_STRUCT_MEMBER_MSIZE(btf_module, btf))) {
+			ERRMSG("Can't get btf_module member btf!\n");
+			goto out;
+		}
+		if (!readmem(VADDR, list - MEMBER_OFF(btf_module, list) +
+				MEMBER_OFF(btf_module, module),
+			&module, GET_KERN_STRUCT_MEMBER_MSIZE(btf_module, module))) {
+			ERRMSG("Can't get btf_module member module!\n");
+			goto out;
+		}
+		if (!readmem(VADDR, module + MEMBER_OFF(module, name),
+			modname, GET_KERN_STRUCT_MEMBER_MSIZE(module, name))) {
+			ERRMSG("Can't get module modname!\n");
+			goto out;
+		}
+		if (!check_ktypes_require_modname(modname, NULL)) {
+			continue;
+		}
+		if (!readmem(VADDR, btf + MEMBER_OFF(btf, data),
+			&data, GET_KERN_STRUCT_MEMBER_MSIZE(btf, data))) {
+			ERRMSG("Can't get module btf address!\n");
+			goto out;
+		}
+		if (!readmem(VADDR, btf + MEMBER_OFF(btf, data_size),
+			&data_size, GET_KERN_STRUCT_MEMBER_MSIZE(btf, data_size))) {
+			ERRMSG("Can't get module btf data size!\n");
+			goto out;
+		}
+		btf_buf = (char *)malloc(data_size);
+		if (!btf_buf)
+			goto no_mem;
+		if (!readmem(VADDR, data, btf_buf, data_size)) {
+			ERRMSG("Can't get module btf data!\n");
+			goto out;
+		}
+		btf_mod = btf__new_split(btf_buf, data_size, btf_arr[0]->btf);
+		free(btf_buf);
+		if (libbpf_get_error(btf_mod) != 0 ||
+		    add_to_btf_arr(btf_mod, strdup(modname)) == false) {
+			ERRMSG("init %s btf fail\n", modname);
+			goto out;
+		}
+	}
+
+	/* OK, we have loaded all needed modules's btf, now resolve the types */
+	for (int i = 0; i < sr_len; i++) {
+		for (p = (struct ktype_info **)(sr[i]->start);
+		     p < (struct ktype_info **)(sr[i]->stop);
+		     p++)
+			get_ktype_info(*p, NULL);
+	}
+
+	ret = true;
+	goto out;
+
+no_mem:
+	ERRMSG("Not enough memory!\n");
+out:
+	if (modname)
+		free(modname);
+	return ret;
+}
+
+static void cleanup_btf_arr(void)
+{
+	for (int i = 0; i < btf_arr_len; i++) {
+		free(btf_arr[i]->module);
+		btf__free(btf_arr[i]->btf);
+		free(btf_arr[i]);
+	}
+	if (btf_arr) {
+		free(btf_arr);
+		btf_arr = NULL;
+	}
+	btf_arr_len = 0;
+	btf_arr_cap = 0;
+}
+
+void cleanup_btf(void)
+{
+	cleanup_btf_arr();
+	cleanup_ktypes_section_range();
+	cleanup_ktypes_modname();
+}
+
 #endif /* EXTENSION */
 
btf_info.h +2 -0
index 8eace67..a6c3bba 100644
@@ -22,6 +22,8 @@ struct ktype_info {
 bool check_ktypes_require_modname(char *modname, int *total);
 bool register_ktype_section(char *start, char *stop);
 bool init_kernel_btf(void);
+bool init_module_btf(void);
+void cleanup_btf(void);
 
 #define _GEN_NAME_PTR_IMPL(PTR, NAME)		PTR##NAME
 #define _GEN_NAME_PTR(PTR, NAME)		_GEN_NAME_PTR_IMPL(PTR, NAME)
Signature · 1 line
2.54.0
author
#9 lore raw

PATCH v6 6/10 Add makedumpfile extensions support

In reply to Tao Liu

The extensions can be specified by makedumpfile cmdline parameter as "--extension", followed by extension's filename or absolute path. If filename is give, then "./", "./extenisons" and "/usr/lib64/makedumpfile/extensions/" will be searched.

The procedures of extensions are as follows:

Step 0: Every extensions will declare which kernel symbol/types they needed during programming. This info will be stored within .init_ksyms/ktypes section. Also extension will have a callback function for makedumpfile to call.

Step 1: Register .init_ksyms and .init_ktypes sections of makedumpfile itself and extension's .so files, then tell kallsyms/btf subcomponent that which kernel symbols/types will be resolved. And callbacks are also registered.

Step 2: Init kernel/module's btf/kallsyms on demand. Any un-needed kenrel modules will be skipped.

Step 3: During btf/kallsyms parsing, the needed info will be filled. For syms/types which are defined via INIT_MOD_OPT(...) macro, these are optinal syms/types, it won't fail at parsing step if any are missing, instead, they need to be checked within extension_init() of each extensions; Otherwise for syms/types which defined via INIT_MOD(...) macro, these are must-have syms/types, if any missing, the extension will fail at this step and as a result this extension will be skipped.

After this step, required kernel symbol value and kernel types size/offset are resolved, the extensions are ready to go.

Step 4: When makedumpfile doing page filtering, in addition to its original filtering mechanism, it will call extensions callbacks for advices whether the page should be included/excluded.

---
 Makefile            |  13 +-
 extension.c         | 355 ++++++++++++++++++++++++++++++++++++++++++++
 extension.h         |  16 ++
 extensions/Makefile |  11 ++
 makedumpfile.c      |  41 ++++-
 makedumpfile.h      |   1 +
 6 files changed, 430 insertions(+), 7 deletions(-)
 create mode 100644 extension.c
 create mode 100644 extension.h
 create mode 100644 extensions/Makefile
Makefile +12 -1
index 1c51802..7e37f13 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -73,6 +73,11 @@ LIBS := -lzstd $(LIBS)
 CFLAGS += -DUSEZSTD
 endif
 
+ifeq ($(EXTENSION), on)
+LIBS := -lbpf $(LIBS)
+CFLAGS += -DEXTENSION
+endif
+
 ifeq ($(DEBUG), on)
 # Requires libasan
 CFLAGS += -fsanitize=address
@@ -126,6 +131,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
 	rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+	$(MAKE) -C extensions clean
 
 install:
 	install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +141,8 @@ install:
 	mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
 	install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
 	install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+	$(MAKE) -C extensions CC=$(CC)
+
extension.c new file +355 -0

355 lines not on this page · raw message

extension.h new file +16 -0
new file mode 100644
index 0000000..ba8d32a
@@ -0,0 +1,16 @@
+#ifndef _EXTENSION_H
+#define _EXTENSION_H
+#include <stdbool.h>
+
+enum {
+	PG_INCLUDE,	// Exntesion will keep the page
+	PG_EXCLUDE,	// Exntesion will discard the page
+	PG_UNDECID,	// Exntesion makes no decision
+};
+int run_extension_callback(unsigned long pfn, const void *pcache);
+void init_extensions(void);
+void cleanup_extensions(void);
+bool add_extension_opts(char *opt);
+bool extension_has_callback(void);
+#endif /* _EXTENSION_H */
+
extensions/Makefile new file +11 -0
new file mode 100644
index 0000000..b23f346
@@ -0,0 +1,11 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+	$(CC) -O2 -g -fPIC -shared -Wl,-T,../makedumpfile.ld -o $@ $^
+
+clean:
+	rm -f $(CONTRIB_SO)
+
makedumpfile.c +35 -6

136 lines not on this page · raw message

makedumpfile.h +1 -0
index 19bb6a5..4f707c7 100644
@@ -2747,6 +2747,7 @@ struct elf_prstatus {
 #define OPT_CHECK_PARAMS        OPT_START+18
 #define OPT_DRY_RUN             OPT_START+19
 #define OPT_SHOW_STATS          OPT_START+20
+#define OPT_EXTENSION           OPT_START+21
 
 /*
  * Function Prototype.
Signature · 1 line
2.54.0
author
#10 lore raw

PATCH v6 7/10 Add sample extension as an example reference

In reply to Tao Liu

This patch will add a sample.c as an example extension for code reference. The extension will do nothing related to page filtering, but print a few kernel symbol/type information.

---
 extensions/Makefile |  2 +-
 extensions/sample.c | 69 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+), 1 deletion(-)
 create mode 100644 extensions/sample.c
extensions/Makefile +1 -1
index b23f346..dbaec4e 100644
@@ -1,5 +1,5 @@
 CC ?= gcc
-CONTRIB_SO :=
+CONTRIB_SO := sample.so
 
 all: $(CONTRIB_SO)
 
extensions/sample.c new file +69 -0

69 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#11 lore raw

PATCH v6 8/10 Add erase_sample extension as an data erase reference

In reply to Tao Liu

This patch will add a erase_sample.c as a data erase reference. It will erase all tasks & threads's comm[] member within their task_struct, as a result, you will see the ps info outputted by crash utility similiar as follows:

crash> ps
...

 1       0  10  ffff96b7c0980000  IN   0.0    25504    15704  XXXXXXXXXXXXXXX
 2       0  15  ffff96b7c0982f40  IN   0.0        0        0  [XXXXXXXXXXXXXXX]
 3       2   2  ffff96b7c09a0000  IN   0.0        0        0  [XXXXXXXXXXXXXXX]
 4       2   0  ffff96b7c09a2f40  ID   0.0        0        0  [XXXXXXXXXXXXXXX]
 5       2   0  ffff96b7c09a8000  ID   0.0        0        0  [XXXXXXXXXXXXXXX]

The comm[] erasure doesn't cause critical info loss and has slight impact on vmcore analyze, so it is safe to take it as an example for developers.

---
 extensions/Makefile       |  2 +-
 extensions/erase_sample.c | 78 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+), 1 deletion(-)
 create mode 100644 extensions/erase_sample.c
extensions/Makefile +1 -1
index dbaec4e..c112d56 100644
@@ -1,5 +1,5 @@
 CC ?= gcc
-CONTRIB_SO := sample.so
+CONTRIB_SO := sample.so erase_sample.so
 
 all: $(CONTRIB_SO)
 
extensions/erase_sample.c new file +78 -0

78 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#12 lore raw

PATCH v6 9/10 Doc: Add --extension option to makedumpfile manual

In reply to Tao Liu
---
 README            |  6 ++++++
 makedumpfile.8.in | 11 ++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)
README +6 -0
index a80b656..b674a10 100644
@@ -65,6 +65,12 @@
     # make USEZSTD=on ; make install
     The user has to prepare zstd library.
 
+  - Build with extension support:
+    # make EXTENSION=on ; make install
+    The user has to prepare bpf library with version(>=v1.4.0).
+    Note: the extension will only work for kernels with DEBUG_INFO_BTF
+    & KALLSYMS enabled.
+
   - Build the extension module for --eppic option.
     # make eppic_makedumpfile.so
     The user has to prepare eppic library from the following site:
makedumpfile.8.in +10 -1
index 1edd0ff..b1a0991 100644
@@ -2,7 +2,7 @@
 .SH NAME
 makedumpfile \- make a small dumpfile of kdump
 .SH SYNOPSIS
-\fBmakedumpfile\fR    [\fIOPTION\fR] [\-x \fIVMLINUX\fR|\-i \fIVMCOREINFO\fR] \fIVMCORE\fR \fIDUMPFILE\fR
+\fBmakedumpfile\fR    [\fIOPTION\fR] [\-x \fIVMLINUX\fR|\-i \fIVMCOREINFO\fR] [--extension \fIEXTENSION1.SO\fR [--extension \fIEXTENSION2.SO\fR ..]] \fIVMCORE\fR \fIDUMPFILE\fR
 .br
 \fBmakedumpfile\fR \-F [\fIOPTION\fR] [\-x \fIVMLINUX\fR|\-i \fIVMCOREINFO\fR] \fIVMCORE\fR
 .br
@@ -664,6 +664,15 @@ This option cannot be used with the --dump-dmesg, --reassemble and -g options.
 Display report messages. This is an alternative to enabling bit 4 in the level
 provided to --message-level.
 
+.TP
+\fB\-\-extension\fR \fIEXTENSION.SO\fR
+Load makedumpfile extensions. By using extensions, users can programmably
+customize mm page keeping / discard, alone with the traditional page-flag
+(-d dump_level) based filtering; as well as kernel data erasure. The option
+can be used multiple times to select various extensions. NOTE: this option
+needs EXTENSION=on when build makedumpfile with libbpf(>=v1.4.0). Also this
+option only works for kernels with DEBUG_INFO_BTF & KALLSYMS enabled.
+
 .SH ENVIRONMENT VARIABLES
 
 .TP 8
Signature · 1 line
2.54.0
author
#13 lore raw

PATCH v6 10/10 Add amdgpu mm pages filtering extension

In reply to Tao Liu

This patch will introduce maple_tree & amdgpu mm page filtering extension, those mm pages allocated to amdgpu will be discarded from vmcore, in order to shrink vmcore size since mm pages allocated to amdgpu are useless to kernel crash and may contain sensitive data.

---
 extensions/Makefile        |   4 +-
 extensions/amdgpu_filter.c | 221 +++++++++++++++++++++++++
 extensions/maple_tree.c    | 328 +++++++++++++++++++++++++++++++++++++
 extensions/maple_tree.h    |   7 +
 4 files changed, 559 insertions(+), 1 deletion(-)
 create mode 100644 extensions/amdgpu_filter.c
 create mode 100644 extensions/maple_tree.c
 create mode 100644 extensions/maple_tree.h
extensions/Makefile +3 -1
index c112d56..9dd4f8e 100644
@@ -1,8 +1,10 @@
 CC ?= gcc
-CONTRIB_SO := sample.so erase_sample.so
+CONTRIB_SO := sample.so erase_sample.so amdgpu_filter.so
 
 all: $(CONTRIB_SO)
 
+amdgpu_filter.so: maple_tree.c
+
 $(CONTRIB_SO): %.so: %.c
 	$(CC) -O2 -g -fPIC -shared -Wl,-T,../makedumpfile.ld -o $@ $^
 
extensions/amdgpu_filter.c new file +221 -0

221 lines not on this page · raw message

extensions/maple_tree.c new file +328 -0

328 lines not on this page · raw message

extensions/maple_tree.h new file +7 -0
new file mode 100644
index 0000000..2e8d029
@@ -0,0 +1,7 @@
+#ifndef _MAPLE_TREE_H
+#define _MAPLE_TREE_H
+#include <stdbool.h>
+unsigned long *mt_dump(unsigned long mt, int *array_len);
+bool maple_init(void);
+#endif /* _MAPLE_TREE_H */
+
Signature · 1 line
2.54.0
#14 lore raw
In reply to Tao Liu

Hi Tao,

I've tested some, no issue was detected.

[without the patchset]
8.09user 1.01system 0:10.04elapsed 90%CPU (0avgtext+0avgdata 4480maxresident)k 18672inputs+763736outputs (0major+268minor)pagefaults 0swaps

[with the patchset without --extension]
8.08user 0.99system 0:09.94elapsed 91%CPU (0avgtext+0avgdata 4608maxresident)k # +128k 19640inputs+763720outputs (4major+277minor)pagefaults 0swaps

[with the patchset with --extension]
6.38user 0.74system 0:07.92elapsed 89%CPU (0avgtext+0avgdata 16984maxresident)k # +12M 17048inputs+725240outputs (0major+3121minor)pagefaults 0swaps

There is also no noticeable rise in max rss without --extension.

Maybe support for compound pages can be debatable in the future, but I think the current patchset is good enough for extension v1. So merged 01/10 - 09/10 patches, thank you for the great work!

Thanks,
Kazu

Quoted 190 lines · Tao Liu

A) This patchset will introduce the following features to makedumpfile:

1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.

B) The purpose of the features are:

1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:

a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.

But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.

2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:

a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.

makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.

With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.

   3) Compile and run makedumpfile extensions:

   $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on EXTENSION=on
   $ make extensions
   
   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/extension.out --extension amdgpu_filter.so
     Loaded extension: ./extensions/amdgpu_filter.so
     makedumpfile Completed.
         User time (seconds): 5.08
         System time (seconds): 0.84
         Percent of CPU this job got: 99%
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.95
         Maximum resident set size (kbytes): 17360
         ...
  
      To contrast with eppic script of v2 [2]:

   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c
     makedumpfile Completed.
         User time (seconds): 8.23
         System time (seconds): 0.88
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
         Maximum resident set size (kbytes): 57128
         ...

   -rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
   -rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
   -rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore

C) Discussion:

1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. rhel8.x is not supported, others are not tested.

D) Testing:

If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:

Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@

Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"

[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df

v6 -> v5:

1) Add erase_sample extension for data erasure sample.
2) Replace fprintf() to MSG()/ERRMSG() in kallsyms.h.
3) Use handle_cbs_len in extension_has_callback() for callback check.
4) Fix bug[5].
5) Add handle_cbs[i]->cb = NULL in init_extensions() to reset the callback if one extension's init() is skipped.
6) The patches are organized as follows:

---- <customization specific> ---
10. Add amdgpu mm pages filtering extension

--- <code should be merged> ---
9. Doc: Add --extension option to makedumpfile manual
8. Add erase_sample extension as an data erase reference
7. Add sample extension as an example reference
6. Add makedumpfile extensions support
5. Implement kernel module's btf resolving
4. Implement kernel module's kallsyms resolving
3. Implement kernel btf resolving
2. Implement kernel kallsyms resolving
1. Reserve sections for makedumpfile and extenions

Patch 10 is customization specific, merging depends on the strategy of maintenance.
Patch 1 ~ 9 are common code which should be merged with makedumpfile.

[5]: https://lore.kernel.org/kexec/[email protected]/

Link to v5: https://lore.kernel.org/kexec/[email protected]/
Link to v4: https://lore.kernel.org/kexec/[email protected]/
Link to v3: https://lore.kernel.org/kexec/[email protected]/
Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/

Tao Liu (10):
Reserve sections for makedumpfile and extenions Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel module's kallsyms resolving Implement kernel module's btf resolving
Add makedumpfile extensions support
Add sample extension as an example reference Add erase_sample extension as an data erase reference
Doc: Add --extension option to makedumpfile manual Add amdgpu mm pages filtering extension

  Makefile                   |  15 +-
  README                     |   6 +
  btf_info.c                 | 371 +++++++++++++++++++++++++
  btf_info.h                 |  77 ++++++
  extension.c                | 355 ++++++++++++++++++++++++
  extension.h                |  16 ++
  extensions/Makefile        |  13 +
  extensions/amdgpu_filter.c | 221 +++++++++++++++
  extensions/erase_sample.c  |  78 ++++++
  extensions/maple_tree.c    | 328 ++++++++++++++++++++++
  extensions/maple_tree.h    |   7 +
  extensions/sample.c        |  69 +++++
  kallsyms.c                 | 550 +++++++++++++++++++++++++++++++++++++
  kallsyms.h                 |  86 ++++++
  makedumpfile.8.in          |  11 +-
  makedumpfile.c             |  44 ++-
  makedumpfile.h             |  12 +
  makedumpfile.ld            |  16 ++
  18 files changed, 2266 insertions(+), 9 deletions(-)

create mode 100644 btf_info.c
create mode 100644 btf_info.h
create mode 100644 extension.c
create mode 100644 extension.h
create mode 100644 extensions/Makefile
create mode 100644 extensions/amdgpu_filter.c
create mode 100644 extensions/erase_sample.c
create mode 100644 extensions/maple_tree.c
create mode 100644 extensions/maple_tree.h
create mode 100644 extensions/sample.c
create mode 100644 kallsyms.c
create mode 100644 kallsyms.h
create mode 100644 makedumpfile.ld

author
#15 lore raw

Hi Kazu,

Quoted 17 lines · HAGIO KAZUHITO(萩尾 一仁)

Hi Tao,

I've tested some, no issue was detected.

[without the patchset]
8.09user 1.01system 0:10.04elapsed 90%CPU (0avgtext+0avgdata 4480maxresident)k 18672inputs+763736outputs (0major+268minor)pagefaults 0swaps

[with the patchset without --extension]
8.08user 0.99system 0:09.94elapsed 91%CPU (0avgtext+0avgdata 4608maxresident)k # +128k 19640inputs+763720outputs (4major+277minor)pagefaults 0swaps

[with the patchset with --extension]
6.38user 0.74system 0:07.92elapsed 89%CPU (0avgtext+0avgdata 16984maxresident)k # +12M 17048inputs+725240outputs (0major+3121minor)pagefaults 0swaps

Quoted 5 lines

There is also no noticeable rise in max rss without --extension.

Maybe support for compound pages can be debatable in the future, but I think the current patchset is good enough for extension v1. So merged 01/10 - 09/10 patches, thank you for the great work!

Thanks for your testing, also many thanks for your code review and suggestions!

Thanks,
Tao Liu

Quoted 197 lines

Thanks,
Kazu

Quoted 190 lines · Tao Liu

A) This patchset will introduce the following features to makedumpfile:

1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.

B) The purpose of the features are:

1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:

a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.

But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.

2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:

a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.

makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.

With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.

   3) Compile and run makedumpfile extensions:

   $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on EXTENSION=on
   $ make extensions

   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/extension.out --extension amdgpu_filter.so
     Loaded extension: ./extensions/amdgpu_filter.so
     makedumpfile Completed.
         User time (seconds): 5.08
         System time (seconds): 0.84
         Percent of CPU this job got: 99%
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.95
         Maximum resident set size (kbytes): 17360
         ...

      To contrast with eppic script of v2 [2]:

   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c
     makedumpfile Completed.
         User time (seconds): 8.23
         System time (seconds): 0.88
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
         Maximum resident set size (kbytes): 57128
         ...

   -rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
   -rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
   -rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore

C) Discussion:

1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. rhel8.x is not supported, others are not tested.

D) Testing:

If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:

Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@

Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"

[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df

v6 -> v5:

1) Add erase_sample extension for data erasure sample.
2) Replace fprintf() to MSG()/ERRMSG() in kallsyms.h.
3) Use handle_cbs_len in extension_has_callback() for callback check.
4) Fix bug[5].
5) Add handle_cbs[i]->cb = NULL in init_extensions() to reset the callback if one extension's init() is skipped.
6) The patches are organized as follows:

---- <customization specific> ---
10. Add amdgpu mm pages filtering extension

--- <code should be merged> ---
9. Doc: Add --extension option to makedumpfile manual
8. Add erase_sample extension as an data erase reference
7. Add sample extension as an example reference
6. Add makedumpfile extensions support
5. Implement kernel module's btf resolving
4. Implement kernel module's kallsyms resolving
3. Implement kernel btf resolving
2. Implement kernel kallsyms resolving
1. Reserve sections for makedumpfile and extenions

Patch 10 is customization specific, merging depends on the strategy of maintenance.
Patch 1 ~ 9 are common code which should be merged with makedumpfile.

[5]: https://lore.kernel.org/kexec/[email protected]/

Link to v5: https://lore.kernel.org/kexec/[email protected]/
Link to v4: https://lore.kernel.org/kexec/[email protected]/
Link to v3: https://lore.kernel.org/kexec/[email protected]/
Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/

Tao Liu (10):
Reserve sections for makedumpfile and extenions Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel module's kallsyms resolving Implement kernel module's btf resolving
Add makedumpfile extensions support
Add sample extension as an example reference Add erase_sample extension as an data erase reference
Doc: Add --extension option to makedumpfile manual Add amdgpu mm pages filtering extension

  Makefile                   |  15 +-
  README                     |   6 +
  btf_info.c                 | 371 +++++++++++++++++++++++++
  btf_info.h                 |  77 ++++++
  extension.c                | 355 ++++++++++++++++++++++++
  extension.h                |  16 ++
  extensions/Makefile        |  13 +
  extensions/amdgpu_filter.c | 221 +++++++++++++++
  extensions/erase_sample.c  |  78 ++++++
  extensions/maple_tree.c    | 328 ++++++++++++++++++++++
  extensions/maple_tree.h    |   7 +
  extensions/sample.c        |  69 +++++
  kallsyms.c                 | 550 +++++++++++++++++++++++++++++++++++++
  kallsyms.h                 |  86 ++++++
  makedumpfile.8.in          |  11 +-
  makedumpfile.c             |  44 ++-
  makedumpfile.h             |  12 +
  makedumpfile.ld            |  16 ++
  18 files changed, 2266 insertions(+), 9 deletions(-)

create mode 100644 btf_info.c
create mode 100644 btf_info.h
create mode 100644 extension.c
create mode 100644 extension.h
create mode 100644 extensions/Makefile
create mode 100644 extensions/amdgpu_filter.c
create mode 100644 extensions/erase_sample.c
create mode 100644 extensions/maple_tree.c
create mode 100644 extensions/maple_tree.h
create mode 100644 extensions/sample.c
create mode 100644 kallsyms.c
create mode 100644 kallsyms.h
create mode 100644 makedumpfile.ld