btf/kallsyms based makedumpfile extension for mm page filtering
Subsystems (3)
v6 cover btf/kallsyms based makedumpfile extension for mm page filtering
A) This patchset will introduce the following features to makedumpfile:
1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.
B) The purpose of the features are:
1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:
a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.
But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.
2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:
a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.
makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.
With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.
3) Compile and run makedumpfile extensions:
$ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on EXTENSION=on $ make extensions
$ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
/tmp/extension.out --extension amdgpu_filter.so
Loaded extension: ./extensions/amdgpu_filter.so
makedumpfile Completed.
User time (seconds): 5.08
System time (seconds): 0.84
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.95
Maximum resident set size (kbytes): 17360
...To contrast with eppic script of v2 [2]:
$ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
/tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c
makedumpfile Completed.
User time (seconds): 8.23
System time (seconds): 0.88
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
Maximum resident set size (kbytes): 57128
... -rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
-rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
-rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore
C) Discussion:
1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. rhel8.x is not supported, others are not tested.
D) Testing:
If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:
Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@
Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"
[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df
v6 -> v5:
1) Add erase_sample extension for data erasure sample.
2) Replace fprintf() to MSG()/ERRMSG() in kallsyms.h.
3) Use handle_cbs_len in extension_has_callback() for callback check.
4) Fix bug[5].
5) Add handle_cbs[i]->cb = NULL in init_extensions() to reset the callback if one extension's init() is skipped.
6) The patches are organized as follows:
---- <customization specific> ---
10. Add amdgpu mm pages filtering extension
--- <code should be merged> ---
9. Doc: Add --extension option to makedumpfile manual
8. Add erase_sample extension as an data erase reference
7. Add sample extension as an example reference
6. Add makedumpfile extensions support
5. Implement kernel module's btf resolving
4. Implement kernel module's kallsyms resolving
3. Implement kernel btf resolving
2. Implement kernel kallsyms resolving
1. Reserve sections for makedumpfile and extenions
Patch 10 is customization specific, merging depends on the strategy of maintenance.
Patch 1 ~ 9 are common code which should be merged with makedumpfile.
[5]: https://lore.kernel.org/kexec/[email protected]/
Link to v5: https://lore.kernel.org/kexec/[email protected]/
Link to v4: https://lore.kernel.org/kexec/[email protected]/
Link to v3: https://lore.kernel.org/kexec/[email protected]/
Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/
Tao Liu (10):
Reserve sections for makedumpfile and extenions Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel module's kallsyms resolving Implement kernel module's btf resolving
Add makedumpfile extensions support
Add sample extension as an example reference Add erase_sample extension as an data erase reference
Doc: Add --extension option to makedumpfile manual Add amdgpu mm pages filtering extension
Makefile | 15 +- README | 6 + btf_info.c | 371 +++++++++++++++++++++++++ btf_info.h | 77 ++++++ extension.c | 355 ++++++++++++++++++++++++ extension.h | 16 ++ extensions/Makefile | 13 + extensions/amdgpu_filter.c | 221 +++++++++++++++ extensions/erase_sample.c | 78 ++++++ extensions/maple_tree.c | 328 ++++++++++++++++++++++ extensions/maple_tree.h | 7 + extensions/sample.c | 69 +++++ kallsyms.c | 550 +++++++++++++++++++++++++++++++++++++ kallsyms.h | 86 ++++++ makedumpfile.8.in | 11 +- makedumpfile.c | 44 ++- makedumpfile.h | 12 + makedumpfile.ld | 16 ++ 18 files changed, 2266 insertions(+), 9 deletions(-) create mode 100644 btf_info.c create mode 100644 btf_info.h create mode 100644 extension.c create mode 100644 extension.h create mode 100644 extensions/Makefile create mode 100644 extensions/amdgpu_filter.c create mode 100644 extensions/erase_sample.c create mode 100644 extensions/maple_tree.c create mode 100644 extensions/maple_tree.h create mode 100644 extensions/sample.c create mode 100644 kallsyms.c create mode 100644 kallsyms.h create mode 100644 makedumpfile.ld
Signature · 1 line
2.54.0
PATCH v6 1/10 Reserve sections for makedumpfile and extenions
This patch makes preparation for btf/kallsyms support of makedumpfile and extensions. Any needed kernel symbols/types will be reserved within a special section, .init_ksyms for kallsyms symbols and .init_ktypes for kernel types. During makedumpfile kallsyms/btf initialization, those missing info will be resolved. A makedumpfile.ld script is introduced for the purpose.
--- Makefile | 2 +- makedumpfile.ld | 16 ++++++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) create mode 100644 makedumpfile.ld
Makefile +1 -1
@@ -113,7 +113,7 @@ $(OBJ_ARCH): $(SRC_ARCH) $(CC) $(CFLAGS_ARCH) -c -o ./$@ $(VPATH)$(@:.o=.c) makedumpfile: $(SRC_BASE) $(OBJ_PART) $(OBJ_ARCH) - $(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -o $@ $< $(LIBS) + $(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -Wl,-T,makedumpfile.ld -o $@ $< $(LIBS) @sed -e "s/@DATE@/$(DATE)/" \ -e "s/@VERSION@/$(VERSION)/" \ $(VPATH)makedumpfile.8.in > $(VPATH)makedumpfile.8
makedumpfile.ld new file +16 -0
@@ -0,0 +1,16 @@ +SECTIONS +{ + .init_ksyms ALIGN(8) : { + __start_init_ksyms = .; + KEEP(*(.init_ksyms*)) + __stop_init_ksyms = .; + } + + .init_ktypes ALIGN(8) : { + __start_init_ktypes = .; + KEEP(*(.init_ktypes*)) + __stop_init_ktypes = .; + } +} +INSERT AFTER .data; +
Signature · 1 line
2.54.0
PATCH v6 2/10 Implement kernel kallsyms resolving
In reply to Tao LiuThis patch will parse kernel's kallsyms data. During the parsing process, the .init_ksyms sections of makedumpfile and the extensions will be iterated, so the kallsyms symbols which belongs to vmlinux can be resolved at this moment.
--- Makefile | 2 +- kallsyms.c | 392 +++++++++++++++++++++++++++++++++++++++++++++++++ kallsyms.h | 83 +++++++++++ makedumpfile.c | 3 + makedumpfile.h | 11 ++ 5 files changed, 490 insertions(+), 1 deletion(-) create mode 100644 kallsyms.c create mode 100644 kallsyms.h
Makefile +1 -1
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32 endif SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h -SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c +SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART)) SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
kallsyms.c new file +392 -0
392 lines not on this page · raw message
kallsyms.h new file +83 -0
83 lines not on this page · raw message
makedumpfile.c +3 -0
@@ -27,6 +27,7 @@#include <limits.h> #include <assert.h> #include <zlib.h> +#include "kallsyms.h" struct symbol_table symbol_table;struct size_table size_table;
@@ -3114,6 +3115,8 @@ read_vmcoreinfo_from_vmcore(off_t offset, unsigned long size, int flag_xen_hv)if (!read_vmcoreinfo ()) goto out; } +read_vmcoreinfo_kallsyms (); +close_vmcoreinfo (); ret = TRUE;
makedumpfile.h +11 -0
@@ -259,6 +259,7 @@ static inline int string_exists(char *s) { return (s ? TRUE : FALSE); }#define UINT (ADDR ) *((unsigned int *)(ADDR))#define ULONG (ADDR ) *((unsigned long *)(ADDR))#define ULONGLONG (ADDR ) *((unsigned long long *)(ADDR)) +#define VOID_PTR (ADDR ) *((void **)(ADDR))/*
@@ -1919,6 +1920,16 @@ struct symbol_table {* symbols on sparc64 arch */ unsigned long long vmemmap_table; + +/* +* kallsyms related +*/ +unsigned long long kallsyms_names; +unsigned long long kallsyms_num_syms; +unsigned long long kallsyms_token_table; +unsigned long long kallsyms_token_index; +unsigned long long kallsyms_offsets; +unsigned long long kallsyms_relative_base; };struct size_table {
Signature · 1 line
2.54.0
Re: [PATCH v6][makedumpfile 02/10] Implement kernel kallsyms resolving
Hi Tao,
Quoted 36 lines · Tao Liu
This patch will parse kernel's kallsyms data. During the parsing process, the .init_ksyms sections of makedumpfile and the extensions will be iterated, so the kallsyms symbols which belongs to vmlinux can be resolved at this moment.
Reviewed-by Stephen Brennan <[email protected]>Signed-off-by Tao Liu <[email protected]>--- Makefile | 2 +- kallsyms.c | 392 +++++++++++++++++++++++++++++++++++++++++++++++++ kallsyms.h | 83 +++++++++++ makedumpfile.c | 3 + makedumpfile.h | 11 ++ 5 files changed, 490 insertions(+), 1 deletion(-)create mode 100644
kallsyms.c
create mode 100644kallsyms.hMakefile +1
-1@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32 endif SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h -SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c +SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART)) SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))kallsyms.c new file +0
-0
kallsyms.c line 1
Quoted 1 line
kallsyms.c new file +1
-0@@ -0,0 +1,392 @@ +#define _GNU_SOURCE
Just for confirmation, is this for what? I can build without this line.
$ git grep _GNU_SOURCE kallsyms.c:#define _GNU_SOURCE
If unnecessary, I would not like to add something new for compatibility. is it ok to remove this when merging?
Thanks,
Kazu
Re: [PATCH v6][makedumpfile 02/10] Implement kernel kallsyms resolving
Hi Kazu,
Yes, it is needed for reallocarray() in kallsyms.c. For "man reallocarray" gives:
reallocarray():
Since glibc 2.29:
_DEFAULT_SOURCE
glibc 2.28 and earlier:
_GNU_SOURCE
For glibc <= 2.28, we need to use _GNU_SOURCE in order to use reallocarray(). E.g I can encounter the compiling warning in rhel8 once removed.
kallsyms.c: In function ‘add_to_arr’:
kallsyms.c:46:9: warning: implicit declaration of function ‘reallocarray’; did you mean ‘realloc’? [-Wimplicit-function-declaration]
tmp = reallocarray(*arr, new_cap, sizeof(void *));
^~~~~~~~~~~~
realloc
kallsyms.c:46:7: warning: assignment to ‘void *’ from ‘int’ makes
pointer from integer without a cast [-Wint-conversion]
tmp = reallocarray(*arr, new_cap, sizeof(void *));
^Thanks,
Tao Liu
Quoted 52 lines · HAGIO KAZUHITO(萩尾 一仁)
Hi Tao,
Quoted 36 lines · Tao Liu
This patch will parse kernel's kallsyms data. During the parsing process, the .init_ksyms sections of makedumpfile and the extensions will be iterated, so the kallsyms symbols which belongs to vmlinux can be resolved at this moment.
Reviewed-by Stephen Brennan <[email protected]>Signed-off-by Tao Liu <[email protected]>--- Makefile | 2 +- kallsyms.c | 392 +++++++++++++++++++++++++++++++++++++++++++++++++ kallsyms.h | 83 +++++++++++ makedumpfile.c | 3 + makedumpfile.h | 11 ++ 5 files changed, 490 insertions(+), 1 deletion(-)create mode 100644
kallsyms.c
create mode 100644kallsyms.hMakefile +1
-1@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32 endif SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h -SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c +SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART)) SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))kallsyms.c new file +0
-0kallsyms.c line 1
Quoted 1 line
kallsyms.c new file +1
-0@@ -0,0 +1,392 @@ +#define _GNU_SOURCEJust for confirmation, is this for what? I can build without this line.
$ git grep _GNU_SOURCE kallsyms.c:#define _GNU_SOURCEIf unnecessary, I would not like to add something new for compatibility. is it ok to remove this when merging?
Thanks,
Kazu
Re: [PATCH v6][makedumpfile 02/10] Implement kernel kallsyms resolving
Quoted 11 lines · Tao Liu
Hi Kazu,
Yes, it is needed for
reallocarray()inkallsyms.c. For "man reallocarray" gives:
reallocarray():
Since glibc 2.29:
_DEFAULT_SOURCE
glibc 2.28 and earlier:
_GNU_SOURCE
Quoted 3 lines
For glibc <= 2.28, we need to use
_GNU_SOURCEin order to usereallocarray(). E.g I can encounter the compiling warning in rhel8 once removed.
Thanks for the info. so let's keep it.
Thanks,
Kazu
PATCH v6 3/10 Implement kernel btf resolving
In reply to Tao LiuThis patch will parse kernel's btf data using libbpf. The kernel's btf data is located between __start_BTF and __stop_BTF symbols which are resolved by kallsyms of the previous patch. Same as the previous one, the .init_ktypes section of makedumpfile and the extensions will be iterated, and any types which belongs to vmlinux can be resolved at this time.
Another primary function implemented in this patch, is recursively diving into anonymous struct/union when encountered any, to find a member by given its name.
--- Makefile | 2 +- btf_info.c | 238 +++++++++++++++++++++++++++++++++++++++++++++++++++++ btf_info.h | 75 +++++++++++++++++ 3 files changed, 314 insertions(+), 1 deletion(-) create mode 100644 btf_info.c create mode 100644 btf_info.h
Makefile +1 -1
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32 endif SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h -SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c +SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART)) SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
btf_info.c new file +238 -0
238 lines not on this page · raw message
btf_info.h new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH v6 4/10 Implement kernel module's kallsyms resolving
In reply to Tao LiuWith kernel's kallsyms and btf ready, we can get any kernel types and symbol addresses. So we can iterate kernel modules' linked list, and parse each one of kernel module's structure to get its kallsyms data. At this time, kernel modules' kallsyms symbol defined within .init_ksyms section will be resolved.
--- kallsyms.c | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++ kallsyms.h | 3 + 2 files changed, 161 insertions(+)
kallsyms.c +158 -0
@@ -6,6 +6,7 @@#include <string.h> #include "makedumpfile.h" #include "kallsyms.h" +#include "btf_info.h" static uint32_t *kallsyms_offsets =NULL ;static uint16_t *kallsyms_token_index =NULL ;
@@ -381,6 +382,163 @@ out: }return ret; } + +INIT_MOD_SYM (vmlinux, modules); + +INIT_MOD_STRUCT_MEMBER (vmlinux, list_head, next); +INIT_MOD_STRUCT_MEMBER (vmlinux, module, list); +INIT_MOD_STRUCT_MEMBER (vmlinux, module, name); +INIT_MOD_STRUCT_MEMBER (vmlinux, module, core_kallsyms); +INIT_MOD_STRUCT_MEMBER (vmlinux, mod_kallsyms, symtab); +INIT_MOD_STRUCT_MEMBER (vmlinux, mod_kallsyms, num_symtab); +INIT_MOD_STRUCT_MEMBER (vmlinux, mod_kallsyms, strtab); +INIT_MOD_STRUCT_MEMBER (vmlinux, elf64_sym, st_name); +INIT_MOD_STRUCT_MEMBER (vmlinux, elf64_sym, st_value); + +#define MEMBER_OFF (S ,M ) \ + GET_MOD_STRUCT_MEMBER_MOFF(vmlinux, S, M) / 8 +#define GET_KERN_STRUCT_MEMBER_MSIZE (S ,M ) \ + GET_MOD_STRUCT_MEMBER_MSIZE(vmlinux, S, M) +#define KERN_STRUCT_MEMBER_EXIST (S ,M ) \ + MOD_STRUCT_MEMBER_EXIST(vmlinux, S, M) +#define GET_KERN_STRUCT_MEMBER_SSIZE (S ,M ) \ + GET_MOD_STRUCT_MEMBER_SSIZE(vmlinux, S, M) +#define GET_KERN_SYM (SYM ) GET_MOD_SYM(vmlinux, SYM) +#define KERN_SYM_EXIST (SYM ) MOD_SYM_EXIST(vmlinux, SYM) + +uint64_t next_list (uint64_t list) +{ +uint64_t next =0 ; + +if (!readmem (VADDR , list +MEMBER_OFF (list_head, next), + &next,GET_KERN_STRUCT_MEMBER_MSIZE (list_head, next))) { +ERRMSG ("Can't get next list!\n" ); + } +return next; +} + +bool init_module_kallsyms (void ) +{ +uint64_t modules, list, value =0 , symtab =0 , strtab =0 ; +uint32_t st_name =0 ; +int num_symtab, i, j; +struct ksym_info **p; +char symname[512 ], ch; +char *modname =NULL ; +bool ret = false; + + modules =GET_KERN_SYM (modules); +if (!KERN_SYM_EXIST (modules)) { +/* Not a failure if no module enabled */ + ret = true; + goto out; + } + +if (!KERN_STRUCT_MEMBER_EXIST (list_head, next) || + !KERN_STRUCT_MEMBER_EXIST (module, list) || + !KERN_STRUCT_MEMBER_EXIST (module, name) || + !KERN_STRUCT_MEMBER_EXIST (module, core_kallsyms) || + !KERN_STRUCT_MEMBER_EXIST (mod_kallsyms, symtab) || + !KERN_STRUCT_MEMBER_EXIST (mod_kallsyms, num_symtab) || + !KERN_STRUCT_MEMBER_EXIST (mod_kallsyms, strtab) || + !KERN_STRUCT_MEMBER_EXIST (elf64_sym, st_name) || + !KERN_STRUCT_MEMBER_EXIST (elf64_sym, st_value)) { +/* Fail when module enabled but any required types not found */ +ERRMSG ("Missing required module syms/types!\n" ); + goto out; + } + + modname = (char *)malloc (GET_KERN_STRUCT_MEMBER_MSIZE (module, name)); +if (!modname) + goto no_mem; + +for (list =next_list (modules); list != modules; list =next_list (list)) { +if (!readmem (VADDR , list -MEMBER_OFF (module, list) + +MEMBER_OFF (module, name), + modname,GET_KERN_STRUCT_MEMBER_MSIZE (module, name))) { +ERRMSG ("Can't get module modname!\n" ); + goto out; + } +if (!check_ksyms_require_modname (modname,NULL )) +continue ; +if (!readmem (VADDR , list -MEMBER_OFF (module, list) + +MEMBER_OFF (module, core_kallsyms) + +MEMBER_OFF (mod_kallsyms, num_symtab), + &num_symtab,GET_KERN_STRUCT_MEMBER_MSIZE (mod_kallsyms, num_symtab))) { +ERRMSG ("Can't get module num_symtab!\n" ); + goto out; + } +if (!readmem (VADDR , list -MEMBER_OFF (module, list) + +MEMBER_OFF (module, core_kallsyms) + +MEMBER_OFF (mod_kallsyms, symtab), + &symtab,GET_KERN_STRUCT_MEMBER_MSIZE (mod_kallsyms, symtab))) { +ERRMSG ("Can't get module symtab!\n" ); + goto out; + } +if (!readmem (VADDR , list -MEMBER_OFF (module, list) + +MEMBER_OFF (module, core_kallsyms) + +MEMBER_OFF (mod_kallsyms, strtab), + &strtab,GET_KERN_STRUCT_MEMBER_MSIZE (mod_kallsyms, strtab))) { +ERRMSG ("Can't get module strtab!\n" ); + goto out; + } +for (i =0 ; i < num_symtab; i++) { + j =0 ; +if (!readmem (VADDR , symtab + i *GET_KERN_STRUCT_MEMBER_SSIZE (elf64_sym, st_value) + +MEMBER_OFF (elf64_sym, st_value), + &value,GET_KERN_STRUCT_MEMBER_MSIZE (elf64_sym, st_value))) { +ERRMSG ("Can't get module st_value!\n" ); + goto out; + } +if (!readmem (VADDR , symtab + i *GET_KERN_STRUCT_MEMBER_SSIZE (elf64_sym, st_name) + +MEMBER_OFF (elf64_sym, st_name), + &st_name,GET_KERN_STRUCT_MEMBER_MSIZE (elf64_sym, st_name))) { +ERRMSG ("Can't get module st_name!\n" ); + goto out; + } +do { +if (!readmem (VADDR , strtab + st_name + j++, &ch,1 )) { +ERRMSG ("Can't get module symname's next char!\n" ); + goto out; + } + }while (ch !='\0' ); +if (j ==1 || j >sizeof (symname)) +/* Skip empty or too long string */ +continue ; +if (!readmem (VADDR , strtab + st_name, symname, j)) { +ERRMSG ("Can't get module symname!\n" ); + goto out; + } + +for (j =0 ; j < sr_len; j++) { +for (p = (struct ksym_info **)(sr[j]->start); + p < (struct ksym_info **)(sr[j]->stop); + p++) { +if (!strcmp ((*p)->modname, modname) && + !strcmp ((*p)->symname, symname)) { + (*p)->value = value; + (*p)->index = i; + } + } + } + } + } + ret = true; + goto out; +no_mem: +ERRMSG ("Not enough memory!\n" ); +out: +if (modname) +free (modname); +return ret; +} + +void cleanup_kallsyms (void ) +{ +cleanup_ksyms_section_range (); +cleanup_ksyms_modname (); +} +#else /* EXTENSION */ bool read_vmcoreinfo_kallsyms (void )
kallsyms.h +3 -0
@@ -79,5 +79,8 @@ bool check_ksyms_require_modname(char *modname, int *total);bool register_ksym_section (char *start,char *stop);bool read_vmcoreinfo_kallsyms (void );bool init_kernel_kallsyms (void ); +uint64_t next_list (uint64_t list); +bool init_module_kallsyms (void ); +void cleanup_kallsyms (void );#endif /* _KALLSYMS_H */
Signature · 1 line
2.54.0
PATCH v6 5/10 Implement kernel module's btf resolving
In reply to Tao LiuSame as the previous patch, with kernel's kallsyms and btf ready, we can locate and iterate all kernel modules' btf data. So kernel modules' types specified within .init_ksyms section will be resolved.
--- btf_info.c | 133 +++++++++++++++++++++++++++++++++++++++++++++++++++++ btf_info.h | 2 + 2 files changed, 135 insertions(+)
btf_info.c +133 -0
@@ -234,5 +234,138 @@ out:free (buf);return ret; } + +INIT_MOD_SYM (vmlinux, btf_modules); + +INIT_MOD_STRUCT_MEMBER (vmlinux, btf_module, list); +INIT_MOD_STRUCT_MEMBER (vmlinux, btf_module, btf); +INIT_MOD_STRUCT_MEMBER (vmlinux, btf_module, module); +DECLARE_MOD_STRUCT_MEMBER (vmlinux, module, name); +INIT_MOD_STRUCT_MEMBER (vmlinux, btf, data); +INIT_MOD_STRUCT_MEMBER (vmlinux, btf, data_size); + +#define KERN_STRUCT_MEMBER_EXIST (S ,M ) MOD_STRUCT_MEMBER_EXIST(vmlinux, S, M) +#define MEMBER_OFF (S ,M ) GET_MOD_STRUCT_MEMBER_MOFF(vmlinux, S, M) / 8 +#define GET_KERN_STRUCT_MEMBER_MSIZE (S ,M ) GET_MOD_STRUCT_MEMBER_MSIZE(vmlinux, S, M) +#define GET_KERN_SYM (SYM ) GET_MOD_SYM(vmlinux, SYM) + +bool init_module_btf (void ) +{ +struct btf *btf_mod; +uint64_t btf_modules, list; +uint64_t btf =0 , data =0 , module =0 ; +int data_size =0 ; +bool ret = false; +char *btf_buf =NULL ; +char *modname =NULL ; +struct ktype_info **p; + + btf_modules =GET_KERN_SYM (btf_modules); +if (!KERN_SYM_EXIST (btf_modules)) +/* Maybe module is not enabled, this is not an error */ +return true; + +if (!KERN_STRUCT_MEMBER_EXIST (btf_module, list) || + !KERN_STRUCT_MEMBER_EXIST (btf_module, btf) || + !KERN_STRUCT_MEMBER_EXIST (btf_module, module) || + !KERN_STRUCT_MEMBER_EXIST (btf, data) || + !KERN_STRUCT_MEMBER_EXIST (btf, data_size)) { +/* Fail when module enabled but any required types not found */ +ERRMSG ("Missing required btf syms/types!\n" ); + goto out; + } + + modname = (char *)malloc (GET_KERN_STRUCT_MEMBER_MSIZE (module, name)); +if (!modname) + goto no_mem; + +for (list =next_list (btf_modules); list != btf_modules; list =next_list (list)) { +if (!readmem (VADDR , list -MEMBER_OFF (btf_module, list) + +MEMBER_OFF (btf_module, btf), + &btf,GET_KERN_STRUCT_MEMBER_MSIZE (btf_module, btf))) { +ERRMSG ("Can't get btf_module member btf!\n" ); + goto out; + } +if (!readmem (VADDR , list -MEMBER_OFF (btf_module, list) + +MEMBER_OFF (btf_module, module), + &module,GET_KERN_STRUCT_MEMBER_MSIZE (btf_module, module))) { +ERRMSG ("Can't get btf_module member module!\n" ); + goto out; + } +if (!readmem (VADDR , module +MEMBER_OFF (module, name), + modname,GET_KERN_STRUCT_MEMBER_MSIZE (module, name))) { +ERRMSG ("Can't get module modname!\n" ); + goto out; + } +if (!check_ktypes_require_modname (modname,NULL )) { +continue ; + } +if (!readmem (VADDR , btf +MEMBER_OFF (btf, data), + &data,GET_KERN_STRUCT_MEMBER_MSIZE (btf, data))) { +ERRMSG ("Can't get module btf address!\n" ); + goto out; + } +if (!readmem (VADDR , btf +MEMBER_OFF (btf, data_size), + &data_size,GET_KERN_STRUCT_MEMBER_MSIZE (btf, data_size))) { +ERRMSG ("Can't get module btf data size!\n" ); + goto out; + } + btf_buf = (char *)malloc (data_size); +if (!btf_buf) + goto no_mem; +if (!readmem (VADDR , data, btf_buf, data_size)) { +ERRMSG ("Can't get module btf data!\n" ); + goto out; + } + btf_mod =btf__new_split (btf_buf, data_size, btf_arr[0 ]->btf); +free (btf_buf); +if (libbpf_get_error (btf_mod) !=0 || +add_to_btf_arr (btf_mod,strdup (modname)) == false) { +ERRMSG ("init %s btf fail\n" , modname); + goto out; + } + } + +/* OK, we have loaded all needed modules's btf, now resolve the types */ +for (int i =0 ; i < sr_len; i++) { +for (p = (struct ktype_info **)(sr[i]->start); + p < (struct ktype_info **)(sr[i]->stop); + p++) +get_ktype_info (*p,NULL ); + } + + ret = true; + goto out; + +no_mem: +ERRMSG ("Not enough memory!\n" ); +out: +if (modname) +free (modname); +return ret; +} + +static void cleanup_btf_arr (void ) +{ +for (int i =0 ; i < btf_arr_len; i++) { +free (btf_arr[i]->module); +btf__free (btf_arr[i]->btf); +free (btf_arr[i]); + } +if (btf_arr) { +free (btf_arr); + btf_arr =NULL ; + } + btf_arr_len =0 ; + btf_arr_cap =0 ; +} + +void cleanup_btf (void ) +{ +cleanup_btf_arr (); +cleanup_ktypes_section_range (); +cleanup_ktypes_modname (); +} +#endif /* EXTENSION */
btf_info.h +2 -0
@@ -22,6 +22,8 @@ struct ktype_info {bool check_ktypes_require_modname (char *modname,int *total);bool register_ktype_section (char *start,char *stop);bool init_kernel_btf (void ); +bool init_module_btf (void ); +void cleanup_btf (void );#define _GEN_NAME_PTR_IMPL (PTR ,NAME ) PTR##NAME#define _GEN_NAME_PTR (PTR ,NAME ) _GEN_NAME_PTR_IMPL(PTR, NAME)
Signature · 1 line
2.54.0
PATCH v6 6/10 Add makedumpfile extensions support
In reply to Tao LiuThe extensions can be specified by makedumpfile cmdline parameter as "--extension", followed by extension's filename or absolute path. If filename is give, then "./", "./extenisons" and "/usr/lib64/makedumpfile/extensions/" will be searched.
The procedures of extensions are as follows:
Step 0: Every extensions will declare which kernel symbol/types they needed during programming. This info will be stored within .init_ksyms/ktypes section. Also extension will have a callback function for makedumpfile to call.
Step 1: Register .init_ksyms and .init_ktypes sections of makedumpfile itself and extension's .so files, then tell kallsyms/btf subcomponent that which kernel symbols/types will be resolved. And callbacks are also registered.
Step 2: Init kernel/module's btf/kallsyms on demand. Any un-needed kenrel modules will be skipped.
Step 3: During btf/kallsyms parsing, the needed info will be filled. For syms/types which are defined via INIT_MOD_OPT(...) macro, these are optinal syms/types, it won't fail at parsing step if any are missing, instead, they need to be checked within extension_init() of each extensions; Otherwise for syms/types which defined via INIT_MOD(...) macro, these are must-have syms/types, if any missing, the extension will fail at this step and as a result this extension will be skipped.
After this step, required kernel symbol value and kernel types size/offset are resolved, the extensions are ready to go.
Step 4: When makedumpfile doing page filtering, in addition to its original filtering mechanism, it will call extensions callbacks for advices whether the page should be included/excluded.
--- Makefile | 13 +- extension.c | 355 ++++++++++++++++++++++++++++++++++++++++++++ extension.h | 16 ++ extensions/Makefile | 11 ++ makedumpfile.c | 41 ++++- makedumpfile.h | 1 + 6 files changed, 430 insertions(+), 7 deletions(-) create mode 100644 extension.c create mode 100644 extension.h create mode 100644 extensions/Makefile
Makefile +12 -1
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32 endif SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h -SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c +SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART)) SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -73,6 +73,11 @@ LIBS := -lzstd $(LIBS) CFLAGS += -DUSEZSTD endif +ifeq ($(EXTENSION), on) +LIBS := -lbpf $(LIBS) +CFLAGS += -DEXTENSION +endif + ifeq ($(DEBUG), on) # Requires libasan CFLAGS += -fsanitize=address
@@ -126,6 +131,7 @@ eppic_makedumpfile.so: extension_eppic.c clean: rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5 + $(MAKE) -C extensions clean install: install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +141,8 @@ install: mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/* + +.PHONY: extensions +extensions: + $(MAKE) -C extensions CC=$(CC) +
extension.c new file +355 -0
355 lines not on this page · raw message
extension.h new file +16 -0
@@ -0,0 +1,16 @@ +#ifndef _EXTENSION_H +#define _EXTENSION_H +#include <stdbool.h> + +enum { +PG_INCLUDE ,// Exntesion will keep the page +PG_EXCLUDE ,// Exntesion will discard the page +PG_UNDECID ,// Exntesion makes no decision +}; +int run_extension_callback (unsigned long pfn,const void *pcache); +void init_extensions (void ); +void cleanup_extensions (void ); +bool add_extension_opts (char *opt); +bool extension_has_callback (void ); +#endif /* _EXTENSION_H */ +
extensions/Makefile new file +11 -0
@@ -0,0 +1,11 @@ +CC ?= gcc +CONTRIB_SO := + +all: $(CONTRIB_SO) + +$(CONTRIB_SO): %.so: %.c + $(CC) -O2 -g -fPIC -shared -Wl,-T,../makedumpfile.ld -o $@ $^ + +clean: + rm -f $(CONTRIB_SO) +
makedumpfile.c +35 -6
136 lines not on this page · raw message
makedumpfile.h +1 -0
@@ -2747,6 +2747,7 @@ struct elf_prstatus {#define OPT_CHECK_PARAMS OPT_START+18#define OPT_DRY_RUN OPT_START+19#define OPT_SHOW_STATS OPT_START+20 +#define OPT_EXTENSION OPT_START+21/* * Function Prototype.
Signature · 1 line
2.54.0
PATCH v6 7/10 Add sample extension as an example reference
In reply to Tao LiuThis patch will add a sample.c as an example extension for code reference. The extension will do nothing related to page filtering, but print a few kernel symbol/type information.
--- extensions/Makefile | 2 +- extensions/sample.c | 69 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 1 deletion(-) create mode 100644 extensions/sample.c
extensions/Makefile +1 -1
@@ -1,5 +1,5 @@ CC ?= gcc -CONTRIB_SO := +CONTRIB_SO := sample.so all: $(CONTRIB_SO)
extensions/sample.c new file +69 -0
69 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH v6 8/10 Add erase_sample extension as an data erase reference
In reply to Tao LiuThis patch will add a erase_sample.c as a data erase reference. It will erase all tasks & threads's comm[] member within their task_struct, as a result, you will see the ps info outputted by crash utility similiar as follows:
crash> ps
...
1 0 10 ffff96b7c0980000 IN 0.0 25504 15704 XXXXXXXXXXXXXXX 2 0 15 ffff96b7c0982f40 IN 0.0 0 0 [XXXXXXXXXXXXXXX] 3 2 2 ffff96b7c09a0000 IN 0.0 0 0 [XXXXXXXXXXXXXXX] 4 2 0 ffff96b7c09a2f40 ID 0.0 0 0 [XXXXXXXXXXXXXXX] 5 2 0 ffff96b7c09a8000 ID 0.0 0 0 [XXXXXXXXXXXXXXX]
The comm[] erasure doesn't cause critical info loss and has slight impact on vmcore analyze, so it is safe to take it as an example for developers.
--- extensions/Makefile | 2 +- extensions/erase_sample.c | 78 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 extensions/erase_sample.c
extensions/Makefile +1 -1
@@ -1,5 +1,5 @@ CC ?= gcc -CONTRIB_SO := sample.so +CONTRIB_SO := sample.so erase_sample.so all: $(CONTRIB_SO)
extensions/erase_sample.c new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH v6 9/10 Doc: Add --extension option to makedumpfile manual
In reply to Tao Liu--- README | 6 ++++++ makedumpfile.8.in | 11 ++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-)
README +6 -0
@@ -65,6 +65,12 @@ # make USEZSTD=on ; make install The user has to prepare zstd library. + - Build with extension support: + # make EXTENSION=on ; make install + The user has to prepare bpf library with version(>=v1.4.0). + Note: the extension will only work for kernels with DEBUG_INFO_BTF + & KALLSYMS enabled. + - Build the extension module for --eppic option. # make eppic_makedumpfile.so The user has to prepare eppic library from the following site:
makedumpfile.8.in +10 -1
@@ -2,7 +2,7 @@ .SH NAME makedumpfile \- make a small dumpfile of kdump .SH SYNOPSIS -\fBmakedumpfile\fR [\fIOPTION\fR] [\-x \fIVMLINUX\fR|\-i \fIVMCOREINFO\fR] \fIVMCORE\fR \fIDUMPFILE\fR +\fBmakedumpfile\fR [\fIOPTION\fR] [\-x \fIVMLINUX\fR|\-i \fIVMCOREINFO\fR] [--extension \fIEXTENSION1.SO\fR [--extension \fIEXTENSION2.SO\fR ..]] \fIVMCORE\fR \fIDUMPFILE\fR .br \fBmakedumpfile\fR \-F [\fIOPTION\fR] [\-x \fIVMLINUX\fR|\-i \fIVMCOREINFO\fR] \fIVMCORE\fR .br
@@ -664,6 +664,15 @@ This option cannot be used with the --dump-dmesg, --reassemble and -g options. Display report messages. This is an alternative to enabling bit 4 in the level provided to --message-level. +.TP +\fB\-\-extension\fR \fIEXTENSION.SO\fR +Load makedumpfile extensions. By using extensions, users can programmably +customize mm page keeping / discard, alone with the traditional page-flag +(-d dump_level) based filtering; as well as kernel data erasure. The option +can be used multiple times to select various extensions. NOTE: this option +needs EXTENSION=on when build makedumpfile with libbpf(>=v1.4.0). Also this +option only works for kernels with DEBUG_INFO_BTF & KALLSYMS enabled. + .SH ENVIRONMENT VARIABLES .TP 8
Signature · 1 line
2.54.0
PATCH v6 10/10 Add amdgpu mm pages filtering extension
In reply to Tao LiuThis patch will introduce maple_tree & amdgpu mm page filtering extension, those mm pages allocated to amdgpu will be discarded from vmcore, in order to shrink vmcore size since mm pages allocated to amdgpu are useless to kernel crash and may contain sensitive data.
--- extensions/Makefile | 4 +- extensions/amdgpu_filter.c | 221 +++++++++++++++++++++++++ extensions/maple_tree.c | 328 +++++++++++++++++++++++++++++++++++++ extensions/maple_tree.h | 7 + 4 files changed, 559 insertions(+), 1 deletion(-) create mode 100644 extensions/amdgpu_filter.c create mode 100644 extensions/maple_tree.c create mode 100644 extensions/maple_tree.h
extensions/Makefile +3 -1
@@ -1,8 +1,10 @@ CC ?= gcc -CONTRIB_SO := sample.so erase_sample.so +CONTRIB_SO := sample.so erase_sample.so amdgpu_filter.so all: $(CONTRIB_SO) +amdgpu_filter.so: maple_tree.c + $(CONTRIB_SO): %.so: %.c $(CC) -O2 -g -fPIC -shared -Wl,-T,../makedumpfile.ld -o $@ $^
extensions/amdgpu_filter.c new file +221 -0
221 lines not on this page · raw message
extensions/maple_tree.c new file +328 -0
328 lines not on this page · raw message
extensions/maple_tree.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef _MAPLE_TREE_H +#define _MAPLE_TREE_H +#include <stdbool.h> +unsigned long *mt_dump (unsigned long mt,int *array_len); +bool maple_init (void ); +#endif /* _MAPLE_TREE_H */ +
Signature · 1 line
2.54.0
Hi Tao,
I've tested some, no issue was detected.
[without the patchset]
8.09user 1.01system 0:10.04elapsed 90%CPU (0avgtext+0avgdata 4480maxresident)k 18672inputs+763736outputs (0major+268minor)pagefaults 0swaps
[with the patchset without --extension]
8.08user 0.99system 0:09.94elapsed 91%CPU (0avgtext+0avgdata 4608maxresident)k # +128k 19640inputs+763720outputs (4major+277minor)pagefaults 0swaps
[with the patchset with --extension]
6.38user 0.74system 0:07.92elapsed 89%CPU (0avgtext+0avgdata 16984maxresident)k # +12M 17048inputs+725240outputs (0major+3121minor)pagefaults 0swaps
There is also no noticeable rise in max rss without --extension.
Maybe support for compound pages can be debatable in the future, but I think the current patchset is good enough for extension v1. So merged 01/10 - 09/10 patches, thank you for the great work!
Thanks,
Kazu
Quoted 190 lines · Tao Liu
A) This patchset will introduce the following features to makedumpfile:
1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.B) The purpose of the features are:
1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:
a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.
2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:
a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.
With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.
3) Compile and run makedumpfile extensions: $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on EXTENSION=on $ make extensions $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/extension.out --extension amdgpu_filter.so Loaded extension: ./extensions/amdgpu_filter.so makedumpfile Completed. User time (seconds): 5.08 System time (seconds): 0.84 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.95 Maximum resident set size (kbytes): 17360 ... To contrast with eppic script of v2 [2]: $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c makedumpfile Completed. User time (seconds): 8.23 System time (seconds): 0.88 Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16 Maximum resident set size (kbytes): 57128 ... -rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out -rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out -rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcoreC) Discussion:
1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. rhel8.x is not supported, others are not tested.D) Testing:
If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:
Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rdffff96b7fa800000
ffff96b7fa800000:405753585a415121!QAZXSW@
crash> rdffff96b87c800000
ffff96b87c800000:405753585a415121!QAZXSW@Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rdffff96b7fa800000
rd: page excluded: kernel virtual address:ffff96b7fa800000type: "64-bit KVADDR" crash> rdffff96b87c800000
rd: page excluded: kernel virtual address:ffff96b87c800000type: "64-bit KVADDR"[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352dfv6 -> v5:
1) Add
erase_sampleextension for data erasure sample.
2) Replacefprintf()to MSG()/ERRMSG() inkallsyms.h.
3) Usehandle_cbs_leninextension_has_callback()for callback check.
4) Fix bug[5].
5) Add handle_cbs[i]->cb = NULL ininit_extensions()to reset the callback if one extension'sinit()is skipped.
6) The patches are organized as follows:---- <customization specific> ---
10. Add amdgpu mm pages filtering extension--- <code should be merged> ---
9. Doc: Add --extension option to makedumpfile manual
8. Adderase_sampleextension as an data erase reference
7. Add sample extension as an example reference
6. Add makedumpfile extensions support
5. Implement kernel module's btf resolving
4. Implement kernel module's kallsyms resolving
3. Implement kernel btf resolving
2. Implement kernel kallsyms resolving
1. Reserve sections for makedumpfile and extenionsPatch 10 is customization specific, merging depends on the strategy of maintenance.
Patch 1 ~ 9 are common code which should be merged with makedumpfile.[5]: https://lore.kernel.org/kexec/[email protected]/
Link to v5: https://lore.kernel.org/kexec/[email protected]/
Link to v4: https://lore.kernel.org/kexec/[email protected]/
Link to v3: https://lore.kernel.org/kexec/[email protected]/
Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/Tao Liu (10):
Reserve sections for makedumpfile and extenions Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel module's kallsyms resolving Implement kernel module's btf resolving
Add makedumpfile extensions support
Add sample extension as an example reference Adderase_sampleextension as an data erase reference
Doc: Add --extension option to makedumpfile manual Add amdgpu mm pages filtering extensionMakefile | 15 +- README | 6 + btf_info.c | 371 +++++++++++++++++++++++++ btf_info.h | 77 ++++++ extension.c | 355 ++++++++++++++++++++++++ extension.h | 16 ++ extensions/Makefile | 13 + extensions/amdgpu_filter.c | 221 +++++++++++++++ extensions/erase_sample.c | 78 ++++++ extensions/maple_tree.c | 328 ++++++++++++++++++++++ extensions/maple_tree.h | 7 + extensions/sample.c | 69 +++++ kallsyms.c | 550 +++++++++++++++++++++++++++++++++++++ kallsyms.h | 86 ++++++ makedumpfile.8.in | 11 +- makedumpfile.c | 44 ++- makedumpfile.h | 12 + makedumpfile.ld | 16 ++ 18 files changed, 2266 insertions(+), 9 deletions(-)create mode 100644
btf_info.c
create mode 100644btf_info.h
create mode 100644extension.c
create mode 100644extension.h
create mode 100644 extensions/Makefile
create mode 100644extensions/amdgpu_filter.c
create mode 100644extensions/erase_sample.c
create mode 100644extensions/maple_tree.c
create mode 100644extensions/maple_tree.h
create mode 100644extensions/sample.c
create mode 100644kallsyms.c
create mode 100644kallsyms.h
create mode 100644 makedumpfile.ld
Hi Kazu,
Quoted 17 lines · HAGIO KAZUHITO(萩尾 一仁)
Hi Tao,
I've tested some, no issue was detected.
[without the patchset]
8.09user 1.01system 0:10.04elapsed 90%CPU (0avgtext+0avgdata 4480maxresident)k 18672inputs+763736outputs (0major+268minor)pagefaults 0swaps[with the patchset without --extension]
8.08user 0.99system 0:09.94elapsed 91%CPU (0avgtext+0avgdata 4608maxresident)k # +128k 19640inputs+763720outputs (4major+277minor)pagefaults 0swaps[with the patchset with --extension]
6.38user 0.74system 0:07.92elapsed 89%CPU (0avgtext+0avgdata 16984maxresident)k # +12M 17048inputs+725240outputs (0major+3121minor)pagefaults 0swaps
Quoted 5 lines
There is also no noticeable rise in max rss without --extension.
Maybe support for compound pages can be debatable in the future, but I think the current patchset is good enough for extension v1. So merged 01/10 - 09/10 patches, thank you for the great work!
Thanks for your testing, also many thanks for your code review and suggestions!
Thanks,
Tao Liu
Quoted 197 lines
Thanks,
KazuQuoted 190 lines · Tao Liu
A) This patchset will introduce the following features to makedumpfile:
1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.B) The purpose of the features are:
1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:
a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.
2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:
a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.
With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.
3) Compile and run makedumpfile extensions: $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on EXTENSION=on $ make extensions $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/extension.out --extension amdgpu_filter.so Loaded extension: ./extensions/amdgpu_filter.so makedumpfile Completed. User time (seconds): 5.08 System time (seconds): 0.84 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.95 Maximum resident set size (kbytes): 17360 ... To contrast with eppic script of v2 [2]: $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c makedumpfile Completed. User time (seconds): 8.23 System time (seconds): 0.88 Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16 Maximum resident set size (kbytes): 57128 ... -rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out -rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out -rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcoreC) Discussion:
1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. rhel8.x is not supported, others are not tested.D) Testing:
If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:
Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rdffff96b7fa800000
ffff96b7fa800000:405753585a415121!QAZXSW@
crash> rdffff96b87c800000
ffff96b87c800000:405753585a415121!QAZXSW@Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rdffff96b7fa800000
rd: page excluded: kernel virtual address:ffff96b7fa800000type: "64-bit KVADDR" crash> rdffff96b87c800000
rd: page excluded: kernel virtual address:ffff96b87c800000type: "64-bit KVADDR"[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352dfv6 -> v5:
1) Add
erase_sampleextension for data erasure sample.
2) Replacefprintf()to MSG()/ERRMSG() inkallsyms.h.
3) Usehandle_cbs_leninextension_has_callback()for callback check.
4) Fix bug[5].
5) Add handle_cbs[i]->cb = NULL ininit_extensions()to reset the callback if one extension'sinit()is skipped.
6) The patches are organized as follows:---- <customization specific> ---
10. Add amdgpu mm pages filtering extension--- <code should be merged> ---
9. Doc: Add --extension option to makedumpfile manual
8. Adderase_sampleextension as an data erase reference
7. Add sample extension as an example reference
6. Add makedumpfile extensions support
5. Implement kernel module's btf resolving
4. Implement kernel module's kallsyms resolving
3. Implement kernel btf resolving
2. Implement kernel kallsyms resolving
1. Reserve sections for makedumpfile and extenionsPatch 10 is customization specific, merging depends on the strategy of maintenance.
Patch 1 ~ 9 are common code which should be merged with makedumpfile.[5]: https://lore.kernel.org/kexec/[email protected]/
Link to v5: https://lore.kernel.org/kexec/[email protected]/
Link to v4: https://lore.kernel.org/kexec/[email protected]/
Link to v3: https://lore.kernel.org/kexec/[email protected]/
Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/Tao Liu (10):
Reserve sections for makedumpfile and extenions Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel module's kallsyms resolving Implement kernel module's btf resolving
Add makedumpfile extensions support
Add sample extension as an example reference Adderase_sampleextension as an data erase reference
Doc: Add --extension option to makedumpfile manual Add amdgpu mm pages filtering extensionMakefile | 15 +- README | 6 + btf_info.c | 371 +++++++++++++++++++++++++ btf_info.h | 77 ++++++ extension.c | 355 ++++++++++++++++++++++++ extension.h | 16 ++ extensions/Makefile | 13 + extensions/amdgpu_filter.c | 221 +++++++++++++++ extensions/erase_sample.c | 78 ++++++ extensions/maple_tree.c | 328 ++++++++++++++++++++++ extensions/maple_tree.h | 7 + extensions/sample.c | 69 +++++ kallsyms.c | 550 +++++++++++++++++++++++++++++++++++++ kallsyms.h | 86 ++++++ makedumpfile.8.in | 11 +- makedumpfile.c | 44 ++- makedumpfile.h | 12 + makedumpfile.ld | 16 ++ 18 files changed, 2266 insertions(+), 9 deletions(-)create mode 100644
btf_info.c
create mode 100644btf_info.h
create mode 100644extension.c
create mode 100644extension.h
create mode 100644 extensions/Makefile
create mode 100644extensions/amdgpu_filter.c
create mode 100644extensions/erase_sample.c
create mode 100644extensions/maple_tree.c
create mode 100644extensions/maple_tree.h
create mode 100644extensions/sample.c
create mode 100644kallsyms.c
create mode 100644kallsyms.h
create mode 100644 makedumpfile.ld