Skip to contentSkip to conversation

btf/kallsyms based makedumpfile extension for mm page filtering

PATCH Superseded Tao Liu 3 participants 23 messages Jan 20 – Mar 17

Subsystems (3)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Tao Liu

author
#0 lore raw

v3 cover btf/kallsyms based makedumpfile extension for mm page filtering

A) This patchset will introduce the following features to makedumpfile:

1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.

B) The purpose of the features are:

1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:

a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.

But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.

2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:

a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.

makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.

With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.

3) Compile and run makedumpfile extensions:

  $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on
  $ make extensions
  $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
    /tmp/extension.out
    Loaded extension: ./extensions/amdgpu_filter.so
    makedumpfile Completed.
        User time (seconds): 6.37
        System time (seconds): 0.70
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.10
        Maximum resident set size (kbytes): 38024
        ...

To contrast with eppic script of v2 [2]:

  $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
    /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c   
    makedumpfile Completed.
        User time (seconds): 8.23
        System time (seconds): 0.88
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
        Maximum resident set size (kbytes): 57128
        ...

-rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
-rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
-rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore

C) Discussion:

1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. Others are not tested.

D) Testing:

If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:

Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@

Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"

[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df

v3 -> v2:

1) Removed btf/kallsyms support for eppic script, and introduced makedumpfile .so extension instead. The reason of removing eppic support is:
a) Native binary code as .so has better performance than scripting, see the time consumption contrast above.
b) Eppic library has memory leaks which hasn't been fixed totally, memeory leaks in 2nd kernel might be fatal.

2) Removed the code of manually parsing btf info, and used libbpf for btf info parsing instead. The reason of removing manually parsing is:
a) Less code modification to makedumpfile, easier to maintain.
b) The performance of using libbpf is as good as manual parsing + hash table indexing, as well as less memory consumption, see time and memory consumption contrast above.

3) The patches are organized as follows:

--- <only for test purpose, don't merge> --- 8.Filter amdgpu mm pages
7.Add maple tree support to makedumpfile extension

--- <code should be merged> ---
6.Add page filtering function
5.Add makedumpfile extension support 4.Implement kernel modules' btf resolving 3.Implement kernel modules' kallsyms resolving 2.Implement kernel btf resolving 1.Implement kernel kallsyms resolving

Patch 7 & 8 are customization specific, which can be maintained separately. Patch 1 ~ 6 are common code which should be integrate with makedumpfile.

Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/

Tao Liu (8):
Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel modules' kallsyms resolving Implement kernel modules' btf resolving Add makedumpfile extension support
Add page filtering function
Add maple tree support to makedumpfile extension Filter amdgpu mm pages

 Makefile                   |   9 +-
 btf_info.c                 | 260 +++++++++++++++++++++++++
 btf_info.h                 |  66 +++++++
 erase_info.c               |  98 ++++++++++
 erase_info.h               |  12 ++
 extension.c                |  82 ++++++++
 extensions/Makefile        |  10 +
 extensions/amdgpu_filter.c |  90 +++++++++
 extensions/maple_tree.c    | 336 +++++++++++++++++++++++++++++++++
 extensions/maple_tree.h    |   6 +
 kallsyms.c                 | 376 +++++++++++++++++++++++++++++++++++++
 kallsyms.h                 |  20 ++
 makedumpfile.c             |  35 +++-
 makedumpfile.h             |  11 ++
 14 files changed, 1405 insertions(+), 6 deletions(-)
 create mode 100644 btf_info.c
 create mode 100644 btf_info.h
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
 create mode 100644 extensions/amdgpu_filter.c
 create mode 100644 extensions/maple_tree.c
 create mode 100644 extensions/maple_tree.h
 create mode 100644 kallsyms.c
 create mode 100644 kallsyms.h
Signature · 1 line
2.47.0
author
#1 lore raw

PATCH v3 1/8 Implement kernel kallsyms resolving

This patch will parse kernel's kallsyms data, and store them into a hash table so they can be referenced later in a fast speed.

---
 Makefile       |   2 +-
 kallsyms.c     | 265 +++++++++++++++++++++++++++++++++++++++++++++++++
 kallsyms.h     |  17 ++++
 makedumpfile.c |   3 +
 makedumpfile.h |  11 ++
 5 files changed, 297 insertions(+), 1 deletion(-)
 create mode 100644 kallsyms.c
 create mode 100644 kallsyms.h
Makefile +1 -1
index 05ab5f2..6c450ac 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
kallsyms.c new file +265 -0

265 lines not on this page · raw message

kallsyms.h new file +17 -0
new file mode 100644
index 0000000..a4fbe10
@@ -0,0 +1,17 @@
+#ifndef _KALLSYMS_H
+#define _KALLSYMS_H
+
+#include <stdint.h>
+#include <stdbool.h>
+
+struct __attribute__((packed)) syment {
+	uint64_t value;
+	char *name;
+	struct syment *name_hash_next;
+};
+
+bool read_vmcoreinfo_kallsyms(void);
+bool init_kernel_kallsyms(void);
+uint64_t get_kallsyms_value_by_name(char *);
+
+#endif /* _KALLSYMS_H */
\ No newline at end of file
makedumpfile.c +3 -0
index 12fb0d8..dba3628 100644
@@ -27,6 +27,7 @@
 #include <limits.h>
 #include <assert.h>
 #include <zlib.h>
+#include "kallsyms.h"
 
 struct symbol_table	symbol_table;
 struct size_table	size_table;
@@ -3105,6 +3106,8 @@ read_vmcoreinfo_from_vmcore(off_t offset, unsigned long size, int flag_xen_hv)
 		if (!read_vmcoreinfo())
 			goto out;
 	}
+	read_vmcoreinfo_kallsyms();
+
 	close_vmcoreinfo();
 
 	ret = TRUE;
makedumpfile.h +11 -0
index 134eb7a..0dec50e 100644
@@ -259,6 +259,7 @@ static inline int string_exists(char *s) { return (s ? TRUE : FALSE); }
 #define UINT(ADDR)	*((unsigned int *)(ADDR))
 #define ULONG(ADDR)	*((unsigned long *)(ADDR))
 #define ULONGLONG(ADDR)	*((unsigned long long *)(ADDR))
+#define VOID_PTR(ADDR)  *((void **)(ADDR))
 
 
 /*
@@ -1919,6 +1920,16 @@ struct symbol_table {
 	 * symbols on sparc64 arch
 	 */
 	unsigned long long		vmemmap_table;
+
+	/*
+	 * kallsyms related
+	 */
+	unsigned long long		kallsyms_names;
+	unsigned long long		kallsyms_num_syms;
+	unsigned long long		kallsyms_token_table;
+	unsigned long long		kallsyms_token_index;
+	unsigned long long		kallsyms_offsets;
+	unsigned long long		kallsyms_relative_base;
 };
 
 struct size_table {
Signature · 1 line
2.47.0
#2 lore raw

Re: [PATCH v3 1/8] Implement kernel kallsyms resolving

Hi Tao,

You've managed to do nearly the same as my implementations of kallsyms, but with far less code! Nice! Thank you for this.

A few comments inline.

Quoted 140 lines · Tao Liu

This patch will parse kernel's kallsyms data, and store them into a hash table so they can be referenced later in a fast speed.

---
 Makefile       |   2 +-
 kallsyms.c     | 265 +++++++++++++++++++++++++++++++++++++++++++++++++
 kallsyms.h     |  17 ++++
 makedumpfile.c |   3 +
 makedumpfile.h |  11 ++
 5 files changed, 297 insertions(+), 1 deletion(-)
 create mode 100644 kallsyms.c
 create mode 100644 kallsyms.h
Makefile +1 -1
index 05ab5f2..6c450ac 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
kallsyms.c new file +107 -0
new file mode 100644
index 0000000..ecf64e0
@@ -0,0 +1,265 @@
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include "makedumpfile.h"
+#include "kallsyms.h"
+
+static uint32_t *kallsyms_offsets = NULL;
+static uint16_t *kallsyms_token_index = NULL;
+static uint8_t  *kallsyms_token_table = NULL;
+static uint8_t  *kallsyms_names = NULL;
+static unsigned long kallsyms_relative_base = 0;
+static unsigned int kallsyms_num_syms = 0;
+
+#define NAME_HASH 512
+static struct syment *name_hash_table[NAME_HASH] = {0};
+
+static uint64_t absolute_percpu(uint64_t base, int32_t val)
+{
+	if (val >= 0)
+		return (uint64_t)val;
+	else
+		return base - 1 - val;
+}
+
+static unsigned int hash_index(const char *name, unsigned int hash_size)
+{
+	unsigned int len, value;
+
+	len = strlen(name);
+	value = name[len - 1] * name[len / 2];
+
+	return (name[0] ^ value) % hash_size;
+}
+
+static void name_hash_install(struct syment *en)
+{
+	unsigned int index = hash_index(en->name, NAME_HASH);
+	struct syment *sp = name_hash_table[index];
+
+	if (sp == NULL) {
+		name_hash_table[index] = en;
+	} else {
+		while (sp) {
+			if (sp->name_hash_next) {
+				sp = sp->name_hash_next;
+			} else {
+				sp->name_hash_next = en;
+				break;
+			}
+		}
+	}
+}
+
+static struct syment *search_kallsyms_by_name(char *name)
+{
+	unsigned int index;
+	struct syment *sp;
+
+	index = hash_index(name, NAME_HASH);
+	for (sp = name_hash_table[index]; sp; sp = sp->name_hash_next) {
+		if (!strcmp(name, sp->name)) {
+			return sp;
+		}
+	}
+	return sp;
+}
+
+static bool is_unwanted_symbol(char *name)
+{
+	const char *unwanted_prefix[] = {
+		"__pfx_",	// CFI symbols
+		"_R",		// Rust symbols
+	};
+	for (int i = 0; i < sizeof(unwanted_prefix) / sizeof(char *); i++) {
+		if (!strncmp(name, unwanted_prefix[i], strlen(unwanted_prefix[i])))
+			return true;
+	}
+	return false;
+}
+
+uint64_t get_kallsyms_value_by_name(char *name)
+{
+	struct syment *sp;
+
+	sp = search_kallsyms_by_name(name);
+	if (!sp)
+		return 0;
+	return sp->value;
+}
+
+#define BUFLEN 1024
+static bool parse_kernel_kallsyms(void)
+{
+	char buf[BUFLEN];
+	int index = 0, i;
+	uint8_t *compressd_data;
+	uint8_t *uncompressd_data;
+	uint64_t stext;
+	uint8_t len, len_old;
+	struct syment *kern_syment;
+	bool skip;
+
+	for (i = 0; i < kallsyms_num_syms; i++) {
+		skip = false;
+		memset(buf, 0, BUFLEN);
+		len = kallsyms_names[index];
+		if (len & 0x80) {

kallsyms.c line 117

Quoted 10 lines
kallsyms.c new file +10 -0
+			index++;
+			len_old = len;
+			len = kallsyms_names[index];
+			if (len & 0x80) {
+				fprintf(stderr, "%s: BUG! Unexpected 3-byte length,"
+					" should be detected in init_kernel_kallsyms()\n",
+					__func__);
+				goto out;
+			}
+			len = (len_old & 0x7F) | (len << 7);

The 2-byte representation was added in commit 73bbb94466fd3 ("kallsyms:
support "big" kernel symbols"), in v6.1. It seems useful to include a comment about that, at a minimum.

It also seems to me that, for older kernel versions, this means lengths 128-255 are ambiguous: for v6.1+, they indicate a long symbol, but for kernel versions prior to that, they are valid lengths.

I guess this is implemented for current kernels, but it might be worth checking the kernel major/minor version for this. Though, I haven't personally witnessed the issue, so maybe it's unnecessary. I will test this on some older kernels and let you know.

Quoted 16 lines
kallsyms.c new file +16 -0
+		}
+		index++;
+
+		compressd_data = &kallsyms_names[index];
+		index += len;
+		while (len--) {
+			uncompressd_data = &kallsyms_token_table[kallsyms_token_index[*compressd_data]];
+			if (strlen(buf) + strlen((char *)uncompressd_data) >= BUFLEN) {
+				skip = true;
+				break;
+			}
+			strcat(buf, (char *)uncompressd_data);
+			compressd_data++;
+		}
+		if (skip || is_unwanted_symbol(&buf[1]))
+			continue;

kallsyms.c line 143

Quoted 10 lines
kallsyms.c new file +10 -0
+		kern_syment = (struct syment *)calloc(1, sizeof(struct syment));
+		if (!kern_syment)
+			goto no_mem;
+		kern_syment->value = kallsyms_offsets[i];
+		kern_syment->name = strdup(&buf[1]);
+		if (!kern_syment->name) {
+			free(kern_syment);
+			goto no_mem;
+		}
+		name_hash_install(kern_syment);

Like I mentioned in a prior email, if we were able to know the list of symbols we care about up-front, we could entirely avoid creating the hash table, and also avoid maintaining a list of symbol prefixes we want to skip loading. I'm not certain that you would want to go that far, but it's a thought.

kallsyms.c line 146

Quoted 3 lines
kallsyms.c new file +3 -0
+	}
+
+	/* Now refresh the absolute each kallsyms address */

I think this could use a better comment. This is my understanding of the history of the kallsyms address encoding history:

Kallsyms originally stored absolute symbol addresses in a plain array called "kallsyms_addresses". This strategy was called "absolute kallsyms". In Linux v4.6, commit 2213e9a66bb87 ("kallsyms:
add support for relative offsets in kallsyms address table"), introduced two ways of storing symbol addresses relative two a base address, so that 64-bit architectures could use 32-bit arrays. These methods were CONFIG_KALLSYMS_BASE_RELATIVE and CONFIG_KALLSYMS_ABSOLUTE_PERCPU. The ABSOLUTE_PERCPU mechanism was used by architectures like x86_64 with a percpu address range near 0x0, but kernel address range in the negative address space. Some architectures, namely tile and ia64, had to continue using absolute kallsyms due to very large gaps in their address spaces.

After both architectures were removed, absolute percpu was dropped in v6.11 commit 64e166099b69b ("kallsyms: get rid of code for absolute kallsyms"). In v6.15, the x86_64 percpu address range was moved away from 0x0, and as a result ABSOLUTE_PERCPU was no longer required. It was dropped in 01157ddc58dc2 ("kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU"), leaving only the BASE_RELATIVE scheme (which no longer has a kconfig entry, since there is no other scheme).

This code implements support for BASE_RELATIVE and ABSOLUTE_PERCPU, but absolute percpu is not supported. The kallsyms symbols themselves were only added to vmcoreinfo in v6.0 with commit f09bddbd86619 ("vmcoreinfo: add kallsyms_num_syms symbol"). At that time, only ia64 would have used the absolute percpu mechanism. Even if these commits were backported to quite old kernels, BASE_RELATIVE and ABSOLUTE_PERCPU would suffice for most other architectures until v4.6 and earlier.

I don't know if all of the context is necessary, but I find it helpful to know.

Thanks,
Stephen

Quoted 195 lines
kallsyms.c new file +119 -0
+	stext = get_kallsyms_value_by_name("_stext");
+	if (SYMBOL(_stext) == absolute_percpu(kallsyms_relative_base, stext)) {
+		for (i = 0; i < NAME_HASH; i++) {
+			for (kern_syment = name_hash_table[i];
+			     kern_syment;
+			     kern_syment = kern_syment->name_hash_next)
+				kern_syment->value = absolute_percpu(kallsyms_relative_base,
+							kern_syment->value);
+		}
+	} else if (SYMBOL(_stext) == kallsyms_relative_base + stext) {
+		for (i = 0; i < NAME_HASH; i++) {
+			for (kern_syment = name_hash_table[i];
+			     kern_syment;
+			     kern_syment = kern_syment->name_hash_next)
+				kern_syment->value += kallsyms_relative_base;
+		}
+	} else {
+		fprintf(stderr, "%s: Wrong calculate kallsyms symbol value!\n", __func__);
+		goto out;
+	}
+
+	return true;
+no_mem:
+	fprintf(stderr, "%s: Not enough memory!\n", __func__);
+out:
+	return false;
+}
+
+static bool vmcore_info_ready = false;
+
+bool read_vmcoreinfo_kallsyms(void)
+{
+	READ_SYMBOL("kallsyms_names", kallsyms_names);
+	READ_SYMBOL("kallsyms_num_syms", kallsyms_num_syms);
+	READ_SYMBOL("kallsyms_token_table", kallsyms_token_table);
+	READ_SYMBOL("kallsyms_token_index", kallsyms_token_index);
+	READ_SYMBOL("kallsyms_offsets", kallsyms_offsets);
+	READ_SYMBOL("kallsyms_relative_base", kallsyms_relative_base);
+	vmcore_info_ready = true;
+	return true;
+}
+
+bool init_kernel_kallsyms(void)
+{
+	const int token_index_size = (UINT8_MAX + 1) * sizeof(uint16_t);
+	uint64_t last_token, len;
+	unsigned char data, data_old;
+	int i;
+	bool ret = false;
+
+	if (vmcore_info_ready == false) {
+		fprintf(stderr, "%s: vmcoreinfo not ready for kallsyms!\n",
+			__func__);
+		return ret;
+	}
+
+	readmem(VADDR, SYMBOL(kallsyms_num_syms), &kallsyms_num_syms,
+		sizeof(kallsyms_num_syms));
+	readmem(VADDR, SYMBOL(kallsyms_relative_base), &kallsyms_relative_base,
+		sizeof(kallsyms_relative_base));
+
+	kallsyms_offsets = malloc(sizeof(uint32_t) * kallsyms_num_syms);
+	if (!kallsyms_offsets)
+		goto no_mem;
+	readmem(VADDR, SYMBOL(kallsyms_offsets), kallsyms_offsets,
+		kallsyms_num_syms * sizeof(uint32_t));
+
+	kallsyms_token_index = malloc(token_index_size);
+	if (!kallsyms_token_index)
+		goto no_mem;
+	readmem(VADDR, SYMBOL(kallsyms_token_index), kallsyms_token_index,
+		token_index_size);
+
+	last_token = SYMBOL(kallsyms_token_table) + kallsyms_token_index[UINT8_MAX];
+	do {
+		readmem(VADDR, last_token++, &data, 1);
+	} while(data);
+	len = last_token - SYMBOL(kallsyms_token_table);
+	kallsyms_token_table = malloc(len);
+	if (!kallsyms_token_table)
+		goto no_mem;
+	readmem(VADDR, SYMBOL(kallsyms_token_table), kallsyms_token_table, len);
+
+	for (len = 0, i = 0; i < kallsyms_num_syms; i++) {
+		readmem(VADDR, SYMBOL(kallsyms_names) + len, &data, 1);
+		if (data & 0x80) {
+			len += 1;
+			data_old = data;
+			readmem(VADDR, SYMBOL(kallsyms_names) + len, &data, 1);
+			if (data & 0x80) {
+				fprintf(stderr, "%s: BUG! Unexpected 3-byte length"
+					" encoding in kallsyms names\n", __func__);
+				goto out;
+			}
+			data = (data_old & 0x7F) | (data << 7);
+		}
+		len += data + 1;
+	}
+	kallsyms_names = malloc(len);
+	if (!kallsyms_names)
+		goto no_mem;
+	readmem(VADDR, SYMBOL(kallsyms_names), kallsyms_names, len);
+
+	ret = parse_kernel_kallsyms();
+	goto out;
+
+no_mem:
+	fprintf(stderr, "%s: Not enough memory!\n", __func__);
+out:
+	if (kallsyms_offsets)
+		free(kallsyms_offsets);
+	if (kallsyms_token_index)
+		free(kallsyms_token_index);
+	if (kallsyms_token_table)
+		free(kallsyms_token_table);
+	if (kallsyms_names)
+		free(kallsyms_names);
+	return ret;
+}
kallsyms.h new file +17 -0
new file mode 100644
index 0000000..a4fbe10
@@ -0,0 +1,17 @@
+#ifndef _KALLSYMS_H
+#define _KALLSYMS_H
+
+#include <stdint.h>
+#include <stdbool.h>
+
+struct __attribute__((packed)) syment {
+	uint64_t value;
+	char *name;
+	struct syment *name_hash_next;
+};
+
+bool read_vmcoreinfo_kallsyms(void);
+bool init_kernel_kallsyms(void);
+uint64_t get_kallsyms_value_by_name(char *);
+
+#endif /* _KALLSYMS_H */
\ No newline at end of file
makedumpfile.c +3 -0
index 12fb0d8..dba3628 100644
@@ -27,6 +27,7 @@
 #include <limits.h>
 #include <assert.h>
 #include <zlib.h>
+#include "kallsyms.h"
 
 struct symbol_table	symbol_table;
 struct size_table	size_table;
@@ -3105,6 +3106,8 @@ read_vmcoreinfo_from_vmcore(off_t offset, unsigned long size, int flag_xen_hv)
 		if (!read_vmcoreinfo())
 			goto out;
 	}
+	read_vmcoreinfo_kallsyms();
+
 	close_vmcoreinfo();
 
 	ret = TRUE;
makedumpfile.h +11 -0
index 134eb7a..0dec50e 100644
@@ -259,6 +259,7 @@ static inline int string_exists(char *s) { return (s ? TRUE : FALSE); }
 #define UINT(ADDR)	*((unsigned int *)(ADDR))
 #define ULONG(ADDR)	*((unsigned long *)(ADDR))
 #define ULONGLONG(ADDR)	*((unsigned long long *)(ADDR))
+#define VOID_PTR(ADDR)  *((void **)(ADDR))
 
 
 /*
@@ -1919,6 +1920,16 @@ struct symbol_table {
 	 * symbols on sparc64 arch
 	 */
 	unsigned long long		vmemmap_table;
+
+	/*
+	 * kallsyms related
+	 */
+	unsigned long long		kallsyms_names;
+	unsigned long long		kallsyms_num_syms;
+	unsigned long long		kallsyms_token_table;
+	unsigned long long		kallsyms_token_index;
+	unsigned long long		kallsyms_offsets;
+	unsigned long long		kallsyms_relative_base;
 };
 
 struct size_table {
Signature · 1 line
2.47.0
author
#3 lore raw

Re: [PATCH v3 1/8] Implement kernel kallsyms resolving

Hi Stephen,

Quoted 7 lines · Stephen Brennan

Hi Tao,

You've managed to do nearly the same as my implementations of kallsyms, but with far less code! Nice! Thank you for this.

A few comments inline.

Thanks again for your detailed information and comments on the patchset. Please give me some time to digest your information and reconsider the features to be implemented for v4.

Thanks,
Tao Liu

Quoted 439 lines
Quoted 140 lines · Tao Liu

This patch will parse kernel's kallsyms data, and store them into a hash table so they can be referenced later in a fast speed.

---
 Makefile       |   2 +-
 kallsyms.c     | 265 +++++++++++++++++++++++++++++++++++++++++++++++++
 kallsyms.h     |  17 ++++
 makedumpfile.c |   3 +
 makedumpfile.h |  11 ++
 5 files changed, 297 insertions(+), 1 deletion(-)
 create mode 100644 kallsyms.c
 create mode 100644 kallsyms.h
Makefile +1 -1
index 05ab5f2..6c450ac 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
kallsyms.c new file +107 -0
new file mode 100644
index 0000000..ecf64e0
@@ -0,0 +1,265 @@
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include "makedumpfile.h"
+#include "kallsyms.h"
+
+static uint32_t *kallsyms_offsets = NULL;
+static uint16_t *kallsyms_token_index = NULL;
+static uint8_t  *kallsyms_token_table = NULL;
+static uint8_t  *kallsyms_names = NULL;
+static unsigned long kallsyms_relative_base = 0;
+static unsigned int kallsyms_num_syms = 0;
+
+#define NAME_HASH 512
+static struct syment *name_hash_table[NAME_HASH] = {0};
+
+static uint64_t absolute_percpu(uint64_t base, int32_t val)
+{
+     if (val >= 0)
+             return (uint64_t)val;
+     else
+             return base - 1 - val;
+}
+
+static unsigned int hash_index(const char *name, unsigned int hash_size)
+{
+     unsigned int len, value;
+
+     len = strlen(name);
+     value = name[len - 1] * name[len / 2];
+
+     return (name[0] ^ value) % hash_size;
+}
+
+static void name_hash_install(struct syment *en)
+{
+     unsigned int index = hash_index(en->name, NAME_HASH);
+     struct syment *sp = name_hash_table[index];
+
+     if (sp == NULL) {
+             name_hash_table[index] = en;
+     } else {
+             while (sp) {
+                     if (sp->name_hash_next) {
+                             sp = sp->name_hash_next;
+                     } else {
+                             sp->name_hash_next = en;
+                             break;
+                     }
+             }
+     }
+}
+
+static struct syment *search_kallsyms_by_name(char *name)
+{
+     unsigned int index;
+     struct syment *sp;
+
+     index = hash_index(name, NAME_HASH);
+     for (sp = name_hash_table[index]; sp; sp = sp->name_hash_next) {
+             if (!strcmp(name, sp->name)) {
+                     return sp;
+             }
+     }
+     return sp;
+}
+
+static bool is_unwanted_symbol(char *name)
+{
+     const char *unwanted_prefix[] = {
+             "__pfx_",       // CFI symbols
+             "_R",           // Rust symbols
+     };
+     for (int i = 0; i < sizeof(unwanted_prefix) / sizeof(char *); i++) {
+             if (!strncmp(name, unwanted_prefix[i], strlen(unwanted_prefix[i])))
+                     return true;
+     }
+     return false;
+}
+
+uint64_t get_kallsyms_value_by_name(char *name)
+{
+     struct syment *sp;
+
+     sp = search_kallsyms_by_name(name);
+     if (!sp)
+             return 0;
+     return sp->value;
+}
+
+#define BUFLEN 1024
+static bool parse_kernel_kallsyms(void)
+{
+     char buf[BUFLEN];
+     int index = 0, i;
+     uint8_t *compressd_data;
+     uint8_t *uncompressd_data;
+     uint64_t stext;
+     uint8_t len, len_old;
+     struct syment *kern_syment;
+     bool skip;
+
+     for (i = 0; i < kallsyms_num_syms; i++) {
+             skip = false;
+             memset(buf, 0, BUFLEN);
+             len = kallsyms_names[index];
+             if (len & 0x80) {

kallsyms.c line 117

Quoted 10 lines
kallsyms.c new file +10 -0
+                     index++;
+                     len_old = len;
+                     len = kallsyms_names[index];
+                     if (len & 0x80) {
+                             fprintf(stderr, "%s: BUG! Unexpected 3-byte length,"
+                                     " should be detected in init_kernel_kallsyms()\n",
+                                     __func__);
+                             goto out;
+                     }
+                     len = (len_old & 0x7F) | (len << 7);

The 2-byte representation was added in commit 73bbb94466fd3 ("kallsyms:
support "big" kernel symbols"), in v6.1. It seems useful to include a comment about that, at a minimum.

It also seems to me that, for older kernel versions, this means lengths 128-255 are ambiguous: for v6.1+, they indicate a long symbol, but for kernel versions prior to that, they are valid lengths.

I guess this is implemented for current kernels, but it might be worth checking the kernel major/minor version for this. Though, I haven't personally witnessed the issue, so maybe it's unnecessary. I will test this on some older kernels and let you know.

Quoted 16 lines
kallsyms.c new file +16 -0
+             }
+             index++;
+
+             compressd_data = &kallsyms_names[index];
+             index += len;
+             while (len--) {
+                     uncompressd_data = &kallsyms_token_table[kallsyms_token_index[*compressd_data]];
+                     if (strlen(buf) + strlen((char *)uncompressd_data) >= BUFLEN) {
+                             skip = true;
+                             break;
+                     }
+                     strcat(buf, (char *)uncompressd_data);
+                     compressd_data++;
+             }
+             if (skip || is_unwanted_symbol(&buf[1]))
+                     continue;

kallsyms.c line 143

Quoted 10 lines
kallsyms.c new file +10 -0
+             kern_syment = (struct syment *)calloc(1, sizeof(struct syment));
+             if (!kern_syment)
+                     goto no_mem;
+             kern_syment->value = kallsyms_offsets[i];
+             kern_syment->name = strdup(&buf[1]);
+             if (!kern_syment->name) {
+                     free(kern_syment);
+                     goto no_mem;
+             }
+             name_hash_install(kern_syment);

Like I mentioned in a prior email, if we were able to know the list of symbols we care about up-front, we could entirely avoid creating the hash table, and also avoid maintaining a list of symbol prefixes we want to skip loading. I'm not certain that you would want to go that far, but it's a thought.

kallsyms.c line 146

Quoted 3 lines
kallsyms.c new file +3 -0
+     }
+
+     /* Now refresh the absolute each kallsyms address */

I think this could use a better comment. This is my understanding of the history of the kallsyms address encoding history:

Kallsyms originally stored absolute symbol addresses in a plain array called "kallsyms_addresses". This strategy was called "absolute kallsyms". In Linux v4.6, commit 2213e9a66bb87 ("kallsyms:
add support for relative offsets in kallsyms address table"), introduced two ways of storing symbol addresses relative two a base address, so that 64-bit architectures could use 32-bit arrays. These methods were CONFIG_KALLSYMS_BASE_RELATIVE and CONFIG_KALLSYMS_ABSOLUTE_PERCPU. The ABSOLUTE_PERCPU mechanism was used by architectures like x86_64 with a percpu address range near 0x0, but kernel address range in the negative address space. Some architectures, namely tile and ia64, had to continue using absolute kallsyms due to very large gaps in their address spaces.

After both architectures were removed, absolute percpu was dropped in v6.11 commit 64e166099b69b ("kallsyms: get rid of code for absolute kallsyms"). In v6.15, the x86_64 percpu address range was moved away from 0x0, and as a result ABSOLUTE_PERCPU was no longer required. It was dropped in 01157ddc58dc2 ("kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU"), leaving only the BASE_RELATIVE scheme (which no longer has a kconfig entry, since there is no other scheme).

This code implements support for BASE_RELATIVE and ABSOLUTE_PERCPU, but absolute percpu is not supported. The kallsyms symbols themselves were only added to vmcoreinfo in v6.0 with commit f09bddbd86619 ("vmcoreinfo: add kallsyms_num_syms symbol"). At that time, only ia64 would have used the absolute percpu mechanism. Even if these commits were backported to quite old kernels, BASE_RELATIVE and ABSOLUTE_PERCPU would suffice for most other architectures until v4.6 and earlier.

I don't know if all of the context is necessary, but I find it helpful to know.

Thanks,
Stephen

Quoted 195 lines
kallsyms.c new file +119 -0
+     stext = get_kallsyms_value_by_name("_stext");
+     if (SYMBOL(_stext) == absolute_percpu(kallsyms_relative_base, stext)) {
+             for (i = 0; i < NAME_HASH; i++) {
+                     for (kern_syment = name_hash_table[i];
+                          kern_syment;
+                          kern_syment = kern_syment->name_hash_next)
+                             kern_syment->value = absolute_percpu(kallsyms_relative_base,
+                                                     kern_syment->value);
+             }
+     } else if (SYMBOL(_stext) == kallsyms_relative_base + stext) {
+             for (i = 0; i < NAME_HASH; i++) {
+                     for (kern_syment = name_hash_table[i];
+                          kern_syment;
+                          kern_syment = kern_syment->name_hash_next)
+                             kern_syment->value += kallsyms_relative_base;
+             }
+     } else {
+             fprintf(stderr, "%s: Wrong calculate kallsyms symbol value!\n", __func__);
+             goto out;
+     }
+
+     return true;
+no_mem:
+     fprintf(stderr, "%s: Not enough memory!\n", __func__);
+out:
+     return false;
+}
+
+static bool vmcore_info_ready = false;
+
+bool read_vmcoreinfo_kallsyms(void)
+{
+     READ_SYMBOL("kallsyms_names", kallsyms_names);
+     READ_SYMBOL("kallsyms_num_syms", kallsyms_num_syms);
+     READ_SYMBOL("kallsyms_token_table", kallsyms_token_table);
+     READ_SYMBOL("kallsyms_token_index", kallsyms_token_index);
+     READ_SYMBOL("kallsyms_offsets", kallsyms_offsets);
+     READ_SYMBOL("kallsyms_relative_base", kallsyms_relative_base);
+     vmcore_info_ready = true;
+     return true;
+}
+
+bool init_kernel_kallsyms(void)
+{
+     const int token_index_size = (UINT8_MAX + 1) * sizeof(uint16_t);
+     uint64_t last_token, len;
+     unsigned char data, data_old;
+     int i;
+     bool ret = false;
+
+     if (vmcore_info_ready == false) {
+             fprintf(stderr, "%s: vmcoreinfo not ready for kallsyms!\n",
+                     __func__);
+             return ret;
+     }
+
+     readmem(VADDR, SYMBOL(kallsyms_num_syms), &kallsyms_num_syms,
+             sizeof(kallsyms_num_syms));
+     readmem(VADDR, SYMBOL(kallsyms_relative_base), &kallsyms_relative_base,
+             sizeof(kallsyms_relative_base));
+
+     kallsyms_offsets = malloc(sizeof(uint32_t) * kallsyms_num_syms);
+     if (!kallsyms_offsets)
+             goto no_mem;
+     readmem(VADDR, SYMBOL(kallsyms_offsets), kallsyms_offsets,
+             kallsyms_num_syms * sizeof(uint32_t));
+
+     kallsyms_token_index = malloc(token_index_size);
+     if (!kallsyms_token_index)
+             goto no_mem;
+     readmem(VADDR, SYMBOL(kallsyms_token_index), kallsyms_token_index,
+             token_index_size);
+
+     last_token = SYMBOL(kallsyms_token_table) + kallsyms_token_index[UINT8_MAX];
+     do {
+             readmem(VADDR, last_token++, &data, 1);
+     } while(data);
+     len = last_token - SYMBOL(kallsyms_token_table);
+     kallsyms_token_table = malloc(len);
+     if (!kallsyms_token_table)
+             goto no_mem;
+     readmem(VADDR, SYMBOL(kallsyms_token_table), kallsyms_token_table, len);
+
+     for (len = 0, i = 0; i < kallsyms_num_syms; i++) {
+             readmem(VADDR, SYMBOL(kallsyms_names) + len, &data, 1);
+             if (data & 0x80) {
+                     len += 1;
+                     data_old = data;
+                     readmem(VADDR, SYMBOL(kallsyms_names) + len, &data, 1);
+                     if (data & 0x80) {
+                             fprintf(stderr, "%s: BUG! Unexpected 3-byte length"
+                                     " encoding in kallsyms names\n", __func__);
+                             goto out;
+                     }
+                     data = (data_old & 0x7F) | (data << 7);
+             }
+             len += data + 1;
+     }
+     kallsyms_names = malloc(len);
+     if (!kallsyms_names)
+             goto no_mem;
+     readmem(VADDR, SYMBOL(kallsyms_names), kallsyms_names, len);
+
+     ret = parse_kernel_kallsyms();
+     goto out;
+
+no_mem:
+     fprintf(stderr, "%s: Not enough memory!\n", __func__);
+out:
+     if (kallsyms_offsets)
+             free(kallsyms_offsets);
+     if (kallsyms_token_index)
+             free(kallsyms_token_index);
+     if (kallsyms_token_table)
+             free(kallsyms_token_table);
+     if (kallsyms_names)
+             free(kallsyms_names);
+     return ret;
+}
kallsyms.h new file +17 -0
new file mode 100644
index 0000000..a4fbe10
@@ -0,0 +1,17 @@
+#ifndef _KALLSYMS_H
+#define _KALLSYMS_H
+
+#include <stdint.h>
+#include <stdbool.h>
+
+struct __attribute__((packed)) syment {
+     uint64_t value;
+     char *name;
+     struct syment *name_hash_next;
+};
+
+bool read_vmcoreinfo_kallsyms(void);
+bool init_kernel_kallsyms(void);
+uint64_t get_kallsyms_value_by_name(char *);
+
+#endif /* _KALLSYMS_H */
\ No newline at end of file
makedumpfile.c +3 -0
index 12fb0d8..dba3628 100644
@@ -27,6 +27,7 @@
 #include <limits.h>
 #include <assert.h>
 #include <zlib.h>
+#include "kallsyms.h"
 
 struct symbol_table  symbol_table;
 struct size_table    size_table;
@@ -3105,6 +3106,8 @@ read_vmcoreinfo_from_vmcore(off_t offset, unsigned long size, int flag_xen_hv)
              if (!read_vmcoreinfo())
                      goto out;
      }
+     read_vmcoreinfo_kallsyms();
+
      close_vmcoreinfo();
 
      ret = TRUE;
makedumpfile.h +11 -1
index 134eb7a..0dec50e 100644
@@ -259,6 +259,7 @@ static inline int string_exists(char *s) { return (s ? TRUE : FALSE); }
 #define UINT(ADDR)   *((unsigned int *)(ADDR))
 #define ULONG(ADDR)  *((unsigned long *)(ADDR))
 #define ULONGLONG(ADDR)      *((unsigned long long *)(ADDR))
+#define VOID_PTR(ADDR)  *((void **)(ADDR))
 
 
 /*
@@ -1919,6 +1920,16 @@ struct symbol_table {
       * symbols on sparc64 arch
       */
      unsigned long long              vmemmap_table;
+
+     /*
+      * kallsyms related
+      */
+     unsigned long long              kallsyms_names;
+     unsigned long long              kallsyms_num_syms;
+     unsigned long long              kallsyms_token_table;
+     unsigned long long              kallsyms_token_index;
+     unsigned long long              kallsyms_offsets;
+     unsigned long long              kallsyms_relative_base;
 };
 
 struct size_table {
--

2.47.0

author
#4 lore raw

PATCH v3 2/8 Implement kernel btf resolving

In reply to Tao Liu

This patch will parse kernel's btf data using libbpf. The kernel's btf data is located between __start_BTF and __stop_BTF symbols which are resolved by kallsyms of the previous patch. The primary function implemented in this patch, is recursively diving into anonymous struct/union when encountered any, to find a member by given its name.

---
 Makefile   |   4 +-
 btf_info.c | 186 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 btf_info.h |  64 ++++++++++++++++++
 3 files changed, 252 insertions(+), 2 deletions(-)
 create mode 100644 btf_info.c
 create mode 100644 btf_info.h
Makefile +2 -2
index 6c450ac..f3f4da8 100644
@@ -45,12 +45,12 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
 
-LIBS = -ldw -lbz2 -ldl -lelf -lz
+LIBS = -ldw -lbz2 -ldl -lelf -lz -lbpf
 ifneq ($(LINKTYPE), dynamic)
 LIBS := -static $(LIBS) -llzma
 endif
btf_info.c new file +186 -0

186 lines not on this page · raw message

btf_info.h new file +64 -0

64 lines not on this page · raw message

Signature · 1 line
2.47.0
author
#5 lore raw

PATCH v3 3/8 Implement kernel modules' kallsyms resolving

In reply to Tao Liu

With kernel's kallsyms and btf ready, we can get any kernel types and symbol addresses. So we can iterate kernel modules' linked list, and parse each one of kernel module's structure to get its kallsyms data.

---
 kallsyms.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 kallsyms.h |   3 ++
 2 files changed, 114 insertions(+)
kallsyms.c +111 -0
index ecf64e0..9069c88 100644
@@ -3,6 +3,7 @@
 #include <string.h>
 #include "makedumpfile.h"
 #include "kallsyms.h"
+#include "btf_info.h"
 
 static uint32_t *kallsyms_offsets = NULL;
 static uint16_t *kallsyms_token_index = NULL;
@@ -263,3 +264,113 @@ out:
 		free(kallsyms_names);
 	return ret;
 }
+
+#define MEMBER_OFF(S, M) \
+	GET_STRUCT_MEMBER_MOFF(S, M) / 8
+
+uint64_t next_list(uint64_t list)
+{
+	static int list_head_next_offset = 0;
+	static int list_head_next_size = 0;
+
+	struct struct_member_info smi;
+	uint64_t next = 0;
+
+	if (!list_head_next_size) {
+		INIT_STRUCT_MEMBER(list_head, next);
+		list_head_next_size = GET_STRUCT_MEMBER_MSIZE(list_head, next);
+		list_head_next_offset = MEMBER_OFF(list_head, next);
+	}
+	readmem(VADDR, list + list_head_next_offset, &next, list_head_next_size);
+	return next;
+}
+
+bool init_module_kallsyms(void)
+{
+	struct struct_member_info smi;
+	uint64_t modules, list, value = 0, symtab = 0, strtab = 0;
+	uint32_t st_name = 0;
+	int num_symtab, i, j;
+	struct syment *mod_syment;
+	char symname[512], ch;
+	bool ret = false;
+
+	modules = get_kallsyms_value_by_name("modules");
+	if (!modules) {
+		/* Not a failure if no module enabled */
+		ret = true;
+		goto out;
+	}
+
+	INIT_STRUCT_MEMBER(module, list);
+	INIT_STRUCT_MEMBER(module, core_kallsyms);
+	INIT_STRUCT_MEMBER(mod_kallsyms, symtab);
+	INIT_STRUCT_MEMBER(mod_kallsyms, num_symtab);
+	INIT_STRUCT_MEMBER(mod_kallsyms, strtab);
+	INIT_STRUCT_MEMBER(elf64_sym, st_name);
+	INIT_STRUCT_MEMBER(elf64_sym, st_value);
+
+	for (list = next_list(modules); list != modules; list = next_list(list)) {
+		readmem(VADDR, list - MEMBER_OFF(module, list) +
+				MEMBER_OFF(module, core_kallsyms) +
+				MEMBER_OFF(mod_kallsyms, num_symtab),
+			&num_symtab, GET_STRUCT_MEMBER_MSIZE(mod_kallsyms, num_symtab));
+		readmem(VADDR, list - MEMBER_OFF(module, list) +
+				MEMBER_OFF(module, core_kallsyms) +
+				MEMBER_OFF(mod_kallsyms, symtab),
+			&symtab, GET_STRUCT_MEMBER_MSIZE(mod_kallsyms, symtab));
+		readmem(VADDR, list - MEMBER_OFF(module, list) +
+				MEMBER_OFF(module, core_kallsyms) +
+				MEMBER_OFF(mod_kallsyms, strtab),
+			&strtab, GET_STRUCT_MEMBER_MSIZE(mod_kallsyms, strtab));
+		for (i = 0; i < num_symtab; i++) {
+			j = 0;
+			readmem(VADDR, symtab + i * GET_STRUCT_MEMBER_SSIZE(elf64_sym, st_value) +
+					MEMBER_OFF(elf64_sym, st_value),
+				&value, GET_STRUCT_MEMBER_MSIZE(elf64_sym, st_value));
+			readmem(VADDR, symtab + i * GET_STRUCT_MEMBER_SSIZE(elf64_sym, st_name) +
+					MEMBER_OFF(elf64_sym, st_name),
+				&st_name, GET_STRUCT_MEMBER_MSIZE(elf64_sym, st_name));
+			do {
+				readmem(VADDR, strtab + st_name + j++, &ch, 1);
+			} while (ch != '\0');
+			if (j == 1 || j > sizeof(symname))
+				/* Skip empty or too long string */
+				continue;
+			readmem(VADDR, strtab + st_name, symname, j);
+			if (is_unwanted_symbol(symname))
+				continue;
+			mod_syment = (struct syment *)calloc(1, sizeof(struct syment));
+			if (!mod_syment)
+				goto no_mem;
+			mod_syment->name = strdup(symname);
+			if (!mod_syment->name) {
+				free(mod_syment);
+				goto no_mem;
+			}
+			mod_syment->value = value;
+			name_hash_install(mod_syment);
+		}
+	}
+	ret = true;
+	goto out;
+no_mem:
+	/* Hashtable will be cleaned later */
+	fprintf(stderr, "%s: Not enough memory!\n", __func__);
+out:
+	return ret;
+}
+
+void cleanup_kallsyms(void)
+{
+	struct syment *en, *en_tmp;
+
+	for (int i = 0; i < NAME_HASH; i++) {
+		for (en = name_hash_table[i]; en;) {
+			en_tmp = en;
+			en = en->name_hash_next;
+			free(en_tmp->name);
+			free(en_tmp);
+		}
+	}
+}
kallsyms.h +3 -0
index a4fbe10..78af4ef 100644
@@ -12,6 +12,9 @@ struct __attribute__((packed)) syment {
 
 bool read_vmcoreinfo_kallsyms(void);
 bool init_kernel_kallsyms(void);
+bool init_module_kallsyms(void);
+void cleanup_kallsyms(void);
+uint64_t next_list(uint64_t);
 uint64_t get_kallsyms_value_by_name(char *);
 
 #endif /* _KALLSYMS_H */
\ No newline at end of file
Signature · 1 line
2.47.0
author
#6 lore raw

PATCH v3 4/8 Implement kernel modules' btf resolving

In reply to Tao Liu

Same as the previous patch, with kernel's kallsyms and btf ready, we can locate and iterate all kernel modules' btf data.

---
 btf_info.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 btf_info.h |  2 ++
 2 files changed, 76 insertions(+)
btf_info.c +74 -0
index e7f8d9a..752661c 100644
@@ -184,3 +184,77 @@ out:
 		free(buf);
 	return ret;
 }
+
+#define MEMBER_OFF(S, M) \
+	GET_STRUCT_MEMBER_MOFF(S, M) / 8
+
+bool init_module_btf(void)
+{
+	struct btf *btf_mod;
+	uint64_t btf_modules, list;
+	struct struct_member_info smi;
+	uint64_t btf = 0, data = 0, module = 0;
+	int data_size = 0;
+	bool ret = false;
+	char *btf_buf = NULL;
+	char *modname = NULL;
+
+	btf_modules = get_kallsyms_value_by_name("btf_modules");
+	if (!btf_modules)
+		/* Maybe module is not enabled, this is not an error */
+		return true;
+
+	INIT_STRUCT_MEMBER(btf_module, list);
+	INIT_STRUCT_MEMBER(btf_module, btf);
+	INIT_STRUCT_MEMBER(btf_module, module);
+	INIT_STRUCT_MEMBER(module, name);
+	INIT_STRUCT_MEMBER(btf, data);
+	INIT_STRUCT_MEMBER(btf, data_size);
+	modname = (char *)malloc(GET_STRUCT_MEMBER_MSIZE(module, name));
+	if (!modname)
+		goto no_mem;
+
+	for (list = next_list(btf_modules); list != btf_modules; list = next_list(list)) {
+		readmem(VADDR, list - MEMBER_OFF(btf_module, list) +
+				MEMBER_OFF(btf_module, btf),
+			&btf, GET_STRUCT_MEMBER_MSIZE(btf_module, btf));
+		readmem(VADDR, list - MEMBER_OFF(btf_module, list) +
+				MEMBER_OFF(btf_module, module),
+			&module, GET_STRUCT_MEMBER_MSIZE(btf_module, module));
+		readmem(VADDR, module + MEMBER_OFF(module, name),
+			modname, GET_STRUCT_MEMBER_MSIZE(module, name));
+		readmem(VADDR, btf + MEMBER_OFF(btf, data),
+			&data, GET_STRUCT_MEMBER_MSIZE(btf, data));
+		readmem(VADDR, btf + MEMBER_OFF(btf, data_size),
+			&data_size, GET_STRUCT_MEMBER_MSIZE(btf, data_size));
+		btf_buf = (char *)malloc(data_size);
+		if (!btf_buf)
+			goto no_mem;
+		readmem(VADDR, data, btf_buf, data_size);
+		btf_mod = btf__new_split(btf_buf, data_size, btf_arr[0].btf);
+		free(btf_buf);
+		if (libbpf_get_error(btf_mod) != 0 ||
+		    add_to_btf_arr(btf_mod, strdup(modname)) == false) {
+			fprintf(stderr, "%s: init %s btf fail\n", __func__, modname);
+			goto out;
+		}
+	}
+	ret = true;
+	goto out;
+
+no_mem:
+	fprintf(stderr, "%s: Not enough memory!\n", __func__);
+out:
+	if (modname)
+		free(modname);
+	return ret;
+}
+
+void cleanup_btf(void)
+{
+	for (int i = 0; i < btf_arr_len; i++) {
+		free(btf_arr[i].module);
+		btf__free(btf_arr[i].btf);
+	}
+	free(btf_arr);
+}
btf_info.h +2 -0
index 1fc6829..d4408c3 100644
@@ -16,7 +16,9 @@ struct struct_member_info {
 };
 
 bool init_kernel_btf(void);
+bool init_module_btf(void);
 bool get_struct_member_by_name(struct struct_member_info *smi);
+void cleanup_btf(void);
 
 struct member_off_size {
 	int m_off;
Signature · 1 line
2.47.0
author
#7 lore raw

PATCH v3 5/8 Add makedumpfile extension support

In reply to Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
 	rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+	$(MAKE) -C extensions clean
 
 install:
 	install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
 	mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
 	install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
 	install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+	$(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +82 -0

83 lines not on this page · raw message

extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+	$(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+	rm -f $(CONTRIB_SO)
makedumpfile.c +4 -0
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
 	}
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
 	if (info->flag_refiltering)
 		update_dump_level();
 
+	run_extensions();
+
 	if ((info->name_filterconfig || info->name_eppic_config)
 			&& !gather_filter_info())
 		return FALSE;
Signature · 1 line
2.47.0
#8 lore raw

Re: [PATCH v3 5/8] Add makedumpfile extension support

Hi Tao,

This series looks really great -- I'm excited to see the switch to native .so extensions instead of epicc. I've applied the series locally and I'll rebuild my userspace stack inclusion feature based on it, to try it out myself.

In the meantime, I'll share some of my feedback on the patches (though I'm not a makedumpfile developer). This seems like the most important patch in terms of design, so I'll start here.

Quoted 78 lines · Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
 	rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+	$(MAKE) -C extensions clean
 
 install:
 	install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
 	mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
 	install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
 	install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+	$(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +21 -0
new file mode 100644
index 0000000..6ee7f4e
@@ -0,0 +1,82 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <dlfcn.h>
+#include <stdbool.h>
+#include "kallsyms.h"
+#include "btf_info.h"
+
+static const char *dirs[] = {
+	"/usr/lib64/makedumpfile/extensions",
+	"./extensions",
+};
+
+/* Will only init once */
+static bool init_kallsyms_btf(void)
+{
+	static bool ret = false;
+	static bool has_inited = false;
+
+	if (has_inited)

extension.c line 31

Quoted 10 lines
extension.c new file +10 -0
+		goto out;
+	if (!init_kernel_kallsyms())
+		goto out;
+	if (!init_kernel_btf())
+		goto out;
+	if (!init_module_kallsyms())
+		goto out;
+	if (!init_module_btf())
+		goto out;
+	ret = true;

I feel it would be good practice to load as little information as is necessary for the task. If "amdgpu" module is required, then load kernel kallsyms, BTF, and then the amdgpu module kallsyms & BTF. If no module debuginfo is required, then just the kernel would suffice.

This would reduce memory usage and runtime, though I don't know if it would show up in profiling. The main benefit could be reliability: by handling less data, there are fewer chances to hit an error.

Quoted 18 lines
extension.c new file +18 -0
+out:
+	has_inited = true;
+	return ret;
+}
+
+static void cleanup_kallsyms_btf(void)
+{
+	cleanup_kallsyms();
+	cleanup_btf();
+}
+
+void run_extensions(void)
+{
+	DIR *dir;
+	struct dirent *entry;
+	size_t len;
+	int i;
+	void *handle;

extension.c line 59

Quoted 10 lines
extension.c new file +10 -0
+	char path[512];
+
+	for (i = 0; i < sizeof(dirs) / sizeof(char *); i++) {
+		if ((dir = opendir(dirs[i])) != NULL)
+			break;
+	}
+
+	if (!dir || i >= sizeof(dirs) / sizeof(char *))
+		/* No extensions found */
+		return;

It could be confusing that makedumpfile would behave differently with the same command-line arguments depending on the presence or absence of these extensions on the filesystem.

I think it may fit users' expectations better if they are required to specify extensions on the command line. Then we could load them by searching each directory in order. This allows:

(a) more expected behavior
(b) multiple extensions can exist without all being enabled, thus more flexibility
(c) extensions can be present in the local "extensions/" directory, or in the system directory

Quoted 6 lines
extension.c new file +6 -0
+	while ((entry = readdir(dir)) != NULL) {
+		len = strlen(entry->d_name);
+		if (len > 3 && strcmp(entry->d_name + len - 3, ".so") == 0) {
+			/* Will only init when .so exist */
+			if (!init_kallsyms_btf())
+				goto out;

extension.c line 76

Quoted 10 lines
extension.c new file +10 -0
+
+			snprintf(path, sizeof(path), "%s/%s", dirs[i], entry->d_name);
+			handle = dlopen(path, RTLD_NOW);
+			if (!handle) {
+				fprintf(stderr, "%s: Failed to load %s: %s\n",
+					__func__, path, dlerror());
+				continue;
+			}
+			printf("Loaded extension: %s\n", path);
+			dlclose(handle);

Using the constructor/destructor of the shared object is clever! But we lose some flexibility: by the time the dlopen() returns, the constructor has executed and the plugin has thus executed.

What if we instead use dlsym() to load some symbols from the DSO? In particular, I think it would be useful if extensions could declare a list of symbols and a list of structure information which they are interested in receiving. We could use these lists to know which kernel/module kallsyms & BTF we should load. We could even load the information into the local variables of the extension, so the extension would not need to manually load it.

Of course this is more complex, but the benefit is:

1. Extensions can be written more simply, and would not need to manually load each symbol & type.
2. We could eliminate the hash tables for kallsyms & BTF, and eliminate the loading of unnecessary module information. Instead, we'd just populate the symbol addresses, struct offsets, and type sizes directly into the local variables which request them.

Again, while I don't want to prematurely optimize -- it's good to avoid loading unnecessary information. I hope I've described my idea well. I would be happy to work on an implementation of it based on your patches here, if you're interested.

Thanks,
Stephen

Quoted 47 lines
extension.c new file +6 -0
+		}
+	}
+out:
+	closedir(dir);
+	cleanup_kallsyms_btf();
+}
\ No newline at end of file
extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+	$(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+	rm -f $(CONTRIB_SO)
makedumpfile.c +4 -0
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
 	}
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
 	if (info->flag_refiltering)
 		update_dump_level();
 
+	run_extensions();
+
 	if ((info->name_filterconfig || info->name_eppic_config)
 			&& !gather_filter_info())
 		return FALSE;
Signature · 1 line
2.47.0
author
#9 lore raw

Re: [PATCH v3 5/8] Add makedumpfile extension support

Hi Stephen,

Thanks a lot for your quick reply and detailed information, I really appreciate it!

Quoted 7 lines · Stephen Brennan

Hi Tao,

This series looks really great -- I'm excited to see the switch to native .so extensions instead of epicc. I've applied the series locally and I'll rebuild my userspace stack inclusion feature based on it, to try it out myself.

Awesome, looking forward to your feedback on the code/API designs etc...

Quoted 100 lines

In the meantime, I'll share some of my feedback on the patches (though I'm not a makedumpfile developer). This seems like the most important patch in terms of design, so I'll start here.

Quoted 78 lines · Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
      rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+     $(MAKE) -C extensions clean
 
 install:
      install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
      mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
      install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
      install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+     $(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +21 -0
new file mode 100644
index 0000000..6ee7f4e
@@ -0,0 +1,82 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <dlfcn.h>
+#include <stdbool.h>
+#include "kallsyms.h"
+#include "btf_info.h"
+
+static const char *dirs[] = {
+     "/usr/lib64/makedumpfile/extensions",
+     "./extensions",
+};
+
+/* Will only init once */
+static bool init_kallsyms_btf(void)
+{
+     static bool ret = false;
+     static bool has_inited = false;
+
+     if (has_inited)

extension.c line 31

Quoted 10 lines
extension.c new file +10 -0
+             goto out;
+     if (!init_kernel_kallsyms())
+             goto out;
+     if (!init_kernel_btf())
+             goto out;
+     if (!init_module_kallsyms())
+             goto out;
+     if (!init_module_btf())
+             goto out;
+     ret = true;

I feel it would be good practice to load as little information as is necessary for the task. If "amdgpu" module is required, then load kernel kallsyms, BTF, and then the amdgpu module kallsyms & BTF. If no module debuginfo is required, then just the kernel would suffice.

Quoted 3 lines

This would reduce memory usage and runtime, though I don't know if it would show up in profiling. The main benefit could be reliability: by handling less data, there are fewer chances to hit an error.

OK, I agree, mandatory kernel btf/kallsyms info + optional kernel module btf/kallsyms info is a reasonable design. So kernel modules' info can be loaded on demand.

Quoted 38 lines
Quoted 19 lines
extension.c new file +18 -0
+out:
+     has_inited = true;
+     return ret;
+}
+
+static void cleanup_kallsyms_btf(void)
+{
+     cleanup_kallsyms();
+     cleanup_btf();
+}
+
+void run_extensions(void)
+{
+     DIR *dir;
+     struct dirent *entry;
+     size_t len;
+     int i;
+     void *handle;

extension.c line 59

Quoted 10 lines
extension.c new file +10 -0
+     char path[512];
+
+     for (i = 0; i < sizeof(dirs) / sizeof(char *); i++) {
+             if ((dir = opendir(dirs[i])) != NULL)
+                     break;
+     }
+
+     if (!dir || i >= sizeof(dirs) / sizeof(char *))
+             /* No extensions found */
+             return;

It could be confusing that makedumpfile would behave differently with the same command-line arguments depending on the presence or absence of these extensions on the filesystem.

I think it may fit users' expectations better if they are required to specify extensions on the command line. Then we could load them by searching each directory in order. This allows:

Quoted 5 lines

(a) more expected behavior
(b) multiple extensions can exist without all being enabled, thus more flexibility
(c) extensions can be present in the local "extensions/" directory, or in the system directory

Sure, it also sounds reasonable. My original thoughts are, user customization on mm filtering are specified in .so, and if user don't need one .so, e.g. amdgpu mm filtering for a nvidia machine, then he doesn't pack the amdgpu_filter.so into kdump's initramfs. I agree adding extra makedumpfile cmdline option to receive those needed .so is a better design.

Quoted 30 lines
Quoted 7 lines
extension.c new file +6 -0
+     while ((entry = readdir(dir)) != NULL) {
+             len = strlen(entry->d_name);
+             if (len > 3 && strcmp(entry->d_name + len - 3, ".so") == 0) {
+                     /* Will only init when .so exist */
+                     if (!init_kallsyms_btf())
+                             goto out;

extension.c line 76

Quoted 10 lines
extension.c new file +10 -0
+
+                     snprintf(path, sizeof(path), "%s/%s", dirs[i], entry->d_name);
+                     handle = dlopen(path, RTLD_NOW);
+                     if (!handle) {
+                             fprintf(stderr, "%s: Failed to load %s: %s\n",
+                                     __func__, path, dlerror());
+                             continue;
+                     }
+                     printf("Loaded extension: %s\n", path);
+                     dlclose(handle);

Using the constructor/destructor of the shared object is clever! But we lose some flexibility: by the time the dlopen() returns, the constructor has executed and the plugin has thus executed.

What if we instead use dlsym() to load some symbols from the DSO? In particular, I think it would be useful if extensions could declare a list of symbols and a list of structure information which they are interested in receiving. We could use these lists to know which kernel/module kallsyms & BTF we should load. We could even load the information into the local variables of the extension, so the extension would not need to manually load it.

Quoted 8 lines

Of course this is more complex, but the benefit is:

1. Extensions can be written more simply, and would not need to manually load each symbol & type.
2. We could eliminate the hash tables for kallsyms & BTF, and eliminate the loading of unnecessary module information. Instead, we'd just populate the symbol addresses, struct offsets, and type sizes directly into the local variables which request them.

It is a clever idea! Though complex for code, I think it is doable.

Quoted 5 lines

Again, while I don't want to prematurely optimize -- it's good to avoid loading unnecessary information. I hope I've described my idea well. I would be happy to work on an implementation of it based on your patches here, if you're interested.

Thanks again for your suggestions! I got your points and I think I can improve the code while waiting for maintainers ideas at the same time. I will let you know when done or encounter blockers if any.

Thanks,
Tao Liu

Quoted 52 lines

Thanks,
Stephen

Quoted 47 lines
extension.c new file +6 -0
+             }
+     }
+out:
+     closedir(dir);
+     cleanup_kallsyms_btf();
+}
\ No newline at end of file
extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+     $(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+     rm -f $(CONTRIB_SO)
makedumpfile.c +4 -1
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
      }
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
      if (info->flag_refiltering)
              update_dump_level();
 
+     run_extensions();
+
      if ((info->name_filterconfig || info->name_eppic_config)
                      && !gather_filter_info())
              return FALSE;
--

2.47.0

author
#10 lore raw

Re: [PATCH v3 5/8] Add makedumpfile extension support

Hi Stephen,

Sorry I took some time to make modifications on v3 code. Please see a drafted v4 [1] for a preview.

I followed your suggestion to let extensions declare the kallsyms symbol/btf types it needed, then during kallsyms/btf initialization, it will only resolve the declared symbol/types, thus getting rid of the hash table. Extensions now don't need to initial needed symbols/types by itself.

In addition, users can specify which extensions to load at makedumpfile cmdline as:

$ ./makedumpfile -d 31 -l
/var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/out --extension
amdgpu_filter.so

Also "--extension" can be used several times, and amdgpu_filter.so can be an absolute path as well. If no extensions are specified, the btf/kallsyms of makedumpfile will not initialize.

I don't know if this is what you wanted, or any suggestions for this? Thanks in advance!

Thanks,
Tao Liu

[1]: https://github.com/liutgnu/makedumpfile/commits/v4/

Quoted 283 lines · Tao Liu

Hi Stephen,

Thanks a lot for your quick reply and detailed information, I really appreciate it!

Quoted 7 lines · Stephen Brennan

Hi Tao,

This series looks really great -- I'm excited to see the switch to native .so extensions instead of epicc. I've applied the series locally and I'll rebuild my userspace stack inclusion feature based on it, to try it out myself.

Awesome, looking forward to your feedback on the code/API designs etc...

Quoted 100 lines

In the meantime, I'll share some of my feedback on the patches (though I'm not a makedumpfile developer). This seems like the most important patch in terms of design, so I'll start here.

Quoted 78 lines · Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
      rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+     $(MAKE) -C extensions clean
 
 install:
      install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
      mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
      install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
      install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+     $(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +21 -0
new file mode 100644
index 0000000..6ee7f4e
@@ -0,0 +1,82 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <dlfcn.h>
+#include <stdbool.h>
+#include "kallsyms.h"
+#include "btf_info.h"
+
+static const char *dirs[] = {
+     "/usr/lib64/makedumpfile/extensions",
+     "./extensions",
+};
+
+/* Will only init once */
+static bool init_kallsyms_btf(void)
+{
+     static bool ret = false;
+     static bool has_inited = false;
+
+     if (has_inited)

extension.c line 31

Quoted 10 lines
extension.c new file +10 -0
+             goto out;
+     if (!init_kernel_kallsyms())
+             goto out;
+     if (!init_kernel_btf())
+             goto out;
+     if (!init_module_kallsyms())
+             goto out;
+     if (!init_module_btf())
+             goto out;
+     ret = true;

I feel it would be good practice to load as little information as is necessary for the task. If "amdgpu" module is required, then load kernel kallsyms, BTF, and then the amdgpu module kallsyms & BTF. If no module debuginfo is required, then just the kernel would suffice.

Quoted 3 lines

This would reduce memory usage and runtime, though I don't know if it would show up in profiling. The main benefit could be reliability: by handling less data, there are fewer chances to hit an error.

OK, I agree, mandatory kernel btf/kallsyms info + optional kernel module btf/kallsyms info is a reasonable design. So kernel modules' info can be loaded on demand.

Quoted 38 lines
Quoted 19 lines
extension.c new file +18 -0
+out:
+     has_inited = true;
+     return ret;
+}
+
+static void cleanup_kallsyms_btf(void)
+{
+     cleanup_kallsyms();
+     cleanup_btf();
+}
+
+void run_extensions(void)
+{
+     DIR *dir;
+     struct dirent *entry;
+     size_t len;
+     int i;
+     void *handle;

extension.c line 59

Quoted 10 lines
extension.c new file +10 -0
+     char path[512];
+
+     for (i = 0; i < sizeof(dirs) / sizeof(char *); i++) {
+             if ((dir = opendir(dirs[i])) != NULL)
+                     break;
+     }
+
+     if (!dir || i >= sizeof(dirs) / sizeof(char *))
+             /* No extensions found */
+             return;

It could be confusing that makedumpfile would behave differently with the same command-line arguments depending on the presence or absence of these extensions on the filesystem.

I think it may fit users' expectations better if they are required to specify extensions on the command line. Then we could load them by searching each directory in order. This allows:

Quoted 5 lines

(a) more expected behavior
(b) multiple extensions can exist without all being enabled, thus more flexibility
(c) extensions can be present in the local "extensions/" directory, or in the system directory

Sure, it also sounds reasonable. My original thoughts are, user customization on mm filtering are specified in .so, and if user don't need one .so, e.g. amdgpu mm filtering for a nvidia machine, then he doesn't pack the amdgpu_filter.so into kdump's initramfs. I agree adding extra makedumpfile cmdline option to receive those needed .so is a better design.

Quoted 30 lines
Quoted 7 lines
extension.c new file +6 -0
+     while ((entry = readdir(dir)) != NULL) {
+             len = strlen(entry->d_name);
+             if (len > 3 && strcmp(entry->d_name + len - 3, ".so") == 0) {
+                     /* Will only init when .so exist */
+                     if (!init_kallsyms_btf())
+                             goto out;

extension.c line 76

Quoted 10 lines
extension.c new file +10 -0
+
+                     snprintf(path, sizeof(path), "%s/%s", dirs[i], entry->d_name);
+                     handle = dlopen(path, RTLD_NOW);
+                     if (!handle) {
+                             fprintf(stderr, "%s: Failed to load %s: %s\n",
+                                     __func__, path, dlerror());
+                             continue;
+                     }
+                     printf("Loaded extension: %s\n", path);
+                     dlclose(handle);

Using the constructor/destructor of the shared object is clever! But we lose some flexibility: by the time the dlopen() returns, the constructor has executed and the plugin has thus executed.

What if we instead use dlsym() to load some symbols from the DSO? In particular, I think it would be useful if extensions could declare a list of symbols and a list of structure information which they are interested in receiving. We could use these lists to know which kernel/module kallsyms & BTF we should load. We could even load the information into the local variables of the extension, so the extension would not need to manually load it.

Quoted 8 lines

Of course this is more complex, but the benefit is:

1. Extensions can be written more simply, and would not need to manually load each symbol & type.
2. We could eliminate the hash tables for kallsyms & BTF, and eliminate the loading of unnecessary module information. Instead, we'd just populate the symbol addresses, struct offsets, and type sizes directly into the local variables which request them.

It is a clever idea! Though complex for code, I think it is doable.

Quoted 5 lines

Again, while I don't want to prematurely optimize -- it's good to avoid loading unnecessary information. I hope I've described my idea well. I would be happy to work on an implementation of it based on your patches here, if you're interested.

Thanks again for your suggestions! I got your points and I think I can improve the code while waiting for maintainers ideas at the same time. I will let you know when done or encounter blockers if any.

Thanks,
Tao Liu

Quoted 52 lines

Thanks,
Stephen

Quoted 47 lines
extension.c new file +6 -0
+             }
+     }
+out:
+     closedir(dir);
+     cleanup_kallsyms_btf();
+}
\ No newline at end of file
extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+     $(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+     rm -f $(CONTRIB_SO)
makedumpfile.c +4 -1
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
      }
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
      if (info->flag_refiltering)
              update_dump_level();
 
+     run_extensions();
+
      if ((info->name_filterconfig || info->name_eppic_config)
                      && !gather_filter_info())
              return FALSE;
--

2.47.0

Reviewed-by
#11 lore raw

Re: [PATCH v3 5/8] Add makedumpfile extension support

Quoted 18 lines · Tao Liu

Hi Stephen,

Sorry I took some time to make modifications on v3 code. Please see a drafted v4 [1] for a preview.

I followed your suggestion to let extensions declare the kallsyms symbol/btf types it needed, then during kallsyms/btf initialization, it will only resolve the declared symbol/types, thus getting rid of the hash table. Extensions now don't need to initial needed symbols/types by itself.

In addition, users can specify which extensions to load at makedumpfile cmdline as:

$ ./makedumpfile -d 31 -l
/var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/out --extension
amdgpu_filter.so
Quoted 6 lines

Also "--extension" can be used several times, and amdgpu_filter.so can be an absolute path as well. If no extensions are specified, the btf/kallsyms of makedumpfile will not initialize.

I don't know if this is what you wanted, or any suggestions for this? Thanks in advance!

Hi Tao,

Please accept my apologies for the long delay on this review. I've gone through each commit with my feedback. This looks like a really excellent job and I think it is ready for the review of the maintainers. I've implemented my own userspace stack extension on top of this support, and I find it to be great for my use case as well, with one small change (adding flags for detecting whether a struct or member is found).

Here is some more detailed feedback on a per-commit level; it's all pretty minor.

d3aee7a ("Reserve sections for makedumpfile and extenions")

* This is just a note, but I've found that the linker script may not be necessary. GCC automatically creates the __start_SECTION and __stop_SECTION symbols for sections it creates.
* Otherwise, this is exactly what I was thinking!

102ae0b ("Implement kernel kallsyms resolving")

* Again, this is looking very close to the design I hoped for, thanks!
* I'm not sure whether is_unwanted_symbol() needs to exist anymore? Given that users opt-in to specific symbols, we don't need to filter out noisy ones that would waste memory. What do you think?
* Unfortunately, upstream has made some changes to the vmlinux kallsyms encoding in 7.0. You may want to check what we did in drgn to support those changes:
https://github.com/osandov/drgn/commit/744f36ec3c3f64d7e1323a0037898158698585c4
* In kallsyms.c:

  +/*
  + * Makedumpfile's .init_ksyms section
  +*/
  +extern struct ksym_info __start_init_ksyms[];
  +extern struct ksym_info __stop_init_ksyms[];

I'm not sure it matters, but the type here is wrong. It should be "extern struct ksym_info *" because you're storing pointers, not the actual struct. That said, the type isn't used so I don't know that it matters.

4187b33 ("Implement kernel btf resolving")

* The get_ktype_info() function fails if either the struct or member is not found. This makes sense in a lot of cases, but there are other cases where we will want to use the presence or absence of a struct/struct member to detect which version of code to use. For example, my userspace stack code will use either maple or rbtree helper code for finding stack VMAs, depending on which is available. We can't know until runtime, when we check whether mm_rb or mm_mt is present.

One solution is to handle this at runtime. We can have a macro like "HAVE_MEMBER(S, M)" and "HAVE_STRUCT(S)", and then each extension can check whether for the members it expects to be present. I think the major downside to this approach is that it requires manual effort, which is likely to be forgotten when writing extensions.

Alternatively, we could set a flag in the struct ktype_info if the type is "required" (or optional), and only fail get_ktype_info() for required structs/members. The concern with this approach is: what if plugin (A) requires a type which is not present, but plugin (B) does not? If both are loaded, the failure of A would cause B not to run. I'm not sure whether we should care about that situation... I don't know if we have a use case for using multiple plugins at the same time. Until we do, we probably won't have a good idea whether it should be allowed for one to fail, but the other to continue.

I've implemented this second alternative in my branch.

* Similar to the previous commit, __start_init_ktypes and __stop_init_ktypes are declared as structs but should probably be declared as pointers.

22097b7 ("Implement kernel module's kallsyms resolving") edfa698 ("Implement kernel module's btf resolving")

* These commits are straightforward:

ede22e8 ("Add makedumpfile extensions support")

* nit: the array "handlers" actually contains "handles" (no r) returned from dlopen(). The word handlers usually implies a function pointer or something like that, called to handle a certain situation. Maybe rename this to "handles".
* The run_extensions() function is called within a retry loop in create_dumpfile(). It seems possible that this could get called multiple times. But many of the global variables in the kallsyms, BTF, and extension code are not safe to be cleaned up and reinitialized. In particular, when array elements are freed, the lengths, capacities, and pointers are not reset to 0/NULL. I think it would be wise to make all the cleanup functions clear the globals, so that they may be reinitialized safely.
* Further, I think it might be helpful to split extension loading, running, and cleanup. Something like this in create_dumpfile():

        load_extensions(); /* loads everything and calls entry() */
    retry:
        /* ... create bitmap and dump ... */
        if (status == NOSPACE) {
            /* ... */
            goto retry;
        }
        /* ... */
        cleanup_extensions();
        return;

For two reasons: (1) this avoids unnecessary work re-loading and re-initializing the BTF, kallsyms, and extensions. (Though I still think it's safer to ensure they can be re-initialized safely.) And (2), this allows for future use of extensions in the rest of the dump operation. For my userspace stack extension, I plan to add a callback which allows extensions to override the decision to filter a page, since my logic can't be easily done via erase info. So the extensions need to remain loaded during the creation of the dumpfile, and cleaned up after. I have tweaked this in my own patches, but I just wanted to share the use case.

c568635 ("btf/kallsyms based makedumpfile extension for mm page filtering")

* This looks good to me!
* I will say that for my userspace stack use case, while the "filter_page(..., false)" mechanism for specifying pages that are retained *looks* useful, I wouldn't be able to use it because I cannot determine the PFNs for each stack VMA. Instead, I have to use the page "mapping" and "index" fields to match the VMA and determine whether the PFN falls in a range I care to save. Of course, just because *I* won't use it doesn't mean it's not useful :)

2b252ec ("Filter amdgpu mm pages")

* I'm no expert in amdgpu, but the overall approach makes sense to me, and the helpers look good.

At a broader level, while the add_to_arr() function is useful, I do think the dynamic array/vector pattern could be captured with a dedicated data structure. For instance, drgn has this excellent LGPL-2.1+ header-only vector library:
https://github.com/osandov/drgn/blob/main/libdrgn/vector.h
I don't think it is high priority or must be addressed. Nor do I mean that this particular implementation choice be used. It's just something to share & think about.

To provide some context on my comments related to the userspace stack tracing, here is a branch of mine which is based on yours, that adds my userspace stack extension and a few tweaks:

https://github.com/brenns10/makedumpfile/commits/stepbren_userstack_v4/

Thank you,
Stephen

Quoted 295 lines

Thanks,
Tao Liu

[1]: https://github.com/liutgnu/makedumpfile/commits/v4/

Quoted 283 lines · Tao Liu

Hi Stephen,

Thanks a lot for your quick reply and detailed information, I really appreciate it!

Quoted 7 lines · Stephen Brennan

Hi Tao,

This series looks really great -- I'm excited to see the switch to native .so extensions instead of epicc. I've applied the series locally and I'll rebuild my userspace stack inclusion feature based on it, to try it out myself.

Awesome, looking forward to your feedback on the code/API designs etc...

Quoted 100 lines

In the meantime, I'll share some of my feedback on the patches (though I'm not a makedumpfile developer). This seems like the most important patch in terms of design, so I'll start here.

Quoted 78 lines · Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
      rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+     $(MAKE) -C extensions clean
 
 install:
      install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
      mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
      install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
      install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+     $(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +21 -0
new file mode 100644
index 0000000..6ee7f4e
@@ -0,0 +1,82 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <dlfcn.h>
+#include <stdbool.h>
+#include "kallsyms.h"
+#include "btf_info.h"
+
+static const char *dirs[] = {
+     "/usr/lib64/makedumpfile/extensions",
+     "./extensions",
+};
+
+/* Will only init once */
+static bool init_kallsyms_btf(void)
+{
+     static bool ret = false;
+     static bool has_inited = false;
+
+     if (has_inited)

extension.c line 31

Quoted 10 lines
extension.c new file +10 -0
+             goto out;
+     if (!init_kernel_kallsyms())
+             goto out;
+     if (!init_kernel_btf())
+             goto out;
+     if (!init_module_kallsyms())
+             goto out;
+     if (!init_module_btf())
+             goto out;
+     ret = true;

I feel it would be good practice to load as little information as is necessary for the task. If "amdgpu" module is required, then load kernel kallsyms, BTF, and then the amdgpu module kallsyms & BTF. If no module debuginfo is required, then just the kernel would suffice.

Quoted 3 lines

This would reduce memory usage and runtime, though I don't know if it would show up in profiling. The main benefit could be reliability: by handling less data, there are fewer chances to hit an error.

OK, I agree, mandatory kernel btf/kallsyms info + optional kernel module btf/kallsyms info is a reasonable design. So kernel modules' info can be loaded on demand.

Quoted 38 lines
Quoted 19 lines
extension.c new file +18 -0
+out:
+     has_inited = true;
+     return ret;
+}
+
+static void cleanup_kallsyms_btf(void)
+{
+     cleanup_kallsyms();
+     cleanup_btf();
+}
+
+void run_extensions(void)
+{
+     DIR *dir;
+     struct dirent *entry;
+     size_t len;
+     int i;
+     void *handle;

extension.c line 59

Quoted 10 lines
extension.c new file +10 -0
+     char path[512];
+
+     for (i = 0; i < sizeof(dirs) / sizeof(char *); i++) {
+             if ((dir = opendir(dirs[i])) != NULL)
+                     break;
+     }
+
+     if (!dir || i >= sizeof(dirs) / sizeof(char *))
+             /* No extensions found */
+             return;

It could be confusing that makedumpfile would behave differently with the same command-line arguments depending on the presence or absence of these extensions on the filesystem.

I think it may fit users' expectations better if they are required to specify extensions on the command line. Then we could load them by searching each directory in order. This allows:

Quoted 5 lines

(a) more expected behavior
(b) multiple extensions can exist without all being enabled, thus more flexibility
(c) extensions can be present in the local "extensions/" directory, or in the system directory

Sure, it also sounds reasonable. My original thoughts are, user customization on mm filtering are specified in .so, and if user don't need one .so, e.g. amdgpu mm filtering for a nvidia machine, then he doesn't pack the amdgpu_filter.so into kdump's initramfs. I agree adding extra makedumpfile cmdline option to receive those needed .so is a better design.

Quoted 30 lines
Quoted 7 lines
extension.c new file +6 -0
+     while ((entry = readdir(dir)) != NULL) {
+             len = strlen(entry->d_name);
+             if (len > 3 && strcmp(entry->d_name + len - 3, ".so") == 0) {
+                     /* Will only init when .so exist */
+                     if (!init_kallsyms_btf())
+                             goto out;

extension.c line 76

Quoted 10 lines
extension.c new file +10 -0
+
+                     snprintf(path, sizeof(path), "%s/%s", dirs[i], entry->d_name);
+                     handle = dlopen(path, RTLD_NOW);
+                     if (!handle) {
+                             fprintf(stderr, "%s: Failed to load %s: %s\n",
+                                     __func__, path, dlerror());
+                             continue;
+                     }
+                     printf("Loaded extension: %s\n", path);
+                     dlclose(handle);

Using the constructor/destructor of the shared object is clever! But we lose some flexibility: by the time the dlopen() returns, the constructor has executed and the plugin has thus executed.

What if we instead use dlsym() to load some symbols from the DSO? In particular, I think it would be useful if extensions could declare a list of symbols and a list of structure information which they are interested in receiving. We could use these lists to know which kernel/module kallsyms & BTF we should load. We could even load the information into the local variables of the extension, so the extension would not need to manually load it.

Quoted 8 lines

Of course this is more complex, but the benefit is:

1. Extensions can be written more simply, and would not need to manually load each symbol & type.
2. We could eliminate the hash tables for kallsyms & BTF, and eliminate the loading of unnecessary module information. Instead, we'd just populate the symbol addresses, struct offsets, and type sizes directly into the local variables which request them.

It is a clever idea! Though complex for code, I think it is doable.

Quoted 5 lines

Again, while I don't want to prematurely optimize -- it's good to avoid loading unnecessary information. I hope I've described my idea well. I would be happy to work on an implementation of it based on your patches here, if you're interested.

Thanks again for your suggestions! I got your points and I think I can improve the code while waiting for maintainers ideas at the same time. I will let you know when done or encounter blockers if any.

Thanks,
Tao Liu

Quoted 52 lines

Thanks,
Stephen

Quoted 47 lines
extension.c new file +6 -0
+             }
+     }
+out:
+     closedir(dir);
+     cleanup_kallsyms_btf();
+}
\ No newline at end of file
extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+     $(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+     rm -f $(CONTRIB_SO)
makedumpfile.c +4 -1
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
      }
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
      if (info->flag_refiltering)
              update_dump_level();
 
+     run_extensions();
+
      if ((info->name_filterconfig || info->name_eppic_config)
                      && !gather_filter_info())
              return FALSE;
--

2.47.0

author
#12 lore raw

Re: [PATCH v3 5/8] Add makedumpfile extension support

Hi Stephen,

Quoted 27 lines · Stephen Brennan
Quoted 18 lines · Tao Liu

Hi Stephen,

Sorry I took some time to make modifications on v3 code. Please see a drafted v4 [1] for a preview.

I followed your suggestion to let extensions declare the kallsyms symbol/btf types it needed, then during kallsyms/btf initialization, it will only resolve the declared symbol/types, thus getting rid of the hash table. Extensions now don't need to initial needed symbols/types by itself.

In addition, users can specify which extensions to load at makedumpfile cmdline as:

$ ./makedumpfile -d 31 -l
/var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/out --extension
amdgpu_filter.so
Quoted 6 lines

Also "--extension" can be used several times, and amdgpu_filter.so can be an absolute path as well. If no extensions are specified, the btf/kallsyms of makedumpfile will not initialize.

I don't know if this is what you wanted, or any suggestions for this? Thanks in advance!

Quoted 8 lines

Hi Tao,

Please accept my apologies for the long delay on this review. I've gone through each commit with my feedback. This looks like a really excellent job and I think it is ready for the review of the maintainers. I've implemented my own userspace stack extension on top of this support, and I find it to be great for my use case as well, with one small change (adding flags for detecting whether a struct or member is found).

No worries at all :) Thanks a lot for your detailed comments. I'm looking through them as well as your code branch in github. This may take a while...

Quoted 9 lines

Here is some more detailed feedback on a per-commit level; it's all pretty minor.

d3aee7a ("Reserve sections for makedumpfile and extenions")

* This is just a note, but I've found that the linker script may not be necessary. GCC automatically creates the __start_SECTION and __stop_SECTION symbols for sections it creates.

I have tried to remove the makedumpfile.ld linker script. The automatic symbol as __start/stop_SECTION are only generated for makedumpfile, aka the main program, but not for .so extensions. This is a breaker because I would expect all .so files have the __start/stop_SECTION symbol so they can be resigtered to main program. I'm not expert in GCC's default behaviours, so I guess I would prefer the current approach of explicit symbol define within the linker script, seems to be more robust... Perhaps you can share your code on this, I will give it a try.

Quoted 8 lines

* Otherwise, this is exactly what I was thinking!

102ae0b ("Implement kernel kallsyms resolving")

* Again, this is looking very close to the design I hoped for, thanks!
* I'm not sure whether is_unwanted_symbol() needs to exist anymore? Given that users opt-in to specific symbols, we don't need to filter out noisy ones that would waste memory. What do you think?

Agreed, I can remove this function.

Quoted 4 lines

* Unfortunately, upstream has made some changes to the vmlinux kallsyms encoding in 7.0. You may want to check what we did in drgn to support those changes:
https://github.com/osandov/drgn/commit/744f36ec3c3f64d7e1323a0037898158698585c4
* In kallsyms.c:

Thanks for the info, I will try this and update the code in v4.

Quoted 10 lines
  +/*
  + * Makedumpfile's .init_ksyms section
  +*/
  +extern struct ksym_info __start_init_ksyms[];
  +extern struct ksym_info __stop_init_ksyms[];

I'm not sure it matters, but the type here is wrong. It should be "extern struct ksym_info *" because you're storing pointers, not the actual struct. That said, the type isn't used so I don't know that it matters.

Right, this is an error. Thanks for pointing that out.

Quoted 11 lines

4187b33 ("Implement kernel btf resolving")

* The get_ktype_info() function fails if either the struct or member is not found. This makes sense in a lot of cases, but there are other cases where we will want to use the presence or absence of a struct/struct member to detect which version of code to use. For example, my userspace stack code will use either maple or rbtree helper code for finding stack VMAs, depending on which is available. We can't know until runtime, when we check whether mm_rb or mm_mt is present.

Quoted 8 lines

One solution is to handle this at runtime. We can have a macro like "HAVE_MEMBER(S, M)" and "HAVE_STRUCT(S)", and then each extension can check whether for the members it expects to be present. I think the major downside to this approach is that it requires manual effort, which is likely to be forgotten when writing extensions.

Alternatively, we could set a flag in the struct ktype_info if the type is "required" (or optional), and only fail get_ktype_info() for required

Agreed, an optional flag looks better.

Quoted 7 lines

structs/members. The concern with this approach is: what if plugin (A) requires a type which is not present, but plugin (B) does not? If both are loaded, the failure of A would cause B not to run. I'm not sure whether we should care about that situation... I don't know if we have a use case for using multiple plugins at the same time. Until we do, we probably won't have a good idea whether it should be allowed for one to fail, but the other to continue.

Great question! I have thought about this previously. I suggest that if plugin(A) fails, it should just fail and allow the execution of any later plugins (B, C....) to continue. Each plugin is responsible for one task, like plugin(A) for dealing with amdgpu's mm page filtering,and plugin(B) for Intel's and plugin(C) for NV's. Plugin(A) certainly will fail if one machine have no amdgpu, thus the amdgpu.ko will never been loaded, so related symbol/types missing. This is expected and shouldn't block the later plugins.

But the "fail" should gentle, not like the ones as segfault, which will crash the entire makedumpfile program. Since currently plugins are native .so libraries, the quality of code is ensured by each plugin authors, rather than makedumpfile maintainers. Idealy the the plugins are well tested in 1st kernel before they are shipped to kdump img, but who knows. From makedumpfile's view, do you think we need to introduce a sandbox to isolate plugins from makedumpfile? This would prevent serious plugin errors from stopping makedumpfile from generating the vmcore.

Quoted 6 lines

I've implemented this second alternative in my branch.

* Similar to the previous commit, __start_init_ktypes and __stop_init_ktypes are declared as structs but should probably be declared as pointers.

Quoted 7 lines

22097b7 ("Implement kernel module's kallsyms resolving") edfa698 ("Implement kernel module's btf resolving")

* These commits are straightforward:

ede22e8 ("Add makedumpfile extensions support")

I will address those later after reading through your code branch. Thanks again for your detailed comments!

Thanks,
Tao Liu

Quoted 365 lines

* nit: the array "handlers" actually contains "handles" (no r) returned from dlopen(). The word handlers usually implies a function pointer or something like that, called to handle a certain situation. Maybe rename this to "handles".
* The run_extensions() function is called within a retry loop in create_dumpfile(). It seems possible that this could get called multiple times. But many of the global variables in the kallsyms, BTF, and extension code are not safe to be cleaned up and reinitialized. In particular, when array elements are freed, the lengths, capacities, and pointers are not reset to 0/NULL. I think it would be wise to make all the cleanup functions clear the globals, so that they may be reinitialized safely.
* Further, I think it might be helpful to split extension loading, running, and cleanup. Something like this in create_dumpfile():

        load_extensions(); /* loads everything and calls entry() */
    retry:
        /* ... create bitmap and dump ... */
        if (status == NOSPACE) {
            /* ... */
            goto retry;
        }
        /* ... */
        cleanup_extensions();
        return;

For two reasons: (1) this avoids unnecessary work re-loading and re-initializing the BTF, kallsyms, and extensions. (Though I still think it's safer to ensure they can be re-initialized safely.) And (2), this allows for future use of extensions in the rest of the dump operation. For my userspace stack extension, I plan to add a callback which allows extensions to override the decision to filter a page, since my logic can't be easily done via erase info. So the extensions need to remain loaded during the creation of the dumpfile, and cleaned up after. I have tweaked this in my own patches, but I just wanted to share the use case.

c568635 ("btf/kallsyms based makedumpfile extension for mm page filtering")

* This looks good to me!
* I will say that for my userspace stack use case, while the "filter_page(..., false)" mechanism for specifying pages that are retained *looks* useful, I wouldn't be able to use it because I cannot determine the PFNs for each stack VMA. Instead, I have to use the page "mapping" and "index" fields to match the VMA and determine whether the PFN falls in a range I care to save. Of course, just because *I* won't use it doesn't mean it's not useful :)

2b252ec ("Filter amdgpu mm pages")

* I'm no expert in amdgpu, but the overall approach makes sense to me, and the helpers look good.

At a broader level, while the add_to_arr() function is useful, I do think the dynamic array/vector pattern could be captured with a dedicated data structure. For instance, drgn has this excellent LGPL-2.1+ header-only vector library:
https://github.com/osandov/drgn/blob/main/libdrgn/vector.h
I don't think it is high priority or must be addressed. Nor do I mean that this particular implementation choice be used. It's just something to share & think about.

To provide some context on my comments related to the userspace stack tracing, here is a branch of mine which is based on yours, that adds my userspace stack extension and a few tweaks:

https://github.com/brenns10/makedumpfile/commits/stepbren_userstack_v4/

Thank you,
Stephen

Quoted 295 lines

Thanks,
Tao Liu

[1]: https://github.com/liutgnu/makedumpfile/commits/v4/

Quoted 283 lines · Tao Liu

Hi Stephen,

Thanks a lot for your quick reply and detailed information, I really appreciate it!

Quoted 7 lines · Stephen Brennan

Hi Tao,

This series looks really great -- I'm excited to see the switch to native .so extensions instead of epicc. I've applied the series locally and I'll rebuild my userspace stack inclusion feature based on it, to try it out myself.

Awesome, looking forward to your feedback on the code/API designs etc...

Quoted 100 lines

In the meantime, I'll share some of my feedback on the patches (though I'm not a makedumpfile developer). This seems like the most important patch in terms of design, so I'll start here.

Quoted 78 lines · Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
      rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+     $(MAKE) -C extensions clean
 
 install:
      install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
      mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
      install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
      install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+     $(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +21 -0
new file mode 100644
index 0000000..6ee7f4e
@@ -0,0 +1,82 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <dlfcn.h>
+#include <stdbool.h>
+#include "kallsyms.h"
+#include "btf_info.h"
+
+static const char *dirs[] = {
+     "/usr/lib64/makedumpfile/extensions",
+     "./extensions",
+};
+
+/* Will only init once */
+static bool init_kallsyms_btf(void)
+{
+     static bool ret = false;
+     static bool has_inited = false;
+
+     if (has_inited)

extension.c line 31

Quoted 10 lines
extension.c new file +10 -0
+             goto out;
+     if (!init_kernel_kallsyms())
+             goto out;
+     if (!init_kernel_btf())
+             goto out;
+     if (!init_module_kallsyms())
+             goto out;
+     if (!init_module_btf())
+             goto out;
+     ret = true;

I feel it would be good practice to load as little information as is necessary for the task. If "amdgpu" module is required, then load kernel kallsyms, BTF, and then the amdgpu module kallsyms & BTF. If no module debuginfo is required, then just the kernel would suffice.

Quoted 3 lines

This would reduce memory usage and runtime, though I don't know if it would show up in profiling. The main benefit could be reliability: by handling less data, there are fewer chances to hit an error.

OK, I agree, mandatory kernel btf/kallsyms info + optional kernel module btf/kallsyms info is a reasonable design. So kernel modules' info can be loaded on demand.

Quoted 38 lines
Quoted 19 lines
extension.c new file +18 -0
+out:
+     has_inited = true;
+     return ret;
+}
+
+static void cleanup_kallsyms_btf(void)
+{
+     cleanup_kallsyms();
+     cleanup_btf();
+}
+
+void run_extensions(void)
+{
+     DIR *dir;
+     struct dirent *entry;
+     size_t len;
+     int i;
+     void *handle;

extension.c line 59

Quoted 10 lines
extension.c new file +10 -0
+     char path[512];
+
+     for (i = 0; i < sizeof(dirs) / sizeof(char *); i++) {
+             if ((dir = opendir(dirs[i])) != NULL)
+                     break;
+     }
+
+     if (!dir || i >= sizeof(dirs) / sizeof(char *))
+             /* No extensions found */
+             return;

It could be confusing that makedumpfile would behave differently with the same command-line arguments depending on the presence or absence of these extensions on the filesystem.

I think it may fit users' expectations better if they are required to specify extensions on the command line. Then we could load them by searching each directory in order. This allows:

Quoted 5 lines

(a) more expected behavior
(b) multiple extensions can exist without all being enabled, thus more flexibility
(c) extensions can be present in the local "extensions/" directory, or in the system directory

Sure, it also sounds reasonable. My original thoughts are, user customization on mm filtering are specified in .so, and if user don't need one .so, e.g. amdgpu mm filtering for a nvidia machine, then he doesn't pack the amdgpu_filter.so into kdump's initramfs. I agree adding extra makedumpfile cmdline option to receive those needed .so is a better design.

Quoted 30 lines
Quoted 7 lines
extension.c new file +6 -0
+     while ((entry = readdir(dir)) != NULL) {
+             len = strlen(entry->d_name);
+             if (len > 3 && strcmp(entry->d_name + len - 3, ".so") == 0) {
+                     /* Will only init when .so exist */
+                     if (!init_kallsyms_btf())
+                             goto out;

extension.c line 76

Quoted 10 lines
extension.c new file +10 -0
+
+                     snprintf(path, sizeof(path), "%s/%s", dirs[i], entry->d_name);
+                     handle = dlopen(path, RTLD_NOW);
+                     if (!handle) {
+                             fprintf(stderr, "%s: Failed to load %s: %s\n",
+                                     __func__, path, dlerror());
+                             continue;
+                     }
+                     printf("Loaded extension: %s\n", path);
+                     dlclose(handle);

Using the constructor/destructor of the shared object is clever! But we lose some flexibility: by the time the dlopen() returns, the constructor has executed and the plugin has thus executed.

What if we instead use dlsym() to load some symbols from the DSO? In particular, I think it would be useful if extensions could declare a list of symbols and a list of structure information which they are interested in receiving. We could use these lists to know which kernel/module kallsyms & BTF we should load. We could even load the information into the local variables of the extension, so the extension would not need to manually load it.

Quoted 8 lines

Of course this is more complex, but the benefit is:

1. Extensions can be written more simply, and would not need to manually load each symbol & type.
2. We could eliminate the hash tables for kallsyms & BTF, and eliminate the loading of unnecessary module information. Instead, we'd just populate the symbol addresses, struct offsets, and type sizes directly into the local variables which request them.

It is a clever idea! Though complex for code, I think it is doable.

Quoted 5 lines

Again, while I don't want to prematurely optimize -- it's good to avoid loading unnecessary information. I hope I've described my idea well. I would be happy to work on an implementation of it based on your patches here, if you're interested.

Thanks again for your suggestions! I got your points and I think I can improve the code while waiting for maintainers ideas at the same time. I will let you know when done or encounter blockers if any.

Thanks,
Tao Liu

Quoted 52 lines

Thanks,
Stephen

Quoted 47 lines
extension.c new file +6 -0
+             }
+     }
+out:
+     closedir(dir);
+     cleanup_kallsyms_btf();
+}
\ No newline at end of file
extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+     $(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+     rm -f $(CONTRIB_SO)
makedumpfile.c +4 -1
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
      }
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
      if (info->flag_refiltering)
              update_dump_level();
 
+     run_extensions();
+
      if ((info->name_filterconfig || info->name_eppic_config)
                      && !gather_filter_info())
              return FALSE;
--

2.47.0

#13 lore raw

Re: [PATCH v3 5/8] Add makedumpfile extension support

Quoted 54 lines · Tao Liu

Hi Stephen,

Quoted 27 lines · Stephen Brennan
Quoted 18 lines · Tao Liu

Hi Stephen,

Sorry I took some time to make modifications on v3 code. Please see a drafted v4 [1] for a preview.

I followed your suggestion to let extensions declare the kallsyms symbol/btf types it needed, then during kallsyms/btf initialization, it will only resolve the declared symbol/types, thus getting rid of the hash table. Extensions now don't need to initial needed symbols/types by itself.

In addition, users can specify which extensions to load at makedumpfile cmdline as:

$ ./makedumpfile -d 31 -l
/var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/out --extension
amdgpu_filter.so
Quoted 6 lines

Also "--extension" can be used several times, and amdgpu_filter.so can be an absolute path as well. If no extensions are specified, the btf/kallsyms of makedumpfile will not initialize.

I don't know if this is what you wanted, or any suggestions for this? Thanks in advance!

Quoted 8 lines

Hi Tao,

Please accept my apologies for the long delay on this review. I've gone through each commit with my feedback. This looks like a really excellent job and I think it is ready for the review of the maintainers. I've implemented my own userspace stack extension on top of this support, and I find it to be great for my use case as well, with one small change (adding flags for detecting whether a struct or member is found).

No worries at all :) Thanks a lot for your detailed comments. I'm looking through them as well as your code branch in github. This may take a while...

Quoted 9 lines

Here is some more detailed feedback on a per-commit level; it's all pretty minor.

d3aee7a ("Reserve sections for makedumpfile and extenions")

* This is just a note, but I've found that the linker script may not be necessary. GCC automatically creates the __start_SECTION and __stop_SECTION symbols for sections it creates.

Quoted 9 lines

I have tried to remove the makedumpfile.ld linker script. The automatic symbol as __start/stop_SECTION are only generated for makedumpfile, aka the main program, but not for .so extensions. This is a breaker because I would expect all .so files have the __start/stop_SECTION symbol so they can be resigtered to main program. I'm not expert in GCC's default behaviours, so I guess I would prefer the current approach of explicit symbol define within the linker script, seems to be more robust... Perhaps you can share your code on this, I will give it a try.

The key to making GCC generate the symbols for the sections is that the section names should themselves be valid C identifiers (i.e. they should not start with a '.'). Here's a patch to give the idea:

From 3618f51869e978328e49f6061eae94469bca43c5 Mon Sep 17 00:00:00 2001
From: Stephen Brennan <[email protected]>
Date: Wed, 11 Mar 2026 09:17:05 -0700
Subject: [PATCH] Use automatically generated section start symbols

GCC will create __start_SECTION and __stop_SECTION only if the section names are valid C identifiers. This means we cannot use a leading dot. Drop the leading dot from the section names so we can avoid defining a custom linker script.

---
 Makefile            |  2 +-
 btf_info.h          |  4 ++--
 extensions/Makefile |  2 +-
 kallsyms.h          |  2 +-
 makedumpfile.ld     | 15 ---------------
 5 files changed, 5 insertions(+), 20 deletions(-)
 delete mode 100644 makedumpfile.ld
Makefile +1 -1
index 8196aa6..e2cde8f 100644
@@ -113,7 +113,7 @@ $(OBJ_ARCH): $(SRC_ARCH)
 	$(CC) $(CFLAGS_ARCH) -c -o ./$@ $(VPATH)$(@:.o=.c)
 
 makedumpfile: $(SRC_BASE) $(OBJ_PART) $(OBJ_ARCH)
-	$(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -Wl,-T,makedumpfile.ld -o $@ $< $(LIBS)
+	$(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -o $@ $< $(LIBS)
 	@sed -e "s/@DATE@/$(DATE)/" \
 	     -e "s/@VERSION@/$(VERSION)/" \
 	     $(VPATH)makedumpfile.8.in > $(VPATH)makedumpfile.8
btf_info.h +2 -2
index 2a78fbf..555f21e 100644
@@ -29,7 +29,7 @@ void cleanup_btf(void);
 	struct ktype_info _##MOD##_##S##_##M = {		\
 		QUATE(MOD), QUATE(S), QUATE(M), 0, 0, 0, 0, R, R\
 	};							\
-	__attribute__((section(".init_ktypes"), used))		\
+	__attribute__((section("init_ktypes"), used))		\
 	struct ktype_info * _ptr_##MOD##_##S##_##M = &_##MOD##_##S##_##M
 
 #define INIT_MOD_STRUCT_MEMBER(MOD, S, M)			\
@@ -60,7 +60,7 @@ void cleanup_btf(void);
 	struct ktype_info _##MOD##_##S = {			\
 		QUATE(MOD), QUATE(S), 0, 0, 0, 0, 0, R, 0	\
 	};							\
-	__attribute__((section(".init_ktypes"), used))		\
+	__attribute__((section("init_ktypes"), used))		\
 	struct ktype_info * _ptr_##MOD##_##S = &_##MOD##_##S
 
 #define INIT_MOD_STRUCT(MOD, S) INIT_MOD_STRUCT_RQD(MOD, S, 1)
extensions/Makefile +1 -1
index fc28577..a4dc86a 100644
@@ -8,7 +8,7 @@ amdgpu_filter.so: maple_tree.c
 userstack.so: maple_tree.c vma_rbtree.c
 
 $(CONTRIB_SO): %.so: %.c
-	$(CC) -O2 -g -fPIC -shared -Wl,-T,../makedumpfile.ld -o $@ $^
+	$(CC) -O2 -g -fPIC -shared -o $@ $^
 
 clean:
 	rm -f $(CONTRIB_SO)
kallsyms.h +1 -1
index d94830e..1989dfe 100644
@@ -17,7 +17,7 @@ struct ksym_info {
 	struct ksym_info _##MOD##_##SYM = {		\
 		QUATE(MOD), QUATE(SYM), 0		\
 	};						\
-	__attribute__((section(".init_ksyms"), used))	\
+	__attribute__((section("init_ksyms"), used))	\
 	struct ksym_info * _ptr_##MOD##_##SYM = &_##MOD##_##SYM
 
 #define GET_MOD_SYM(MOD, SYM) (_##MOD##_##SYM.value)
makedumpfile.ld deleted +0 -15
deleted file mode 100644
index 231a162..0000000
@@ -1,15 +0,0 @@
-SECTIONS
-{
-	.init_ksyms ALIGN(8) : {
-		__start_init_ksyms = .;
-		KEEP(*(.init_ksyms*))
-		__stop_init_ksyms = .;
-	}
-
-	.init_ktypes ALIGN(8) : {
-		__start_init_ktypes = .;
-		KEEP(*(.init_ktypes*))
-		__stop_init_ktypes = .;
-	}
-}
-INSERT AFTER .data;
\ No newline at end of file
Signature · 1 line
2.47.3
Quoted 70 lines
Quoted 8 lines

* Otherwise, this is exactly what I was thinking!

102ae0b ("Implement kernel kallsyms resolving")

* Again, this is looking very close to the design I hoped for, thanks!
* I'm not sure whether is_unwanted_symbol() needs to exist anymore? Given that users opt-in to specific symbols, we don't need to filter out noisy ones that would waste memory. What do you think?

Agreed, I can remove this function.

Quoted 4 lines

* Unfortunately, upstream has made some changes to the vmlinux kallsyms encoding in 7.0. You may want to check what we did in drgn to support those changes:
https://github.com/osandov/drgn/commit/744f36ec3c3f64d7e1323a0037898158698585c4
* In kallsyms.c:

Thanks for the info, I will try this and update the code in v4.

Quoted 10 lines
  +/*
  + * Makedumpfile's .init_ksyms section
  +*/
  +extern struct ksym_info __start_init_ksyms[];
  +extern struct ksym_info __stop_init_ksyms[];

I'm not sure it matters, but the type here is wrong. It should be "extern struct ksym_info *" because you're storing pointers, not the actual struct. That said, the type isn't used so I don't know that it matters.

Right, this is an error. Thanks for pointing that out.

Quoted 11 lines

4187b33 ("Implement kernel btf resolving")

* The get_ktype_info() function fails if either the struct or member is not found. This makes sense in a lot of cases, but there are other cases where we will want to use the presence or absence of a struct/struct member to detect which version of code to use. For example, my userspace stack code will use either maple or rbtree helper code for finding stack VMAs, depending on which is available. We can't know until runtime, when we check whether mm_rb or mm_mt is present.

Quoted 8 lines

One solution is to handle this at runtime. We can have a macro like "HAVE_MEMBER(S, M)" and "HAVE_STRUCT(S)", and then each extension can check whether for the members it expects to be present. I think the major downside to this approach is that it requires manual effort, which is likely to be forgotten when writing extensions.

Alternatively, we could set a flag in the struct ktype_info if the type is "required" (or optional), and only fail get_ktype_info() for required

Agreed, an optional flag looks better.

Quoted 7 lines

structs/members. The concern with this approach is: what if plugin (A) requires a type which is not present, but plugin (B) does not? If both are loaded, the failure of A would cause B not to run. I'm not sure whether we should care about that situation... I don't know if we have a use case for using multiple plugins at the same time. Until we do, we probably won't have a good idea whether it should be allowed for one to fail, but the other to continue.

Great question! I have thought about this previously. I suggest that if plugin(A) fails, it should just fail and allow the execution of any later plugins (B, C....) to continue. Each plugin is responsible for one task, like plugin(A) for dealing with amdgpu's mm page filtering,and plugin(B) for Intel's and plugin(C) for NV's. Plugin(A) certainly will fail if one machine have no amdgpu, thus the amdgpu.ko will never been loaded, so related symbol/types missing. This is expected and shouldn't block the later plugins.

Quoted 2 lines

But the "fail" should gentle, not like the ones as segfault, which will crash the entire makedumpfile program.

Totally agreed up to here! I think it makes sense for plugin failures to be independent.

It seems like the BTF loading process should not fail if any type or member is not found. But once we've tried to load everything, we can then go through each plugin's list, and if we find a "required" type or member which is not present, we log that failure and don't run the plugin. This shouldn't be much work, because we already have all the types in an array for each plugin.

Quoted 8 lines

Since currently plugins
are native .so libraries, the quality of code is ensured by each plugin authors, rather than makedumpfile maintainers. Idealy the the plugins are well tested in 1st kernel before they are shipped to kdump img, but who knows. From makedumpfile's view, do you think we need to introduce a sandbox to isolate plugins from makedumpfile? This would prevent serious plugin errors from stopping makedumpfile from generating the vmcore.

Personally, I don't believe we need sandbox. The plugins should be written to handle the clear error cases:

- Optional types or symbols may not be present, so validate them.
- Values read from /proc/vmcore may not be valid, or the read may fail, so check all return values.

That said, I'm not set against it, so if you believe it's necessary I could be convinced :)

Thanks again,
Stephen

Quoted 385 lines
Quoted 6 lines

I've implemented this second alternative in my branch.

* Similar to the previous commit, __start_init_ktypes and __stop_init_ktypes are declared as structs but should probably be declared as pointers.

Quoted 7 lines

22097b7 ("Implement kernel module's kallsyms resolving") edfa698 ("Implement kernel module's btf resolving")

* These commits are straightforward:

ede22e8 ("Add makedumpfile extensions support")

I will address those later after reading through your code branch. Thanks again for your detailed comments!

Thanks,
Tao Liu

Quoted 365 lines

* nit: the array "handlers" actually contains "handles" (no r) returned from dlopen(). The word handlers usually implies a function pointer or something like that, called to handle a certain situation. Maybe rename this to "handles".
* The run_extensions() function is called within a retry loop in create_dumpfile(). It seems possible that this could get called multiple times. But many of the global variables in the kallsyms, BTF, and extension code are not safe to be cleaned up and reinitialized. In particular, when array elements are freed, the lengths, capacities, and pointers are not reset to 0/NULL. I think it would be wise to make all the cleanup functions clear the globals, so that they may be reinitialized safely.
* Further, I think it might be helpful to split extension loading, running, and cleanup. Something like this in create_dumpfile():

        load_extensions(); /* loads everything and calls entry() */
    retry:
        /* ... create bitmap and dump ... */
        if (status == NOSPACE) {
            /* ... */
            goto retry;
        }
        /* ... */
        cleanup_extensions();
        return;

For two reasons: (1) this avoids unnecessary work re-loading and re-initializing the BTF, kallsyms, and extensions. (Though I still think it's safer to ensure they can be re-initialized safely.) And (2), this allows for future use of extensions in the rest of the dump operation. For my userspace stack extension, I plan to add a callback which allows extensions to override the decision to filter a page, since my logic can't be easily done via erase info. So the extensions need to remain loaded during the creation of the dumpfile, and cleaned up after. I have tweaked this in my own patches, but I just wanted to share the use case.

c568635 ("btf/kallsyms based makedumpfile extension for mm page filtering")

* This looks good to me!
* I will say that for my userspace stack use case, while the "filter_page(..., false)" mechanism for specifying pages that are retained *looks* useful, I wouldn't be able to use it because I cannot determine the PFNs for each stack VMA. Instead, I have to use the page "mapping" and "index" fields to match the VMA and determine whether the PFN falls in a range I care to save. Of course, just because *I* won't use it doesn't mean it's not useful :)

2b252ec ("Filter amdgpu mm pages")

* I'm no expert in amdgpu, but the overall approach makes sense to me, and the helpers look good.

At a broader level, while the add_to_arr() function is useful, I do think the dynamic array/vector pattern could be captured with a dedicated data structure. For instance, drgn has this excellent LGPL-2.1+ header-only vector library:
https://github.com/osandov/drgn/blob/main/libdrgn/vector.h
I don't think it is high priority or must be addressed. Nor do I mean that this particular implementation choice be used. It's just something to share & think about.

To provide some context on my comments related to the userspace stack tracing, here is a branch of mine which is based on yours, that adds my userspace stack extension and a few tweaks:

https://github.com/brenns10/makedumpfile/commits/stepbren_userstack_v4/

Thank you,
Stephen

Quoted 295 lines

Thanks,
Tao Liu

[1]: https://github.com/liutgnu/makedumpfile/commits/v4/

Quoted 283 lines · Tao Liu

Hi Stephen,

Thanks a lot for your quick reply and detailed information, I really appreciate it!

Quoted 7 lines · Stephen Brennan

Hi Tao,

This series looks really great -- I'm excited to see the switch to native .so extensions instead of epicc. I've applied the series locally and I'll rebuild my userspace stack inclusion feature based on it, to try it out myself.

Awesome, looking forward to your feedback on the code/API designs etc...

Quoted 100 lines

In the meantime, I'll share some of my feedback on the patches (though I'm not a makedumpfile developer). This seems like the most important patch in terms of design, so I'll start here.

Quoted 78 lines · Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
      rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+     $(MAKE) -C extensions clean
 
 install:
      install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
      mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
      install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
      install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+     $(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +21 -0
new file mode 100644
index 0000000..6ee7f4e
@@ -0,0 +1,82 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <dlfcn.h>
+#include <stdbool.h>
+#include "kallsyms.h"
+#include "btf_info.h"
+
+static const char *dirs[] = {
+     "/usr/lib64/makedumpfile/extensions",
+     "./extensions",
+};
+
+/* Will only init once */
+static bool init_kallsyms_btf(void)
+{
+     static bool ret = false;
+     static bool has_inited = false;
+
+     if (has_inited)

extension.c line 31

Quoted 10 lines
extension.c new file +10 -0
+             goto out;
+     if (!init_kernel_kallsyms())
+             goto out;
+     if (!init_kernel_btf())
+             goto out;
+     if (!init_module_kallsyms())
+             goto out;
+     if (!init_module_btf())
+             goto out;
+     ret = true;

I feel it would be good practice to load as little information as is necessary for the task. If "amdgpu" module is required, then load kernel kallsyms, BTF, and then the amdgpu module kallsyms & BTF. If no module debuginfo is required, then just the kernel would suffice.

Quoted 3 lines

This would reduce memory usage and runtime, though I don't know if it would show up in profiling. The main benefit could be reliability: by handling less data, there are fewer chances to hit an error.

OK, I agree, mandatory kernel btf/kallsyms info + optional kernel module btf/kallsyms info is a reasonable design. So kernel modules' info can be loaded on demand.

Quoted 38 lines
Quoted 19 lines
extension.c new file +18 -0
+out:
+     has_inited = true;
+     return ret;
+}
+
+static void cleanup_kallsyms_btf(void)
+{
+     cleanup_kallsyms();
+     cleanup_btf();
+}
+
+void run_extensions(void)
+{
+     DIR *dir;
+     struct dirent *entry;
+     size_t len;
+     int i;
+     void *handle;

extension.c line 59

Quoted 10 lines
extension.c new file +10 -0
+     char path[512];
+
+     for (i = 0; i < sizeof(dirs) / sizeof(char *); i++) {
+             if ((dir = opendir(dirs[i])) != NULL)
+                     break;
+     }
+
+     if (!dir || i >= sizeof(dirs) / sizeof(char *))
+             /* No extensions found */
+             return;

It could be confusing that makedumpfile would behave differently with the same command-line arguments depending on the presence or absence of these extensions on the filesystem.

I think it may fit users' expectations better if they are required to specify extensions on the command line. Then we could load them by searching each directory in order. This allows:

Quoted 5 lines

(a) more expected behavior
(b) multiple extensions can exist without all being enabled, thus more flexibility
(c) extensions can be present in the local "extensions/" directory, or in the system directory

Sure, it also sounds reasonable. My original thoughts are, user customization on mm filtering are specified in .so, and if user don't need one .so, e.g. amdgpu mm filtering for a nvidia machine, then he doesn't pack the amdgpu_filter.so into kdump's initramfs. I agree adding extra makedumpfile cmdline option to receive those needed .so is a better design.

Quoted 30 lines
Quoted 7 lines
extension.c new file +6 -0
+     while ((entry = readdir(dir)) != NULL) {
+             len = strlen(entry->d_name);
+             if (len > 3 && strcmp(entry->d_name + len - 3, ".so") == 0) {
+                     /* Will only init when .so exist */
+                     if (!init_kallsyms_btf())
+                             goto out;

extension.c line 75

Quoted 10 lines
extension.c new file +10 -0
+
+                     snprintf(path, sizeof(path), "%s/%s", dirs[i], entry->d_name);
+                     handle = dlopen(path, RTLD_NOW);
+                     if (!handle) {
+                             fprintf(stderr, "%s: Failed to load %s: %s\n",
+                                     __func__, path, dlerror());
+                             continue;
+                     }
+                     printf("Loaded extension: %s\n", path);
+                     dlclose(handle);

Using the constructor/destructor of the shared object is clever! But we lose some flexibility: by the time the dlopen() returns, the constructor has executed and the plugin has thus executed.

What if we instead use dlsym() to load some symbols from the DSO? In particular, I think it would be useful if extensions could declare a list of symbols and a list of structure information which they are interested in receiving. We could use these lists to know which kernel/module kallsyms & BTF we should load. We could even load the information into the local variables of the extension, so the extension would not need to manually load it.

Quoted 8 lines

Of course this is more complex, but the benefit is:

1. Extensions can be written more simply, and would not need to manually load each symbol & type.
2. We could eliminate the hash tables for kallsyms & BTF, and eliminate the loading of unnecessary module information. Instead, we'd just populate the symbol addresses, struct offsets, and type sizes directly into the local variables which request them.

It is a clever idea! Though complex for code, I think it is doable.

Quoted 5 lines

Again, while I don't want to prematurely optimize -- it's good to avoid loading unnecessary information. I hope I've described my idea well. I would be happy to work on an implementation of it based on your patches here, if you're interested.

Thanks again for your suggestions! I got your points and I think I can improve the code while waiting for maintainers ideas at the same time. I will let you know when done or encounter blockers if any.

Thanks,
Tao Liu

Quoted 52 lines

Thanks,
Stephen

Quoted 47 lines
extension.c new file +6 -0
+             }
+     }
+out:
+     closedir(dir);
+     cleanup_kallsyms_btf();
+}
\ No newline at end of file
extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+     $(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+     rm -f $(CONTRIB_SO)
makedumpfile.c +4 -1
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
      }
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
      if (info->flag_refiltering)
              update_dump_level();
 
+     run_extensions();
+
      if ((info->name_filterconfig || info->name_eppic_config)
                      && !gather_filter_info())
              return FALSE;
--

2.47.0

author
#14 lore raw

Re: [PATCH v3 5/8] Add makedumpfile extension support

Hi Stephen,

Thanks a lot for your comments, I have made plenty of code improvements based on your suggestions and your github code repo. Sorry I didn't inline reply to all your comments, please check out the latest v4 patch I posted upstream.

Some notable improvements:

1) I adjusted the extension callback function to amdgpu_filter, so a same callback interface can serve both our extensions.
2) I followed your implementation of the INIT_OPT_(...) macro for optional sym/types and the INIT_(...) macro for must-have sym/types. If extension missing any must-have sym/types, it will load-fail and exit as early(e.g. a machine doesn't have amdgpu). And optional symbols checking during extension_init() (e.g. existence of kernel symbol for kernel version detection)
3) kallsyms address calc for v7.0 kernel

Quoted 175 lines · Stephen Brennan
Quoted 54 lines · Tao Liu

Hi Stephen,

Quoted 27 lines · Stephen Brennan
Quoted 18 lines · Tao Liu

Hi Stephen,

Sorry I took some time to make modifications on v3 code. Please see a drafted v4 [1] for a preview.

I followed your suggestion to let extensions declare the kallsyms symbol/btf types it needed, then during kallsyms/btf initialization, it will only resolve the declared symbol/types, thus getting rid of the hash table. Extensions now don't need to initial needed symbols/types by itself.

In addition, users can specify which extensions to load at makedumpfile cmdline as:

$ ./makedumpfile -d 31 -l
/var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore /tmp/out --extension
amdgpu_filter.so
Quoted 6 lines

Also "--extension" can be used several times, and amdgpu_filter.so can be an absolute path as well. If no extensions are specified, the btf/kallsyms of makedumpfile will not initialize.

I don't know if this is what you wanted, or any suggestions for this? Thanks in advance!

Quoted 8 lines

Hi Tao,

Please accept my apologies for the long delay on this review. I've gone through each commit with my feedback. This looks like a really excellent job and I think it is ready for the review of the maintainers. I've implemented my own userspace stack extension on top of this support, and I find it to be great for my use case as well, with one small change (adding flags for detecting whether a struct or member is found).

No worries at all :) Thanks a lot for your detailed comments. I'm looking through them as well as your code branch in github. This may take a while...

Quoted 9 lines

Here is some more detailed feedback on a per-commit level; it's all pretty minor.

d3aee7a ("Reserve sections for makedumpfile and extenions")

* This is just a note, but I've found that the linker script may not be necessary. GCC automatically creates the __start_SECTION and __stop_SECTION symbols for sections it creates.

Quoted 9 lines

I have tried to remove the makedumpfile.ld linker script. The automatic symbol as __start/stop_SECTION are only generated for makedumpfile, aka the main program, but not for .so extensions. This is a breaker because I would expect all .so files have the __start/stop_SECTION symbol so they can be resigtered to main program. I'm not expert in GCC's default behaviours, so I guess I would prefer the current approach of explicit symbol define within the linker script, seems to be more robust... Perhaps you can share your code on this, I will give it a try.

The key to making GCC generate the symbols for the sections is that the section names should themselves be valid C identifiers (i.e. they should not start with a '.'). Here's a patch to give the idea:

From 3618f51869e978328e49f6061eae94469bca43c5 Mon Sep 17 00:00:00 2001
From: Stephen Brennan <[email protected]>
Date: Wed, 11 Mar 2026 09:17:05 -0700
Subject: [PATCH] Use automatically generated section start symbols

GCC will create __start_SECTION and __stop_SECTION only if the section names are valid C identifiers. This means we cannot use a leading dot. Drop the leading dot from the section names so we can avoid defining a custom linker script.

---
 Makefile            |  2 +-
 btf_info.h          |  4 ++--
 extensions/Makefile |  2 +-
 kallsyms.h          |  2 +-
 makedumpfile.ld     | 15 ---------------
 5 files changed, 5 insertions(+), 20 deletions(-)
 delete mode 100644 makedumpfile.ld
Makefile +1 -1
index 8196aa6..e2cde8f 100644
@@ -113,7 +113,7 @@ $(OBJ_ARCH): $(SRC_ARCH)
        $(CC) $(CFLAGS_ARCH) -c -o ./$@ $(VPATH)$(@:.o=.c)
 
 makedumpfile: $(SRC_BASE) $(OBJ_PART) $(OBJ_ARCH)
-       $(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -Wl,-T,makedumpfile.ld -o $@ $< $(LIBS)
+       $(CC) $(CFLAGS) $(LDFLAGS) $(OBJ_PART) $(OBJ_ARCH) -rdynamic -o $@ $< $(LIBS)
        @sed -e "s/@DATE@/$(DATE)/" \
             -e "s/@VERSION@/$(VERSION)/" \
             $(VPATH)makedumpfile.8.in > $(VPATH)makedumpfile.8
btf_info.h +2 -2
index 2a78fbf..555f21e 100644
@@ -29,7 +29,7 @@ void cleanup_btf(void);
        struct ktype_info _##MOD##_##S##_##M = {                \
                QUATE(MOD), QUATE(S), QUATE(M), 0, 0, 0, 0, R, R\
        };                                                      \
-       __attribute__((section(".init_ktypes"), used))          \
+       __attribute__((section("init_ktypes"), used))           \
        struct ktype_info * _ptr_##MOD##_##S##_##M = &_##MOD##_##S##_##M
 
 #define INIT_MOD_STRUCT_MEMBER(MOD, S, M)                      \
@@ -60,7 +60,7 @@ void cleanup_btf(void);
        struct ktype_info _##MOD##_##S = {                      \
                QUATE(MOD), QUATE(S), 0, 0, 0, 0, 0, R, 0       \
        };                                                      \
-       __attribute__((section(".init_ktypes"), used))          \
+       __attribute__((section("init_ktypes"), used))           \
        struct ktype_info * _ptr_##MOD##_##S = &_##MOD##_##S
 
 #define INIT_MOD_STRUCT(MOD, S) INIT_MOD_STRUCT_RQD(MOD, S, 1)
extensions/Makefile +1 -1
index fc28577..a4dc86a 100644
@@ -8,7 +8,7 @@ amdgpu_filter.so: maple_tree.c
 userstack.so: maple_tree.c vma_rbtree.c
 
 $(CONTRIB_SO): %.so: %.c
-       $(CC) -O2 -g -fPIC -shared -Wl,-T,../makedumpfile.ld -o $@ $^
+       $(CC) -O2 -g -fPIC -shared -o $@ $^
 
 clean:
        rm -f $(CONTRIB_SO)
kallsyms.h +1 -1
index d94830e..1989dfe 100644
@@ -17,7 +17,7 @@ struct ksym_info {
        struct ksym_info _##MOD##_##SYM = {             \
                QUATE(MOD), QUATE(SYM), 0               \
        };                                              \
-       __attribute__((section(".init_ksyms"), used))   \
+       __attribute__((section("init_ksyms"), used))    \
        struct ksym_info * _ptr_##MOD##_##SYM = &_##MOD##_##SYM
 
 #define GET_MOD_SYM(MOD, SYM) (_##MOD##_##SYM.value)
makedumpfile.ld deleted +0 -16
deleted file mode 100644
index 231a162..0000000
@@ -1,15 +0,0 @@
-SECTIONS
-{
-       .init_ksyms ALIGN(8) : {
-               __start_init_ksyms = .;
-               KEEP(*(.init_ksyms*))
-               __stop_init_ksyms = .;
-       }
-
-       .init_ktypes ALIGN(8) : {
-               __start_init_ktypes = .;
-               KEEP(*(.init_ktypes*))
-               __stop_init_ktypes = .;
-       }
-}
-INSERT AFTER .data;
\ No newline at end of file
--
Quoted 1 line

2.47.3

Unfortunately with the patch applied I cannot see the __start_init_ksyms/ktypes symbol for amdgpu_filter.so. So I had to keep the linker script in v4...

Thanks,
Tao Liu

Quoted 494 lines
Quoted 73 lines
Quoted 11 lines

* Otherwise, this is exactly what I was thinking!

102ae0b ("Implement kernel kallsyms resolving")

* Again, this is looking very close to the design I hoped for, thanks!
* I'm not sure whether is_unwanted_symbol() needs to exist anymore? Given that users opt-in to specific symbols, we don't need to filter out noisy ones that would waste memory. What do you think?

Agreed, I can remove this function.

Quoted 4 lines

* Unfortunately, upstream has made some changes to the vmlinux kallsyms encoding in 7.0. You may want to check what we did in drgn to support those changes:
https://github.com/osandov/drgn/commit/744f36ec3c3f64d7e1323a0037898158698585c4
* In kallsyms.c:

Thanks for the info, I will try this and update the code in v4.

Quoted 10 lines
  +/*
  + * Makedumpfile's .init_ksyms section
  +*/
  +extern struct ksym_info __start_init_ksyms[];
  +extern struct ksym_info __stop_init_ksyms[];

I'm not sure it matters, but the type here is wrong. It should be "extern struct ksym_info *" because you're storing pointers, not the actual struct. That said, the type isn't used so I don't know that it matters.

Right, this is an error. Thanks for pointing that out.

Quoted 11 lines

4187b33 ("Implement kernel btf resolving")

* The get_ktype_info() function fails if either the struct or member is not found. This makes sense in a lot of cases, but there are other cases where we will want to use the presence or absence of a struct/struct member to detect which version of code to use. For example, my userspace stack code will use either maple or rbtree helper code for finding stack VMAs, depending on which is available. We can't know until runtime, when we check whether mm_rb or mm_mt is present.

Quoted 8 lines

One solution is to handle this at runtime. We can have a macro like "HAVE_MEMBER(S, M)" and "HAVE_STRUCT(S)", and then each extension can check whether for the members it expects to be present. I think the major downside to this approach is that it requires manual effort, which is likely to be forgotten when writing extensions.

Alternatively, we could set a flag in the struct ktype_info if the type is "required" (or optional), and only fail get_ktype_info() for required

Agreed, an optional flag looks better.

Quoted 7 lines

structs/members. The concern with this approach is: what if plugin (A) requires a type which is not present, but plugin (B) does not? If both are loaded, the failure of A would cause B not to run. I'm not sure whether we should care about that situation... I don't know if we have a use case for using multiple plugins at the same time. Until we do, we probably won't have a good idea whether it should be allowed for one to fail, but the other to continue.

Great question! I have thought about this previously. I suggest that if plugin(A) fails, it should just fail and allow the execution of any later plugins (B, C....) to continue. Each plugin is responsible for one task, like plugin(A) for dealing with amdgpu's mm page filtering,and plugin(B) for Intel's and plugin(C) for NV's. Plugin(A) certainly will fail if one machine have no amdgpu, thus the amdgpu.ko will never been loaded, so related symbol/types missing. This is expected and shouldn't block the later plugins.

Quoted 2 lines

But the "fail" should gentle, not like the ones as segfault, which will crash the entire makedumpfile program.

Totally agreed up to here! I think it makes sense for plugin failures to be independent.

It seems like the BTF loading process should not fail if any type or member is not found. But once we've tried to load everything, we can then go through each plugin's list, and if we find a "required" type or member which is not present, we log that failure and don't run the plugin. This shouldn't be much work, because we already have all the types in an array for each plugin.

Quoted 8 lines

Since currently plugins
are native .so libraries, the quality of code is ensured by each plugin authors, rather than makedumpfile maintainers. Idealy the the plugins are well tested in 1st kernel before they are shipped to kdump img, but who knows. From makedumpfile's view, do you think we need to introduce a sandbox to isolate plugins from makedumpfile? This would prevent serious plugin errors from stopping makedumpfile from generating the vmcore.

Personally, I don't believe we need sandbox. The plugins should be written to handle the clear error cases:

- Optional types or symbols may not be present, so validate them.
- Values read from /proc/vmcore may not be valid, or the read may fail, so check all return values.

That said, I'm not set against it, so if you believe it's necessary I could be convinced :)

Thanks again,
Stephen

Quoted 385 lines
Quoted 6 lines

I've implemented this second alternative in my branch.

* Similar to the previous commit, __start_init_ktypes and __stop_init_ktypes are declared as structs but should probably be declared as pointers.

Quoted 7 lines

22097b7 ("Implement kernel module's kallsyms resolving") edfa698 ("Implement kernel module's btf resolving")

* These commits are straightforward:

ede22e8 ("Add makedumpfile extensions support")

I will address those later after reading through your code branch. Thanks again for your detailed comments!

Thanks,
Tao Liu

Quoted 365 lines

* nit: the array "handlers" actually contains "handles" (no r) returned from dlopen(). The word handlers usually implies a function pointer or something like that, called to handle a certain situation. Maybe rename this to "handles".
* The run_extensions() function is called within a retry loop in create_dumpfile(). It seems possible that this could get called multiple times. But many of the global variables in the kallsyms, BTF, and extension code are not safe to be cleaned up and reinitialized. In particular, when array elements are freed, the lengths, capacities, and pointers are not reset to 0/NULL. I think it would be wise to make all the cleanup functions clear the globals, so that they may be reinitialized safely.
* Further, I think it might be helpful to split extension loading, running, and cleanup. Something like this in create_dumpfile():

        load_extensions(); /* loads everything and calls entry() */
    retry:
        /* ... create bitmap and dump ... */
        if (status == NOSPACE) {
            /* ... */
            goto retry;
        }
        /* ... */
        cleanup_extensions();
        return;

For two reasons: (1) this avoids unnecessary work re-loading and re-initializing the BTF, kallsyms, and extensions. (Though I still think it's safer to ensure they can be re-initialized safely.) And (2), this allows for future use of extensions in the rest of the dump operation. For my userspace stack extension, I plan to add a callback which allows extensions to override the decision to filter a page, since my logic can't be easily done via erase info. So the extensions need to remain loaded during the creation of the dumpfile, and cleaned up after. I have tweaked this in my own patches, but I just wanted to share the use case.

c568635 ("btf/kallsyms based makedumpfile extension for mm page filtering")

* This looks good to me!
* I will say that for my userspace stack use case, while the "filter_page(..., false)" mechanism for specifying pages that are retained *looks* useful, I wouldn't be able to use it because I cannot determine the PFNs for each stack VMA. Instead, I have to use the page "mapping" and "index" fields to match the VMA and determine whether the PFN falls in a range I care to save. Of course, just because *I* won't use it doesn't mean it's not useful :)

2b252ec ("Filter amdgpu mm pages")

* I'm no expert in amdgpu, but the overall approach makes sense to me, and the helpers look good.

At a broader level, while the add_to_arr() function is useful, I do think the dynamic array/vector pattern could be captured with a dedicated data structure. For instance, drgn has this excellent LGPL-2.1+ header-only vector library:
https://github.com/osandov/drgn/blob/main/libdrgn/vector.h
I don't think it is high priority or must be addressed. Nor do I mean that this particular implementation choice be used. It's just something to share & think about.

To provide some context on my comments related to the userspace stack tracing, here is a branch of mine which is based on yours, that adds my userspace stack extension and a few tweaks:

https://github.com/brenns10/makedumpfile/commits/stepbren_userstack_v4/

Thank you,
Stephen

Quoted 295 lines

Thanks,
Tao Liu

[1]: https://github.com/liutgnu/makedumpfile/commits/v4/

Quoted 283 lines · Tao Liu

Hi Stephen,

Thanks a lot for your quick reply and detailed information, I really appreciate it!

Quoted 7 lines · Stephen Brennan

Hi Tao,

This series looks really great -- I'm excited to see the switch to native .so extensions instead of epicc. I've applied the series locally and I'll rebuild my userspace stack inclusion feature based on it, to try it out myself.

Awesome, looking forward to your feedback on the code/API designs etc...

Quoted 100 lines

In the meantime, I'll share some of my feedback on the patches (though I'm not a makedumpfile developer). This seems like the most important patch in terms of design, so I'll start here.

Quoted 78 lines · Tao Liu

This patch will add .so extension support to makedumpfile, similar to crash extension to crash utility. Currently only "/usr/lib64/makedumpfile/extensions" and "./extensions" are searched for extensions. Once found, kallsyms and btf will be initialized so all extensions can benifit from it (Currently makedumpfile doesn't use these info, we can move the kallsyms/btf init code else where later if makedumpfile needs them).

The makedumpfile extension is to help users to customize mm page filtering upon traditional mm page flag filtering, without make code modification on makedumpfile itself.

---
 Makefile            |  7 +++-
 extension.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++
 extensions/Makefile | 10 ++++++
 makedumpfile.c      |  4 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
Makefile +6 -1
index f3f4da8..7e29220 100644
@@ -45,7 +45,7 @@ CFLAGS_ARCH += -m32
 endif
 
 SRC_BASE = makedumpfile.c makedumpfile.h diskdump_mod.h sadump_mod.h sadump_info.h
-SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c
+SRC_PART = print_info.c dwarf_info.c elf_info.c erase_info.c sadump_info.c cache.c tools.c printk.c detect_cycle.c kallsyms.c btf_info.c extension.c
 OBJ_PART=$(patsubst %.c,%.o,$(SRC_PART))
 SRC_ARCH = arch/arm.c arch/arm64.c arch/x86.c arch/x86_64.c arch/ia64.c arch/ppc64.c arch/s390x.c arch/ppc.c arch/sparc64.c arch/mips64.c arch/loongarch64.c arch/riscv64.c
 OBJ_ARCH=$(patsubst %.c,%.o,$(SRC_ARCH))
@@ -126,6 +126,7 @@ eppic_makedumpfile.so: extension_eppic.c
 
 clean:
      rm -f $(OBJ) $(OBJ_PART) $(OBJ_ARCH) makedumpfile makedumpfile.8 makedumpfile.conf.5
+     $(MAKE) -C extensions clean
 
 install:
      install -m 755 -d ${DESTDIR}/${SBINDIR} ${DESTDIR}/usr/share/man/man5 ${DESTDIR}/usr/share/man/man8
@@ -135,3 +136,7 @@ install:
      mkdir -p ${DESTDIR}/usr/share/makedumpfile/eppic_scripts
      install -m 644 -D $(VPATH)makedumpfile.conf ${DESTDIR}/usr/share/makedumpfile/makedumpfile.conf.sample
      install -m 644 -t ${DESTDIR}/usr/share/makedumpfile/eppic_scripts/ $(VPATH)eppic_scripts/*
+
+.PHONY: extensions
+extensions:
+     $(MAKE) -C extensions CC=$(CC)
\ No newline at end of file
extension.c new file +21 -0
new file mode 100644
index 0000000..6ee7f4e
@@ -0,0 +1,82 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <dirent.h>
+#include <dlfcn.h>
+#include <stdbool.h>
+#include "kallsyms.h"
+#include "btf_info.h"
+
+static const char *dirs[] = {
+     "/usr/lib64/makedumpfile/extensions",
+     "./extensions",
+};
+
+/* Will only init once */
+static bool init_kallsyms_btf(void)
+{
+     static bool ret = false;
+     static bool has_inited = false;
+
+     if (has_inited)

extension.c line 31

Quoted 10 lines
extension.c new file +10 -0
+             goto out;
+     if (!init_kernel_kallsyms())
+             goto out;
+     if (!init_kernel_btf())
+             goto out;
+     if (!init_module_kallsyms())
+             goto out;
+     if (!init_module_btf())
+             goto out;
+     ret = true;

I feel it would be good practice to load as little information as is necessary for the task. If "amdgpu" module is required, then load kernel kallsyms, BTF, and then the amdgpu module kallsyms & BTF. If no module debuginfo is required, then just the kernel would suffice.

Quoted 3 lines

This would reduce memory usage and runtime, though I don't know if it would show up in profiling. The main benefit could be reliability: by handling less data, there are fewer chances to hit an error.

OK, I agree, mandatory kernel btf/kallsyms info + optional kernel module btf/kallsyms info is a reasonable design. So kernel modules' info can be loaded on demand.

Quoted 38 lines
Quoted 19 lines
extension.c new file +18 -0
+out:
+     has_inited = true;
+     return ret;
+}
+
+static void cleanup_kallsyms_btf(void)
+{
+     cleanup_kallsyms();
+     cleanup_btf();
+}
+
+void run_extensions(void)
+{
+     DIR *dir;
+     struct dirent *entry;
+     size_t len;
+     int i;
+     void *handle;

extension.c line 59

Quoted 10 lines
extension.c new file +10 -0
+     char path[512];
+
+     for (i = 0; i < sizeof(dirs) / sizeof(char *); i++) {
+             if ((dir = opendir(dirs[i])) != NULL)
+                     break;
+     }
+
+     if (!dir || i >= sizeof(dirs) / sizeof(char *))
+             /* No extensions found */
+             return;

It could be confusing that makedumpfile would behave differently with the same command-line arguments depending on the presence or absence of these extensions on the filesystem.

I think it may fit users' expectations better if they are required to specify extensions on the command line. Then we could load them by searching each directory in order. This allows:

Quoted 5 lines

(a) more expected behavior
(b) multiple extensions can exist without all being enabled, thus more flexibility
(c) extensions can be present in the local "extensions/" directory, or in the system directory

Sure, it also sounds reasonable. My original thoughts are, user customization on mm filtering are specified in .so, and if user don't need one .so, e.g. amdgpu mm filtering for a nvidia machine, then he doesn't pack the amdgpu_filter.so into kdump's initramfs. I agree adding extra makedumpfile cmdline option to receive those needed .so is a better design.

Quoted 30 lines
Quoted 7 lines
extension.c new file +6 -0
+     while ((entry = readdir(dir)) != NULL) {
+             len = strlen(entry->d_name);
+             if (len > 3 && strcmp(entry->d_name + len - 3, ".so") == 0) {
+                     /* Will only init when .so exist */
+                     if (!init_kallsyms_btf())
+                             goto out;

extension.c line 75

Quoted 10 lines
extension.c new file +10 -0
+
+                     snprintf(path, sizeof(path), "%s/%s", dirs[i], entry->d_name);
+                     handle = dlopen(path, RTLD_NOW);
+                     if (!handle) {
+                             fprintf(stderr, "%s: Failed to load %s: %s\n",
+                                     __func__, path, dlerror());
+                             continue;
+                     }
+                     printf("Loaded extension: %s\n", path);
+                     dlclose(handle);

Using the constructor/destructor of the shared object is clever! But we lose some flexibility: by the time the dlopen() returns, the constructor has executed and the plugin has thus executed.

What if we instead use dlsym() to load some symbols from the DSO? In particular, I think it would be useful if extensions could declare a list of symbols and a list of structure information which they are interested in receiving. We could use these lists to know which kernel/module kallsyms & BTF we should load. We could even load the information into the local variables of the extension, so the extension would not need to manually load it.

Quoted 8 lines

Of course this is more complex, but the benefit is:

1. Extensions can be written more simply, and would not need to manually load each symbol & type.
2. We could eliminate the hash tables for kallsyms & BTF, and eliminate the loading of unnecessary module information. Instead, we'd just populate the symbol addresses, struct offsets, and type sizes directly into the local variables which request them.

It is a clever idea! Though complex for code, I think it is doable.

Quoted 5 lines

Again, while I don't want to prematurely optimize -- it's good to avoid loading unnecessary information. I hope I've described my idea well. I would be happy to work on an implementation of it based on your patches here, if you're interested.

Thanks again for your suggestions! I got your points and I think I can improve the code while waiting for maintainers ideas at the same time. I will let you know when done or encounter blockers if any.

Thanks,
Tao Liu

Quoted 52 lines

Thanks,
Stephen

Quoted 47 lines
extension.c new file +6 -0
+             }
+     }
+out:
+     closedir(dir);
+     cleanup_kallsyms_btf();
+}
\ No newline at end of file
extensions/Makefile new file +10 -0
new file mode 100644
index 0000000..afbc61e
@@ -0,0 +1,10 @@
+CC ?= gcc
+CONTRIB_SO :=
+
+all: $(CONTRIB_SO)
+
+$(CONTRIB_SO): %.so: %.c
+     $(CC) -O2 -g -fPIC -shared -o $@ $^
+
+clean:
+     rm -f $(CONTRIB_SO)
makedumpfile.c +4 -1
index dba3628..ca8ed8a 100644
@@ -10847,6 +10847,8 @@ update_dump_level(void)
      }
 }
 
+void run_extensions(void);
+
 int
 create_dumpfile(void)
 {
@@ -10884,6 +10886,8 @@ retry:
      if (info->flag_refiltering)
              update_dump_level();
 
+     run_extensions();
+
      if ((info->name_filterconfig || info->name_eppic_config)
                      && !gather_filter_info())
              return FALSE;
--

2.47.0

author
#15 lore raw

PATCH v3 6/8 Add page filtering function

In reply to Tao Liu

pfn and num is the data which extensions give to makedumpfile for mm page filtering. Since makedumpfile will iterate the pfn in an ascending order in __exclude_unnecessary_pages(), pfn and num are stored within ft_page_info linked lists and organized in an ascending order by pfn, so if one pfn
is hit by one list, the next pfn is most likely to be hit either by this list again, or the one follows, so a cur variable is used for saving the current list position to speedup the pfn checking process.

In addition, 2 ft_page_info linked list chains are used, one for mm page discarding and the other for page keeping.

---
 erase_info.c   | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++
 erase_info.h   | 12 +++++++
 makedumpfile.c | 28 ++++++++++++---
 3 files changed, 134 insertions(+), 4 deletions(-)
erase_info.c +98 -0
index b67d1d0..8838bea 100644
@@ -2466,3 +2466,101 @@ get_size_eraseinfo(void)
 	return size_eraseinfo;
 }
 
+/* Pages to be discarded */
+static struct ft_page_info *ft_head_discard = NULL;
+/* Pages to be keeped */
+static struct ft_page_info *ft_head_keep = NULL;
+
+/*
+ * Insert the ft_page_info blocks into ft_head by ascending pfn.
+ */
+bool
+update_filter_pages_info(unsigned long pfn, unsigned long num, bool to_discard)
+{
+	struct ft_page_info *p, **ft_head;
+	struct ft_page_info *new_p = malloc(sizeof(struct ft_page_info));
+
+	ft_head = to_discard ? &ft_head_discard : &ft_head_keep;
+
+	if (!new_p) {
+		ERRMSG("Can't allocate memory for ft_page_info at %lx\n", pfn);
+		return false;
+	}
+	new_p->pfn = pfn;
+	new_p->num = num;
+	new_p->next = NULL;
+
+	if (!(*ft_head) || (*ft_head)->pfn > new_p->pfn) {
+		new_p->next = (*ft_head);
+		(*ft_head) = new_p;
+		return true;
+	}
+
+	p = (*ft_head);
+	while (p->next != NULL && p->next->pfn < new_p->pfn) {
+		p = p->next;
+	}
+
+	new_p->next = p->next;
+	p->next = new_p;
+	return true;
+}
+
+/*
+ * Check if the pfn hit ft_page_info block.
+ *
+ * pfn and ft_head are in ascending order, so save the current ft_page_info
+ * block into **p because it is likely to hit again next time.
+ */
+bool
+filter_page(unsigned long pfn, struct ft_page_info **p, bool handle_discard)
+{
+	struct ft_page_info *ft_head;
+
+	ft_head = handle_discard ? ft_head_discard : ft_head_keep;
+
+	if (ft_head == NULL)
+		return false;
+
+	if (*p == NULL)
+		*p = ft_head;
+
+	/* The gap before 1st block */
+	if (pfn >= 0 && pfn < ft_head->pfn)
+		return false;
+
+	/* Handle 1~(n-1) blocks and following gaps */
+	while ((*p)->next) {
+		if (pfn >= (*p)->pfn && pfn < (*p)->pfn + (*p)->num)
+			return true; // hit the block
+		if (pfn >= (*p)->pfn + (*p)->num && pfn < (*p)->next->pfn)
+			return false; // the gap after the block
+		*p = (*p)->next;
+	}
+
+	/* The last block and gap */
+	if (pfn >= (*p)->pfn + (*p)->num)
+		return false;
+	else
+		return true;
+}
+
+static void
+do_cleanup(struct ft_page_info **ft_head)
+{
+	struct ft_page_info *p, *p_tmp;
+
+	for (p = *ft_head; p;) {
+		p_tmp = p;
+		p = p->next;
+		free(p_tmp);
+	}
+	*ft_head = NULL;
+}
+
+void
+cleanup_filter_pages_info(void)
+{
+	do_cleanup(&ft_head_discard);
+	do_cleanup(&ft_head_keep);
+}
erase_info.h +12 -0
index b363a40..6c60706 100644
@@ -20,6 +20,7 @@
 #define _ERASE_INFO_H
 
 #define MAX_SIZE_STR_LEN (26)
+#include <stdbool.h>
 
 /*
  * Erase information, original symbol expressions.
@@ -65,5 +66,16 @@ void filter_data_buffer_parallel(unsigned char *buf, unsigned long long paddr,
 unsigned long get_size_eraseinfo(void);
 int update_filter_info_raw(unsigned long long, int, int);
 
+bool update_filter_pages_info(unsigned long, unsigned long, bool);
+
+struct ft_page_info {
+	unsigned long pfn;
+	unsigned long num;
+	struct ft_page_info *next;
+} __attribute__((packed));
+
+bool filter_page(unsigned long, struct ft_page_info **p, bool handle_discard);
+void cleanup_filter_pages_info(void);
+
 #endif /* _ERASE_INFO_H */
 
makedumpfile.c +24 -4
index ca8ed8a..ebac8da 100644
@@ -102,6 +102,7 @@ mdf_pfn_t pfn_free;
 mdf_pfn_t pfn_hwpoison;
 mdf_pfn_t pfn_offline;
 mdf_pfn_t pfn_elf_excluded;
+mdf_pfn_t pfn_extension;
 
 mdf_pfn_t num_dumped;
 
@@ -6459,6 +6460,8 @@ __exclude_unnecessary_pages(unsigned long mem_map,
 	unsigned int order_offset, dtor_offset;
 	unsigned long flags, mapping, private = 0;
 	unsigned long compound_dtor, compound_head = 0;
+	struct ft_page_info *cur_discard = NULL;
+	struct ft_page_info *cur_keep = NULL;
 
 	/*
 	 * If a multi-page exclusion is pending, do it first
@@ -6495,6 +6498,13 @@ __exclude_unnecessary_pages(unsigned long mem_map,
 		if (info->flag_cyclic && !is_cyclic_region(pfn, cycle))
 			continue;
 
+		/*
+		 * Keep pages that specified by user via
+		 * makedumpfile extensions
+		 */
+		if (filter_page(pfn, &cur_keep, false))
+			continue;
+
 		/*
 		 * Exclude the memory hole.
 		 */
@@ -6687,6 +6697,14 @@ check_order:
 		else if (isOffline(flags, _mapcount)) {
 			pfn_counter = &pfn_offline;
 		}
+		/*
+		 * Exclude pages that specified by user via
+		 * makedumpfile extensions
+		 */
+		else if (filter_page(pfn, &cur_discard, true)) {
+			nr_pages = 1;
+			pfn_counter = &pfn_extension;
+		}
 		/*
 		 * Unexcludable page
 		 */
@@ -6748,6 +6766,7 @@ exclude_unnecessary_pages(struct cycle *cycle)
 		print_progress(PROGRESS_UNN_PAGES, info->num_mem_map, info->num_mem_map, NULL);
 		print_execution_time(PROGRESS_UNN_PAGES, &ts_start);
 	}
+	cleanup_filter_pages_info();
 
 	return TRUE;
 }
@@ -8234,7 +8253,7 @@ write_elf_pages_cyclic(struct cache_data *cd_header, struct cache_data *cd_page)
 	 */
 	if (info->flag_cyclic) {
 		pfn_zero = pfn_cache = pfn_cache_private = 0;
-		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = 0;
+		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = pfn_extension = 0;
 		pfn_memhole = info->max_mapnr;
 	}
 
@@ -9579,7 +9598,7 @@ write_kdump_pages_and_bitmap_cyclic(struct cache_data *cd_header, struct cache_d
 		 * Reset counter for debug message.
 		 */
 		pfn_zero = pfn_cache = pfn_cache_private = 0;
-		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = 0;
+		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = pfn_extension = 0;
 		pfn_memhole = info->max_mapnr;
 
 		/*
@@ -10528,7 +10547,7 @@ print_report(void)
 	pfn_original = info->max_mapnr - pfn_memhole;
 
 	pfn_excluded = pfn_zero + pfn_cache + pfn_cache_private
-	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline;
+	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline + pfn_extension;
 
 	REPORT_MSG("\n");
 	REPORT_MSG("Original pages  : 0x%016llx\n", pfn_original);
@@ -10544,6 +10563,7 @@ print_report(void)
 	REPORT_MSG("    Free pages              : 0x%016llx\n", pfn_free);
 	REPORT_MSG("    Hwpoison pages          : 0x%016llx\n", pfn_hwpoison);
 	REPORT_MSG("    Offline pages           : 0x%016llx\n", pfn_offline);
+	REPORT_MSG("    Extension filter pages  : 0x%016llx\n", pfn_extension);
 	REPORT_MSG("  Remaining pages  : 0x%016llx\n",
 	    pfn_original - pfn_excluded);
 
@@ -10584,7 +10604,7 @@ print_mem_usage(void)
 	pfn_original = info->max_mapnr - pfn_memhole;
 
 	pfn_excluded = pfn_zero + pfn_cache + pfn_cache_private
-	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline;
+	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline + pfn_extension;
 	shrinking = (pfn_original - pfn_excluded) * 100;
 	shrinking = shrinking / pfn_original;
 	total_size = info->page_size * pfn_original;
Signature · 1 line
2.47.0
#16 lore raw

Re: [PATCH v3 6/8] Add page filtering function

Quoted 7 lines · Tao Liu

pfn and num is the data which extensions give to makedumpfile for mm page filtering. Since makedumpfile will iterate the pfn in an ascending order in __exclude_unnecessary_pages(), pfn and num are stored within ft_page_info linked lists and organized in an ascending order by pfn, so if one pfn
is hit by one list, the next pfn is most likely to be hit either by this list again, or the one follows, so a cur variable is used for saving the current list position to speedup the pfn checking process.

I'm wondering about the trade-off for using a linked list versus an array. Using the linked list, we are forced to maintain the sorted order as we construct the list, which is an O(N^2) insertion sort.

If instead we used an array, we could sort it with qsort() once, at the end. Then we could merge any overlapping ranges. Lookup could be implemented cheaply with bsearch(), and we could continue to use the optimization where we maintain a "cur" pointer. I believe the overall runtime complexity of the array approach would be O(N*log(N)) without requiring hand-implementing anything too complex, compared to O(N^2).

Depending on the number of pages (and how fragmented they are), this may or may not be an issue.

In my testing for userspace tasks, the number of pages retained can be on the order of ~100k. However -- my use case can't really use a list of PFNs, which I'll explain below. So my use case doesn't really matter too much here -- maybe your use case has relatively few page ranges, so the cost of O(N^2) is not bad.

So I guess I don't have a strong preference - but it's worth considering.

Quoted 45 lines

In addition, 2 ft_page_info linked list chains are used, one for mm page discarding and the other for page keeping.

---
 erase_info.c   | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++
 erase_info.h   | 12 +++++++
 makedumpfile.c | 28 ++++++++++++---
 3 files changed, 134 insertions(+), 4 deletions(-)
erase_info.c +27 -0
index b67d1d0..8838bea 100644
@@ -2466,3 +2466,101 @@ get_size_eraseinfo(void)
 	return size_eraseinfo;
 }
 
+/* Pages to be discarded */
+static struct ft_page_info *ft_head_discard = NULL;
+/* Pages to be keeped */
+static struct ft_page_info *ft_head_keep = NULL;
+
+/*
+ * Insert the ft_page_info blocks into ft_head by ascending pfn.
+ */
+bool
+update_filter_pages_info(unsigned long pfn, unsigned long num, bool to_discard)
+{
+	struct ft_page_info *p, **ft_head;
+	struct ft_page_info *new_p = malloc(sizeof(struct ft_page_info));
+
+	ft_head = to_discard ? &ft_head_discard : &ft_head_keep;
+
+	if (!new_p) {
+		ERRMSG("Can't allocate memory for ft_page_info at %lx\n", pfn);
+		return false;
+	}
+	new_p->pfn = pfn;
+	new_p->num = num;
+	new_p->next = NULL;
+
+	if (!(*ft_head) || (*ft_head)->pfn > new_p->pfn) {
+		new_p->next = (*ft_head);
+		(*ft_head) = new_p;

erase_info.c get_size_eraseinfo(void) · line 2505

Quoted 10 lines
erase_info.c +10 -0
+		return true;
+	}
+
+	p = (*ft_head);
+	while (p->next != NULL && p->next->pfn < new_p->pfn) {
+		p = p->next;
+	}
+
+	new_p->next = p->next;
+	p->next = new_p;

It might be wise to defensively handle the case of overlapping PFN ranges by merging them.

Quoted 112 lines
erase_info.c +61 -0
+	return true;
+}
+
+/*
+ * Check if the pfn hit ft_page_info block.
+ *
+ * pfn and ft_head are in ascending order, so save the current ft_page_info
+ * block into **p because it is likely to hit again next time.
+ */
+bool
+filter_page(unsigned long pfn, struct ft_page_info **p, bool handle_discard)
+{
+	struct ft_page_info *ft_head;
+
+	ft_head = handle_discard ? ft_head_discard : ft_head_keep;
+
+	if (ft_head == NULL)
+		return false;
+
+	if (*p == NULL)
+		*p = ft_head;
+
+	/* The gap before 1st block */
+	if (pfn >= 0 && pfn < ft_head->pfn)
+		return false;
+
+	/* Handle 1~(n-1) blocks and following gaps */
+	while ((*p)->next) {
+		if (pfn >= (*p)->pfn && pfn < (*p)->pfn + (*p)->num)
+			return true; // hit the block
+		if (pfn >= (*p)->pfn + (*p)->num && pfn < (*p)->next->pfn)
+			return false; // the gap after the block
+		*p = (*p)->next;
+	}
+
+	/* The last block and gap */
+	if (pfn >= (*p)->pfn + (*p)->num)
+		return false;
+	else
+		return true;
+}
+
+static void
+do_cleanup(struct ft_page_info **ft_head)
+{
+	struct ft_page_info *p, *p_tmp;
+
+	for (p = *ft_head; p;) {
+		p_tmp = p;
+		p = p->next;
+		free(p_tmp);
+	}
+	*ft_head = NULL;
+}
+
+void
+cleanup_filter_pages_info(void)
+{
+	do_cleanup(&ft_head_discard);
+	do_cleanup(&ft_head_keep);
+}
erase_info.h +12 -0
index b363a40..6c60706 100644
@@ -20,6 +20,7 @@
 #define _ERASE_INFO_H
 
 #define MAX_SIZE_STR_LEN (26)
+#include <stdbool.h>
 
 /*
  * Erase information, original symbol expressions.
@@ -65,5 +66,16 @@ void filter_data_buffer_parallel(unsigned char *buf, unsigned long long paddr,
 unsigned long get_size_eraseinfo(void);
 int update_filter_info_raw(unsigned long long, int, int);
 
+bool update_filter_pages_info(unsigned long, unsigned long, bool);
+
+struct ft_page_info {
+	unsigned long pfn;
+	unsigned long num;
+	struct ft_page_info *next;
+} __attribute__((packed));
+
+bool filter_page(unsigned long, struct ft_page_info **p, bool handle_discard);
+void cleanup_filter_pages_info(void);
+
 #endif /* _ERASE_INFO_H */
 
makedumpfile.c +3 -0
index ca8ed8a..ebac8da 100644
@@ -102,6 +102,7 @@ mdf_pfn_t pfn_free;
 mdf_pfn_t pfn_hwpoison;
 mdf_pfn_t pfn_offline;
 mdf_pfn_t pfn_elf_excluded;
+mdf_pfn_t pfn_extension;
 
 mdf_pfn_t num_dumped;
 
@@ -6459,6 +6460,8 @@ __exclude_unnecessary_pages(unsigned long mem_map,
 	unsigned int order_offset, dtor_offset;
 	unsigned long flags, mapping, private = 0;
 	unsigned long compound_dtor, compound_head = 0;
+	struct ft_page_info *cur_discard = NULL;
+	struct ft_page_info *cur_keep = NULL;
 
 	/*
 	 * If a multi-page exclusion is pending, do it first

makedumpfile.c __exclude_unnecessary_pages(unsigned long mem_map, · line 6507

Quoted 10 lines
makedumpfile.c +7 -0
@@ -6495,6 +6498,13 @@ __exclude_unnecessary_pages(unsigned long mem_map,
 		if (info->flag_cyclic && !is_cyclic_region(pfn, cycle))
 			continue;
 
+		/*
+		 * Keep pages that specified by user via
+		 * makedumpfile extensions
+		 */
+		if (filter_page(pfn, &cur_keep, false))
+			continue;
+

It makes sense to allow plugins to enumerate a list of PFNs to override and include. I like that - it's simple enough. But it's not flexible enough for my use case with userspace stacks :(

The userspace stack region is an anon_vma. My plugin can enumerate the anon_vmas that it wants to save, but it's prohibitively expensive and complex to enumerate the list of pages associated with each anon_vma. We would need to do a page table walk for each process.

There's a simpler way: from the struct page mapping and index fields, it's possible to determine which anon_vma the page is associated with, and what index it has within the VMA. And from this, we can make the determination of whether to include a page or not. This is what I had implemented in this patch:

https://github.com/brenns10/makedumpfile/commit/1c0a828ef80962480f771915c2d494272721b659#diff-2593512d7ec329b34b1ca5686a7b6b073d0ca636df8ff20fea04684da2c8e063R6692-R12150

So, I wonder if it makes sense to allow a plugin to register a callback to be called here, so the plugin can make the more complex decision? This would keep the logic outside of the core makedumpfile code, but allow the necessary flexibility.

Something like:

if (plugin_keep_page_callback && plugin_keep_page_callback(pfn, pcache))

    continue;

And then the extension system could allow an extension to register that callback. It would need to keep the extension loaded for the duration of the execution of makedumpfile (rather than calling dlclose() immediately).

What do you think about this? I'm happy to implement this part of it separate from your patch series -- you could simply drop the stuff related to page inclusion, and I can add the necessary pieces when I submit my extension patches.

Thanks,
Stephen

Quoted 72 lines
makedumpfile.c +14 -4
 		/*
 		 * Exclude the memory hole.
 		 */
@@ -6687,6 +6697,14 @@ check_order:
 		else if (isOffline(flags, _mapcount)) {
 			pfn_counter = &pfn_offline;
 		}
+		/*
+		 * Exclude pages that specified by user via
+		 * makedumpfile extensions
+		 */
+		else if (filter_page(pfn, &cur_discard, true)) {
+			nr_pages = 1;
+			pfn_counter = &pfn_extension;
+		}
 		/*
 		 * Unexcludable page
 		 */
@@ -6748,6 +6766,7 @@ exclude_unnecessary_pages(struct cycle *cycle)
 		print_progress(PROGRESS_UNN_PAGES, info->num_mem_map, info->num_mem_map, NULL);
 		print_execution_time(PROGRESS_UNN_PAGES, &ts_start);
 	}
+	cleanup_filter_pages_info();
 
 	return TRUE;
 }
@@ -8234,7 +8253,7 @@ write_elf_pages_cyclic(struct cache_data *cd_header, struct cache_data *cd_page)
 	 */
 	if (info->flag_cyclic) {
 		pfn_zero = pfn_cache = pfn_cache_private = 0;
-		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = 0;
+		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = pfn_extension = 0;
 		pfn_memhole = info->max_mapnr;
 	}
 
@@ -9579,7 +9598,7 @@ write_kdump_pages_and_bitmap_cyclic(struct cache_data *cd_header, struct cache_d
 		 * Reset counter for debug message.
 		 */
 		pfn_zero = pfn_cache = pfn_cache_private = 0;
-		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = 0;
+		pfn_user = pfn_free = pfn_hwpoison = pfn_offline = pfn_extension = 0;
 		pfn_memhole = info->max_mapnr;
 
 		/*
@@ -10528,7 +10547,7 @@ print_report(void)
 	pfn_original = info->max_mapnr - pfn_memhole;
 
 	pfn_excluded = pfn_zero + pfn_cache + pfn_cache_private
-	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline;
+	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline + pfn_extension;
 
 	REPORT_MSG("\n");
 	REPORT_MSG("Original pages  : 0x%016llx\n", pfn_original);
@@ -10544,6 +10563,7 @@ print_report(void)
 	REPORT_MSG("    Free pages              : 0x%016llx\n", pfn_free);
 	REPORT_MSG("    Hwpoison pages          : 0x%016llx\n", pfn_hwpoison);
 	REPORT_MSG("    Offline pages           : 0x%016llx\n", pfn_offline);
+	REPORT_MSG("    Extension filter pages  : 0x%016llx\n", pfn_extension);
 	REPORT_MSG("  Remaining pages  : 0x%016llx\n",
 	    pfn_original - pfn_excluded);
 
@@ -10584,7 +10604,7 @@ print_mem_usage(void)
 	pfn_original = info->max_mapnr - pfn_memhole;
 
 	pfn_excluded = pfn_zero + pfn_cache + pfn_cache_private
-	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline;
+	    + pfn_user + pfn_free + pfn_hwpoison + pfn_offline + pfn_extension;
 	shrinking = (pfn_original - pfn_excluded) * 100;
 	shrinking = shrinking / pfn_original;
 	total_size = info->page_size * pfn_original;
Signature · 1 line
2.47.0
author
#17 lore raw

Re: [PATCH v3 6/8] Add page filtering function

Hi Stephen,

Quoted 247 lines · Stephen Brennan
Quoted 7 lines · Tao Liu

pfn and num is the data which extensions give to makedumpfile for mm page filtering. Since makedumpfile will iterate the pfn in an ascending order in __exclude_unnecessary_pages(), pfn and num are stored within ft_page_info linked lists and organized in an ascending order by pfn, so if one pfn
is hit by one list, the next pfn is most likely to be hit either by this list again, or the one follows, so a cur variable is used for saving the current list position to speedup the pfn checking process.

I'm wondering about the trade-off for using a linked list versus an array. Using the linked list, we are forced to maintain the sorted order as we construct the list, which is an O(N^2) insertion sort.

If instead we used an array, we could sort it with qsort() once, at the end. Then we could merge any overlapping ranges. Lookup could be implemented cheaply with bsearch(), and we could continue to use the optimization where we maintain a "cur" pointer. I believe the overall runtime complexity of the array approach would be O(N*log(N)) without requiring hand-implementing anything too complex, compared to O(N^2).

Depending on the number of pages (and how fragmented they are), this may or may not be an issue.

In my testing for userspace tasks, the number of pages retained can be on the order of ~100k. However -- my use case can't really use a list of PFNs, which I'll explain below. So my use case doesn't really matter too much here -- maybe your use case has relatively few page ranges, so the cost of O(N^2) is not bad.

So I guess I don't have a strong preference - but it's worth considering.

Quoted 45 lines

In addition, 2 ft_page_info linked list chains are used, one for mm page discarding and the other for page keeping.

---
 erase_info.c   | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++
 erase_info.h   | 12 +++++++
 makedumpfile.c | 28 ++++++++++++---
 3 files changed, 134 insertions(+), 4 deletions(-)
erase_info.c +27 -0
index b67d1d0..8838bea 100644
@@ -2466,3 +2466,101 @@ get_size_eraseinfo(void)
      return size_eraseinfo;
 }
 
+/* Pages to be discarded */
+static struct ft_page_info *ft_head_discard = NULL;
+/* Pages to be keeped */
+static struct ft_page_info *ft_head_keep = NULL;
+
+/*
+ * Insert the ft_page_info blocks into ft_head by ascending pfn.
+ */
+bool
+update_filter_pages_info(unsigned long pfn, unsigned long num, bool to_discard)
+{
+     struct ft_page_info *p, **ft_head;
+     struct ft_page_info *new_p = malloc(sizeof(struct ft_page_info));
+
+     ft_head = to_discard ? &ft_head_discard : &ft_head_keep;
+
+     if (!new_p) {
+             ERRMSG("Can't allocate memory for ft_page_info at %lx\n", pfn);
+             return false;
+     }
+     new_p->pfn = pfn;
+     new_p->num = num;
+     new_p->next = NULL;
+
+     if (!(*ft_head) || (*ft_head)->pfn > new_p->pfn) {
+             new_p->next = (*ft_head);
+             (*ft_head) = new_p;

erase_info.c get_size_eraseinfo(void) · line 2505

Quoted 10 lines
erase_info.c +10 -0
+             return true;
+     }
+
+     p = (*ft_head);
+     while (p->next != NULL && p->next->pfn < new_p->pfn) {
+             p = p->next;
+     }
+
+     new_p->next = p->next;
+     p->next = new_p;

It might be wise to defensively handle the case of overlapping PFN ranges by merging them.

Quoted 112 lines
erase_info.c +61 -0
+     return true;
+}
+
+/*
+ * Check if the pfn hit ft_page_info block.
+ *
+ * pfn and ft_head are in ascending order, so save the current ft_page_info
+ * block into **p because it is likely to hit again next time.
+ */
+bool
+filter_page(unsigned long pfn, struct ft_page_info **p, bool handle_discard)
+{
+     struct ft_page_info *ft_head;
+
+     ft_head = handle_discard ? ft_head_discard : ft_head_keep;
+
+     if (ft_head == NULL)
+             return false;
+
+     if (*p == NULL)
+             *p = ft_head;
+
+     /* The gap before 1st block */
+     if (pfn >= 0 && pfn < ft_head->pfn)
+             return false;
+
+     /* Handle 1~(n-1) blocks and following gaps */
+     while ((*p)->next) {
+             if (pfn >= (*p)->pfn && pfn < (*p)->pfn + (*p)->num)
+                     return true; // hit the block
+             if (pfn >= (*p)->pfn + (*p)->num && pfn < (*p)->next->pfn)
+                     return false; // the gap after the block
+             *p = (*p)->next;
+     }
+
+     /* The last block and gap */
+     if (pfn >= (*p)->pfn + (*p)->num)
+             return false;
+     else
+             return true;
+}
+
+static void
+do_cleanup(struct ft_page_info **ft_head)
+{
+     struct ft_page_info *p, *p_tmp;
+
+     for (p = *ft_head; p;) {
+             p_tmp = p;
+             p = p->next;
+             free(p_tmp);
+     }
+     *ft_head = NULL;
+}
+
+void
+cleanup_filter_pages_info(void)
+{
+     do_cleanup(&ft_head_discard);
+     do_cleanup(&ft_head_keep);
+}
erase_info.h +12 -0
index b363a40..6c60706 100644
@@ -20,6 +20,7 @@
 #define _ERASE_INFO_H
 
 #define MAX_SIZE_STR_LEN (26)
+#include <stdbool.h>
 
 /*
  * Erase information, original symbol expressions.
@@ -65,5 +66,16 @@ void filter_data_buffer_parallel(unsigned char *buf, unsigned long long paddr,
 unsigned long get_size_eraseinfo(void);
 int update_filter_info_raw(unsigned long long, int, int);
 
+bool update_filter_pages_info(unsigned long, unsigned long, bool);
+
+struct ft_page_info {
+     unsigned long pfn;
+     unsigned long num;
+     struct ft_page_info *next;
+} __attribute__((packed));
+
+bool filter_page(unsigned long, struct ft_page_info **p, bool handle_discard);
+void cleanup_filter_pages_info(void);
+
 #endif /* _ERASE_INFO_H */
 
makedumpfile.c +3 -0
index ca8ed8a..ebac8da 100644
@@ -102,6 +102,7 @@ mdf_pfn_t pfn_free;
 mdf_pfn_t pfn_hwpoison;
 mdf_pfn_t pfn_offline;
 mdf_pfn_t pfn_elf_excluded;
+mdf_pfn_t pfn_extension;
 
 mdf_pfn_t num_dumped;
 
@@ -6459,6 +6460,8 @@ __exclude_unnecessary_pages(unsigned long mem_map,
      unsigned int order_offset, dtor_offset;
      unsigned long flags, mapping, private = 0;
      unsigned long compound_dtor, compound_head = 0;
+     struct ft_page_info *cur_discard = NULL;
+     struct ft_page_info *cur_keep = NULL;
 
      /*
       * If a multi-page exclusion is pending, do it first

makedumpfile.c __exclude_unnecessary_pages(unsigned long mem_map, · line 6507

Quoted 10 lines
makedumpfile.c +7 -0
@@ -6495,6 +6498,13 @@ __exclude_unnecessary_pages(unsigned long mem_map,
              if (info->flag_cyclic && !is_cyclic_region(pfn, cycle))
                      continue;
 
+             /*
+              * Keep pages that specified by user via
+              * makedumpfile extensions
+              */
+             if (filter_page(pfn, &cur_keep, false))
+                     continue;
+

It makes sense to allow plugins to enumerate a list of PFNs to override and include. I like that - it's simple enough. But it's not flexible enough for my use case with userspace stacks :(

The userspace stack region is an anon_vma. My plugin can enumerate the anon_vmas that it wants to save, but it's prohibitively expensive and complex to enumerate the list of pages associated with each anon_vma. We would need to do a page table walk for each process.

There's a simpler way: from the struct page mapping and index fields, it's possible to determine which anon_vma the page is associated with, and what index it has within the VMA. And from this, we can make the determination of whether to include a page or not. This is what I had implemented in this patch:

https://github.com/brenns10/makedumpfile/commit/1c0a828ef80962480f771915c2d494272721b659#diff-2593512d7ec329b34b1ca5686a7b6b073d0ca636df8ff20fea04684da2c8e063R6692-R12150

So, I wonder if it makes sense to allow a plugin to register a callback to be called here, so the plugin can make the more complex decision? This would keep the logic outside of the core makedumpfile code, but allow the necessary flexibility.

Something like:

if (plugin_keep_page_callback && plugin_keep_page_callback(pfn, pcache))

    continue;

And then the extension system could allow an extension to register that callback. It would need to keep the extension loaded for the duration of the execution of makedumpfile (rather than calling dlclose() immediately).

Quoted 4 lines

What do you think about this? I'm happy to implement this part of it separate from your patch series -- you could simply drop the stuff related to page inclusion, and I can add the necessary pieces when I submit my extension patches.

Thanks a lot for your comments and suggestions! For my GPU mm filtering case, the pages are often allocated as a large block set, so one linked list like {discard: true, start_pfn: 100, pfn_nums: 100000} could hit a large range of pages' pfn, thus linked lists aren't much long and overlap/merge isn't a performance bottleneck to me. I agree with your case that this can be a problem for scattered pages.

Here is my plan, in v4, I will re-design the extensions part, that extensions will declare which kallsyms/btf symbol it is interested in, and remove the hash tables for kallsyms as you suggested (Many thanks for that, I think it is a brilliant idea to save the kallsyms memory!). And I will leave this linked list / array page (ex)inclusion function to you, so we can cooperate on this. Thanks again for your support and help :)

Thanks,
Tao Liu

Quoted 77 lines

Thanks,
Stephen

Quoted 72 lines
makedumpfile.c +14 -5
              /*
               * Exclude the memory hole.
               */
@@ -6687,6 +6697,14 @@ check_order:
              else if (isOffline(flags, _mapcount)) {
                      pfn_counter = &pfn_offline;
              }
+             /*
+              * Exclude pages that specified by user via
+              * makedumpfile extensions
+              */
+             else if (filter_page(pfn, &cur_discard, true)) {
+                     nr_pages = 1;
+                     pfn_counter = &pfn_extension;
+             }
              /*
               * Unexcludable page
               */
@@ -6748,6 +6766,7 @@ exclude_unnecessary_pages(struct cycle *cycle)
              print_progress(PROGRESS_UNN_PAGES, info->num_mem_map, info->num_mem_map, NULL);
              print_execution_time(PROGRESS_UNN_PAGES, &ts_start);
      }
+     cleanup_filter_pages_info();
 
      return TRUE;
 }
@@ -8234,7 +8253,7 @@ write_elf_pages_cyclic(struct cache_data *cd_header, struct cache_data *cd_page)
       */
      if (info->flag_cyclic) {
              pfn_zero = pfn_cache = pfn_cache_private = 0;
-             pfn_user = pfn_free = pfn_hwpoison = pfn_offline = 0;
+             pfn_user = pfn_free = pfn_hwpoison = pfn_offline = pfn_extension = 0;
              pfn_memhole = info->max_mapnr;
      }
 
@@ -9579,7 +9598,7 @@ write_kdump_pages_and_bitmap_cyclic(struct cache_data *cd_header, struct cache_d
               * Reset counter for debug message.
               */
              pfn_zero = pfn_cache = pfn_cache_private = 0;
-             pfn_user = pfn_free = pfn_hwpoison = pfn_offline = 0;
+             pfn_user = pfn_free = pfn_hwpoison = pfn_offline = pfn_extension = 0;
              pfn_memhole = info->max_mapnr;
 
              /*
@@ -10528,7 +10547,7 @@ print_report(void)
      pfn_original = info->max_mapnr - pfn_memhole;
 
      pfn_excluded = pfn_zero + pfn_cache + pfn_cache_private
-         + pfn_user + pfn_free + pfn_hwpoison + pfn_offline;
+         + pfn_user + pfn_free + pfn_hwpoison + pfn_offline + pfn_extension;
 
      REPORT_MSG("\n");
      REPORT_MSG("Original pages  : 0x%016llx\n", pfn_original);
@@ -10544,6 +10563,7 @@ print_report(void)
      REPORT_MSG("    Free pages              : 0x%016llx\n", pfn_free);
      REPORT_MSG("    Hwpoison pages          : 0x%016llx\n", pfn_hwpoison);
      REPORT_MSG("    Offline pages           : 0x%016llx\n", pfn_offline);
+     REPORT_MSG("    Extension filter pages  : 0x%016llx\n", pfn_extension);
      REPORT_MSG("  Remaining pages  : 0x%016llx\n",
          pfn_original - pfn_excluded);
 
@@ -10584,7 +10604,7 @@ print_mem_usage(void)
      pfn_original = info->max_mapnr - pfn_memhole;
 
      pfn_excluded = pfn_zero + pfn_cache + pfn_cache_private
-         + pfn_user + pfn_free + pfn_hwpoison + pfn_offline;
+         + pfn_user + pfn_free + pfn_hwpoison + pfn_offline + pfn_extension;
      shrinking = (pfn_original - pfn_excluded) * 100;
      shrinking = shrinking / pfn_original;
      total_size = info->page_size * pfn_original;
--

2.47.0

author
#18 lore raw

PATCH v3 7/8 Add maple tree support to makedumpfile extension

In reply to Tao Liu
---
 extensions/maple_tree.c | 336 ++++++++++++++++++++++++++++++++++++++++
 extensions/maple_tree.h |   6 +
 2 files changed, 342 insertions(+)
 create mode 100644 extensions/maple_tree.c
 create mode 100644 extensions/maple_tree.h
extensions/maple_tree.c new file +336 -0
new file mode 100644
index 0000000..0cc65bc
@@ -0,0 +1,336 @@
+#include <stdio.h>
+#include <stdbool.h>
+#include "../btf_info.h"
+#include "../kallsyms.h"
+#include "../makedumpfile.h"
+
+static unsigned char mt_slots[4] = {0};
+static unsigned char mt_pivots[4] = {0};
+static unsigned long mt_max[4] = {0};
+
+static int maple_tree_size;
+static int maple_node_size;
+static int maple_tree_ma_root;
+static int maple_tree_ma_flags;
+static int maple_node_parent;
+static int maple_node_ma64;
+static int maple_node_mr64;
+static int maple_node_slot;
+static int maple_arange_64_pivot;
+static int maple_arange_64_slot;
+static int maple_arange_64_gap;
+static int maple_arange_64_meta;
+static int maple_range_64_pivot;
+static int maple_range_64_slot;
+static int maple_metadata_end;
+static int maple_metadata_gap;
+
+#define MAPLE_BUFSIZE			512
+
+enum {
+	maple_dense_enum,
+	maple_leaf_64_enum,
+	maple_range_64_enum,
+	maple_arange_64_enum,
+};
+
+#define MAPLE_NODE_MASK			255UL
+#define MAPLE_NODE_TYPE_MASK		0x0F
+#define MAPLE_NODE_TYPE_SHIFT		0x03
+#define XA_ZERO_ENTRY			xa_mk_internal(257)
+
+static unsigned long xa_mk_internal(unsigned long v)
+{
+	return (v << 2) | 2;
+}
+
+static bool xa_is_internal(unsigned long entry)
+{
+	return (entry & 3) == 2;
+}
+
+static bool xa_is_node(unsigned long entry)
+{
+	return xa_is_internal(entry) && entry > 4096;
+}
+
+static bool xa_is_value(unsigned long entry)
+{
+	return entry & 1;
+}
+
+static bool xa_is_zero(unsigned long entry)
+{
+	return entry == XA_ZERO_ENTRY;
+}
+
+static unsigned long xa_to_internal(unsigned long entry)
+{
+	return entry >> 2;
+}
+
+static unsigned long xa_to_value(unsigned long entry)
+{
+	return entry >> 1;
+}
+
+static unsigned long mte_to_node(unsigned long entry)
+{
+        return entry & ~MAPLE_NODE_MASK;
+}
+
+static unsigned long mte_node_type(unsigned long maple_enode_entry)
+{
+	return (maple_enode_entry >> MAPLE_NODE_TYPE_SHIFT) &
+		MAPLE_NODE_TYPE_MASK;
+}
+
+static unsigned long mt_slot(void **slots, unsigned char offset)
+{
+       return (unsigned long)slots[offset];
+}
+
+static bool ma_is_leaf(unsigned long type)
+{
+	return type < maple_range_64_enum;
+}
+
+static bool mte_is_leaf(unsigned long maple_enode_entry)
+{
+       return ma_is_leaf(mte_node_type(maple_enode_entry));
+}
+
+static void mt_dump_entry(unsigned long entry, unsigned long min,
+			unsigned long max, unsigned int depth,
+			unsigned long **array_out, int *array_len,
+			int *array_cap)
+{
+	unsigned long *tmp;
+	int new_cap = 0;
+
+	if (entry == 0)
+		return;
+	if (*array_out == NULL) {
+		*array_len = 0;
+		new_cap = 4;
+	} else if (*array_len >= *array_cap) {
+		new_cap = *array_cap + (*array_cap >> 1);
+	}
+
+	if (new_cap) {
+		tmp = reallocarray(*array_out, new_cap, sizeof(unsigned long));
+		if (!tmp)
+			goto no_mem;
+		*array_out = tmp;
+		*array_cap = new_cap;
+	}
+
+	(*array_out)[(*array_len)++] = entry;
+	return;
+
+no_mem:
+	printf("%s: Not enough memory!\n", __func__);
+}
+
+static void mt_dump_node(unsigned long entry, unsigned long min,
+			unsigned long max, unsigned int depth,
+			unsigned long **array_out, int *array_len,
+			int *array_cap);
+
+static void mt_dump_range64(unsigned long entry, unsigned long min,
+			unsigned long max, unsigned int depth,
+			unsigned long **array_out, int *array_len,
+			int *array_cap)
+{
+	unsigned long maple_node_m_node = mte_to_node(entry);
+	char node_buf[MAPLE_BUFSIZE];
+	bool leaf = mte_is_leaf(entry);
+	unsigned long first = min, last;
+	int i;
+	char *mr64_buf;
+
+	readmem(VADDR, maple_node_m_node, node_buf, maple_node_size);
+	mr64_buf = node_buf + maple_node_mr64;
+
+	for (i = 0; i < mt_slots[maple_range_64_enum]; i++) {
+		last = max;
+
+		if (i < (mt_slots[maple_range_64_enum] - 1))
+			last = ULONG(mr64_buf + maple_range_64_pivot +
+				     sizeof(ulong) * i);
+
+		else if (!VOID_PTR(mr64_buf + maple_range_64_slot +
+			  sizeof(void *) * i) &&
+			 max != mt_max[mte_node_type(entry)])
+			break;
+		if (last == 0 && i > 0)
+			break;
+		if (leaf)
+			mt_dump_entry(mt_slot((void **)(mr64_buf +
+						      maple_range_64_slot), i),
+				first, last, depth + 1, array_out, array_len, array_cap);
+		else if (VOID_PTR(mr64_buf + maple_range_64_slot +
+				  sizeof(void *) * i)) {
+			mt_dump_node(mt_slot((void **)(mr64_buf +
+						     maple_range_64_slot), i),
+				first, last, depth + 1, array_out, array_len, array_cap);
+		}
+
+		if (last == max)
+			break;
+		if (last > max) {
+			printf("node %p last (%lu) > max (%lu) at pivot %d!\n",
+				mr64_buf, last, max, i);
+			break;
+		}
+		first = last + 1;
+	}
+}
+
+static void mt_dump_arange64(unsigned long entry, unsigned long min,
+			unsigned long max, unsigned int depth,
+			unsigned long **array_out, int *array_len,
+			int *array_cap)
+{
+	unsigned long maple_node_m_node = mte_to_node(entry);
+	char node_buf[MAPLE_BUFSIZE];
+	unsigned long first = min, last;
+	int i;
+	char *ma64_buf;
+
+	readmem(VADDR, maple_node_m_node, node_buf, maple_node_size);
+	ma64_buf = node_buf + maple_node_ma64;
+
+	for (i = 0; i < mt_slots[maple_arange_64_enum]; i++) {
+		last = max;
+
+		if (i < (mt_slots[maple_arange_64_enum] - 1))
+			last = ULONG(ma64_buf + maple_arange_64_pivot +
+				     sizeof(void *) * i);
+		else if (!VOID_PTR(ma64_buf + maple_arange_64_slot +
+				   sizeof(void *) * i))
+			break;
+		if (last == 0 && i > 0)
+			break;
+
+		if (ULONG(ma64_buf + maple_arange_64_slot + sizeof(void *) * i))
+			mt_dump_node(mt_slot((void **)(ma64_buf +
+						      maple_arange_64_slot), i),
+				first, last, depth + 1, array_out, array_len, array_cap);
+
+		if (last == max)
+			break;
+		if (last > max) {
+			printf("node %p last (%lu) > max (%lu) at pivot %d!\n",
+				ma64_buf, last, max, i);
+			break;
+		}
+		first = last + 1;
+	}
+}
+
+static void mt_dump_node(unsigned long entry, unsigned long min,
+			unsigned long max, unsigned int depth,
+			unsigned long **array_out, int *array_len,
+			int *array_cap)
+{
+	unsigned long maple_node = mte_to_node(entry);
+	unsigned long type = mte_node_type(entry);
+	int i;
+	char node_buf[MAPLE_BUFSIZE];
+
+	readmem(VADDR, maple_node, node_buf, maple_node_size);
+
+	switch (type) {
+	case maple_dense_enum:
+		for (i = 0; i < mt_slots[maple_dense_enum]; i++) {
+			if (min + i > max)
+				printf("OUT OF RANGE: ");
+			mt_dump_entry(mt_slot((void **)(node_buf + maple_node_slot), i),
+				min + i, min + i, depth, array_out, array_len, array_cap);
+		}
+		break;
+	case maple_leaf_64_enum:
+	case maple_range_64_enum:
+		mt_dump_range64(entry, min, max, depth, array_out, array_len, array_cap);
+		break;
+	case maple_arange_64_enum:
+		mt_dump_arange64(entry, min, max, depth, array_out, array_len, array_cap);
+		break;
+	default:
+		printf(" UNKNOWN TYPE\n");
+	}	
+}
+
+unsigned long *mt_dump(unsigned long mt, int *array_len)
+{
+	char tree_buf[MAPLE_BUFSIZE];
+	unsigned long entry;
+	unsigned long *array_out = NULL;
+	int array_cap = 0;
+	*array_len = 0;
+
+	readmem(VADDR, mt, tree_buf, maple_tree_size);
+	entry = ULONG(tree_buf + maple_tree_ma_root);
+
+	if (xa_is_node(entry))
+		mt_dump_node(entry, 0, mt_max[mte_node_type(entry)], 0,
+				&array_out, array_len, &array_cap);
+	else if (entry)
+		mt_dump_entry(entry, 0, 0, 0, &array_out, array_len, &array_cap);
+	else
+		printf("(empty)\n");
+
+	return array_out;
+}
+
+#define MAPLE_STRUCT_SIZE(S) \
+	({INIT_STRUCT(S); GET_STRUCT_SSIZE(S);})
+#define MAPLE_STRUCT_MEMBER_OFFSET(S, M) \
+	({INIT_STRUCT_MEMBER(S, M); GET_STRUCT_MEMBER_MOFF(S, M) / 8;})
+
+bool maple_init(void)
+{
+	unsigned long mt_slots_ptr;
+	unsigned long mt_pivots_ptr;
+	struct struct_member_info smi;
+
+	maple_tree_size = MAPLE_STRUCT_SIZE(maple_tree);
+	maple_node_size = MAPLE_STRUCT_SIZE(maple_node);
+	maple_tree_ma_root = MAPLE_STRUCT_MEMBER_OFFSET(maple_tree, ma_root);
+	maple_tree_ma_flags = MAPLE_STRUCT_MEMBER_OFFSET(maple_tree, ma_flags);
+	maple_node_parent = MAPLE_STRUCT_MEMBER_OFFSET(maple_node, parent);
+	maple_node_ma64 = MAPLE_STRUCT_MEMBER_OFFSET(maple_node, ma64);
+	maple_node_mr64 = MAPLE_STRUCT_MEMBER_OFFSET(maple_node, mr64);
+	maple_node_slot = MAPLE_STRUCT_MEMBER_OFFSET(maple_node, slot);
+	maple_arange_64_pivot = MAPLE_STRUCT_MEMBER_OFFSET(maple_arange_64, pivot);
+	maple_arange_64_slot = MAPLE_STRUCT_MEMBER_OFFSET(maple_arange_64, slot);
+	maple_arange_64_gap = MAPLE_STRUCT_MEMBER_OFFSET(maple_arange_64, gap);
+	maple_arange_64_meta = MAPLE_STRUCT_MEMBER_OFFSET(maple_arange_64, meta);
+	maple_range_64_pivot = MAPLE_STRUCT_MEMBER_OFFSET(maple_range_64, pivot);
+	maple_range_64_slot = MAPLE_STRUCT_MEMBER_OFFSET(maple_range_64, slot);
+	maple_metadata_end = MAPLE_STRUCT_MEMBER_OFFSET(maple_metadata, end);
+	maple_metadata_gap = MAPLE_STRUCT_MEMBER_OFFSET(maple_metadata, gap);
+
+	mt_slots_ptr = get_kallsyms_value_by_name("mt_slots");
+	mt_pivots_ptr = get_kallsyms_value_by_name("mt_pivots");
+	if (mt_slots_ptr == 0 || mt_pivots_ptr == 0) {
+		printf("Invalid mt_slots/mt_pivots address\n");
+		return false;
+	}
+	if (maple_tree_size > MAPLE_BUFSIZE ||
+	    maple_node_size > MAPLE_BUFSIZE) {
+		printf("MAPLE_BUFSIZE should be larger than maple_node/tree struct\n");
+		return false;
+	}
+
+	readmem(VADDR, mt_slots_ptr, mt_slots, sizeof(mt_slots));
+	readmem(VADDR, mt_pivots_ptr, mt_pivots, sizeof(mt_pivots));
+
+	mt_max[maple_dense_enum]           = mt_slots[maple_dense_enum];
+	mt_max[maple_leaf_64_enum]         = ULONG_MAX;
+	mt_max[maple_range_64_enum]        = ULONG_MAX;
+	mt_max[maple_arange_64_enum]       = ULONG_MAX;
+
+	return true;
+}
extensions/maple_tree.h new file +6 -0
new file mode 100644
index 0000000..c96624c
@@ -0,0 +1,6 @@
+#ifndef _MAPLE_TREE_H
+#define _MAPLE_TREE_H
+#include <stdbool.h>
+unsigned long *mt_dump(unsigned long mt, int *array_len);
+bool maple_init(void);
+#endif /* _MAPLE_TREE_H */
\ No newline at end of file
Signature · 1 line
2.47.0
author
#19 lore raw

PATCH v3 8/8 Filter amdgpu mm pages

In reply to Tao Liu

This patch will introduce amdgpu mm page filtering extension, those mm pages allocated to amdgpu will be discarded from vmcore, in order to shrink vmcore size since mm pages allocated to amdgpu are useless to kernel crash and may contain sensitive data.

---
 extensions/Makefile        |  4 +-
 extensions/amdgpu_filter.c | 90 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+), 2 deletions(-)
 create mode 100644 extensions/amdgpu_filter.c
extensions/Makefile +2 -2
index afbc61e..81dda31 100644
@@ -1,9 +1,9 @@
 CC ?= gcc
-CONTRIB_SO :=
+CONTRIB_SO := amdgpu_filter.so
 
 all: $(CONTRIB_SO)
 
-$(CONTRIB_SO): %.so: %.c
+$(CONTRIB_SO): %.so: %.c maple_tree.c
 	$(CC) -O2 -g -fPIC -shared -o $@ $^
 
 clean:
extensions/amdgpu_filter.c new file +90 -0

91 lines not on this page · raw message

Signature · 1 line
2.47.0
author
#20 lore raw
In reply to Tao Liu

Forget to mention, this patchset is for makedumpfile only.

Quoted 184 lines · Tao Liu

A) This patchset will introduce the following features to makedumpfile:

1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.

B) The purpose of the features are:

1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:

a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.

But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.

2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:

a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.

makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.

With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.

3) Compile and run makedumpfile extensions:

  $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on
  $ make extensions
  $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
    /tmp/extension.out
    Loaded extension: ./extensions/amdgpu_filter.so
    makedumpfile Completed.
        User time (seconds): 6.37
        System time (seconds): 0.70
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.10
        Maximum resident set size (kbytes): 38024
        ...

To contrast with eppic script of v2 [2]:

  $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
    /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c
    makedumpfile Completed.
        User time (seconds): 8.23
        System time (seconds): 0.88
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
        Maximum resident set size (kbytes): 57128
        ...

-rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
-rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
-rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore

C) Discussion:

1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. Others are not tested.

D) Testing:

If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:

Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@

Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"

[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df

v3 -> v2:

1) Removed btf/kallsyms support for eppic script, and introduced makedumpfile .so extension instead. The reason of removing eppic support is:
a) Native binary code as .so has better performance than scripting, see the time consumption contrast above.
b) Eppic library has memory leaks which hasn't been fixed totally, memeory leaks in 2nd kernel might be fatal.

2) Removed the code of manually parsing btf info, and used libbpf for btf info parsing instead. The reason of removing manually parsing is:
a) Less code modification to makedumpfile, easier to maintain.
b) The performance of using libbpf is as good as manual parsing + hash table indexing, as well as less memory consumption, see time and memory consumption contrast above.

3) The patches are organized as follows:

--- <only for test purpose, don't merge> --- 8.Filter amdgpu mm pages
7.Add maple tree support to makedumpfile extension

--- <code should be merged> ---
6.Add page filtering function
5.Add makedumpfile extension support 4.Implement kernel modules' btf resolving 3.Implement kernel modules' kallsyms resolving 2.Implement kernel btf resolving 1.Implement kernel kallsyms resolving

Patch 7 & 8 are customization specific, which can be maintained separately. Patch 1 ~ 6 are common code which should be integrate with makedumpfile.

Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/

Tao Liu (8):
Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel modules' kallsyms resolving Implement kernel modules' btf resolving Add makedumpfile extension support
Add page filtering function
Add maple tree support to makedumpfile extension Filter amdgpu mm pages

 Makefile                   |   9 +-
 btf_info.c                 | 260 +++++++++++++++++++++++++
 btf_info.h                 |  66 +++++++
 erase_info.c               |  98 ++++++++++
 erase_info.h               |  12 ++
 extension.c                |  82 ++++++++
 extensions/Makefile        |  10 +
 extensions/amdgpu_filter.c |  90 +++++++++
 extensions/maple_tree.c    | 336 +++++++++++++++++++++++++++++++++
 extensions/maple_tree.h    |   6 +
 kallsyms.c                 | 376 +++++++++++++++++++++++++++++++++++++
 kallsyms.h                 |  20 ++
 makedumpfile.c             |  35 +++-
 makedumpfile.h             |  11 ++
 14 files changed, 1405 insertions(+), 6 deletions(-)
 create mode 100644 btf_info.c
 create mode 100644 btf_info.h
 create mode 100644 extension.c
 create mode 100644 extensions/Makefile
 create mode 100644 extensions/amdgpu_filter.c
 create mode 100644 extensions/maple_tree.c
 create mode 100644 extensions/maple_tree.h
 create mode 100644 kallsyms.c
 create mode 100644 kallsyms.h

--
2.47.0

#21 lore raw
In reply to Tao Liu
Quoted 170 lines · Tao Liu

A) This patchset will introduce the following features to makedumpfile:

1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.

B) The purpose of the features are:

1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:

a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.

But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.

2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:

a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.

makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.

With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.

   3) Compile and run makedumpfile extensions:

   $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on
   $ make extensions
   
   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/extension.out
     Loaded extension: ./extensions/amdgpu_filter.so
     makedumpfile Completed.
         User time (seconds): 6.37
         System time (seconds): 0.70
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.10
         Maximum resident set size (kbytes): 38024
         ...
  
      To contrast with eppic script of v2 [2]:

   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c
     makedumpfile Completed.
         User time (seconds): 8.23
         System time (seconds): 0.88
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
         Maximum resident set size (kbytes): 57128
         ...

   -rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
   -rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
   -rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore

C) Discussion:

1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. Others are not tested.

D) Testing:

If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:

Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@

Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"

[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df

v3 -> v2:

1) Removed btf/kallsyms support for eppic script, and introduced makedumpfile .so extension instead. The reason of removing eppic support is:
a) Native binary code as .so has better performance than scripting, see the time consumption contrast above.
b) Eppic library has memory leaks which hasn't been fixed totally, memeory leaks in 2nd kernel might be fatal.

2) Removed the code of manually parsing btf info, and used libbpf for btf info parsing instead. The reason of removing manually parsing is:
a) Less code modification to makedumpfile, easier to maintain.
b) The performance of using libbpf is as good as manual parsing + hash table indexing, as well as less memory consumption, see time and memory consumption contrast above.

3) The patches are organized as follows:

--- <only for test purpose, don't merge> --- 8.Filter amdgpu mm pages
7.Add maple tree support to makedumpfile extension

--- <code should be merged> ---
6.Add page filtering function
5.Add makedumpfile extension support 4.Implement kernel modules' btf resolving 3.Implement kernel modules' kallsyms resolving 2.Implement kernel btf resolving 1.Implement kernel kallsyms resolving

Patch 7 & 8 are customization specific, which can be maintained separately. Patch 1 ~ 6 are common code which should be integrate with makedumpfile.

Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/

Tao Liu (8):
Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel modules' kallsyms resolving Implement kernel modules' btf resolving Add makedumpfile extension support
Add page filtering function
Add maple tree support to makedumpfile extension Filter amdgpu mm pages

  Makefile                   |   9 +-
  btf_info.c                 | 260 +++++++++++++++++++++++++
  btf_info.h                 |  66 +++++++
  erase_info.c               |  98 ++++++++++
  erase_info.h               |  12 ++
  extension.c                |  82 ++++++++
  extensions/Makefile        |  10 +
  extensions/amdgpu_filter.c |  90 +++++++++
  extensions/maple_tree.c    | 336 +++++++++++++++++++++++++++++++++
  extensions/maple_tree.h    |   6 +
  kallsyms.c                 | 376 +++++++++++++++++++++++++++++++++++++
  kallsyms.h                 |  20 ++
  makedumpfile.c             |  35 +++-
  makedumpfile.h             |  11 ++
  14 files changed, 1405 insertions(+), 6 deletions(-)
Quoted 9 lines

create mode 100644 btf_info.c
create mode 100644 btf_info.h
create mode 100644 extension.c
create mode 100644 extensions/Makefile
create mode 100644 extensions/amdgpu_filter.c
create mode 100644 extensions/maple_tree.c
create mode 100644 extensions/maple_tree.h
create mode 100644 kallsyms.c
create mode 100644 kallsyms.h

Thank you for your the v3 patch. And thanks for the appropriate improvements.

I believe the results will be significantly improved by stopping use of eppic and using libbpf instead.

My concerns are as follows:

* Performance has improved compared to the version using eppic, but I would like to see a comparison of performance without these patches applied.

* This mechanism is enabled when .so (shared libraries) are present, but I think it would be better to have an option to turn this feature on or off. If you don't use shared libraries, you have to delete or move them,
 which can be inconvenient.

I'm looking forward to v4, which will improve things like calculation Order.

Masa

author
#22 lore raw

Hi YAMAZAKI,

Sorry for the late reply.

Quoted 185 lines · YAMAZAKI MASAMITSU(山崎 真光)
Quoted 170 lines · Tao Liu

A) This patchset will introduce the following features to makedumpfile:

1) Add .so extension support to makedumpfile
2) Enable btf and kallsyms for symbol type and address resolving.

B) The purpose of the features are:

1) Currently makedumpfile filters mm pages based on page flags, because flags can help to determine one page's usage. But this page-flag-checking method lacks of flexibility in certain cases, e.g. if we want to filter those mm pages occupied by GPU during vmcore dumping due to:

a) GPU may be taking a large memory and contains sensitive data;
b) GPU mm pages have no relations to kernel crash and useless for vmcore analysis.

But there is no GPU mm page specific flags, and apparently we don't need to create one just for kdump use. A programmable filtering tool is more suitable for such cases. In addition, different GPU vendors may use different ways for mm pages allocating, programmable filtering is better than hard coding these GPU specific logics into makedumpfile in this case.

2) Currently makedumpfile already contains a programmable filtering tool, aka eppic script, which allows user to write customized code for data erasing. However it has the following drawbacks:

a) cannot do mm page filtering.
b) need to access to debuginfo of both kernel and modules, which is not applicable in the 2nd kernel.
c) eppic library has memory leaks which are not all resolved [1]. This is not acceptable in 2nd kernel.

makedumpfile need to resolve the dwarf data from debuginfo, to get symbols types and addresses. In recent kernel there are dwarf alternatives such as btf/kallsyms which can be used for this purpose. And btf/kallsyms info are already packed within vmcore, so we can use it directly.

With these, this patchset introduces makedumpfile extensions, which is based on btf/kallsyms symbol resolving, and is programmable for mm page filtering. The following section shows its usage and performance, please note the tests are performed in 1st kernel.

   3) Compile and run makedumpfile extensions:

   $ make LINKTYPE=dynamic USELZO=on USESNAPPY=on USEZSTD=on
   $ make extensions

   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/extension.out
     Loaded extension: ./extensions/amdgpu_filter.so
     makedumpfile Completed.
         User time (seconds): 6.37
         System time (seconds): 0.70
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.10
         Maximum resident set size (kbytes): 38024
         ...

      To contrast with eppic script of v2 [2]:

   $ /usr/bin/time -v ./makedumpfile -d 31 -l /var/crash/127.0.0.1-2025-06-10-18\:03\:12/vmcore
     /tmp/eppic.out --eppic eppic_scripts/filter_amdgpu_mm_pages.c
     makedumpfile Completed.
         User time (seconds): 8.23
         System time (seconds): 0.88
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.16
         Maximum resident set size (kbytes): 57128
         ...

   -rw------- 1 root root 367475074 Jan 19 19:01 /tmp/extension.out
   -rw------- 1 root root 367475074 Jan 19 19:48 /tmp/eppic.out
   -rw------- 1 root root 387181418 Jun 10 18:03 /var/crash/127.0.0.1-2025-06-10-18:03:12/vmcore

C) Discussion:

1) GPU types: Currently only tested with amdgpu's mm page filtering, others are not tested.
2) OS: The code can work on rhel-10+/rhel9.5+ on x86_64/arm64/s390/ppc64. Others are not tested.

D) Testing:

If you don't want to create your vmcore, you can find a vmcore which I created with amdgpu mm pages unfiltered [3], the amdgpu mm pages are allocated by program [4]. You can use the vmcore in 1st kernel to filter the amdgpu mm pages by the previous performance testing cmdline. To verify the pages are filtered in crash:

Unfiltered:
crash> search -c "!QAZXSW@#EDC"
ffff96b7fa800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ffff96b87c800000: !QAZXSW@#EDCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX crash> rd ffff96b7fa800000
ffff96b7fa800000: 405753585a415121 !QAZXSW@
crash> rd ffff96b87c800000
ffff96b87c800000: 405753585a415121 !QAZXSW@

Filtered:
crash> search -c "!QAZXSW@#EDC"
crash> rd ffff96b7fa800000
rd: page excluded: kernel virtual address: ffff96b7fa800000 type: "64-bit KVADDR" crash> rd ffff96b87c800000
rd: page excluded: kernel virtual address: ffff96b87c800000 type: "64-bit KVADDR"

[1]: https://github.com/lucchouina/eppic/pull/32
[2]: https://lore.kernel.org/kexec/[email protected]/
[3]: https://people.redhat.com/~ltao/core/vmcore
[4]: https://gist.github.com/liutgnu/a8cbce1c666452f1530e1410d1f352df

v3 -> v2:

1) Removed btf/kallsyms support for eppic script, and introduced makedumpfile .so extension instead. The reason of removing eppic support is:
a) Native binary code as .so has better performance than scripting, see the time consumption contrast above.
b) Eppic library has memory leaks which hasn't been fixed totally, memeory leaks in 2nd kernel might be fatal.

2) Removed the code of manually parsing btf info, and used libbpf for btf info parsing instead. The reason of removing manually parsing is:
a) Less code modification to makedumpfile, easier to maintain.
b) The performance of using libbpf is as good as manual parsing + hash table indexing, as well as less memory consumption, see time and memory consumption contrast above.

3) The patches are organized as follows:

--- <only for test purpose, don't merge> --- 8.Filter amdgpu mm pages
7.Add maple tree support to makedumpfile extension

--- <code should be merged> ---
6.Add page filtering function
5.Add makedumpfile extension support 4.Implement kernel modules' btf resolving 3.Implement kernel modules' kallsyms resolving 2.Implement kernel btf resolving 1.Implement kernel kallsyms resolving

Patch 7 & 8 are customization specific, which can be maintained separately. Patch 1 ~ 6 are common code which should be integrate with makedumpfile.

Link to v2: https://lore.kernel.org/kexec/[email protected]/
Link to v1: https://lore.kernel.org/kexec/[email protected]/

Tao Liu (8):
Implement kernel kallsyms resolving Implement kernel btf resolving
Implement kernel modules' kallsyms resolving Implement kernel modules' btf resolving Add makedumpfile extension support
Add page filtering function
Add maple tree support to makedumpfile extension Filter amdgpu mm pages

  Makefile                   |   9 +-
  btf_info.c                 | 260 +++++++++++++++++++++++++
  btf_info.h                 |  66 +++++++
  erase_info.c               |  98 ++++++++++
  erase_info.h               |  12 ++
  extension.c                |  82 ++++++++
  extensions/Makefile        |  10 +
  extensions/amdgpu_filter.c |  90 +++++++++
  extensions/maple_tree.c    | 336 +++++++++++++++++++++++++++++++++
  extensions/maple_tree.h    |   6 +
  kallsyms.c                 | 376 +++++++++++++++++++++++++++++++++++++
  kallsyms.h                 |  20 ++
  makedumpfile.c             |  35 +++-
  makedumpfile.h             |  11 ++
  14 files changed, 1405 insertions(+), 6 deletions(-)
Quoted 9 lines

create mode 100644 btf_info.c
create mode 100644 btf_info.h
create mode 100644 extension.c
create mode 100644 extensions/Makefile
create mode 100644 extensions/amdgpu_filter.c
create mode 100644 extensions/maple_tree.c
create mode 100644 extensions/maple_tree.h
create mode 100644 kallsyms.c
create mode 100644 kallsyms.h

Thank you for your the v3 patch. And thanks for the appropriate improvements.

Quoted 8 lines

I believe the results will be significantly improved by stopping use of eppic and using libbpf instead.

My concerns are as follows:

* Performance has improved compared to the version using eppic, but I would like to see a comparison of performance without these patches applied.

Sure, I will attach the performance comparison for v4. Currently I'm cooperating with Stephen on a drafted v4. Once we are OK with it, I will post the final v4 to upstream later.

Quoted 6 lines

* This mechanism is enabled when .so (shared libraries) are present, but I think it would be better to have an option to turn this feature on or off. If you don't use shared libraries, you have to delete or move them,
which can be inconvenient.

I guess your meaning is, when no .so extensions are needed, then btf/kallsyms code won't get executed. Currently I'm introducing a "--extension" option to makedumpfile. Only when specified, the btf/kallsyms of makedumpfile will be initialized.

Quoted 2 lines

I'm looking forward to v4, which will improve things like calculation Order.

Sure, no problem. Thanks again for your suggestions!

Thanks,
Tao Liu

Quoted 2 lines

Masa