Skip to contentSkip to conversation

bpftool: Make the LLVM disassembler an optional runtime dependency

PATCH Superseded Mark Korenberg 2 participants 4 messages Oct 29

Subsystems (2)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover bpftool: Make the LLVM disassembler an optional runtime dependency

libLLVM (~140 MiB on Fedora) is a hard dependency of bpftool, although it is only used by "bpftool prog dump jited". This series moves the LLVM disassembler to a plugin that bpftool loads on demand, so that distributions can make it optional. Patch 1 fixes a leak found on the way.

Changes in v2:
- Move the LLVMDisasmDispose() fix to a separate patch (Quentin)
- Look for the plugin next to the bpftool binary when it is not installed, so that bpftool works from its build directory (Quentin, Sashiko). Not in the current directory: bpftool usually runs as root.
- Export a single versioned structure of callbacks from the plugin (Quentin)
- Only report "llvm" in "bpftool version" if the plugin loads (Quentin)
- Handle dlerror() returning NULL (Quentin)
- Rebuild jit_disasm.o if libdir changes between "make" and
"make install" (Quentin)
- uninstall: use QUIET_UNINST, remove $(libdir)/bpftool (Quentin)
- With a static libLLVM, which a static bpftool requires, link the disassembler into bpftool as before (Sashiko). This drops
LLVM_LINK_STATIC: static LLVM archives are never linked into a shared object.
- Commit log and comment fixes (Quentin, Sashiko)
- Rebase on bpf-next
- Link to v1: https://lore.kernel.org/r/[email protected]

Tested on Fedora 44 x86_64, LLVM 22: plugin, built-in and SKIP_LLVM builds; plugin lookup (installed, build directory, missing, wrong ABI); install/uninstall; "prog dump jited" (plain and JSON) on live programs, and disassembly of aarch64 code through the plugin. Not tested: a fully static build (no static libraries available).

The v2 changes were written with the help of an AI assistant (Claude), from the review comments; I reviewed and tested them.

---
Mark Korenberg (2):
bpftool: Fix disposal of the LLVM disassembler context
bpftool: Make the LLVM disassembler an optional runtime dependency

 tools/bpf/bpftool/Makefile      |  38 +++++++++-
 tools/bpf/bpftool/jit_disasm.c  | 161 ++++++++++++++++++++++++++++------------
 tools/bpf/bpftool/llvm_disasm.c |  84 +++++++++++++++++++++
 tools/bpf/bpftool/llvm_disasm.h |  41 ++++++++++
 tools/bpf/bpftool/main.c        |   3 +-
 tools/bpf/bpftool/main.h        |   3 +
 6 files changed, 279 insertions(+), 51 deletions(-)

---
base-commit: d80373a8e70d3193151232c3062abbbe367ff5ad
change-id: 20260603-bpftool-plugin-c994bc3e0643

Best regards,

Signature · 1 line
Mark Korenberg <[email protected]>
#1 lore raw

PATCH v2 1/2 bpftool: Fix disposal of the LLVM disassembler context

destroy_context() frees the disassembler context with LLVMDisposeMessage(), which is a plain free() meant for strings returned by LLVM. The context is therefore never destroyed, leaking about 28 KiB per disassembled function. Use LLVMDisasmDispose() instead.

Fixes eb9d1acf634b ("bpftool: Add LLVM as default library for disassembling JIT-ed programs")
---
 tools/bpf/bpftool/jit_disasm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/bpf/bpftool/jit_disasm.c +1 -1
index 04541155e9cc..5ff144ce84a1 100644
@@ -115,7 +115,7 @@ init_context(disasm_ctx_t *ctx, const char *arch,
 
 static void destroy_context(disasm_ctx_t *ctx)
 {
-	LLVMDisposeMessage(*ctx);
+	LLVMDisasmDispose(*ctx);
 }
 
 static int
Signature · 1 line
2.56.0
#2 lore raw

PATCH v2 2/2 bpftool: Make the LLVM disassembler an optional runtime dependency

In reply to Mark Korenberg

libLLVM is only needed for "bpftool prog dump jited", yet it is a hard dependency of bpftool: on Fedora 43, installing the ~730 KiB bpftool pulls in ~140 MiB of llvm20-libs.

Move the LLVM disassembler to a plugin, $(libdir)/bpftool/bpftool-llvm.so, that bpftool loads with dlopen() only when disassembling JITed programs. Distributions can then ship it as an optional package.

The plugin exports a single structure of callbacks, with an ABI version checked by bpftool. bpftool looks for the plugin in its install directory, then next to its own binary, but not in the current directory since it usually runs as root. "bpftool version" reports "llvm" only if the plugin can be loaded.

If libLLVM is static, as required for a static bpftool, the disassembler is linked into bpftool as before.

---
 tools/bpf/bpftool/Makefile      |  38 +++++++++-
 tools/bpf/bpftool/jit_disasm.c  | 161 ++++++++++++++++++++++++++++------------
 tools/bpf/bpftool/llvm_disasm.c |  84 +++++++++++++++++++++
 tools/bpf/bpftool/llvm_disasm.h |  41 ++++++++++
 tools/bpf/bpftool/main.c        |   3 +-
 tools/bpf/bpftool/main.h        |   3 +
 6 files changed, 279 insertions(+), 51 deletions(-)
tools/bpf/bpftool/Makefile +35 -3
index b0f7168e7943..dc8e4be86296 100644
@@ -63,6 +63,7 @@ $(LIBBPF_BOOTSTRAP)-clean: FORCE | $(LIBBPF_BOOTSTRAP_OUTPUT)
 	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) clean >/dev/null
 
 prefix ?= /usr/local
+libdir ?= $(prefix)/lib
 bash_compdir ?= /usr/share/bash-completion/completions
 
 CFLAGS += -O2
@@ -170,7 +171,7 @@ include $(wildcard $(OUTPUT)*.d)
 
 all: $(OUTPUT)bpftool
 
-SRCS := $(wildcard *.c)
+SRCS := $(filter-out llvm_disasm.c,$(wildcard *.c))
 
 ifeq ($(feature-llvm),1)
 ifneq ($(SKIP_LLVM),1)
@@ -186,10 +187,19 @@ ifeq ($(HAS_LLVM),1)
   # Makefile includes lib.mk which sets -D_GNU_SOURCE) which would cause
   # compilation error due to redefinition. Let's filter it out here.
   CFLAGS  += $(filter-out -D_GNU_SOURCE,$(shell $(LLVM_CONFIG) --cflags))
-  LIBS    += $(shell $(LLVM_CONFIG) --libs $(LLVM_CONFIG_LIB_COMPONENTS))
+  LLVM_LIBS := $(shell $(LLVM_CONFIG) --libs $(LLVM_CONFIG_LIB_COMPONENTS))
   ifeq ($(shell $(LLVM_CONFIG) --shared-mode),static)
+    # No libLLVM.so dependency to avoid, link the disassembler into bpftool
+    SRCS += llvm_disasm.c
+    LIBS += $(LLVM_LIBS)
     LIBS += $(shell $(LLVM_CONFIG) --system-libs $(LLVM_CONFIG_LIB_COMPONENTS))
     LIBS += -lstdc++
+  else
+    # Build the disassembler as a plugin, the only object linked against
+    # libLLVM, that bpftool loads only to disassemble JITed programs
+    LLVM_PLUGIN := $(OUTPUT)bpftool-llvm.so
+    CFLAGS += -DLLVM_PLUGIN_DIR='"$(libdir)/bpftool"'
+    LIBS += -ldl
   endif
   LDFLAGS += $(shell $(LLVM_CONFIG) --ldflags)
 else
@@ -290,6 +300,21 @@ $(BPFTOOL_BOOTSTRAP): $(BOOTSTRAP_OBJS) $(LIBBPF_BOOTSTRAP)
 $(OUTPUT)bpftool: $(OBJS) $(LIBBPF)
 	$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) $(OBJS) $(LIBS) -o $@
 
+ifneq ($(LLVM_PLUGIN),)
+all: $(LLVM_PLUGIN)
+
+$(LLVM_PLUGIN): CFLAGS += -fPIC
+$(LLVM_PLUGIN): $(OUTPUT)llvm_disasm.o
+	$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -shared $< $(LLVM_LIBS) -o $@
+
+# The plugin directory, $(libdir)/bpftool, is compiled into jit_disasm.o.
+# Recompile it if libdir changes, e.g. on "make" then "make install libdir=...".
+$(OUTPUT)llvm-plugin-dir: FORCE
+	$(Q)echo '$(libdir)/bpftool' | cmp -s - $@ || echo '$(libdir)/bpftool' > $@
+
+$(OUTPUT)jit_disasm.o: $(OUTPUT)llvm-plugin-dir
+endif
+
 $(BOOTSTRAP_OUTPUT)%.o: %.c $(LIBBPF_BOOTSTRAP_INTERNAL_HDRS) | $(BOOTSTRAP_OUTPUT)
 	$(QUIET_CC)$(HOSTCC) $(HOST_CFLAGS) -c -MMD $< -o $@
 
@@ -303,16 +328,21 @@ feature-detect-clean:
 clean: $(LIBBPF)-clean $(LIBBPF_BOOTSTRAP)-clean feature-detect-clean
 	$(call QUIET_CLEAN, bpftool)
 	$(Q)$(RM) -- $(OUTPUT)bpftool $(OUTPUT)*.o $(OUTPUT)*.d
+	$(Q)$(RM) -- $(OUTPUT)bpftool-llvm.so $(OUTPUT)llvm-plugin-dir
 	$(Q)$(RM) -- $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
 	$(Q)$(RM) -r -- $(LIBBPF_OUTPUT) $(BOOTSTRAP_OUTPUT)
 	$(call QUIET_CLEAN, core-gen)
 	$(Q)$(RM) -- $(OUTPUT)FEATURE-DUMP.bpftool
 	$(Q)$(RM) -r -- $(OUTPUT)feature/
 
-install-bin: $(OUTPUT)bpftool
+install-bin: $(OUTPUT)bpftool $(LLVM_PLUGIN)
 	$(call QUIET_INSTALL, bpftool)
 	$(Q)$(INSTALL) -m 0755 -d $(DESTDIR)$(prefix)/sbin
 	$(Q)$(INSTALL) $(OUTPUT)bpftool $(DESTDIR)$(prefix)/sbin/bpftool
+ifneq ($(LLVM_PLUGIN),)
+	$(call QUIET_INSTALL, bpftool-llvm.so)
+	$(Q)$(INSTALL) -m 0755 -D -t $(DESTDIR)$(libdir)/bpftool $(LLVM_PLUGIN)
+endif
 
 install: install-bin
 	$(Q)$(INSTALL) -m 0755 -d $(DESTDIR)$(bash_compdir)
@@ -321,6 +351,8 @@ install: install-bin
 uninstall:
 	$(call QUIET_UNINST, bpftool)
 	$(Q)$(RM) -- $(DESTDIR)$(prefix)/sbin/bpftool
+	$(call QUIET_UNINST, bpftool-llvm.so)
+	$(Q)$(RM) -r -- $(DESTDIR)$(libdir)/bpftool
 	$(Q)$(RM) -- $(DESTDIR)$(bash_compdir)/bpftool
 
 doc:
tools/bpf/bpftool/jit_disasm.c +114 -47

205 lines not on this page · raw message

tools/bpf/bpftool/llvm_disasm.c new file +84 -0

84 lines not on this page · raw message

tools/bpf/bpftool/llvm_disasm.h new file +41 -0

41 lines not on this page · raw message

tools/bpf/bpftool/main.c +2 -1
index d15fec71b64e..321e928d651c 100644
@@ -124,7 +124,8 @@ static int do_version(int argc, char **argv)
 	const bool has_libbfd = false;
 #endif
 #ifdef HAVE_LLVM_SUPPORT
-	const bool has_llvm = true;
+	/* The LLVM disassembler may be a plugin that is not installed */
+	const bool has_llvm = disasm_llvm_available();
 #else
 	const bool has_llvm = false;
 #endif
tools/bpf/bpftool/main.h +3 -0
index b1eb4b5d991b..96d7b147aa9d 100644
@@ -214,6 +214,9 @@ static inline int disasm_init(void)
 	return -1;
 }
 #endif
+#ifdef HAVE_LLVM_SUPPORT
+bool disasm_llvm_available(void);
+#endif
 void print_data_json(uint8_t *data, size_t len);
 void print_hex_data_json(uint8_t *data, size_t len);
 
Signature · 1 line
2.56.0
#3 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] bpftool: llvm_plugin_find() misses staged plugins and prefers system plugin --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpftool: llvm_plugin_find() misses staged plugins and prefers system plugin --

commit aa662f293876360fc9cf9139e86022f88a5bbb64
Author: Mark Korenberg <[email protected]>

bpftool: Make the LLVM disassembler an optional runtime dependency

This commit moves the libLLVM-based JIT disassembler into an optional shared library plugin (bpftool-llvm.so) loaded on demand via dlopen() when disassembling JITed programs or checking feature availability.

Quoted 4 lines
tools/bpf/bpftool/jit_disasm.c +0 -0
index 5ff144ce84a1c..2691252a115b9 100644

[ ... ]

Quoted 1 line
tools/bpf/bpftool/jit_disasm.c +0 -0
@@ -45,7 +45,105 @@ static int oper_count;

[ ... ]

Quoted 21 lines
tools/bpf/bpftool/jit_disasm.c +21 -0
+/*
+ * Look for the plugin in its install directory, then next to the bpftool
+ * binary so that bpftool also works from its build directory. Do not look
+ * in the current directory: bpftool usually runs as root, and would then
+ * load any library planted in the directory it is run from.
+ */
+static int llvm_plugin_find(char *path, size_t size)
+{
+	char exe[PATH_MAX];
+	ssize_t len;
+	char *dir;
+	int n;
+
+	snprintf(path, size, "%s/%s", LLVM_PLUGIN_DIR, LLVM_PLUGIN_NAME);
+	if (!access(path, F_OK))
+		return 0;
+
+	len = readlink("/proc/self/exe", exe, sizeof(exe) - 1);
+	if (len < 0)
+		return -1;
+	exe[len] = '\0';

tools/bpf/bpftool/jit_disasm.c static int oper_count; · line 93

Quoted 10 lines
tools/bpf/bpftool/jit_disasm.c +10 -0
+	dir = strrchr(exe, '/');
+	if (!dir)
+		return -1;
+	*dir = '\0';
+
+	n = snprintf(path, size, "%s/%s", exe, LLVM_PLUGIN_NAME);
+	if (n < 0 || (size_t)n >= size)
+		return -1;
+	return access(path, F_OK);
+}

[Severity: Medium]
Can llvm_plugin_find() fail to locate bpftool-llvm.so when bpftool is installed into a staging directory via DESTDIR, such as in the BPF selftests build?

In tools/testing/selftests/bpf/Makefile, $(DEFAULT_BPFTOOL) and $(CROSS_BPFTOOL) are built and staged with:

tools/testing/selftests/bpf/Makefile:

	$(Q)$(MAKE) $(submake_extras)  -C $(BPFTOOLDIR)			       \
		    ...
		    OUTPUT=$(HOST_BUILD_DIR)/bpftool/			       \
		    ...
		    prefix= DESTDIR=$(HOST_SCRATCH_DIR)/ install-bin

Because prefix= is empty and libdir is not overridden, tools/bpf/bpftool/Makefile sets libdir to /lib and compiles jit_disasm.o with:

tools/bpf/bpftool/Makefile:

	prefix ?= /usr/local
	libdir ?= $(prefix)/lib
	...
	CFLAGS += -DLLVM_PLUGIN_DIR='"$(libdir)/bpftool"'

Meanwhile, the install-bin target places the binary and plugin in separate directories under $(DESTDIR):

tools/bpf/bpftool/Makefile:

	install-bin: $(OUTPUT)bpftool $(LLVM_PLUGIN)
		$(call QUIET_INSTALL, bpftool)
		$(Q)$(INSTALL) -m 0755 -d $(DESTDIR)$(prefix)/sbin
		$(Q)$(INSTALL) $(OUTPUT)bpftool $(DESTDIR)$(prefix)/sbin/bpftool
	ifneq ($(LLVM_PLUGIN),)
		$(call QUIET_INSTALL, bpftool-llvm.so)
		$(Q)$(INSTALL) -m 0755 -D -t $(DESTDIR)$(libdir)/bpftool $(LLVM_PLUGIN)
	endif

This installs bpftool to $(HOST_SCRATCH_DIR)/sbin/bpftool and the plugin to $(HOST_SCRATCH_DIR)/lib/bpftool/bpftool-llvm.so. When the staged bpftool runs, llvm_plugin_find() checks /lib/bpftool/bpftool-llvm.so and $(HOST_SCRATCH_DIR)/sbin/bpftool-llvm.so, missing the staged plugin in $(HOST_SCRATCH_DIR)/lib/bpftool/bpftool-llvm.so (causing "bpftool prog dump jited" to fail and "bpftool version" to omit the llvm feature).

Additionally, because llvm_plugin_find() checks LLVM_PLUGIN_DIR before the directory of /proc/self/exe, wouldn't running an uninstalled ./bpftool from its build directory on a host where $(LLVM_PLUGIN_DIR)/bpftool-llvm.so is already installed load the system plugin instead of ./bpftool-llvm.so next to the binary (and fail without falling back if the installed plugin has a different BPFTOOL_LLVM_ABI_VERSION or incompatible sanitizer flags)?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2