Building bpftool with OUTPUT set breaks
Subsystems (4)
Hello,
On 5.2 and current bpf-next, the following fails:
$ make -C tools/bpf/bpftool OUTPUT=/tmp/tmp.NUSttIbAYw/ make: Entering directory '/home/lorenz/dev/bpf-next/tools/bpf/bpftool'
Auto-detecting system features:
... libbfd: OFF
... disassembler-four-args: OFF
CC /tmp/tmp.NUSttIbAYw/map_perf_ring.o
<snip>
CC /tmp/tmp.NUSttIbAYw/disasm.o
make[1]: Entering directory '/home/lorenz/dev/bpf-next/tools/lib/bpf'
Auto-detecting system features:
... libelf: on
... bpf: on
CC /tmp/tmp.NUSttIbAYw/libbpf.o
<snip>
CC /tmp/tmp.NUSttIbAYw/btf_dump.o LD /tmp/tmp.NUSttIbAYw/libbpf-in.o LINK /tmp/tmp.NUSttIbAYw/libbpf.a
make[1]: Leaving directory '/home/lorenz/dev/bpf-next/tools/lib/bpf'
LINK /tmp/tmp.NUSttIbAYw/bpftool/usr/bin/ld: /tmp/tmp.NUSttIbAYw/libbpf.a(libbpf-in.o): in function `do_btf':
(.text+0x105b0): multiple definition of `do_btf'; /tmp/tmp.NUSttIbAYw/btf.o:btf.c:(.text+0x11f0): first defined here <snip>
I think the problem is that objects for both bpftool and libbpf end up in the same directory
if OUTPUT is set. Does anybody know how to fix this?
Best
Lorenz
Signature · 4 lines
Lorenz Bauer | Systems Engineer 6th Floor, County Hall/The Riverside Building, SE1 7PB, UK www.cloudflare.com
PATCH 1/1 tools/bpf: fix bpftool build with OUTPUT set
Hi Lorenz,
I've been using the following patch for quite some time now. Please let me know if it works for you.
Best regards,
Ilya
---
When OUTPUT is set, bpftool and libbpf put their objects into the same directory, and since some of them have the same names, the collision happens.
Fix by invoking libbpf build in a manner similar to $(call descend) - descend itself cannot be used, since libbpf is a sibling, and not a child, of bpftool.
Also, don't link bpftool with libbpf.a twice.
--- tools/bpf/bpftool/Makefile | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
tools/bpf/bpftool/Makefile +6 -11
@@ -15,23 +15,18 @@ else endif BPF_DIR = $(srctree)/tools/lib/bpf/ - -ifneq ($(OUTPUT),) - BPF_PATH = $(OUTPUT) -else - BPF_PATH = $(BPF_DIR) -endif - -LIBBPF = $(BPF_PATH)libbpf.a +BPF_PATH = $(objtree)/tools/lib/bpf +LIBBPF = $(BPF_PATH)/libbpf.a BPFTOOL_VERSION := $(shell make --no-print-directory -sC ../../.. kernelversion) $(LIBBPF): FORCE - $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(OUTPUT) $(OUTPUT)libbpf.a + $(Q)mkdir -p $(BPF_PATH) + $(Q)$(MAKE) $(COMMAND_O) subdir=tools/lib/bpf -C $(BPF_DIR) $(LIBBPF) $(LIBBPF)-clean: $(call QUIET_CLEAN, libbpf) - $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(OUTPUT) clean >/dev/null + $(Q)$(MAKE) $(COMMAND_O) subdir=tools/lib/bpf -C $(BPF_DIR) clean >/dev/null prefix ?= /usr/local bash_compdir ?= /usr/share/bash-completion/completions
@@ -112,7 +107,7 @@ $(OUTPUT)disasm.o: $(srctree)/kernel/bpf/disasm.c $(QUIET_CC)$(COMPILE.c) -MMD -o $@ $< $(OUTPUT)bpftool: $(OBJS) $(LIBBPF) - $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $^ $(LIBS) + $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS) $(OUTPUT)%.o: %.c $(QUIET_CC)$(COMPILE.c) -MMD -o $@ $<
Signature · 1 line
2.21.0
Re: [PATCH bpf] tools/bpf: fix bpftool build with OUTPUT set
Quoted 32 lines · Ilya Leoshkevich
Hi Lorenz,
I've been using the following patch for quite some time now. Please let me know if it works for you.
Best regards,
Ilya---
When OUTPUT is set, bpftool and libbpf put their objects into the same directory, and since some of them have the same names, the collision happens.
Fix by invoking libbpf build in a manner similar to $(call descend) - descend itself cannot be used, since libbpf is a sibling, and not a child, of bpftool.
Also, don't link bpftool with libbpf.a twice.
Signed-off-by Ilya Leoshkevich <[email protected]>--- tools/bpf/bpftool/Makefile | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)tools/bpf/bpftool/Makefile +0
-0@@ -15,23 +15,18 @@ else endif
tools/bpf/bpftool/Makefile else · line 18
Quoted 10 lines
tools/bpf/bpftool/Makefile +1
-8BPF_DIR = $(srctree)/tools/lib/bpf/ - -ifneq ($(OUTPUT),) - BPF_PATH = $(OUTPUT) -else - BPF_PATH = $(BPF_DIR) -endif - -LIBBPF = $(BPF_PATH)libbpf.a +BPF_PATH = $(objtree)/tools/lib/bpf
objtree won't be set for simple make in the directory. Perhaps we should stick to using OUTPUT and srctree?
We should probably make a script with all the ways of calling make should work. Otherwise we can lose track too easily.
# thru kbuild make tools/bpf
T=$(mktemp -d)
make tools/bpf OUTPUT=$T
rm -rf $T
# from kernel source tree make -C tools/bpf/bpftool
T=$(mktemp -d)
make -C tools/bpf/bpftool OUTPUT=$T
rm -rf $T
# from tools cd tools/ make bpf
T=$(mktemp -d)
make bpf OUTPUT=$T
rm -rf $T
# from bpftool's dir cd bpf/bpftool make
T=$(mktemp -d)
make OUTPUT=$T
rm -rf $T
.. add your own.
Quoted 25 lines
tools/bpf/bpftool/Makefile +5
-3+LIBBPF = $(BPF_PATH)/libbpf.a BPFTOOL_VERSION := $(shell make --no-print-directory -sC ../../.. kernelversion) $(LIBBPF): FORCE - $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(OUTPUT) $(OUTPUT)libbpf.a + $(Q)mkdir -p $(BPF_PATH) + $(Q)$(MAKE) $(COMMAND_O) subdir=tools/lib/bpf -C $(BPF_DIR) $(LIBBPF) $(LIBBPF)-clean: $(call QUIET_CLEAN, libbpf) - $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(OUTPUT) clean >/dev/null + $(Q)$(MAKE) $(COMMAND_O) subdir=tools/lib/bpf -C $(BPF_DIR) clean >/dev/null prefix ?= /usr/local bash_compdir ?= /usr/share/bash-completion/completions@@ -112,7 +107,7 @@ $(OUTPUT)disasm.o: $(srctree)/kernel/bpf/disasm.c $(QUIET_CC)$(COMPILE.c) -MMD -o $@ $< $(OUTPUT)bpftool: $(OBJS) $(LIBBPF) - $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $^ $(LIBS) + $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS) $(OUTPUT)%.o: %.c $(QUIET_CC)$(COMPILE.c) -MMD -o $@ $<
Re: [PATCH bpf] tools/bpf: fix bpftool build with OUTPUT set
Quoted 4 lines
Am 18.07.2019 um 20:51 schrieb Jakub Kicinski <[email protected]>:
We should probably make a script with all the ways of calling make should work. Otherwise we can lose track too easily.
Thanks for the script!
I’m trying to make it all pass now, and hitting a weird issue in the Kbuild case. The build prints "No rule to make target 'scripts/Makefile.ubsan.o'" and proceeds with an empty BPFTOOL_VERSION, which causes problems later on.
I've found that this is caused by sub_make_done=1 environment variable, and unsetting it indeed fixes the problem, since the root Makefile no longer uses the implicit %.o rule.
However, I wonder if that would be acceptable in the final version of the patch, and whether there is a cleaner way to achieve the same effect?
Best regards,
Ilya
Re: [PATCH bpf] tools/bpf: fix bpftool build with OUTPUT set
Quoted 11 lines · Ilya Leoshkevich
Quoted 4 lines
Am 18.07.2019 um 20:51 schrieb Jakub Kicinski <[email protected]>:
We should probably make a script with all the ways of calling make should work. Otherwise we can lose track too easily.
Thanks for the script!
I’m trying to make it all pass now, and hitting a weird issue in the
Kbuildcase. The build prints "No rule to make target 'scripts/Makefile.ubsan.o'" and proceeds with an emptyBPFTOOL_VERSION, which causes problems later on.
Does it only break with UBSAN enabled?
Quoted 7 lines
I've found that this is caused by sub_make_done=1 environment variable, and unsetting it indeed fixes the problem, since the root
Makefileno longer uses the implicit %.o rule.However, I wonder if that would be acceptable in the final version of the patch, and whether there is a cleaner way to achieve the same effect?
I'm not sure to be honest. Did you check how perf deals with that?
My goal was primarily to make sure we don't regress, so maybe if some corner cases don't work that's not the end of the world. I think a good rule of the thumb would be "if it works for perf it should work for bpftool" ;) Perf gets a lot more build testing.
Re: [PATCH bpf] tools/bpf: fix bpftool build with OUTPUT set
Quoted 8 lines
Am 19.07.2019 um 20:17 schrieb Jakub Kicinski <[email protected]>:
Quoted 5 lines · Ilya Leoshkevich
Quoted 4 lines
Am 18.07.2019 um 20:51 schrieb Jakub Kicinski <[email protected]>:
We should probably make a script with all the ways of calling make should work. Otherwise we can lose track too easily.
Quoted 8 lines
Quoted 6 lines
Thanks for the script!
I’m trying to make it all pass now, and hitting a weird issue in the
Kbuildcase. The build prints "No rule to make target 'scripts/Makefile.ubsan.o'" and proceeds with an emptyBPFTOOL_VERSION, which causes problems later on.Does it only break with UBSAN enabled?
No, all the time. I think this is a coincidence - make happens to scan scripts/Makefile.ubsan first.
Quoted 10 lines
Quoted 8 lines
I've found that this is caused by sub_make_done=1 environment variable, and unsetting it indeed fixes the problem, since the root
Makefileno longer uses the implicit %.o rule.However, I wonder if that would be acceptable in the final version of the patch, and whether there is a cleaner way to achieve the same effect?
I'm not sure to be honest. Did you check how perf deals with that?
perf obtains the version using "git describe". However, if we are building it from a tarball, it falls back to "make kernelversion" and fails in a similar way:
linux-5.3-rc1$ make defconfig
linux-5.3-rc1$ make tools/perf
<snip>
make[6]: Circular scripts/Makefile.ubsan.mod <- scripts/Makefile.ubsan.o dependency dropped. make[6]: m2c: Command not found
make[6]: *** [<builtin>: scripts/Makefile.ubsan.o] Error 127
make[5]: *** [Makefile:1765: scripts/Makefile.ubsan.o] Error 2
<snip>
The same trick helps:
perf/util/PERF-VERSION-GEN renamed from perf/util/PERF-VERSION-GEN.orig +1 -1
@@ -26,7 +26,7 @@ fi if test -z "$TAG" then - TAG=$(MAKEFLAGS= make -sC ../.. kernelversion) + TAG=$(MAKEFLAGS= sub_make_done= make -sC ../.. kernelversion) fi VN="$TAG$CID" if test -n "$CID"
Re: [PATCH bpf] tools/bpf: fix bpftool build with OUTPUT set
In reply to Ilya LeoshkevichHi Ilya,
Thanks for your patch! I tried it but had problems with cross-compilation. Not sure if this is related to the patch or not though, I haven't had the time to follow up.
Best
Lorenz
Quoted 71 lines · Ilya Leoshkevich
Hi Lorenz,
I've been using the following patch for quite some time now. Please let me know if it works for you.
Best regards,
Ilya---
When OUTPUT is set, bpftool and libbpf put their objects into the same directory, and since some of them have the same names, the collision happens.
Fix by invoking libbpf build in a manner similar to $(call descend) - descend itself cannot be used, since libbpf is a sibling, and not a child, of bpftool.
Also, don't link bpftool with libbpf.a twice.
Signed-off-by Ilya Leoshkevich <[email protected]>--- tools/bpf/bpftool/Makefile | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)tools/bpf/bpftool/Makefile +6
-12@@ -15,23 +15,18 @@ else endif BPF_DIR = $(srctree)/tools/lib/bpf/ - -ifneq ($(OUTPUT),) - BPF_PATH = $(OUTPUT) -else - BPF_PATH = $(BPF_DIR) -endif - -LIBBPF = $(BPF_PATH)libbpf.a +BPF_PATH = $(objtree)/tools/lib/bpf +LIBBPF = $(BPF_PATH)/libbpf.a BPFTOOL_VERSION := $(shell make --no-print-directory -sC ../../.. kernelversion) $(LIBBPF): FORCE - $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(OUTPUT) $(OUTPUT)libbpf.a + $(Q)mkdir -p $(BPF_PATH) + $(Q)$(MAKE) $(COMMAND_O) subdir=tools/lib/bpf -C $(BPF_DIR) $(LIBBPF) $(LIBBPF)-clean: $(call QUIET_CLEAN, libbpf) - $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(OUTPUT) clean >/dev/null + $(Q)$(MAKE) $(COMMAND_O) subdir=tools/lib/bpf -C $(BPF_DIR) clean >/dev/null prefix ?= /usr/local bash_compdir ?= /usr/share/bash-completion/completions@@ -112,7 +107,7 @@ $(OUTPUT)disasm.o: $(srctree)/kernel/bpf/disasm.c $(QUIET_CC)$(COMPILE.c) -MMD -o $@ $< $(OUTPUT)bpftool: $(OBJS) $(LIBBPF) - $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $^ $(LIBS) + $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS) $(OUTPUT)%.o: %.c $(QUIET_CC)$(COMPILE.c) -MMD -o $@ $< --2.21.0
Signature · 4 lines
Lorenz Bauer | Systems Engineer 6th Floor, County Hall/The Riverside Building, SE1 7PB, UK www.cloudflare.com