Skip to contentSkip to conversation

libbpf: fix build issue with llvm-readelf

PATCH Superseded Yonghong Song 2 participants 3 messages Feb 4, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libbpf: fix build issue with llvm-readelf

There are cases where clang compiler is packaged in a way readelf is a symbolic link to llvm-readelf. In such cases, llvm-readelf will be used instead of default binutils readelf, and the following error will appear during libbpf build:

Warning: Num of global symbols in
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/sharedobjs/libbpf-in.o (367) does NOT match with num of versioned symbols in /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.so libbpf.map (383). Please make sure all LIBBPF_API symbols are versioned in libbpf.map.
--- /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf_global_syms.tmp ... +++ /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf_versioned_syms.tmp ... @@ -324,6 +324,22 @@
btf__str_by_offset
btf__type_by_id
btf__type_cnt
+LIBBPF_0.0.1
+LIBBPF_0.0.2
+LIBBPF_0.0.3
+LIBBPF_0.0.4
+LIBBPF_0.0.5
+LIBBPF_0.0.6
+LIBBPF_0.0.7
+LIBBPF_0.0.8
+LIBBPF_0.0.9
+LIBBPF_0.1.0
+LIBBPF_0.2.0
+LIBBPF_0.3.0
+LIBBPF_0.4.0
+LIBBPF_0.5.0
+LIBBPF_0.6.0
+LIBBPF_0.7.0
libbpf_attach_type_by_name
libbpf_find_kernel_btf
libbpf_find_vmlinux_btf_id
make[2]: *** [Makefile:184: check_abi] Error 1
make[1]: *** [Makefile:140: all] Error 2

The above failure is due to different printouts for some ABS versioned symbols. For example, with the same libbpf.so,

  $ /bin/readelf --dyn-syms --wide tools/lib/bpf/libbpf.so | grep "LIBBPF" | grep ABS
     134: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBBPF_0.5.0
     202: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBBPF_0.6.0
     ...
  $ /opt/llvm/bin/readelf --dyn-syms --wide tools/lib/bpf/libbpf.so | grep "LIBBPF" | grep ABS
     134: 0000000000000000     0 OBJECT  GLOBAL DEFAULT   ABS LIBBPF_0.5.0@@LIBBPF_0.5.0
     202: 0000000000000000     0 OBJECT  GLOBAL DEFAULT   ABS LIBBPF_0.6.0@@LIBBPF_0.6.0
     ...
The binutils readelf doesn't print out the symbol LIBBPF_* version and llvm-readelf does.
Such a difference caused libbpf build failure with llvm-readelf.

To fix the issue, let us do proper filtering for LIBBPF_*@@LIBBPF_* symbols. The proposed fix works for both binutils readelf and llvm-readelf.

---
 tools/lib/bpf/Makefile | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/lib/bpf/Makefile +4 -2
index f947b61b2107..d1577c26c16b 100644
@@ -132,7 +132,8 @@ GLOBAL_SYM_COUNT = $(shell readelf -s --wide $(BPF_IN_SHARED) | \
 VERSIONED_SYM_COUNT = $(shell readelf --dyn-syms --wide $(OUTPUT)libbpf.so | \
 			      sed 's/\[.*\]//' | \
 			      awk '/GLOBAL/ && /DEFAULT/ && !/UND/ {print $$NF}' | \
-			      grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 | sort -u | wc -l)
+			      grep -Eo '[^ ]+@LIBBPF_' | grep -Ev '^LIBBPF_' | \
+			      cut -d@ -f1 | sort -u | wc -l)
 
 CMD_TARGETS = $(LIB_TARGET) $(PC_FILE)
 
@@ -195,7 +196,8 @@ check_abi: $(OUTPUT)libbpf.so $(VERSION_SCRIPT)
 		readelf --dyn-syms --wide $(OUTPUT)libbpf.so |		 \
 		    sed 's/\[.*\]//' |					 \
 		    awk '/GLOBAL/ && /DEFAULT/ && !/UND/ {print $$NF}'|  \
-		    grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 |		 \
+		    grep -Eo '[^ ]+@LIBBPF_' | grep -Ev '^LIBBPF_' |	 \
+		    cut -d@ -f1 |					 \
 		    sort -u > $(OUTPUT)libbpf_versioned_syms.tmp; 	 \
 		diff -u $(OUTPUT)libbpf_global_syms.tmp			 \
 		     $(OUTPUT)libbpf_versioned_syms.tmp;		 \
Signature · 1 line
2.30.2
#1 lore raw
Quoted 66 lines · Yonghong Song

There are cases where clang compiler is packaged in a way readelf is a symbolic link to llvm-readelf. In such cases, llvm-readelf will be used instead of default binutils readelf, and the following error will appear during libbpf build:

Warning: Num of global symbols in
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/sharedobjs/libbpf-in.o (367) does NOT match with num of versioned symbols in /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.so libbpf.map (383). Please make sure all LIBBPF_API symbols are versioned in libbpf.map.
--- /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf_global_syms.tmp ... +++ /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf_versioned_syms.tmp ... @@ -324,6 +324,22 @@
btf__str_by_offset
btf__type_by_id
btf__type_cnt
+LIBBPF_0.0.1
+LIBBPF_0.0.2
+LIBBPF_0.0.3
+LIBBPF_0.0.4
+LIBBPF_0.0.5
+LIBBPF_0.0.6
+LIBBPF_0.0.7
+LIBBPF_0.0.8
+LIBBPF_0.0.9
+LIBBPF_0.1.0
+LIBBPF_0.2.0
+LIBBPF_0.3.0
+LIBBPF_0.4.0
+LIBBPF_0.5.0
+LIBBPF_0.6.0
+LIBBPF_0.7.0
libbpf_attach_type_by_name
libbpf_find_kernel_btf
libbpf_find_vmlinux_btf_id
make[2]: *** [Makefile:184: check_abi] Error 1
make[1]: *** [Makefile:140: all] Error 2

The above failure is due to different printouts for some ABS versioned symbols. For example, with the same libbpf.so,

  $ /bin/readelf --dyn-syms --wide tools/lib/bpf/libbpf.so | grep "LIBBPF" | grep ABS
     134: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBBPF_0.5.0
     202: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBBPF_0.6.0
     ...
  $ /opt/llvm/bin/readelf --dyn-syms --wide tools/lib/bpf/libbpf.so | grep "LIBBPF" | grep ABS
     134: 0000000000000000     0 OBJECT  GLOBAL DEFAULT   ABS LIBBPF_0.5.0@@LIBBPF_0.5.0
     202: 0000000000000000     0 OBJECT  GLOBAL DEFAULT   ABS LIBBPF_0.6.0@@LIBBPF_0.6.0
     ...
The binutils readelf doesn't print out the symbol LIBBPF_* version and llvm-readelf does.
Such a difference caused libbpf build failure with llvm-readelf.

To fix the issue, let us do proper filtering for LIBBPF_*@@LIBBPF_* symbols. The proposed fix works for both binutils readelf and llvm-readelf.

---
 tools/lib/bpf/Makefile | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/lib/bpf/Makefile +0 -0
index f947b61b2107..d1577c26c16b 100644

tools/lib/bpf/Makefile GLOBAL_SYM_COUNT = $(shell readelf -s --wide $(BPF_IN_SHARED) | \ · line 134

Quoted 3 lines
tools/lib/bpf/Makefile +0 -0
@@ -132,7 +132,8 @@ GLOBAL_SYM_COUNT = $(shell readelf -s --wide $(BPF_IN_SHARED) | \
 VERSIONED_SYM_COUNT = $(shell readelf --dyn-syms --wide $(OUTPUT)libbpf.so | \
                              sed 's/\[.*\]//' | \
                              awk '/GLOBAL/ && /DEFAULT/ && !/UND/ {print $$NF}' | \

have you tried just doing !/UND|ABS/ here?

Quoted 19 lines
tools/lib/bpf/Makefile +4 -3
-                             grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 | sort -u | wc -l)
+                             grep -Eo '[^ ]+@LIBBPF_' | grep -Ev '^LIBBPF_' | \
+                             cut -d@ -f1 | sort -u | wc -l)
 
 CMD_TARGETS = $(LIB_TARGET) $(PC_FILE)
 
@@ -195,7 +196,8 @@ check_abi: $(OUTPUT)libbpf.so $(VERSION_SCRIPT)
                readelf --dyn-syms --wide $(OUTPUT)libbpf.so |           \
                    sed 's/\[.*\]//' |                                   \
                    awk '/GLOBAL/ && /DEFAULT/ && !/UND/ {print $$NF}'|  \
-                   grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 |             \
+                   grep -Eo '[^ ]+@LIBBPF_' | grep -Ev '^LIBBPF_' |     \
+                   cut -d@ -f1 |                                        \
                    sort -u > $(OUTPUT)libbpf_versioned_syms.tmp;        \
                diff -u $(OUTPUT)libbpf_global_syms.tmp                  \
                     $(OUTPUT)libbpf_versioned_syms.tmp;                 \
--

2.30.2

#2 lore raw
Quoted 62 lines · Andrii Nakryiko
Quoted 61 lines · Yonghong Song

There are cases where clang compiler is packaged in a way readelf is a symbolic link to llvm-readelf. In such cases, llvm-readelf will be used instead of default binutils readelf, and the following error will appear during libbpf build:

   Warning: Num of global symbols in
    /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/sharedobjs/libbpf-in.o (367)
    does NOT match with num of versioned symbols in
    /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.so libbpf.map (383).
    Please make sure all LIBBPF_API symbols are versioned in libbpf.map.
   --- /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf_global_syms.tmp ...
   +++ /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf_versioned_syms.tmp ...
   @@ -324,6 +324,22 @@
    btf__str_by_offset
    btf__type_by_id
    btf__type_cnt
   +LIBBPF_0.0.1
   +LIBBPF_0.0.2
   +LIBBPF_0.0.3
   +LIBBPF_0.0.4
   +LIBBPF_0.0.5
   +LIBBPF_0.0.6
   +LIBBPF_0.0.7
   +LIBBPF_0.0.8
   +LIBBPF_0.0.9
   +LIBBPF_0.1.0
   +LIBBPF_0.2.0
   +LIBBPF_0.3.0
   +LIBBPF_0.4.0
   +LIBBPF_0.5.0
   +LIBBPF_0.6.0
   +LIBBPF_0.7.0
    libbpf_attach_type_by_name
    libbpf_find_kernel_btf
    libbpf_find_vmlinux_btf_id
   make[2]: *** [Makefile:184: check_abi] Error 1
   make[1]: *** [Makefile:140: all] Error 2

The above failure is due to different printouts for some ABS versioned symbols. For example, with the same libbpf.so,

   $ /bin/readelf --dyn-syms --wide tools/lib/bpf/libbpf.so | grep "LIBBPF" | grep ABS
      134: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBBPF_0.5.0
      202: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBBPF_0.6.0
      ...
   $ /opt/llvm/bin/readelf --dyn-syms --wide tools/lib/bpf/libbpf.so | grep "LIBBPF" | grep ABS
      134: 0000000000000000     0 OBJECT  GLOBAL DEFAULT   ABS LIBBPF_0.5.0@@LIBBPF_0.5.0
      202: 0000000000000000     0 OBJECT  GLOBAL DEFAULT   ABS LIBBPF_0.6.0@@LIBBPF_0.6.0
      ...
The binutils readelf doesn't print out the symbol LIBBPF_* version and llvm-readelf does.
Such a difference caused libbpf build failure with llvm-readelf.

To fix the issue, let us do proper filtering for LIBBPF_*@@LIBBPF_* symbols. The proposed fix works for both binutils readelf and llvm-readelf.

---
  tools/lib/bpf/Makefile | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
Quoted 10 lines

tools/lib/bpf/Makefile GLOBAL_SYM_COUNT = $(shell readelf -s --wide $(BPF_IN_SHARED) | \ · line 134

Quoted 8 lines
tools/lib/bpf/Makefile +0 -0
index f947b61b2107..d1577c26c16b 100644
@@ -132,7 +132,8 @@ GLOBAL_SYM_COUNT = $(shell readelf -s --wide $(BPF_IN_SHARED) | \
  VERSIONED_SYM_COUNT = $(shell readelf --dyn-syms --wide $(OUTPUT)libbpf.so | \
                               sed 's/\[.*\]//' | \
                               awk '/GLOBAL/ && /DEFAULT/ && !/UND/ {print $$NF}' | \

have you tried just doing !/UND|ABS/ here?

It works and even simpler.
Thanks for suggestion. Will send v2.

Quoted 20 lines
Quoted 20 lines
tools/lib/bpf/Makefile +4 -3
-                             grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 | sort -u | wc -l)
+                             grep -Eo '[^ ]+@LIBBPF_' | grep -Ev '^LIBBPF_' | \
+                             cut -d@ -f1 | sort -u | wc -l)
 
  CMD_TARGETS = $(LIB_TARGET) $(PC_FILE)
 
@@ -195,7 +196,8 @@ check_abi: $(OUTPUT)libbpf.so $(VERSION_SCRIPT)
                 readelf --dyn-syms --wide $(OUTPUT)libbpf.so |           \
                     sed 's/\[.*\]//' |                                   \
                     awk '/GLOBAL/ && /DEFAULT/ && !/UND/ {print $$NF}'|  \
-                   grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 |             \
+                   grep -Eo '[^ ]+@LIBBPF_' | grep -Ev '^LIBBPF_' |     \
+                   cut -d@ -f1 |                                        \
                     sort -u > $(OUTPUT)libbpf_versioned_syms.tmp;        \
                 diff -u $(OUTPUT)libbpf_global_syms.tmp                  \
                      $(OUTPUT)libbpf_versioned_syms.tmp;                 \
--

2.30.2