Skip to contentSkip to conversation

RISC-V: Fix the VDSO symbol generaton for binutils-2.35+

PATCH In review Palmer Dabbelt 4 participants 7 messages Oct 24, 2020 – Nov 6, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 RISC-V: Fix the VDSO symbol generaton for binutils-2.35+

We were relying on GNU ld's ability to re-link executable files in order to extract our VDSO symbols. This behavior was deemed a bug as of binutils-2.35 (specifically the binutils-gdb commit a87e1817a4 ("Have the linker fail if any attempt to link in an executable is made."), but as that has been backported to at least Debian's binutils-2.34 in may manifest in other places.

The previous version of this was a bit of a mess: we were linking a static executable version of the VDSO, containing only a subset of the input symbols, which we then linked into the kernel. This worked, but certainly wasn't a supported path through the toolchain. Instead this new version parses the textual output of nm to produce a symbol table. Both rely on near-zero addresses being linkable, but as we rely on weak undefined symbols being linkable elsewhere I don't view this as a major issue.

Fixes e2c0cdfba7f6 ("RISC-V: User-facing API")

---

Changes since v2 <[email protected]>:

* Uses $(srctree)/$(src) to allow for out-of-tree builds.

Changes since v1 <[email protected]>:

* Uses $(NM) instead of $(CROSS_COMPILE)nm. We use the $(CROSS_COMPILE) form elsewhere in this file, but we'll fix that later.
* Removed the unnecesary .map file creation.

---
 arch/riscv/kernel/vdso/.gitignore |  1 +
 arch/riscv/kernel/vdso/Makefile   | 17 ++++++++---------
 arch/riscv/kernel/vdso/so2s.sh    |  6 ++++++
 3 files changed, 15 insertions(+), 9 deletions(-)
 create mode 100755 arch/riscv/kernel/vdso/so2s.sh
arch/riscv/kernel/vdso/.gitignore +1 -0
index 11ebee9e4c1d..3a19def868ec 100644
@@ -1,3 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 vdso.lds
 *.tmp
+vdso-syms.S
arch/riscv/kernel/vdso/Makefile +8 -9
index 478e7338ddc1..a8ecf102e09b 100644
@@ -43,19 +43,14 @@ $(obj)/vdso.o: $(obj)/vdso.so
 SYSCFLAGS_vdso.so.dbg = $(c_flags)
 $(obj)/vdso.so.dbg: $(src)/vdso.lds $(obj-vdso) FORCE
 	$(call if_changed,vdsold)
+SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
+	-Wl,--build-id -Wl,--hash-style=both
 
 # We also create a special relocatable object that should mirror the symbol
 # table and layout of the linked DSO. With ld --just-symbols we can then
 # refer to these symbols in the kernel code rather than hand-coded addresses.
-
-SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
-	-Wl,--build-id -Wl,--hash-style=both
-$(obj)/vdso-dummy.o: $(src)/vdso.lds $(obj)/rt_sigreturn.o FORCE
-	$(call if_changed,vdsold)
-
-LDFLAGS_vdso-syms.o := -r --just-symbols
-$(obj)/vdso-syms.o: $(obj)/vdso-dummy.o FORCE
-	$(call if_changed,ld)
+$(obj)/vdso-syms.S: $(obj)/vdso.so FORCE
+	$(call if_changed,so2s)
 
 # strip rule for the .so file
 $(obj)/%.so: OBJCOPYFLAGS := -S
@@ -73,6 +68,10 @@ quiet_cmd_vdsold = VDSOLD  $@
                            $(patsubst %, -G __vdso_%, $(vdso-syms)) [email protected] $@ && \
                    rm [email protected]
 
+# Extracts
+quiet_cmd_so2s = SO2S    $@
+      cmd_so2s = $(NM) -D $< | $(srctree)/$(src)/so2s.sh > $@
+
 # install commands for the unstripped file
 quiet_cmd_vdso_install = INSTALL $@
       cmd_vdso_install = cp $(obj)/[email protected] $(MODLIB)/vdso/$@
arch/riscv/kernel/vdso/so2s.sh new file +6 -0
new file mode 100755
index 000000000000..3c5b43207658
@@ -0,0 +1,6 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright 2020 Palmer Dabbelt <[email protected]>
+
+sed 's!\([0-9a-f]*\) T \([a-z0-9_]*\)@@LINUX_4.15!.global \2\n.set \2,0x\1!' \
+| grep '^\.'
Signature · 1 line
2.29.0.rc1.297.gfa9743e501-goog
#1 lore raw
Quoted 17 lines · Palmer Dabbelt' via Clang Built Linux

We were relying on GNU ld's ability to re-link executable files in order to extract our VDSO symbols. This behavior was deemed a bug as of binutils-2.35 (specifically the binutils-gdb commit a87e1817a4 ("Have the linker fail if any attempt to link in an executable is made."), but as that has been backported to at least Debian's binutils-2.34 in may manifest in other places.

The previous version of this was a bit of a mess: we were linking a static executable version of the VDSO, containing only a subset of the input symbols, which we then linked into the kernel. This worked, but certainly wasn't a supported path through the toolchain. Instead this new version parses the textual output of nm to produce a symbol table. Both rely on near-zero addresses being linkable, but as we rely on weak undefined symbols being linkable elsewhere I don't view this as a major issue.

Quoted 4 lines
Fixes e2c0cdfba7f6 ("RISC-V: User-facing API")

Any way to improve the error message if/when this fails?
https://travis-ci.com/github/ClangBuiltLinux/continuous-integration/jobs/407165683

Quoted 87 lines

---

Changes since v2 <[email protected]>:

* Uses $(srctree)/$(src) to allow for out-of-tree builds.

Changes since v1 <[email protected]>:

* Uses $(NM) instead of $(CROSS_COMPILE)nm. We use the $(CROSS_COMPILE) form elsewhere in this file, but we'll fix that later.
* Removed the unnecesary .map file creation.

---
 arch/riscv/kernel/vdso/.gitignore |  1 +
 arch/riscv/kernel/vdso/Makefile   | 17 ++++++++---------
 arch/riscv/kernel/vdso/so2s.sh    |  6 ++++++
 3 files changed, 15 insertions(+), 9 deletions(-)
 create mode 100755 arch/riscv/kernel/vdso/so2s.sh
arch/riscv/kernel/vdso/.gitignore +1 -0
index 11ebee9e4c1d..3a19def868ec 100644
@@ -1,3 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 vdso.lds
 *.tmp
+vdso-syms.S
arch/riscv/kernel/vdso/Makefile +8 -9
index 478e7338ddc1..a8ecf102e09b 100644
@@ -43,19 +43,14 @@ $(obj)/vdso.o: $(obj)/vdso.so
 SYSCFLAGS_vdso.so.dbg = $(c_flags)
 $(obj)/vdso.so.dbg: $(src)/vdso.lds $(obj-vdso) FORCE
        $(call if_changed,vdsold)
+SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
+       -Wl,--build-id -Wl,--hash-style=both
 
 # We also create a special relocatable object that should mirror the symbol
 # table and layout of the linked DSO. With ld --just-symbols we can then
 # refer to these symbols in the kernel code rather than hand-coded addresses.
-
-SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
-       -Wl,--build-id -Wl,--hash-style=both
-$(obj)/vdso-dummy.o: $(src)/vdso.lds $(obj)/rt_sigreturn.o FORCE
-       $(call if_changed,vdsold)
-
-LDFLAGS_vdso-syms.o := -r --just-symbols
-$(obj)/vdso-syms.o: $(obj)/vdso-dummy.o FORCE
-       $(call if_changed,ld)
+$(obj)/vdso-syms.S: $(obj)/vdso.so FORCE
+       $(call if_changed,so2s)
 
 # strip rule for the .so file
 $(obj)/%.so: OBJCOPYFLAGS := -S
@@ -73,6 +68,10 @@ quiet_cmd_vdsold = VDSOLD  $@
                            $(patsubst %, -G __vdso_%, $(vdso-syms)) [email protected] $@ && \
                    rm [email protected]
 
+# Extracts
+quiet_cmd_so2s = SO2S    $@
+      cmd_so2s = $(NM) -D $< | $(srctree)/$(src)/so2s.sh > $@
+
 # install commands for the unstripped file
 quiet_cmd_vdso_install = INSTALL $@
       cmd_vdso_install = cp $(obj)/[email protected] $(MODLIB)/vdso/$@
arch/riscv/kernel/vdso/so2s.sh new file +6 -1
new file mode 100755
index 000000000000..3c5b43207658
@@ -0,0 +1,6 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright 2020 Palmer Dabbelt <[email protected]>
+
+sed 's!\([0-9a-f]*\) T \([a-z0-9_]*\)@@LINUX_4.15!.global \2\n.set \2,0x\1!' \
+| grep '^\.'
--

2.29.0.rc1.297.gfa9743e501-goog

--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20201024045046.3018271-1-palmerdabbelt%40google.com.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#2 lore raw
Quoted 19 lines · Nick Desaulniers
Quoted 17 lines · Palmer Dabbelt' via Clang Built Linux

We were relying on GNU ld's ability to re-link executable files in order to extract our VDSO symbols. This behavior was deemed a bug as of binutils-2.35 (specifically the binutils-gdb commit a87e1817a4 ("Have the linker fail if any attempt to link in an executable is made."), but as that has been backported to at least Debian's binutils-2.34 in may manifest in other places.

The previous version of this was a bit of a mess: we were linking a static executable version of the VDSO, containing only a subset of the input symbols, which we then linked into the kernel. This worked, but certainly wasn't a supported path through the toolchain. Instead this new version parses the textual output of nm to produce a symbol table. Both rely on near-zero addresses being linkable, but as we rely on weak undefined symbols being linkable elsewhere I don't view this as a major issue.

Quoted 7 lines
Quoted 4 lines
Fixes e2c0cdfba7f6 ("RISC-V: User-facing API")

Any way to improve the error message if/when this fails?
https://travis-ci.com/github/ClangBuiltLinux/continuous-integration/jobs/407165683

Probably, but I can't get that command to actually run this stuff. I tried pulling the commands, but I'm getting some weirdness

$ rm -f arch/riscv/kernel/vdso/vdso-syms.S
$ make ARCH=riscv defconfig
$ make -j2 AR=llvm-ar 'CC=clang' 'HOSTCC=clang' HOSTLD=ld KCFLAGS=-Wno-implicit-fallthrough LD=riscv64-linux-gnu-ld LLVM_IAS=1 NM=llvm-nm OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump OBJSIZE=llvm-size READELF=llvm-readelf STRIP=llvm-strip ARCH=riscv Image
 SYNC    include/config/auto.conf.cmd
 HOSTCC  scripts/basic/fixdep
 HOSTCC  scripts/kconfig/conf.o
 HOSTCC  scripts/kconfig/confdata.o
 HOSTCC  scripts/kconfig/expr.o
 HOSTCC  scripts/kconfig/lexer.lex.o
 HOSTCC  scripts/kconfig/parser.tab.o
 HOSTCC  scripts/kconfig/preprocess.o
 HOSTCC  scripts/kconfig/symbol.o
 HOSTCC  scripts/kconfig/util.o
 HOSTLD  scripts/kconfig/conf
*
* Restart config...
*
*
* Memory initialization
*
Initialize kernel stack variables at function entry
Quoted 1 line

1. no automatic initialization (weakest) (INIT_STACK_NONE)

2. 0xAA-init everything on the stack (strongest) (INIT_STACK_ALL_PATTERN) (NEW)
3. zero-init everything on the stack (strongest and safest) (INIT_STACK_ALL_ZERO) (NEW)
choice[1-3?]: Enable heap memory zeroing on allocation by default (INIT_ON_ALLOC_DEFAULT_ON) [N/y/?] n Enable heap memory zeroing on free by default (INIT_ON_FREE_DEFAULT_ON) [N/y/?] n

 HOSTCC  scripts/dtc/dtc.o
 HOSTCC  scripts/dtc/flattree.o
 HOSTCC  scripts/dtc/fstree.o
 HOSTCC  scripts/dtc/data.o
 HOSTCC  scripts/dtc/livetree.o
 HOSTCC  scripts/dtc/treesource.o
 HOSTCC  scripts/dtc/srcpos.o
 HOSTCC  scripts/dtc/checks.o
 HOSTCC  scripts/dtc/util.o
 HOSTCC  scripts/dtc/dtc-lexer.lex.o
 HOSTCC  scripts/dtc/dtc-parser.tab.o
 HOSTLD  scripts/dtc/dtc
 HOSTCC  scripts/kallsyms
 CC      scripts/mod/empty.o
 HOSTCC  scripts/mod/mk_elfconfig

error: invalid value 'medany' in '-mcode-model medany' make[1]: *** [scripts/Makefile.build:283: scripts/mod/empty.o] Error 1 make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1199: prepare0] Error 2

I have no idea where the space in '-mcode-model medany' comes from.

Does this fail in general for LLVM, or is the issue just the error message? I've put this on fixes assuming it's just the error message, but LMK if it's actually not working in which case I won't send it out as I don't want to break stuff that was working.

Either way I'd be happy to fix it if I can reproduce it. I always just guess at regexes until they work for me, so I bet there's something subtly different in LLVM. This splits out the calls, which might be enough to sort it out (I've fixed the comment on fixes):

arch/riscv/kernel/vdso/.gitignore +0 -0
index 3a19def868ec..88206dd8b472 100644
@@ -2,3 +2,4 @@

vdso.lds
*.tmp
vdso-syms.S
+vdso-syms.nm

arch/riscv/kernel/vdso/Makefile +0 -0
index a8ecf102e09b..fe5c969a6bf4 100644
@@ -49,8 +49,11 @@ SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \

# We also create a special relocatable object that should mirror the symbol

# table and layout of the linked DSO. With ld --just-symbols we can then
# refer to these symbols in the kernel code rather than hand-coded addresses.
-$(obj)/vdso-syms.S: $(obj)/vdso.so FORCE
-	$(call if_changed,so2s)
+$(obj)/vdso-syms.nm: $(obj)/vdso.so
+	$(call if_changed,nm_d)
+
+$(obj)/vdso-syms.S: $(obj)/vdso-syms.nm
+	$(call if_changed,nm2s)
# strip rule for the .so file
$(obj)/%.so: OBJCOPYFLAGS := -S
@@ -68,9 +71,13 @@ quiet_cmd_vdsold = VDSOLD  $@
                           $(patsubst %, -G __vdso_%, $(vdso-syms)) [email protected] $@ && \
                   rm [email protected]

-# Extracts
-quiet_cmd_so2s = SO2S $@
- cmd_so2s = $(NM) -D $< | $(srctree)/$(src)/so2s.sh > $@
+# Extracts symbol offsets from the VDSO, converting them into an assembly file +# that contains the same symbols at the same offsets.
+quiet_cmd_nm_d = NM -D $@
+ cmd_nm_d = $(NM) -D $< > $@
+
+quiet_cmd_nm2s = SYMS2S $@
+ cmd_nm2s = cat $< | $(srctree)/$(src)/so2s.sh > $@

# install commands for the unstripped file
quiet_cmd_vdso_install = INSTALL $@

For reference, here's the output of nmo for me:

$ cat arch/riscv/kernel/vdso/vdso-syms.nm 0000000000000000 A LINUX_4.15
00000000000009e0 T __vdso_clock_getres@@LINUX_4.15
000000000000080a T __vdso_clock_gettime@@LINUX_4.15
0000000000000a48 T __vdso_flush_icache@@LINUX_4.15
0000000000000a3c T __vdso_getcpu@@LINUX_4.15
0000000000000916 T __vdso_gettimeofday@@LINUX_4.15
0000000000000800 T __vdso_rt_sigreturn@@LINUX_4.15
Quoted 86 lines
Quoted 86 lines

---

Changes since v2 <[email protected]>:

* Uses $(srctree)/$(src) to allow for out-of-tree builds.

Changes since v1 <[email protected]>:

* Uses $(NM) instead of $(CROSS_COMPILE)nm. We use the $(CROSS_COMPILE) form elsewhere in this file, but we'll fix that later.
* Removed the unnecesary .map file creation.

---
 arch/riscv/kernel/vdso/.gitignore |  1 +
 arch/riscv/kernel/vdso/Makefile   | 17 ++++++++---------
 arch/riscv/kernel/vdso/so2s.sh    |  6 ++++++
 3 files changed, 15 insertions(+), 9 deletions(-)
 create mode 100755 arch/riscv/kernel/vdso/so2s.sh
arch/riscv/kernel/vdso/.gitignore +1 -0
index 11ebee9e4c1d..3a19def868ec 100644
@@ -1,3 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 vdso.lds
 *.tmp
+vdso-syms.S
arch/riscv/kernel/vdso/Makefile +8 -9
index 478e7338ddc1..a8ecf102e09b 100644
@@ -43,19 +43,14 @@ $(obj)/vdso.o: $(obj)/vdso.so
 SYSCFLAGS_vdso.so.dbg = $(c_flags)
 $(obj)/vdso.so.dbg: $(src)/vdso.lds $(obj-vdso) FORCE
        $(call if_changed,vdsold)
+SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
+       -Wl,--build-id -Wl,--hash-style=both
 
 # We also create a special relocatable object that should mirror the symbol
 # table and layout of the linked DSO. With ld --just-symbols we can then
 # refer to these symbols in the kernel code rather than hand-coded addresses.
-
-SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
-       -Wl,--build-id -Wl,--hash-style=both
-$(obj)/vdso-dummy.o: $(src)/vdso.lds $(obj)/rt_sigreturn.o FORCE
-       $(call if_changed,vdsold)
-
-LDFLAGS_vdso-syms.o := -r --just-symbols
-$(obj)/vdso-syms.o: $(obj)/vdso-dummy.o FORCE
-       $(call if_changed,ld)
+$(obj)/vdso-syms.S: $(obj)/vdso.so FORCE
+       $(call if_changed,so2s)
 
 # strip rule for the .so file
 $(obj)/%.so: OBJCOPYFLAGS := -S
@@ -73,6 +68,10 @@ quiet_cmd_vdsold = VDSOLD  $@
                            $(patsubst %, -G __vdso_%, $(vdso-syms)) [email protected] $@ && \
                    rm [email protected]
 
+# Extracts
+quiet_cmd_so2s = SO2S    $@
+      cmd_so2s = $(NM) -D $< | $(srctree)/$(src)/so2s.sh > $@
+
 # install commands for the unstripped file
 quiet_cmd_vdso_install = INSTALL $@
       cmd_vdso_install = cp $(obj)/[email protected] $(MODLIB)/vdso/$@
arch/riscv/kernel/vdso/so2s.sh new file +6 -1
new file mode 100755
index 000000000000..3c5b43207658
@@ -0,0 +1,6 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright 2020 Palmer Dabbelt <[email protected]>
+
+sed 's!\([0-9a-f]*\) T \([a-z0-9_]*\)@@LINUX_4.15!.global \2\n.set \2,0x\1!' \
+| grep '^\.'
--

2.29.0.rc1.297.gfa9743e501-goog

--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20201024045046.3018271-1-palmerdabbelt%40google.com.

#3 lore raw
Quoted 28 lines · Palmer Dabbelt' via Clang Built Linux
Quoted 19 lines · Nick Desaulniers
Quoted 17 lines · Palmer Dabbelt' via Clang Built Linux

We were relying on GNU ld's ability to re-link executable files in order to extract our VDSO symbols. This behavior was deemed a bug as of binutils-2.35 (specifically the binutils-gdb commit a87e1817a4 ("Have the linker fail if any attempt to link in an executable is made."), but as that has been backported to at least Debian's binutils-2.34 in may manifest in other places.

The previous version of this was a bit of a mess: we were linking a static executable version of the VDSO, containing only a subset of the input symbols, which we then linked into the kernel. This worked, but certainly wasn't a supported path through the toolchain. Instead this new version parses the textual output of nm to produce a symbol table. Both rely on near-zero addresses being linkable, but as we rely on weak undefined symbols being linkable elsewhere I don't view this as a major issue.

Quoted 7 lines
Quoted 4 lines
Fixes e2c0cdfba7f6 ("RISC-V: User-facing API")

Any way to improve the error message if/when this fails?
https://travis-ci.com/github/ClangBuiltLinux/continuous-integration/jobs/407165683

Quoted 6 lines

Probably, but I can't get that command to actually run this stuff. I tried pulling the commands, but I'm getting some weirdness

$ rm -f arch/riscv/kernel/vdso/vdso-syms.S
$ make ARCH=riscv defconfig
$ make -j2 AR=llvm-ar 'CC=clang' 'HOSTCC=clang' HOSTLD=ld KCFLAGS=-Wno-implicit-fallthrough LD=riscv64-linux-gnu-ld LLVM_IAS=1 NM=llvm-nm OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump OBJSIZE=llvm-size READELF=llvm-readelf STRIP=llvm-strip ARCH=riscv Image

This command is simpler and reproduces it for me locally on next-20201105.

$ make -skj"$(nproc)" ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- LD=riscv64-linux-gnu-ld LLVM=1 distclean defconfig Image
clang-12: warning: argument unused during compilation: '-no-pie' [-Wunused-command-line-argument]
make[4]: *** [arch/riscv/kernel/vdso/Makefile:53: arch/riscv/kernel/vdso/vdso-syms.S] Error 1
make[4]: *** Deleting file 'arch/riscv/kernel/vdso/vdso-syms.S'
make[4]: Target '__build' not remade because of errors.
make[3]: *** [scripts/Makefile.build:500: arch/riscv/kernel/vdso] Error 2
make[3]: Target '__build' not remade because of errors.
make[2]: *** [scripts/Makefile.build:500: arch/riscv/kernel] Error 2
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1797: arch/riscv] Error 2
make[1]: Target 'Image' not remade because of errors.
make: *** [Makefile:335: __build_one_by_one] Error 2
make: Target 'distclean' not remade because of errors.
make: Target 'defconfig' not remade because of errors.
make: Target 'Image' not remade because of errors.

<snip>

Quoted 47 lines
arch/riscv/kernel/vdso/.gitignore +0 -0
index 3a19def868ec..88206dd8b472 100644
@@ -2,3 +2,4 @@

vdso.lds
*.tmp
vdso-syms.S

arch/riscv/kernel/vdso/.gitignore +1 -0
+vdso-syms.nm
arch/riscv/kernel/vdso/Makefile +5 -2
index a8ecf102e09b..fe5c969a6bf4 100644
@@ -49,8 +49,11 @@ SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
 # We also create a special relocatable object that should mirror the symbol
 # table and layout of the linked DSO. With ld --just-symbols we can then
 # refer to these symbols in the kernel code rather than hand-coded addresses.
-$(obj)/vdso-syms.S: $(obj)/vdso.so FORCE
-	$(call if_changed,so2s)
+$(obj)/vdso-syms.nm: $(obj)/vdso.so
+	$(call if_changed,nm_d)
+
+$(obj)/vdso-syms.S: $(obj)/vdso-syms.nm
+	$(call if_changed,nm2s)
 
 # strip rule for the .so file

$(obj)/%.so: OBJCOPYFLAGS := -S

arch/riscv/kernel/vdso/Makefile +7 -3
@@ -68,9 +71,13 @@ quiet_cmd_vdsold = VDSOLD  $@
                           $(patsubst %, -G __vdso_%, $(vdso-syms)) [email protected] $@ && \
                   rm [email protected]
 
-# Extracts
-quiet_cmd_so2s = SO2S    $@
-      cmd_so2s = $(NM) -D $< | $(srctree)/$(src)/so2s.sh > $@
+# Extracts symbol offsets from the VDSO, converting them into an assembly file
+# that contains the same symbols at the same offsets.
+quiet_cmd_nm_d = NM -D   $@
+      cmd_nm_d = $(NM) -D $< > $@
+
+quiet_cmd_nm2s = SYMS2S  $@
+      cmd_nm2s = cat $< | $(srctree)/$(src)/so2s.sh > $@
 
 # install commands for the unstripped file

quiet_cmd_vdso_install = INSTALL $@

For reference, here's the output of nmo for me:

Quoted 7 lines
$ cat arch/riscv/kernel/vdso/vdso-syms.nm 0000000000000000 A LINUX_4.15
00000000000009e0 T __vdso_clock_getres@@LINUX_4.15
000000000000080a T __vdso_clock_gettime@@LINUX_4.15
0000000000000a48 T __vdso_flush_icache@@LINUX_4.15
0000000000000a3c T __vdso_getcpu@@LINUX_4.15
0000000000000916 T __vdso_gettimeofday@@LINUX_4.15
0000000000000800 T __vdso_rt_sigreturn@@LINUX_4.15

This diff does not solve the issue for me with the above command.

Cheers,
Nathan

#4 lore raw
Quoted 103 lines · On Thu, 05 Nov 2020 20:59:02 PST (-0800), [email protected] wrote:
Quoted 28 lines · Palmer Dabbelt' via Clang Built Linux
Quoted 19 lines · Nick Desaulniers
Quoted 17 lines · Palmer Dabbelt' via Clang Built Linux

We were relying on GNU ld's ability to re-link executable files in order to extract our VDSO symbols. This behavior was deemed a bug as of binutils-2.35 (specifically the binutils-gdb commit a87e1817a4 ("Have the linker fail if any attempt to link in an executable is made."), but as that has been backported to at least Debian's binutils-2.34 in may manifest in other places.

The previous version of this was a bit of a mess: we were linking a static executable version of the VDSO, containing only a subset of the input symbols, which we then linked into the kernel. This worked, but certainly wasn't a supported path through the toolchain. Instead this new version parses the textual output of nm to produce a symbol table. Both rely on near-zero addresses being linkable, but as we rely on weak undefined symbols being linkable elsewhere I don't view this as a major issue.

Quoted 7 lines
Quoted 4 lines
Fixes e2c0cdfba7f6 ("RISC-V: User-facing API")

Any way to improve the error message if/when this fails?
https://travis-ci.com/github/ClangBuiltLinux/continuous-integration/jobs/407165683

Quoted 6 lines

Probably, but I can't get that command to actually run this stuff. I tried pulling the commands, but I'm getting some weirdness

$ rm -f arch/riscv/kernel/vdso/vdso-syms.S
$ make ARCH=riscv defconfig
$ make -j2 AR=llvm-ar 'CC=clang' 'HOSTCC=clang' HOSTLD=ld KCFLAGS=-Wno-implicit-fallthrough LD=riscv64-linux-gnu-ld LLVM_IAS=1 NM=llvm-nm OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump OBJSIZE=llvm-size READELF=llvm-readelf STRIP=llvm-strip ARCH=riscv Image

This command is simpler and reproduces it for me locally on next-20201105.

$ make -skj"$(nproc)" ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- LD=riscv64-linux-gnu-ld LLVM=1 distclean defconfig Image
clang-12: warning: argument unused during compilation: '-no-pie' [-Wunused-command-line-argument]
make[4]: *** [arch/riscv/kernel/vdso/Makefile:53: arch/riscv/kernel/vdso/vdso-syms.S] Error 1
make[4]: *** Deleting file 'arch/riscv/kernel/vdso/vdso-syms.S'
make[4]: Target '__build' not remade because of errors.
make[3]: *** [scripts/Makefile.build:500: arch/riscv/kernel/vdso] Error 2
make[3]: Target '__build' not remade because of errors.
make[2]: *** [scripts/Makefile.build:500: arch/riscv/kernel] Error 2
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1797: arch/riscv] Error 2
make[1]: Target 'Image' not remade because of errors.
make: *** [Makefile:335: __build_one_by_one] Error 2
make: Target 'distclean' not remade because of errors.
make: Target 'defconfig' not remade because of errors.
make: Target 'Image' not remade because of errors.

<snip>

Quoted 47 lines
arch/riscv/kernel/vdso/.gitignore +0 -0
index 3a19def868ec..88206dd8b472 100644
@@ -2,3 +2,4 @@

vdso.lds
*.tmp
vdso-syms.S

arch/riscv/kernel/vdso/.gitignore +1 -0
+vdso-syms.nm
arch/riscv/kernel/vdso/Makefile +5 -2
index a8ecf102e09b..fe5c969a6bf4 100644
@@ -49,8 +49,11 @@ SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
 # We also create a special relocatable object that should mirror the symbol
 # table and layout of the linked DSO. With ld --just-symbols we can then
 # refer to these symbols in the kernel code rather than hand-coded addresses.
-$(obj)/vdso-syms.S: $(obj)/vdso.so FORCE
-	$(call if_changed,so2s)
+$(obj)/vdso-syms.nm: $(obj)/vdso.so
+	$(call if_changed,nm_d)
+
+$(obj)/vdso-syms.S: $(obj)/vdso-syms.nm
+	$(call if_changed,nm2s)
 
 # strip rule for the .so file

$(obj)/%.so: OBJCOPYFLAGS := -S

arch/riscv/kernel/vdso/Makefile +7 -3
@@ -68,9 +71,13 @@ quiet_cmd_vdsold = VDSOLD  $@
                           $(patsubst %, -G __vdso_%, $(vdso-syms)) [email protected] $@ && \
                   rm [email protected]
 
-# Extracts
-quiet_cmd_so2s = SO2S    $@
-      cmd_so2s = $(NM) -D $< | $(srctree)/$(src)/so2s.sh > $@
+# Extracts symbol offsets from the VDSO, converting them into an assembly file
+# that contains the same symbols at the same offsets.
+quiet_cmd_nm_d = NM -D   $@
+      cmd_nm_d = $(NM) -D $< > $@
+
+quiet_cmd_nm2s = SYMS2S  $@
+      cmd_nm2s = cat $< | $(srctree)/$(src)/so2s.sh > $@
 
 # install commands for the unstripped file

quiet_cmd_vdso_install = INSTALL $@

For reference, here's the output of nmo for me:

Quoted 9 lines
Quoted 7 lines
$ cat arch/riscv/kernel/vdso/vdso-syms.nm 0000000000000000 A LINUX_4.15
00000000000009e0 T __vdso_clock_getres@@LINUX_4.15
000000000000080a T __vdso_clock_gettime@@LINUX_4.15
0000000000000a48 T __vdso_flush_icache@@LINUX_4.15
0000000000000a3c T __vdso_getcpu@@LINUX_4.15
0000000000000916 T __vdso_gettimeofday@@LINUX_4.15
0000000000000800 T __vdso_rt_sigreturn@@LINUX_4.15

This diff does not solve the issue for me with the above command.

It wasn't really meant to solve anything, just split the commands up a touch more so we could see what's going on.

I just installed Debian's toolchain, which is LLVM 9. IIRC that's pretty ancient WRT RISC-V, so my guess is that it's just a long way from building Linux. Looks like llvm-nm on my system doesn't put the @@LINUX_4.15 after the symbol names, I think this should do it?

arch/riscv/kernel/vdso/so2s.sh +0 -0
index 3c5b43207658..e64cb6d9440e 100755
@@ -2,5 +2,5 @@

# SPDX-License-Identifier: GPL-2.0+
# Copyright 2020 Palmer Dabbelt <[email protected]>

-sed 's!\([0-9a-f]*\) T \([a-z0-9_]*\)@@LINUX_4.15!.global \2\n.set \2,0x\1!' \
+sed 's!\([0-9a-f]*\) T \([a-z0-9_]*\)\(@@LINUX_4.15\)*!.global \2\n.set \2,0x\1!' \
| grep '^\.'

It works for me, at least for this specific problem.

$ llvm-nm-9 -D arch/riscv/kernel/vdso/vdso.so | ./arch/riscv/kernel/vdso/so2s.sh .global __vdso_clock_getres
.set __vdso_clock_getres,0x00000000000009e0
.global __vdso_clock_gettime
.set __vdso_clock_gettime,0x000000000000080a
.global __vdso_flush_icache
.set __vdso_flush_icache,0x0000000000000a48
.global __vdso_getcpu
.set __vdso_getcpu,0x0000000000000a3c
.global __vdso_gettimeofday
.set __vdso_gettimeofday,0x0000000000000916
.global __vdso_rt_sigreturn
.set __vdso_rt_sigreturn,0x0000000000000800
$ nm -D arch/riscv/kernel/vdso/vdso.so | ./arch/riscv/kernel/vdso/so2s.sh .global __vdso_clock_getres
.set __vdso_clock_getres,0x00000000000009e0
.global __vdso_clock_gettime
.set __vdso_clock_gettime,0x000000000000080a
.global __vdso_flush_icache
.set __vdso_flush_icache,0x0000000000000a48
.global __vdso_getcpu
.set __vdso_getcpu,0x0000000000000a3c
.global __vdso_gettimeofday
.set __vdso_gettimeofday,0x0000000000000916
.global __vdso_rt_sigreturn
.set __vdso_rt_sigreturn,0x0000000000000800
Quoted 3 lines

Cheers,
Nathan

#5 lore raw
Quoted 134 lines · Palmer Dabbelt
Quoted 103 lines · On Thu, 05 Nov 2020 20:59:02 PST (-0800), [email protected] wrote:
Quoted 28 lines · Palmer Dabbelt' via Clang Built Linux
Quoted 19 lines · Nick Desaulniers
Quoted 17 lines · Palmer Dabbelt' via Clang Built Linux

We were relying on GNU ld's ability to re-link executable files in order to extract our VDSO symbols. This behavior was deemed a bug as of binutils-2.35 (specifically the binutils-gdb commit a87e1817a4 ("Have the linker fail if any attempt to link in an executable is made."), but as that has been backported to at least Debian's binutils-2.34 in may manifest in other places.

The previous version of this was a bit of a mess: we were linking a static executable version of the VDSO, containing only a subset of the input symbols, which we then linked into the kernel. This worked, but certainly wasn't a supported path through the toolchain. Instead this new version parses the textual output of nm to produce a symbol table. Both rely on near-zero addresses being linkable, but as we rely on weak undefined symbols being linkable elsewhere I don't view this as a major issue.

Quoted 7 lines
Quoted 4 lines
Fixes e2c0cdfba7f6 ("RISC-V: User-facing API")

Any way to improve the error message if/when this fails?
https://travis-ci.com/github/ClangBuiltLinux/continuous-integration/jobs/407165683

Quoted 6 lines

Probably, but I can't get that command to actually run this stuff. I tried pulling the commands, but I'm getting some weirdness

$ rm -f arch/riscv/kernel/vdso/vdso-syms.S
$ make ARCH=riscv defconfig
$ make -j2 AR=llvm-ar 'CC=clang' 'HOSTCC=clang' HOSTLD=ld KCFLAGS=-Wno-implicit-fallthrough LD=riscv64-linux-gnu-ld LLVM_IAS=1 NM=llvm-nm OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump OBJSIZE=llvm-size READELF=llvm-readelf STRIP=llvm-strip ARCH=riscv Image

This command is simpler and reproduces it for me locally on next-20201105.

$ make -skj"$(nproc)" ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- LD=riscv64-linux-gnu-ld LLVM=1 distclean defconfig Image
clang-12: warning: argument unused during compilation: '-no-pie' [-Wunused-command-line-argument]
make[4]: *** [arch/riscv/kernel/vdso/Makefile:53: arch/riscv/kernel/vdso/vdso-syms.S] Error 1
make[4]: *** Deleting file 'arch/riscv/kernel/vdso/vdso-syms.S'
make[4]: Target '__build' not remade because of errors.
make[3]: *** [scripts/Makefile.build:500: arch/riscv/kernel/vdso] Error 2
make[3]: Target '__build' not remade because of errors.
make[2]: *** [scripts/Makefile.build:500: arch/riscv/kernel] Error 2
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1797: arch/riscv] Error 2
make[1]: Target 'Image' not remade because of errors.
make: *** [Makefile:335: __build_one_by_one] Error 2
make: Target 'distclean' not remade because of errors.
make: Target 'defconfig' not remade because of errors.
make: Target 'Image' not remade because of errors.

<snip>

Quoted 47 lines
arch/riscv/kernel/vdso/.gitignore +0 -0
index 3a19def868ec..88206dd8b472 100644
@@ -2,3 +2,4 @@

vdso.lds
*.tmp
vdso-syms.S

arch/riscv/kernel/vdso/.gitignore +1 -0
+vdso-syms.nm
arch/riscv/kernel/vdso/Makefile +5 -2
index a8ecf102e09b..fe5c969a6bf4 100644
@@ -49,8 +49,11 @@ SYSCFLAGS_vdso.so.dbg = -shared -s -Wl,-soname=linux-vdso.so.1 \
 # We also create a special relocatable object that should mirror the symbol
 # table and layout of the linked DSO. With ld --just-symbols we can then
 # refer to these symbols in the kernel code rather than hand-coded addresses.
-$(obj)/vdso-syms.S: $(obj)/vdso.so FORCE
-	$(call if_changed,so2s)
+$(obj)/vdso-syms.nm: $(obj)/vdso.so
+	$(call if_changed,nm_d)
+
+$(obj)/vdso-syms.S: $(obj)/vdso-syms.nm
+	$(call if_changed,nm2s)
 
 # strip rule for the .so file

$(obj)/%.so: OBJCOPYFLAGS := -S

arch/riscv/kernel/vdso/Makefile +7 -3
@@ -68,9 +71,13 @@ quiet_cmd_vdsold = VDSOLD  $@
                           $(patsubst %, -G __vdso_%, $(vdso-syms)) [email protected] $@ && \
                   rm [email protected]
 
-# Extracts
-quiet_cmd_so2s = SO2S    $@
-      cmd_so2s = $(NM) -D $< | $(srctree)/$(src)/so2s.sh > $@
+# Extracts symbol offsets from the VDSO, converting them into an assembly file
+# that contains the same symbols at the same offsets.
+quiet_cmd_nm_d = NM -D   $@
+      cmd_nm_d = $(NM) -D $< > $@
+
+quiet_cmd_nm2s = SYMS2S  $@
+      cmd_nm2s = cat $< | $(srctree)/$(src)/so2s.sh > $@
 
 # install commands for the unstripped file

quiet_cmd_vdso_install = INSTALL $@

For reference, here's the output of nmo for me:

Quoted 9 lines
Quoted 7 lines
$ cat arch/riscv/kernel/vdso/vdso-syms.nm 0000000000000000 A LINUX_4.15
00000000000009e0 T __vdso_clock_getres@@LINUX_4.15
000000000000080a T __vdso_clock_gettime@@LINUX_4.15
0000000000000a48 T __vdso_flush_icache@@LINUX_4.15
0000000000000a3c T __vdso_getcpu@@LINUX_4.15
0000000000000916 T __vdso_gettimeofday@@LINUX_4.15
0000000000000800 T __vdso_rt_sigreturn@@LINUX_4.15

This diff does not solve the issue for me with the above command.

It wasn't really meant to solve anything, just split the commands up a touch more so we could see what's going on.

I just installed Debian's toolchain, which is LLVM 9. IIRC that's pretty ancient WRT RISC-V, so my guess is that it's just a long way from building Linux. Looks like llvm-nm on my system doesn't put the @@LINUX_4.15 after the symbol names, I think this should do it?

arch/riscv/kernel/vdso/so2s.sh +1 -1
index 3c5b43207658..e64cb6d9440e 100755
@@ -2,5 +2,5 @@
 # SPDX-License-Identifier: GPL-2.0+
 # Copyright 2020 Palmer Dabbelt <[email protected]>
 
-sed 's!\([0-9a-f]*\) T \([a-z0-9_]*\)@@LINUX_4.15!.global \2\n.set \2,0x\1!' \
+sed 's!\([0-9a-f]*\) T \([a-z0-9_]*\)\(@@LINUX_4.15\)*!.global \2\n.set \2,0x\1!' \

| grep '^\.'

Quoted 1 line

It works for me, at least for this specific problem.

Same here. defconfig minus CONFIG_EFI builds and boots in QEMU with that diff (plus LLVM_IAS=1 in the make command, I forgot that earlier).

Quoted 30 lines
$ llvm-nm-9 -D arch/riscv/kernel/vdso/vdso.so |
./arch/riscv/kernel/vdso/so2s.sh .global __vdso_clock_getres
.set __vdso_clock_getres,0x00000000000009e0
.global __vdso_clock_gettime
.set __vdso_clock_gettime,0x000000000000080a
.global __vdso_flush_icache
.set __vdso_flush_icache,0x0000000000000a48
.global __vdso_getcpu
.set __vdso_getcpu,0x0000000000000a3c
.global __vdso_gettimeofday
.set __vdso_gettimeofday,0x0000000000000916
.global __vdso_rt_sigreturn
.set __vdso_rt_sigreturn,0x0000000000000800
$ nm -D arch/riscv/kernel/vdso/vdso.so | ./arch/riscv/kernel/vdso/so2s.sh
.global __vdso_clock_getres
.set __vdso_clock_getres,0x00000000000009e0
.global __vdso_clock_gettime
.set __vdso_clock_gettime,0x000000000000080a
.global __vdso_flush_icache
.set __vdso_flush_icache,0x0000000000000a48
.global __vdso_getcpu
.set __vdso_getcpu,0x0000000000000a3c
.global __vdso_gettimeofday
.set __vdso_gettimeofday,0x0000000000000916
.global __vdso_rt_sigreturn
.set __vdso_rt_sigreturn,0x0000000000000800
Quoted 3 lines

Cheers,
Nathan

#6 lore raw
In reply to Palmer Dabbelt
Quoted 3 lines · Palmer Dabbelt

I just installed Debian's toolchain, which is LLVM 9. IIRC that's pretty ancient WRT RISC-V, so my guess is that it's just a long way from building Linux.

Hello Palmer,

I haven't used them myself yet, but Debian has clang-11/llvm-11 packages, although only in the "experimental" suite which isn't enabled by default.

You can install them by adding a corresponding entry to /etc/apt/sources.list, such as

deb http://deb.debian.org/debian-ports/ experimental main (on a riscv64 system)

respectively

deb http://deb.debian.org/debian/ experimental main (on an amd64 system)

and running

sudo apt-get update && sudo apt-get -t experimental clang llvm

Regards,
Karsten

Signature · 3 lines
Hiermit widerspreche ich ausdrücklich der Nutzung sowie der Weitergabe
meiner personenbezogenen Daten für Zwecke der Werbung sowie der Markt-
oder Meinungsforschung.