Skip to contentSkip to conversation

add an elfnote with type BUILD_COMPILER_LTO_INFO

PATCH In review Yonghong Song 2 participants 5 messages Apr 1, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover add an elfnote with type BUILD_COMPILER_LTO_INFO

Currently, clang LTO built vmlinux won't work with pahole. LTO introduced cross-cu dwarf tag references and broke current pahole model which handles one cu as a time. The solution is to merge all cu's as one pahole cu as in [1]. We would like to do this merging only if cross-cu dwarf references happens. The LTO build mode is a pretty good indication for that.

In earlier version of this patch ([2]), clang flag -grecord-gcc-switches is proposed to add to compilation flags so pahole could detect "-flto" and then merging cu's. This will increate the binary size of 1% without LTO though.

Arnaldo suggested to use a note to indicate the vmlinux is built with LTO. Such a cheap way to get whether the vmlinux is built with LTO or not helps pahole but is also useful for tracing as LTO may inline/delete/demote global functions, promote static functions, etc.

This patch set added an elfnote with type BUILD_COMPILER_LTO_INFO. The owner of the note is "Linux". Patch #1 did some refactoring and Patch #2 added the elfnote with BUILD_COMPILER_LTO_INFO to indicate whether vmlinux is built with LTO or not.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/

Yonghong Song (2):
kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO

 include/linux/build-salt.h | 2 --
 include/linux/compiler.h   | 8 ++++++++
 include/linux/elfnote.h    | 6 ++++++
 init/version.c             | 2 ++
 scripts/mod/modpost.c      | 1 +
 5 files changed, 17 insertions(+), 2 deletions(-)
Signature · 1 line
2.30.2
#1 lore raw

PATCH v3 1/2 kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h

Move LINUX_ELFNOTE_BUILD_SALT to elfnote.h so we have a central place for all types with "Linux" owner. This will make add more types easier.

---
 include/linux/build-salt.h | 2 --
 include/linux/elfnote.h    | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)
include/linux/build-salt.h +0 -2
index bb007bd05e7a..6ab1db1d7f44 100644
@@ -3,8 +3,6 @@
 
 #include <linux/elfnote.h>
 
-#define LINUX_ELFNOTE_BUILD_SALT       0x100
-
 #ifdef __ASSEMBLER__
 
 #define BUILD_SALT \
include/linux/elfnote.h +5 -0
index 69b136e4dd2b..04af7ac40b1a 100644
@@ -96,4 +96,9 @@
 #define ELFNOTE64(name, type, desc) ELFNOTE(64, name, type, desc)
 #endif	/* __ASSEMBLER__ */
 
+/*
+ * The types for "Linux" owned notes.
+ */
+#define LINUX_ELFNOTE_BUILD_SALT	0x100
+
 #endif /* _LINUX_ELFNOTE_H */
Signature · 1 line
2.30.2
#2 lore raw

PATCH v3 2/2 kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO

In reply to Yonghong Song

Currently, clang LTO built vmlinux won't work with pahole. LTO introduced cross-cu dwarf tag references and broke current pahole model which handles one cu as a time. The solution is to merge all cu's as one pahole cu as in [1]. We would like to do this merging only if cross-cu dwarf references happens. The LTO build mode is a pretty good indication for that.

In earlier version of this patch ([2]), clang flag -grecord-gcc-switches is proposed to add to compilation flags so pahole could detect "-flto" and then merging cu's. This will increate the binary size of 1% without LTO though.

Arnaldo suggested to use a note to indicate the vmlinux is built with LTO. Such a cheap way to get whether the vmlinux is built with LTO or not helps pahole but is also useful for tracing as LTO may inline/delete/demote global functions, promote static functions, etc.

So this patch added an elfnote with type BUILD_COMPILER_LTO_INFO. The owner of the note is "Linux".

With gcc 8.4.1 and clang trunk, without LTO, I got

  $ readelf -n vmlinux
  Displaying notes found in: .notes
    Owner                Data size        Description
  ...
    Linux                0x00000004       func
     description data: 00 00 00 00
  ...
With "readelf -x ".notes" vmlinux", I can verify the above "func"
with type code 0x101.

With clang thin-LTO, I got the same as above except the following:

     description data: 01 00 00 00

which indicates the vmlinux is built with LTO.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/compiler.h | 8 ++++++++
 include/linux/elfnote.h  | 1 +
 init/version.c           | 2 ++
 scripts/mod/modpost.c    | 1 +
 4 files changed, 12 insertions(+)
include/linux/compiler.h +8 -0
index df5b405e6305..b92930877277 100644
@@ -245,6 +245,14 @@ static inline void *offset_to_ptr(const int *off)
  */
 #define prevent_tail_call_optimization()	mb()
 
+#include <linux/elfnote.h>
+
+#ifdef CONFIG_LTO
+#define BUILD_COMPILER_LTO_INFO ELFNOTE32("Linux", LINUX_ELFNOTE_BUILD_LTO, 1)
+#else
+#define BUILD_COMPILER_LTO_INFO ELFNOTE32("Linux", LINUX_ELFNOTE_BUILD_LTO, 0)
+#endif
+
 #include <asm/rwonce.h>
 
 #endif /* __LINUX_COMPILER_H */
include/linux/elfnote.h +1 -0
index 04af7ac40b1a..f5ec2b50ab7d 100644
@@ -100,5 +100,6 @@
  * The types for "Linux" owned notes.
  */
 #define LINUX_ELFNOTE_BUILD_SALT	0x100
+#define LINUX_ELFNOTE_BUILD_LTO		0x101
 
 #endif /* _LINUX_ELFNOTE_H */
init/version.c +2 -0
index 92afc782b043..a4f74b06fe78 100644
@@ -9,6 +9,7 @@
 
 #include <generated/compile.h>
 #include <linux/build-salt.h>
+#include <linux/compiler.h>
 #include <linux/export.h>
 #include <linux/uts.h>
 #include <linux/utsname.h>
@@ -45,3 +46,4 @@ const char linux_proc_banner[] =
 	" (" LINUX_COMPILER ") %s\n";
 
 BUILD_SALT;
+BUILD_COMPILER_LTO_INFO;
scripts/mod/modpost.c +1 -0
index 24725e50c7b4..713c0d5d5525 100644
@@ -2195,6 +2195,7 @@ static void add_header(struct buffer *b, struct module *mod)
 	buf_printf(b, "#include <linux/compiler.h>\n");
 	buf_printf(b, "\n");
 	buf_printf(b, "BUILD_SALT;\n");
+	buf_printf(b, "BUILD_COMPILER_LTO_INFO;\n");
 	buf_printf(b, "\n");
 	buf_printf(b, "MODULE_INFO(vermagic, VERMAGIC_STRING);\n");
 	buf_printf(b, "MODULE_INFO(name, KBUILD_MODNAME);\n");
Signature · 1 line
2.30.2
#3 lore raw

Re: [PATCH kbuild v3 2/2] kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO

Quoted 54 lines · Yonghong Song

Currently, clang LTO built vmlinux won't work with pahole. LTO introduced cross-cu dwarf tag references and broke current pahole model which handles one cu as a time. The solution is to merge all cu's as one pahole cu as in [1]. We would like to do this merging only if cross-cu dwarf references happens. The LTO build mode is a pretty good indication for that.

In earlier version of this patch ([2]), clang flag -grecord-gcc-switches is proposed to add to compilation flags so pahole could detect "-flto" and then merging cu's. This will increate the binary size of 1% without LTO though.

Arnaldo suggested to use a note to indicate the vmlinux is built with LTO. Such a cheap way to get whether the vmlinux is built with LTO or not helps pahole but is also useful for tracing as LTO may inline/delete/demote global functions, promote static functions, etc.

So this patch added an elfnote with type BUILD_COMPILER_LTO_INFO. The owner of the note is "Linux".

With gcc 8.4.1 and clang trunk, without LTO, I got

  $ readelf -n vmlinux
  Displaying notes found in: .notes
    Owner                Data size        Description
  ...
    Linux                0x00000004       func
     description data: 00 00 00 00
  ...
With "readelf -x ".notes" vmlinux", I can verify the above "func"
with type code 0x101.

With clang thin-LTO, I got the same as above except the following:

     description data: 01 00 00 00

which indicates the vmlinux is built with LTO.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/compiler.h | 8 ++++++++
 include/linux/elfnote.h  | 1 +
 init/version.c           | 2 ++
 scripts/mod/modpost.c    | 1 +
 4 files changed, 12 insertions(+)
include/linux/compiler.h +0 -0
index df5b405e6305..b92930877277 100644

include/linux/compiler.h static inline void *offset_to_ptr(const int *off) · line 254

Quoted 10 lines
include/linux/compiler.h +7 -0
@@ -245,6 +245,14 @@ static inline void *offset_to_ptr(const int *off)
  */
 #define prevent_tail_call_optimization()       mb()
 
+#include <linux/elfnote.h>
+
+#ifdef CONFIG_LTO
+#define BUILD_COMPILER_LTO_INFO ELFNOTE32("Linux", LINUX_ELFNOTE_BUILD_LTO, 1)
+#else
+#define BUILD_COMPILER_LTO_INFO ELFNOTE32("Linux", LINUX_ELFNOTE_BUILD_LTO, 0)
+#endif

With this approach BUILD_COMPILER_LTO_INFO won't be available #ifdef __ASSEMBLER__; we don't need it today, and perhaps YAGNI, but I think I prefer how include/linux/build-salt.h defines LINUX_ELFNOTE_BUILD_SALT and keeps it isolated there. Similarly, I think it would be better to create a new header, say include/linux/elfnote-lto.h that is basically a copy of include/linux/build-salt.h, but with the relevant defines replaced with the LTO identifiers you add above. Then init/version.c and scripts/mod/modpost.c can include include/linux/elfnote-lto.h and you don't have to touch include/linux/build-salt.h and we can keep the elfnote "types" isolated to their respective headers (otherwise this approach reduces the usefulness of include/linux/build-salt.h even existing, IMO. Feels like it should just be merged into include/linux/elfnote.h entirely at that point).

But, this is a much nicer approach! I forgot that elf notes were a thing!

Quoted 47 lines
include/linux/compiler.h +1 -0
+
 #include <asm/rwonce.h>
 
 #endif /* __LINUX_COMPILER_H */
include/linux/elfnote.h +1 -0
index 04af7ac40b1a..f5ec2b50ab7d 100644
@@ -100,5 +100,6 @@
  * The types for "Linux" owned notes.
  */
 #define LINUX_ELFNOTE_BUILD_SALT       0x100
+#define LINUX_ELFNOTE_BUILD_LTO                0x101
 
 #endif /* _LINUX_ELFNOTE_H */
init/version.c +2 -0
index 92afc782b043..a4f74b06fe78 100644
@@ -9,6 +9,7 @@
 
 #include <generated/compile.h>
 #include <linux/build-salt.h>
+#include <linux/compiler.h>
 #include <linux/export.h>
 #include <linux/uts.h>
 #include <linux/utsname.h>
@@ -45,3 +46,4 @@ const char linux_proc_banner[] =
        " (" LINUX_COMPILER ") %s\n";
 
 BUILD_SALT;
+BUILD_COMPILER_LTO_INFO;
scripts/mod/modpost.c +1 -1
index 24725e50c7b4..713c0d5d5525 100644
@@ -2195,6 +2195,7 @@ static void add_header(struct buffer *b, struct module *mod)
        buf_printf(b, "#include <linux/compiler.h>\n");
        buf_printf(b, "\n");
        buf_printf(b, "BUILD_SALT;\n");
+       buf_printf(b, "BUILD_COMPILER_LTO_INFO;\n");
        buf_printf(b, "\n");
        buf_printf(b, "MODULE_INFO(vermagic, VERMAGIC_STRING);\n");
        buf_printf(b, "MODULE_INFO(name, KBUILD_MODNAME);\n");
--

2.30.2

Signature · 2 lines
Thanks,
~Nick Desaulniers
#4 lore raw

Re: [PATCH kbuild v3 2/2] kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO

Quoted 58 lines · Nick Desaulniers
Quoted 57 lines · Yonghong Song

Currently, clang LTO built vmlinux won't work with pahole. LTO introduced cross-cu dwarf tag references and broke current pahole model which handles one cu as a time. The solution is to merge all cu's as one pahole cu as in [1]. We would like to do this merging only if cross-cu dwarf references happens. The LTO build mode is a pretty good indication for that.

In earlier version of this patch ([2]), clang flag -grecord-gcc-switches is proposed to add to compilation flags so pahole could detect "-flto" and then merging cu's. This will increate the binary size of 1% without LTO though.

Arnaldo suggested to use a note to indicate the vmlinux is built with LTO. Such a cheap way to get whether the vmlinux is built with LTO or not helps pahole but is also useful for tracing as LTO may inline/delete/demote global functions, promote static functions, etc.

So this patch added an elfnote with type BUILD_COMPILER_LTO_INFO. The owner of the note is "Linux".

With gcc 8.4.1 and clang trunk, without LTO, I got

   $ readelf -n vmlinux
   Displaying notes found in: .notes
     Owner                Data size        Description
   ...
     Linux                0x00000004       func
      description data: 00 00 00 00
   ...
With "readelf -x ".notes" vmlinux", I can verify the above "func"
with type code 0x101.

With clang thin-LTO, I got the same as above except the following:

      description data: 01 00 00 00

which indicates the vmlinux is built with LTO.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/

---
  include/linux/compiler.h | 8 ++++++++
  include/linux/elfnote.h  | 1 +
  init/version.c           | 2 ++
  scripts/mod/modpost.c    | 1 +
  4 files changed, 12 insertions(+)
include/linux/compiler.h +0 -0
index df5b405e6305..b92930877277 100644
@@ -245,6 +245,14 @@ static inline void *offset_to_ptr(const int *off)
   */
  #define prevent_tail_call_optimization()       mb()
 
Quoted 10 lines

include/linux/compiler.h static inline void *offset_to_ptr(const int *off) · line 254

Quoted 7 lines
include/linux/compiler.h +7 -0
+#include <linux/elfnote.h>
+
+#ifdef CONFIG_LTO
+#define BUILD_COMPILER_LTO_INFO ELFNOTE32("Linux", LINUX_ELFNOTE_BUILD_LTO, 1)
+#else
+#define BUILD_COMPILER_LTO_INFO ELFNOTE32("Linux", LINUX_ELFNOTE_BUILD_LTO, 0)
+#endif

With this approach BUILD_COMPILER_LTO_INFO won't be available #ifdef __ASSEMBLER__; we don't need it today, and perhaps YAGNI, but I think

That is true. I didn't add it since I don't feel it. BUILD_SALT also added to vdso binary which I feel we don't need it today.

Quoted 5 lines

I prefer how include/linux/build-salt.h defines LINUX_ELFNOTE_BUILD_SALT and keeps it isolated there. Similarly, I think it would be better to create a new header, say include/linux/elfnote-lto.h that is basically a copy of include/linux/build-salt.h, but with the relevant defines replaced

Having a separate header like elfnote-lto.h sounds okay. Originally I am reluctant to add a new header file, but maybe a new header file is much cleaner than otherwise.

Quoted 7 lines

with the LTO identifiers you add above. Then init/version.c and scripts/mod/modpost.c can include include/linux/elfnote-lto.h and you don't have to touch include/linux/build-salt.h and we can keep the elfnote "types" isolated to their respective headers (otherwise this approach reduces the usefulness of include/linux/build-salt.h even existing, IMO. Feels like it should just be merged into include/linux/elfnote.h entirely at that point).

The only "drawback" is the type values are scattered in different files which I am not really comfortable with it. But with consistent naming convention, all values can be easily searched so we may not have issue at all.

Quoted 52 lines

But, this is a much nicer approach! I forgot that elf notes were a thing!

Quoted 47 lines
include/linux/compiler.h +1 -0
+
  #include <asm/rwonce.h>
 
  #endif /* __LINUX_COMPILER_H */
include/linux/elfnote.h +1 -0
index 04af7ac40b1a..f5ec2b50ab7d 100644
@@ -100,5 +100,6 @@
   * The types for "Linux" owned notes.
   */
  #define LINUX_ELFNOTE_BUILD_SALT       0x100
+#define LINUX_ELFNOTE_BUILD_LTO                0x101
 
  #endif /* _LINUX_ELFNOTE_H */
init/version.c +2 -0
index 92afc782b043..a4f74b06fe78 100644
@@ -9,6 +9,7 @@
 
  #include <generated/compile.h>
  #include <linux/build-salt.h>
+#include <linux/compiler.h>
  #include <linux/export.h>
  #include <linux/uts.h>
  #include <linux/utsname.h>
@@ -45,3 +46,4 @@ const char linux_proc_banner[] =
         " (" LINUX_COMPILER ") %s\n";
 
  BUILD_SALT;
+BUILD_COMPILER_LTO_INFO;
scripts/mod/modpost.c +1 -1
index 24725e50c7b4..713c0d5d5525 100644
@@ -2195,6 +2195,7 @@ static void add_header(struct buffer *b, struct module *mod)
         buf_printf(b, "#include <linux/compiler.h>\n");
         buf_printf(b, "\n");
         buf_printf(b, "BUILD_SALT;\n");
+       buf_printf(b, "BUILD_COMPILER_LTO_INFO;\n");
         buf_printf(b, "\n");
         buf_printf(b, "MODULE_INFO(vermagic, VERMAGIC_STRING);\n");
         buf_printf(b, "MODULE_INFO(name, KBUILD_MODNAME);\n");
--

2.30.2