Skip to contentSkip to conversation

dwarf_loader: improve cus__merging_cu()

PATCH In review Yonghong Song 8 participants 33 messages Apr 1, 2021 – Apr 6, 2021 BPF

lore
Share & tools
lore
#0 lore raw

cover dwarf_loader: improve cus__merging_cu()

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

 dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 21 deletions(-)
Signature · 1 line
2.30.2
#1 lore raw

PATCH 1/2 dwarf_loader: check .debug_abbrev for cross-cu references

Commit 39227909db3c checked compilation flags to see whether the binary is built with lto or not (-flto). Currently, for clang lto build, default setting won't put compilation flags in dwarf due to size concern.

David Blaikie suggested in [1] to scan through .debug_abbrev for DW_FORM_ref_addr which should be most faster than scanning through cu's. This patch implemented this suggestion and replaced the previous compilation flag matching approach. Indeed, it seems that the overhead for this approach is indeed managable.

I did some change to measure the overhead of cus_merging_cu():
@@ -2650,7 +2652,15 @@ static int cus__load_module(struct cus *cus, struct conf_load *conf,

                  }
          }
  -       if (cus__merging_cu(dw)) {
  +       bool do_merging;
  +       struct timeval start, end;
  +       gettimeofday(&start, NULL);
  +       do_merging = cus__merging_cu(dw);
  +       gettimeofday(&end, NULL);
  +       fprintf(stderr, "%ld %ld -> %ld %ld\n", start.tv_sec, start.tv_usec,

+ end.tv_sec, end.tv_usec);
+
+ if (do_merging) {
res = cus__merge_and_process_cu(cus, conf, mod, dw, elf, filename,
build_id, build_id_len, type_cu ? &type_dcu : NULL);

For lto vmlinux, the cus__merging_cu() checking takes
130us over total "pahole -J vmlinux" time 65sec as the function bail out earlier due to detecting a merging cu condition.
For non-lto vmlinux, the cus__merging_cu() checking takes ~171368us over total pahole time 36sec, roughly 0.5% overhead.

[1] https://lore.kernel.org/bpf/[email protected]/

---
 dwarf_loader.c | 43 ++++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 19 deletions(-)
dwarf_loader.c +24 -19
index c1ca1a3..bd23751 100644
@@ -2503,35 +2503,40 @@ static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,
 
 static bool cus__merging_cu(Dwarf *dw)
 {
-	uint8_t pointer_size, offset_size;
 	Dwarf_Off off = 0, noff;
 	size_t cuhl;
-	int cnt = 0;
 
-	/*
-	 * Just checking the first cu is not enough.
-	 * In linux, some C files may have LTO is disabled, e.g.,
-	 *   e242db40be27  x86, vdso: disable LTO only for vDSO
-	 *   d2dcd3e37475  x86, cpu: disable LTO for cpu.c
-	 * Fortunately, disabling LTO for a particular file in a LTO build
-	 * is rather an exception. Iterating 5 cu's to check whether
-	 * LTO is used or not should be enough.
-	 */
-	while (dwarf_nextcu(dw, off, &noff, &cuhl, NULL, &pointer_size,
-			    &offset_size) == 0) {
+	while (dwarf_nextcu (dw, off, &noff, &cuhl, NULL, NULL, NULL) == 0) {
 		Dwarf_Die die_mem;
 		Dwarf_Die *cu_die = dwarf_offdie(dw, off + cuhl, &die_mem);
 
 		if (cu_die == NULL)
 			break;
 
-		if (++cnt > 5)
-			break;
+		Dwarf_Off offset = 0;
+		while (true) {
+			size_t length;
+			Dwarf_Abbrev *abbrev = dwarf_getabbrev (cu_die, offset, &length);
+			if (abbrev == NULL || abbrev == DWARF_END_ABBREV)
+				break;
 
-		const char *producer = attr_string(cu_die, DW_AT_producer);
-		if (strstr(producer, "clang version") != NULL &&
-		    strstr(producer, "-flto") != NULL)
-			return true;
+			size_t attrcnt;
+			if (dwarf_getattrcnt (abbrev, &attrcnt) != 0)
+				return false;
+
+			unsigned int attr_num, attr_form;
+			Dwarf_Off aboffset;
+			size_t j;
+			for (j = 0; j < attrcnt; ++j) {
+				if (dwarf_getabbrevattr (abbrev, j, &attr_num, &attr_form,
+							 &aboffset))
+					return false;
+				if (attr_form == DW_FORM_ref_addr)
+					return true;
+			}
+
+			offset += length;
+		}
 
 		off = noff;
 	}
Signature · 1 line
2.30.2
#2 lore raw

Re: [PATCH dwarves 1/2] dwarf_loader: check .debug_abbrev for cross-cu references

Quoted 33 lines · Yonghong Song

Commit 39227909db3c checked compilation flags to see whether the binary is built with lto or not (-flto). Currently, for clang lto build, default setting won't put compilation flags in dwarf due to size concern.

David Blaikie suggested in [1] to scan through .debug_abbrev for DW_FORM_ref_addr which should be most faster than scanning through cu's. This patch implemented this suggestion and replaced the previous compilation flag matching approach. Indeed, it seems that the overhead for this approach is indeed managable.

I did some change to measure the overhead of cus_merging_cu():
@@ -2650,7 +2652,15 @@ static int cus__load_module(struct cus *cus, struct conf_load *conf,

                  }
          }
  -       if (cus__merging_cu(dw)) {
  +       bool do_merging;
  +       struct timeval start, end;
  +       gettimeofday(&start, NULL);
  +       do_merging = cus__merging_cu(dw);
  +       gettimeofday(&end, NULL);
  +       fprintf(stderr, "%ld %ld -> %ld %ld\n", start.tv_sec, start.tv_usec,

+ end.tv_sec, end.tv_usec);
+
+ if (do_merging) {
res = cus__merge_and_process_cu(cus, conf, mod, dw, elf, filename,
build_id, build_id_len, type_cu ? &type_dcu : NULL);

Quoted 8 lines

For lto vmlinux, the cus__merging_cu() checking takes
130us over total "pahole -J vmlinux" time 65sec as the function bail out earlier due to detecting a merging cu condition.
For non-lto vmlinux, the cus__merging_cu() checking takes ~171368us over total pahole time 36sec, roughly 0.5% overhead.

[1] https://lore.kernel.org/bpf/[email protected]/

It might be a nice little touch to add:

Quoted 72 lines
---
 dwarf_loader.c | 43 ++++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 19 deletions(-)
dwarf_loader.c +24 -20
index c1ca1a3..bd23751 100644
@@ -2503,35 +2503,40 @@ static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,
 
 static bool cus__merging_cu(Dwarf *dw)
 {
-       uint8_t pointer_size, offset_size;
        Dwarf_Off off = 0, noff;
        size_t cuhl;
-       int cnt = 0;
 
-       /*
-        * Just checking the first cu is not enough.
-        * In linux, some C files may have LTO is disabled, e.g.,
-        *   e242db40be27  x86, vdso: disable LTO only for vDSO
-        *   d2dcd3e37475  x86, cpu: disable LTO for cpu.c
-        * Fortunately, disabling LTO for a particular file in a LTO build
-        * is rather an exception. Iterating 5 cu's to check whether
-        * LTO is used or not should be enough.
-        */
-       while (dwarf_nextcu(dw, off, &noff, &cuhl, NULL, &pointer_size,
-                           &offset_size) == 0) {
+       while (dwarf_nextcu (dw, off, &noff, &cuhl, NULL, NULL, NULL) == 0) {
                Dwarf_Die die_mem;
                Dwarf_Die *cu_die = dwarf_offdie(dw, off + cuhl, &die_mem);
 
                if (cu_die == NULL)
                        break;
 
-               if (++cnt > 5)
-                       break;
+               Dwarf_Off offset = 0;
+               while (true) {
+                       size_t length;
+                       Dwarf_Abbrev *abbrev = dwarf_getabbrev (cu_die, offset, &length);
+                       if (abbrev == NULL || abbrev == DWARF_END_ABBREV)
+                               break;
 
-               const char *producer = attr_string(cu_die, DW_AT_producer);
-               if (strstr(producer, "clang version") != NULL &&
-                   strstr(producer, "-flto") != NULL)
-                       return true;
+                       size_t attrcnt;
+                       if (dwarf_getattrcnt (abbrev, &attrcnt) != 0)
+                               return false;
+
+                       unsigned int attr_num, attr_form;
+                       Dwarf_Off aboffset;
+                       size_t j;
+                       for (j = 0; j < attrcnt; ++j) {
+                               if (dwarf_getabbrevattr (abbrev, j, &attr_num, &attr_form,
+                                                        &aboffset))
+                                       return false;
+                               if (attr_form == DW_FORM_ref_addr)
+                                       return true;
+                       }
+
+                       offset += length;
+               }
 
                off = noff;
        }
--

2.30.2

Signature · 2 lines
Thanks,
~Nick Desaulniers
#3 lore raw

Re: [PATCH dwarves 1/2] dwarf_loader: check .debug_abbrev for cross-cu references

Quoted 46 lines · Nick Desaulniers
Quoted 14 lines · Yonghong Song

Commit 39227909db3c checked compilation flags to see whether the binary is built with lto or not (-flto). Currently, for clang lto build, default setting won't put compilation flags in dwarf due to size concern.

David Blaikie suggested in [1] to scan through .debug_abbrev for DW_FORM_ref_addr which should be most faster than scanning through cu's. This patch implemented this suggestion and replaced the previous compilation flag matching approach. Indeed, it seems that the overhead for this approach is indeed managable.

Quoted 2 lines

I did some change to measure the overhead of cus_merging_cu():
@@ -2650,7 +2652,15 @@ static int cus__load_module(struct cus *cus,

struct conf_load *conf,

Quoted 10 lines
                  }
          }
  -       if (cus__merging_cu(dw)) {
  +       bool do_merging;
  +       struct timeval start, end;
  +       gettimeofday(&start, NULL);
  +       do_merging = cus__merging_cu(dw);
  +       gettimeofday(&end, NULL);
  +       fprintf(stderr, "%ld %ld -> %ld %ld\n", start.tv_sec,

start.tv_usec,

Quoted 4 lines

+ end.tv_sec, end.tv_usec);
+
+ if (do_merging) {
res = cus__merge_and_process_cu(cus, conf, mod, dw,

elf, filename,

Quoted 1 line

build_id,

build_id_len,

Quoted 1 line

type_cu ? &type_dcu

: NULL);

Quoted 3 lines

For lto vmlinux, the cus__merging_cu() checking takes 130us over total "pahole -J vmlinux" time 65sec as the function bail

out

Quoted 4 lines

earlier due to detecting a merging cu condition.
For non-lto vmlinux, the cus__merging_cu() checking takes ~171368us over total pahole time 36sec, roughly 0.5% overhead.

Quoted 6 lines
Quoted 2 lines

[1] https://lore.kernel.org/bpf/[email protected]/

It might be a nice little touch to add:

Sure, this is something that is becoming the norm, be it from patch submitters, or, that being somehow lost, by the maintainer.

I think this is not just fair, but documents what actually happened and encourage people to share ideas more freely and quickly.

I'll do it in this specific case.

if I failed to do so I'm the past, I'm sorry.

- Arnaldo

Quoted 82 lines

dwarf_loader.c static int cus__load_debug_types(struct cus

Quoted 11 lines
---
 dwarf_loader.c | 43 ++++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 19 deletions(-)
dwarf_loader.c +0 -0
index c1ca1a3..bd23751 100644
@@ -2503,35 +2503,40 @@ static int cus__load_debug_types(struct cus

*cus, struct conf_load *conf,

Quoted 4 lines
dwarf_loader.c +0 -1
 static bool cus__merging_cu(Dwarf *dw)
 {
-       uint8_t pointer_size, offset_size;

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 10 lines
dwarf_loader.c +0 -7
        Dwarf_Off off = 0, noff;
        size_t cuhl;
-       int cnt = 0;
 
-       /*
-        * Just checking the first cu is not enough.
-        * In linux, some C files may have LTO is disabled, e.g.,
-        *   e242db40be27  x86, vdso: disable LTO only for vDSO
-        *   d2dcd3e37475  x86, cpu: disable LTO for cpu.c
-        * Fortunately, disabling LTO for a particular file in a LTO

build

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 4 lines
dwarf_loader.c +0 -4
-        * is rather an exception. Iterating 5 cu's to check whether
-        * LTO is used or not should be enough.
-        */
-       while (dwarf_nextcu(dw, off, &noff, &cuhl, NULL,

&pointer_size,

dwarf_loader.c static int cus__load_debug_types(struct cus · line 2508

Quoted 2 lines
dwarf_loader.c +1 -1
-                           &offset_size) == 0) {
+       while (dwarf_nextcu (dw, off, &noff, &cuhl, NULL, NULL, NULL)

== 0) {

dwarf_loader.c static int cus__load_debug_types(struct cus · line 2510

Quoted 2 lines
dwarf_loader.c +0 -0
                Dwarf_Die die_mem;
                Dwarf_Die *cu_die = dwarf_offdie(dw, off + cuhl,

&die_mem);

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 10 lines
dwarf_loader.c +4 -2
                if (cu_die == NULL)
                        break;
 
-               if (++cnt > 5)
-                       break;
+               Dwarf_Off offset = 0;
+               while (true) {
+                       size_t length;
+                       Dwarf_Abbrev *abbrev = dwarf_getabbrev

(cu_die, offset, &length);

dwarf_loader.c static int cus__load_debug_types(struct cus · line 2518

Quoted 1 line
dwarf_loader.c +1 -0
+                       if (abbrev == NULL || abbrev ==

DWARF_END_ABBREV)

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 3 lines
dwarf_loader.c +1 -1
+                               break;
 
-               const char *producer = attr_string(cu_die,

DW_AT_producer);

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 12 lines
dwarf_loader.c +9 -3
-               if (strstr(producer, "clang version") != NULL &&
-                   strstr(producer, "-flto") != NULL)
-                       return true;
+                       size_t attrcnt;
+                       if (dwarf_getattrcnt (abbrev, &attrcnt) != 0)
+                               return false;
+
+                       unsigned int attr_num, attr_form;
+                       Dwarf_Off aboffset;
+                       size_t j;
+                       for (j = 0; j < attrcnt; ++j) {
+                               if (dwarf_getabbrevattr (abbrev, j,

&attr_num, &attr_form,

Quoted 14 lines
dwarf_loader.c +8 -1
+                                                        &aboffset))
+                                       return false;
+                               if (attr_form == DW_FORM_ref_addr)
+                                       return true;
+                       }
+
+                       offset += length;
+               }
 
                off = noff;
        }
--

2.30.2

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#4 lore raw

Re: [PATCH dwarves 1/2] dwarf_loader: check .debug_abbrev for cross-cu references

Quoted 54 lines · Arnaldo
Quoted 51 lines · Nick Desaulniers
Quoted 14 lines · Yonghong Song

Commit 39227909db3c checked compilation flags to see whether the binary is built with lto or not (-flto). Currently, for clang lto build, default setting won't put compilation flags in dwarf due to size concern.

David Blaikie suggested in [1] to scan through .debug_abbrev for DW_FORM_ref_addr which should be most faster than scanning through cu's. This patch implemented this suggestion and replaced the previous compilation flag matching approach. Indeed, it seems that the overhead for this approach is indeed managable.

Quoted 2 lines

I did some change to measure the overhead of cus_merging_cu():

   @@ -2650,7 +2652,15 @@ static int cus__load_module(struct cus *cus,

struct conf_load *conf,

Quoted 10 lines
                   }
           }
   -       if (cus__merging_cu(dw)) {
   +       bool do_merging;
   +       struct timeval start, end;
   +       gettimeofday(&start, NULL);
   +       do_merging = cus__merging_cu(dw);
   +       gettimeofday(&end, NULL);
   +       fprintf(stderr, "%ld %ld -> %ld %ld\n", start.tv_sec,

start.tv_usec,

Quoted 4 lines

+ end.tv_sec, end.tv_usec);
+
+ if (do_merging) {
res = cus__merge_and_process_cu(cus, conf, mod, dw,

elf, filename,

Quoted 1 line

build_id,

build_id_len,

Quoted 1 line

type_cu ? &type_dcu

: NULL);

Quoted 3 lines

For lto vmlinux, the cus__merging_cu() checking takes 130us over total "pahole -J vmlinux" time 65sec as the function bail

out

Quoted 6 lines

earlier due to detecting a merging cu condition.
For non-lto vmlinux, the cus__merging_cu() checking takes ~171368us over total pahole time 36sec, roughly 0.5% overhead.

[1] https://lore.kernel.org/bpf/[email protected]/

It might be a nice little touch to add:

Quoted 7 lines
Quoted 1 line

Sure, this is something that is becoming the norm, be it from patch submitters, or, that being somehow lost, by the maintainer.

I think this is not just fair, but documents what actually happened and encourage people to share ideas more freely and quickly.

I'll do it in this specific case.

Thanks, Arnaldo. Sometimes I indeed missed to add necessary tags like Suggested-by, Reported-by, etc, esp. if I want to push the patch out ASAP. Thanks for correcting this!

Quoted 87 lines

if I failed to do so I'm the past, I'm sorry.

- Arnaldo

Quoted 82 lines

dwarf_loader.c static int cus__load_debug_types(struct cus

Quoted 11 lines
---
  dwarf_loader.c | 43 ++++++++++++++++++++++++-------------------
  1 file changed, 24 insertions(+), 19 deletions(-)
dwarf_loader.c +0 -0
index c1ca1a3..bd23751 100644
@@ -2503,35 +2503,40 @@ static int cus__load_debug_types(struct cus

*cus, struct conf_load *conf,

Quoted 4 lines
dwarf_loader.c +0 -1
  static bool cus__merging_cu(Dwarf *dw)
  {
-       uint8_t pointer_size, offset_size;

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 10 lines
dwarf_loader.c +0 -7
         Dwarf_Off off = 0, noff;
         size_t cuhl;
-       int cnt = 0;
 
-       /*
-        * Just checking the first cu is not enough.
-        * In linux, some C files may have LTO is disabled, e.g.,
-        *   e242db40be27  x86, vdso: disable LTO only for vDSO
-        *   d2dcd3e37475  x86, cpu: disable LTO for cpu.c
-        * Fortunately, disabling LTO for a particular file in a LTO

build

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 4 lines
dwarf_loader.c +0 -4
-        * is rather an exception. Iterating 5 cu's to check whether
-        * LTO is used or not should be enough.
-        */
-       while (dwarf_nextcu(dw, off, &noff, &cuhl, NULL,

&pointer_size,

dwarf_loader.c static int cus__load_debug_types(struct cus · line 2508

Quoted 2 lines
dwarf_loader.c +1 -1
-                           &offset_size) == 0) {
+       while (dwarf_nextcu (dw, off, &noff, &cuhl, NULL, NULL, NULL)

== 0) {

dwarf_loader.c static int cus__load_debug_types(struct cus · line 2510

Quoted 2 lines
dwarf_loader.c +0 -0
                 Dwarf_Die die_mem;
                 Dwarf_Die *cu_die = dwarf_offdie(dw, off + cuhl,

&die_mem);

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 10 lines
dwarf_loader.c +4 -2
                 if (cu_die == NULL)
                         break;
 
-               if (++cnt > 5)
-                       break;
+               Dwarf_Off offset = 0;
+               while (true) {
+                       size_t length;
+                       Dwarf_Abbrev *abbrev = dwarf_getabbrev

(cu_die, offset, &length);

dwarf_loader.c static int cus__load_debug_types(struct cus · line 2518

Quoted 1 line
dwarf_loader.c +1 -0
+                       if (abbrev == NULL || abbrev ==

DWARF_END_ABBREV)

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 3 lines
dwarf_loader.c +1 -1
+                               break;
 
-               const char *producer = attr_string(cu_die,

DW_AT_producer);

dwarf_loader.c static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,

Quoted 12 lines
dwarf_loader.c +9 -3
-               if (strstr(producer, "clang version") != NULL &&
-                   strstr(producer, "-flto") != NULL)
-                       return true;
+                       size_t attrcnt;
+                       if (dwarf_getattrcnt (abbrev, &attrcnt) != 0)
+                               return false;
+
+                       unsigned int attr_num, attr_form;
+                       Dwarf_Off aboffset;
+                       size_t j;
+                       for (j = 0; j < attrcnt; ++j) {
+                               if (dwarf_getabbrevattr (abbrev, j,

&attr_num, &attr_form,

Quoted 14 lines
dwarf_loader.c +8 -1
+                                                        &aboffset))
+                                       return false;
+                               if (attr_form == DW_FORM_ref_addr)
+                                       return true;
+                       }
+
+                       offset += length;
+               }
 
                 off = noff;
         }
--

2.30.2

#5 lore raw

PATCH 2/2 dwarf_loader: check .notes section for lto build info

In reply to Yonghong Song

In [1], linux introduced a note with type BUILD_COMPILER_LTO_INFO. If this note type exist, there is no need to scan .debug_abbrev section for cross-cu reference. With a kernel built with [1], the cus__merging_cu() overhead is about 3us for either lto or non-lto vmlinux.

[1] https://lore.kernel.org/bpf/[email protected]/

---
 dwarf_loader.c | 33 +++++++++++++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)
dwarf_loader.c +31 -2
index bd23751..026d137 100644
@@ -2501,8 +2501,37 @@ static int cus__load_debug_types(struct cus *cus, struct conf_load *conf,
 	return 0;
 }
 
-static bool cus__merging_cu(Dwarf *dw)
+/* Match the define in linux:include/linux/elfnote.h */
+#define LINUX_ELFNOTE_BUILD_LTO		0x101
+
+static bool cus__merging_cu(Dwarf *dw, Elf *elf)
 {
+	Elf_Scn *section = NULL;
+	while ((section = elf_nextscn(elf, section)) != 0) {
+		GElf_Shdr header;
+		if (!gelf_getshdr(section, &header))
+			continue;
+
+		if (header.sh_type != SHT_NOTE)
+			continue;
+
+		Elf_Data *data = NULL;
+		while ((data = elf_getdata(section, data)) != 0) {
+			size_t name_off, desc_off, offset = 0;
+			GElf_Nhdr hdr;
+			while ((offset = gelf_getnote(data, offset, &hdr, &name_off, &desc_off)) != 0) {
+				if (hdr.n_type != LINUX_ELFNOTE_BUILD_LTO)
+					continue;
+
+				/* owner is Linux */
+				if (strcmp((char *)data->d_buf + name_off, "Linux") != 0)
+					continue;
+
+				return *(int *)(data->d_buf + desc_off) != 0;
+			}
+		}
+	}
+
 	Dwarf_Off off = 0, noff;
 	size_t cuhl;
 
@@ -2649,7 +2678,7 @@ static int cus__load_module(struct cus *cus, struct conf_load *conf,
 		}
 	}
 
-	if (cus__merging_cu(dw)) {
+	if (cus__merging_cu(dw, elf)) {
 		res = cus__merge_and_process_cu(cus, conf, mod, dw, elf, filename,
 						build_id, build_id_len,
 						type_cu ? &type_dcu : NULL);
Signature · 1 line
2.30.2
#6 lore raw
In reply to Yonghong Song

Em Wed, Mar 31, 2021 at 07:58:15PM -0700, Yonghong Song escreveu:

Quoted 9 lines

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

Great work! Reviewing/testing right now.

- Arnaldo

Quoted 14 lines

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

 dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 21 deletions(-)
Signature · 1 line
2.30.2
Signature · 1 line
- Arnaldo
Reviewed-by Tested-by on the v1 series
#7 lore raw
In reply to Yonghong Song
Quoted 11 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

Quoted 3 lines

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

For the series:

(Noting for others on our mailing list:
https://lore.kernel.org/linux-kbuild/[email protected]/
is a series of kernel patches required to test this. I had feedback on the kernel patches, but this approach in pahole LGTM since I think using simple notes in ELF is a good approach).

Quoted 11 lines

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

 dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 21 deletions(-)

--
2.30.2

--
Thanks,
~Nick Desaulniers

Tested-by on the v1 series
#8 lore raw
In reply to Yonghong Song
Quoted 15 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Quoted 7 lines

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

 dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

Thanks!
-bw

#9 lore raw
Quoted 21 lines · Bill Wendling
Quoted 19 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 3 lines
 dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

I did notice these warnings following the "pahole -J .tmp_vmlinux.btf" command. I don't know the severity of them, but it might be good to investigate.

$ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
  BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 355, 8746 - using 355
WARN: multiple IDs found for 'file': 588, 8779 - using 588
WARN: multiple IDs found for 'path': 411, 8780 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 8836 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 8873 - using 538
WARN: multiple IDs found for 'task_struct': 28, 8880 - using 28
WARN: multiple IDs found for 'inode': 355, 9484 - using 355
WARN: multiple IDs found for 'file': 588, 9517 - using 588
WARN: multiple IDs found for 'path': 411, 9518 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 9578 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 9615 - using 538
WARN: multiple IDs found for 'task_struct': 28, 9622 - using 28
WARN: multiple IDs found for 'seq_file': 1414, 12223 - using 1414
WARN: multiple IDs found for 'file': 588, 12237 - using 588
WARN: multiple IDs found for 'path': 411, 12238 - using 411
...

-bw

#10 lore raw
Quoted 44 lines · Bill Wendling
Quoted 21 lines · Bill Wendling
Quoted 19 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 3 lines
  dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

I did notice these warnings following the "pahole -J .tmp_vmlinux.btf" command. I don't know the severity of them, but it might be good to investigate.

$ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
   BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 355, 8746 - using 355
WARN: multiple IDs found for 'file': 588, 8779 - using 588
WARN: multiple IDs found for 'path': 411, 8780 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 8836 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 8873 - using 538
WARN: multiple IDs found for 'task_struct': 28, 8880 - using 28
WARN: multiple IDs found for 'inode': 355, 9484 - using 355
WARN: multiple IDs found for 'file': 588, 9517 - using 588
Quoted 8 lines
WARN: multiple IDs found for 'path': 411, 9518 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 9578 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 9615 - using 538
WARN: multiple IDs found for 'task_struct': 28, 9622 - using 28
WARN: multiple IDs found for 'seq_file': 1414, 12223 - using 1414
WARN: multiple IDs found for 'file': 588, 12237 - using 588
WARN: multiple IDs found for 'path': 411, 12238 - using 411
...

I didn't see it with my config. Maybe you can share your config file?

Quoted 3 lines

-bw

#11 lore raw

I tried porting the .config we're using to the official branch and couldn't replicate the problem. It's probably something local.

-bw

Quoted 62 lines · Yonghong Song
Quoted 44 lines · Bill Wendling
Quoted 21 lines · Bill Wendling
Quoted 19 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 3 lines
  dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

I did notice these warnings following the "pahole -J .tmp_vmlinux.btf" command. I don't know the severity of them, but it might be good to investigate.

$ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
   BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 355, 8746 - using 355
WARN: multiple IDs found for 'file': 588, 8779 - using 588
WARN: multiple IDs found for 'path': 411, 8780 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 8836 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 8873 - using 538
WARN: multiple IDs found for 'task_struct': 28, 8880 - using 28
WARN: multiple IDs found for 'inode': 355, 9484 - using 355
WARN: multiple IDs found for 'file': 588, 9517 - using 588
Quoted 8 lines
WARN: multiple IDs found for 'path': 411, 9518 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 9578 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 9615 - using 538
WARN: multiple IDs found for 'task_struct': 28, 9622 - using 28
WARN: multiple IDs found for 'seq_file': 1414, 12223 - using 1414
WARN: multiple IDs found for 'file': 588, 12237 - using 588
WARN: multiple IDs found for 'path': 411, 12238 - using 411
...

I didn't see it with my config. Maybe you can share your config file?

Quoted 3 lines

-bw

#12 lore raw

Em Fri, Apr 02, 2021 at 12:44:50PM -0700, Bill Wendling escreveu:

Quoted 59 lines

I tried porting the .config we're using to the official branch and couldn't replicate the problem. It's probably something local.

Quoted 55 lines · Yonghong Song
Quoted 44 lines · Bill Wendling
Quoted 21 lines · Bill Wendling
Quoted 19 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 3 lines
  dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

I did notice these warnings following the "pahole -J .tmp_vmlinux.btf" command. I don't know the severity of them, but it might be good to investigate.

$ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
   BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 355, 8746 - using 355
WARN: multiple IDs found for 'file': 588, 8779 - using 588
WARN: multiple IDs found for 'path': 411, 8780 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 8836 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 8873 - using 538
WARN: multiple IDs found for 'task_struct': 28, 8880 - using 28
WARN: multiple IDs found for 'inode': 355, 9484 - using 355
WARN: multiple IDs found for 'file': 588, 9517 - using 588
Quoted 8 lines
WARN: multiple IDs found for 'path': 411, 9518 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 9578 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 9615 - using 538
WARN: multiple IDs found for 'task_struct': 28, 9622 - using 28
WARN: multiple IDs found for 'seq_file': 1414, 12223 - using 1414
WARN: multiple IDs found for 'file': 588, 12237 - using 588
WARN: multiple IDs found for 'path': 411, 12238 - using 411
...

I didn't see it with my config. Maybe you can share your config file?

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

  CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
  CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

I'll try after a 'make mrproper'

- Arnaldo

#13 lore raw

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 143 lines

Em Fri, Apr 02, 2021 at 12:44:50PM -0700, Bill Wendling escreveu:

Quoted 59 lines

I tried porting the .config we're using to the official branch and couldn't replicate the problem. It's probably something local.

Quoted 55 lines · Yonghong Song
Quoted 44 lines · Bill Wendling
Quoted 21 lines · Bill Wendling
Quoted 19 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 3 lines
  dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

I did notice these warnings following the "pahole -J .tmp_vmlinux.btf" command. I don't know the severity of them, but it might be good to investigate.

$ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
   BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 355, 8746 - using 355
WARN: multiple IDs found for 'file': 588, 8779 - using 588
WARN: multiple IDs found for 'path': 411, 8780 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 8836 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 8873 - using 538
WARN: multiple IDs found for 'task_struct': 28, 8880 - using 28
WARN: multiple IDs found for 'inode': 355, 9484 - using 355
WARN: multiple IDs found for 'file': 588, 9517 - using 588
Quoted 8 lines
WARN: multiple IDs found for 'path': 411, 9518 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 9578 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 9615 - using 538
WARN: multiple IDs found for 'task_struct': 28, 9622 - using 28
WARN: multiple IDs found for 'seq_file': 1414, 12223 - using 1414
WARN: multiple IDs found for 'file': 588, 12237 - using 588
WARN: multiple IDs found for 'path': 411, 12238 - using 411
...

I didn't see it with my config. Maybe you can share your config file?

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

  CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
  CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

Quoted 1 line

I'll try after a 'make mrproper'

Also:

[acme@five pahole]$ git log --oneline -10
ae0b7dde1fd50b12 (HEAD -> master) dwarf_loader: Handle DWARF5 DW_OP_addrx properly 9adb014930f31c66 dwarf_loader: Handle subprogram ret type with abstract_origin properly 5752d1951d081a80 dwarf_loader: Check .notes section for LTO build info 209e45424ff4a22d dwarf_loader: Check .debug_abbrev for cross-CU references 39227909db3cc2c2 dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary 763475ca1101ccfe dwarf_loader: Factor out common code to initialize a cu d0d3fbd4744953e8 dwarf_loader: Permit a flexible HASHTAGS__BITS ffe0ef4d73906c18 btf: Add --btf_gen_all flag
de708b33114d42c2 btf: Add support for the floating-point types 4b7f8c04d009942b fprintf: Honour conf_fprintf.hex when printing enumerations [acme@five pahole]$

#14 lore raw
In reply to Arnaldo Carvalho de Melo

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 143 lines

Em Fri, Apr 02, 2021 at 12:44:50PM -0700, Bill Wendling escreveu:

Quoted 59 lines

I tried porting the .config we're using to the official branch and couldn't replicate the problem. It's probably something local.

Quoted 55 lines · Yonghong Song
Quoted 44 lines · Bill Wendling
Quoted 21 lines · Bill Wendling
Quoted 19 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 3 lines
  dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

I did notice these warnings following the "pahole -J .tmp_vmlinux.btf" command. I don't know the severity of them, but it might be good to investigate.

$ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
   BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 355, 8746 - using 355
WARN: multiple IDs found for 'file': 588, 8779 - using 588
WARN: multiple IDs found for 'path': 411, 8780 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 8836 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 8873 - using 538
WARN: multiple IDs found for 'task_struct': 28, 8880 - using 28
WARN: multiple IDs found for 'inode': 355, 9484 - using 355
WARN: multiple IDs found for 'file': 588, 9517 - using 588
Quoted 8 lines
WARN: multiple IDs found for 'path': 411, 9518 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 9578 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 9615 - using 538
WARN: multiple IDs found for 'task_struct': 28, 9622 - using 28
WARN: multiple IDs found for 'seq_file': 1414, 12223 - using 1414
WARN: multiple IDs found for 'file': 588, 12237 - using 588
WARN: multiple IDs found for 'path': 411, 12238 - using 411
...

I didn't see it with my config. Maybe you can share your config file?

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

  CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
  CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

Quoted 1 line

I'll try after a 'make mrproper'

Same thing, trying now with gcc.

- Arnaldo

#15 lore raw

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 83 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 82 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

  CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
  CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

Quoted 3 lines
Quoted 1 line

I'll try after a 'make mrproper'

Same thing, trying now with gcc.

With gcc no such messages:

  CC [M]  drivers/gpu/drm/nouveau/nv84_fence.o
  CC [M]  drivers/gpu/drm/nouveau/nvc0_fence.o
  LD [M]  drivers/gpu/drm/nouveau/nouveau.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  HOSTCC  arch/x86/tools/insn_decoder_test
  HOSTCC  arch/x86/tools/insn_sanity
  MODPOST Module.symvers
  TEST    posttest
  CC [M]  arch/x86/crypto/aegis128-aesni.mod.o
  CC [M]  arch/x86/crypto/blake2s-x86_64.mod.o

Now will try with clang non-LTO.

- Arnaldo

#16 lore raw

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 120 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 83 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 82 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

  CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
  CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

Quoted 3 lines
Quoted 1 line

I'll try after a 'make mrproper'

Same thing, trying now with gcc.

With gcc no such messages:

  CC [M]  drivers/gpu/drm/nouveau/nv84_fence.o
  CC [M]  drivers/gpu/drm/nouveau/nvc0_fence.o
  LD [M]  drivers/gpu/drm/nouveau/nouveau.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  HOSTCC  arch/x86/tools/insn_decoder_test
  HOSTCC  arch/x86/tools/insn_sanity
  MODPOST Module.symvers
  TEST    posttest
  CC [M]  arch/x86/crypto/aegis128-aesni.mod.o
  CC [M]  arch/x86/crypto/blake2s-x86_64.mod.o
Quoted 1 line

Now will try with clang non-LTO.

Works:

  AR      drivers/usb/built-in.a
  AR      lib/built-in.a
  AR      drivers/md/built-in.a
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_no_lto'

[acme@five bpf]$ grep LTO ../build/bpf_clang_no_lto/.config CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y CONFIG_LTO_NONE=y
CONFIG_HID_WALTOP=m
[acme@five bpf]$

This works, gcc as well, but this produces those BTFID warnings:

[acme@five bpf]$ grep LTO ../build/bpf_clang_thin_lto/.config CONFIG_LTO=y
CONFIG_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y CONFIG_HAS_LTO_CLANG=y
# CONFIG_LTO_NONE is not set
# CONFIG_LTO_CLANG_FULL is not set CONFIG_LTO_CLANG_THIN=y
CONFIG_HID_WALTOP=m
[acme@five bpf]$

Trying with clang full LTO.

- Arnaldo

#17 lore raw

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 149 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 120 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 83 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 82 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

  CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
  CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

Quoted 3 lines
Quoted 1 line

I'll try after a 'make mrproper'

Same thing, trying now with gcc.

With gcc no such messages:

  CC [M]  drivers/gpu/drm/nouveau/nv84_fence.o
  CC [M]  drivers/gpu/drm/nouveau/nvc0_fence.o
  LD [M]  drivers/gpu/drm/nouveau/nouveau.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  HOSTCC  arch/x86/tools/insn_decoder_test
  HOSTCC  arch/x86/tools/insn_sanity
  MODPOST Module.symvers
  TEST    posttest
  CC [M]  arch/x86/crypto/aegis128-aesni.mod.o
  CC [M]  arch/x86/crypto/blake2s-x86_64.mod.o
Quoted 1 line

Now will try with clang non-LTO.

Works:

  AR      drivers/usb/built-in.a
  AR      lib/built-in.a
  AR      drivers/md/built-in.a
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_no_lto'

Quoted 6 lines

[acme@five bpf]$ grep LTO ../build/bpf_clang_no_lto/.config CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y CONFIG_LTO_NONE=y
CONFIG_HID_WALTOP=m
[acme@five bpf]$

Sorry, I forgot to use clang on this no-lto build... doing it now with:

[acme@five bpf]$ grep LTO ../build/bpf_clang_no_lto/.config CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_HAS_LTO_CLANG=y
CONFIG_LTO_NONE=y
# CONFIG_LTO_CLANG_FULL is not set
# CONFIG_LTO_CLANG_THIN is not set
CONFIG_HID_WALTOP=m
[acme@five bpf]$
[acme@five bpf]$ make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_no_lto/ vmlinux

- Arnaldo

#18 lore raw

Em Tue, Apr 06, 2021 at 02:03:41PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 12 lines

Sorry, I forgot to use clang on this no-lto build... doing it now with:

[acme@five bpf]$ grep LTO ../build/bpf_clang_no_lto/.config CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_HAS_LTO_CLANG=y
CONFIG_LTO_NONE=y
# CONFIG_LTO_CLANG_FULL is not set
# CONFIG_LTO_CLANG_THIN is not set
CONFIG_HID_WALTOP=m
[acme@five bpf]$
[acme@five bpf]$ make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_no_lto/ vmlinux

Works:

[acme@five bpf]$ make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_no_lto/ vmlinux

  CC      drivers/acpi/acpica/utxfmutex.o
  AR      fs/built-in.a
  AR      drivers/acpi/acpica/built-in.a
  AR      drivers/acpi/built-in.a
  AR      drivers/md/built-in.a
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_no_lto' [acme@five bpf]$ grep LTO ../build/bpf_clang_thin_lto/.config CONFIG_LTO=y
CONFIG_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y CONFIG_HAS_LTO_CLANG=y
# CONFIG_LTO_NONE is not set
# CONFIG_LTO_CLANG_FULL is not set CONFIG_LTO_CLANG_THIN=y
CONFIG_HID_WALTOP=m
[acme@five bpf]$

#19 lore raw

Em Tue, Apr 06, 2021 at 02:07:26PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 15 lines

Em Tue, Apr 06, 2021 at 02:03:41PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 12 lines

Sorry, I forgot to use clang on this no-lto build... doing it now with:

[acme@five bpf]$ grep LTO ../build/bpf_clang_no_lto/.config CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_HAS_LTO_CLANG=y
CONFIG_LTO_NONE=y
# CONFIG_LTO_CLANG_FULL is not set
# CONFIG_LTO_CLANG_THIN is not set
CONFIG_HID_WALTOP=m
[acme@five bpf]$
[acme@five bpf]$ make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_no_lto/ vmlinux

Quoted 3 lines

Works:

[acme@five bpf]$ make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_no_lto/ vmlinux

Now with FULL LTO:

  CC      fs/mbcache.o
  CC      fs/posix_acl.o
  CC      fs/coredump.o
  CC      fs/drop_caches.o
  CC      fs/fhandle.o
  AR      drivers/soundwire/built-in.a
  AR      drivers/hid/built-in.a
  AR      drivers/acpi/built-in.a
  AR      drivers/built-in.a
  AR      fs/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo
GEN modules.builtin
LD .tmp_vmlinux.btf

  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux

SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_full_lto'

real 8m36.854s
user 50m44.832s
sys 4m0.946s
[acme@five bpf]$

[acme@five bpf]$ cat /etc/fedora-release
Fedora release 33 (Thirty Three)
[acme@five bpf]$ grep LTO ../build/bpf_clang_full_lto/.config CONFIG_LTO=y
CONFIG_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y CONFIG_HAS_LTO_CLANG=y
# CONFIG_LTO_NONE is not set CONFIG_LTO_CLANG_FULL=y
# CONFIG_LTO_CLANG_THIN is not set CONFIG_HID_WALTOP=m
[acme@five bpf]$

Probably this will go away if I update clang/llvm? :)

Will try.

- Arnaldo

#20 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 61 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 60 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 59 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 58 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
   CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

This could be due to the compiler. The clang 11 is used here. Sedat is
using clang 12 and didn't see warnings and I am using clang dev branch (clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Quoted 111 lines
Quoted 88 lines
Quoted 60 lines
Quoted 24 lines
Quoted 24 lines

Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

Quoted 3 lines
Quoted 1 line

I'll try after a 'make mrproper'

Same thing, trying now with gcc.

With gcc no such messages:

   CC [M]  drivers/gpu/drm/nouveau/nv84_fence.o
   CC [M]  drivers/gpu/drm/nouveau/nvc0_fence.o
   LD [M]  drivers/gpu/drm/nouveau/nouveau.o
   AR      drivers/gpu/built-in.a
   AR      drivers/built-in.a
   GEN     .version
   CHK     include/generated/compile.h
   LD      vmlinux.o
   MODPOST vmlinux.symvers
   MODINFO modules.builtin.modinfo
   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux
   SORTTAB vmlinux
   SYSMAP  System.map
   HOSTCC  arch/x86/tools/insn_decoder_test
   HOSTCC  arch/x86/tools/insn_sanity
   MODPOST Module.symvers
   TEST    posttest
   CC [M]  arch/x86/crypto/aegis128-aesni.mod.o
   CC [M]  arch/x86/crypto/blake2s-x86_64.mod.o
Quoted 1 line

Now will try with clang non-LTO.

Works:

   AR      drivers/usb/built-in.a
   AR      lib/built-in.a
   AR      drivers/md/built-in.a
   AR      drivers/built-in.a
   GEN     .version
   CHK     include/generated/compile.h
   LD      vmlinux.o
   MODPOST vmlinux.symvers
   MODINFO modules.builtin.modinfo
   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux

SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_no_lto'

Quoted 6 lines

[acme@five bpf]$ grep LTO ../build/bpf_clang_no_lto/.config CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y CONFIG_LTO_NONE=y
CONFIG_HID_WALTOP=m
[acme@five bpf]$

Sorry, I forgot to use clang on this no-lto build... doing it now with:

[acme@five bpf]$ grep LTO ../build/bpf_clang_no_lto/.config CONFIG_ARCH_SUPPORTS_LTO_CLANG=y CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_HAS_LTO_CLANG=y
CONFIG_LTO_NONE=y
# CONFIG_LTO_CLANG_FULL is not set
# CONFIG_LTO_CLANG_THIN is not set
CONFIG_HID_WALTOP=m
[acme@five bpf]$
[acme@five bpf]$ make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_no_lto/ vmlinux

- Arnaldo

#21 lore raw

Em Tue, Apr 06, 2021 at 10:23:37AM -0700, Yonghong Song escreveu:

Quoted 66 lines · Arnaldo Carvalho de Melo
Quoted 62 lines

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 61 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 60 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 59 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 58 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
   CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

Quoted 7 lines

This could be due to the compiler. The clang 11 is used here. Sedat is using clang 12 and didn't see warnings and I am using clang dev branch (clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Right, I'm now at:

$ git clone https://github.com/llvm/llvm-project.git
$ cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm

:-)

- Arnaldo

#22 lore raw

Em Tue, Apr 06, 2021 at 02:34:01PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 67 lines

Em Tue, Apr 06, 2021 at 10:23:37AM -0700, Yonghong Song escreveu:

Quoted 66 lines · Arnaldo Carvalho de Melo
Quoted 62 lines

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 61 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 60 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 59 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 58 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
   CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

This could be due to the compiler. The clang 11 is used here. Sedat is using clang 12 and didn't see warnings and I am using clang dev branch

Cool, clang 12 is the one in Fedora 34 beta.

- Arnaldo

Quoted 10 lines
Quoted 5 lines

(clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Right, I'm now at:

$ git clone https://github.com/llvm/llvm-project.git
$ cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm
#23 lore raw
In reply to Yonghong Song
Quoted 66 lines · Yonghong Song
Quoted 62 lines · Arnaldo Carvalho de Melo

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 61 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 60 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 59 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 58 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu*
[acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
   GEN     Makefile
   DESCEND  objtool
   DESCEND  bpf/resolve_btfids
   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
   CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o
   OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment MODPOST vmlinux.symvers
   MODINFO modules.builtin.modinfo
   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
   SORTTAB vmlinux
   SYSMAP  System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

Quoted 6 lines

This could be due to the compiler. The clang 11 is used here. Sedat is
using clang 12 and didn't see warnings and I am using clang dev branch (clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Yes, I can see it with llvm11:

LD vmlinux

BTFIDS vmlinux

WARN: multiple IDs found for 'inode': 245, 36255 - using 245

WARN: multiple IDs found for 'file': 390, 36288 - using 390

WARN: multiple IDs found for 'path': 395, 36289 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36362 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36399 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36498 - using 524

WARN: multiple IDs found for 'inode': 245, 36784 - using 245

WARN: multiple IDs found for 'file': 390, 36868 - using 390

WARN: multiple IDs found for 'path': 395, 36869 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36910 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36920 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36951 - using 524

SORTTAB vmlinux

SYSMAP System.map

LTO [M] crypto/crypto_engine.lto.o

LTO [M] drivers/crypto/virtio/virtio_crypto.lto.o

$ clang --version
clang version 11.1.0 (https://github.com/llvm/llvm-project.git 1fdec59bffc11ae37eb51a1b9869f0696bfd5312)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin

clang12 is okay:

LTO vmlinux.o
OBJTOOL vmlinux.o
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
$ clang --version
clang version 12.0.0 (https://github.com/llvm/llvm-project.git 31001be371e8f2c74470e727e54503fb2aabec8b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin

I think we do not need to fix pahole for llvm11.
When linus tree 5.12 is out. clang 12 should have been released or very close, we can just recommend clang 12 and later.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 25 lines

Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/10 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10 Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@five pahole]$

[acme@five bpf]$ git log --oneline -10
49b9da70941c3c8a (HEAD -> bpf_perf_enable) kbuild: add an elfnote with type BUILD_COMPILER_LTO_INFO 5c4f082a143c786e kbuild: move LINUX_ELFNOTE_BUILD_SALT to elfnote.h
42c8b565decb3662 bpf: Introduce helpers to enable/disable perf event fds in a map
f73ea1eb4cce6637 (bpf-next/master, bpf-next/for-next) bpf: selftests: Specify CONFIG_DYNAMIC_FTRACE in the testing config f07669df4c8df0b7 libbpf: Remove redundant semi-colon
6ac4c6f887f5a8ef bpf: Remove repeated struct btf_type declaration
2daae89666ad2532 bpf, cgroup: Delete repeated struct bpf_prog declaration
2ec9898e9c70b93a bpf: Remove unused parameter from ___bpf_prog_run
007bdc12d4b46656 bpf, selftests: test_maps generating unrecognized data section
82506665179209e4 tcp: reorder tcp_congestion_ops for better cache locality
[acme@five bpf]$

Quoted 1 line

I'll try after a 'make mrproper'

Same thing, trying now with gcc.

[...]

#24 lore raw

Em Tue, Apr 06, 2021 at 10:48:22AM -0700, Yonghong Song escreveu:

Quoted 182 lines · Yonghong Song
Quoted 87 lines · Arnaldo Carvalho de Melo
Quoted 83 lines

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 81 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 79 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 77 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo
Carvalho de Melo escreveu:

Quoted 75 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu* [acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux
make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto' GEN     Makefile
   DESCEND  objtool
   DESCEND  bpf/resolve_btfids
   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o
   OBJTOOL vmlinux.o
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen2()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen4()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen4()+0x12: unsupported
stack pointer realignment
   MODPOST vmlinux.symvers
   MODINFO modules.builtin.modinfo
   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510 SORTTAB vmlinux
   SYSMAP  System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

Quoted 7 lines

This could be due to the compiler. The clang 11 is used here. Sedat is using clang 12 and didn't see warnings and I am using clang dev branch (clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Yes, I can see it with llvm11:

LD vmlinux

BTFIDS vmlinux

WARN: multiple IDs found for 'inode': 245, 36255 - using 245

WARN: multiple IDs found for 'file': 390, 36288 - using 390

WARN: multiple IDs found for 'path': 395, 36289 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36362 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36399 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36498 - using 524

WARN: multiple IDs found for 'inode': 245, 36784 - using 245

WARN: multiple IDs found for 'file': 390, 36868 - using 390

WARN: multiple IDs found for 'path': 395, 36869 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36910 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36920 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36951 - using 524

SORTTAB vmlinux

SYSMAP System.map

LTO [M] crypto/crypto_engine.lto.o

LTO [M] drivers/crypto/virtio/virtio_crypto.lto.o

$ clang --version
clang version 11.1.0 (https://github.com/llvm/llvm-project.git
1fdec59bffc11ae37eb51a1b9869f0696bfd5312)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin

clang12 is okay:

LTO vmlinux.o
OBJTOOL vmlinux.o
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
$ clang --version
clang version 12.0.0 (https://github.com/llvm/llvm-project.git
31001be371e8f2c74470e727e54503fb2aabec8b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin
Quoted 3 lines

I think we do not need to fix pahole for llvm11.
When linus tree 5.12 is out. clang 12 should have been released or very close, we can just recommend clang 12 and later.

Agreed, and it is just for _thin_ LTO, those warnings don't pop up when building for full LTO with clang 11, the one in Fedora 33.

And Fedora 34 beta has clang/llvm 12.0, so we're good.

/me goes back to building clang/llvm HEAD, reducing the number of linker instances to 1 as I have just 32GB of ram in this Ryzen machine... ;-)

- Arnaldo

#25 lore raw

Em Tue, Apr 06, 2021 at 03:03:29PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 2 lines

/me goes back to building clang/llvm HEAD, reducing the number of linker instances to 1 as I have just 32GB of ram in this Ryzen machine... ;-)

And guess what takes a long time?

[ 61%] Linking CXX executable ../../bin/llvm-lto

;-)

- Arnaldo

#26 lore raw
Quoted 8 lines · Arnaldo Carvalho de Melo

Em Tue, Apr 06, 2021 at 03:03:29PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 2 lines

/me goes back to building clang/llvm HEAD, reducing the number of linker instances to 1 as I have just 32GB of ram in this Ryzen machine... ;-)

And guess what takes a long time?

[ 61%] Linking CXX executable ../../bin/llvm-lto

During configuration of LLVM (cmake):

$ cmake ... -DLLVM_ENABLE_LLD=ON

to link the binaries with LLD, which should be able to link using all cores.
https://llvm.org/docs/CMake.html

Signature · 2 lines
Thanks,
~Nick Desaulniers
#27 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 188 lines · Arnaldo Carvalho de Melo

Em Tue, Apr 06, 2021 at 10:48:22AM -0700, Yonghong Song escreveu:

Quoted 182 lines · Yonghong Song
Quoted 87 lines · Arnaldo Carvalho de Melo
Quoted 83 lines

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 81 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 79 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 77 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo
Carvalho de Melo escreveu:

Quoted 75 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu* [acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux
make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
   CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen2()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen4()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen4()+0x12: unsupported
stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

Quoted 7 lines

This could be due to the compiler. The clang 11 is used here. Sedat is using clang 12 and didn't see warnings and I am using clang dev branch (clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Yes, I can see it with llvm11:

LD vmlinux

BTFIDS vmlinux

WARN: multiple IDs found for 'inode': 245, 36255 - using 245

WARN: multiple IDs found for 'file': 390, 36288 - using 390

WARN: multiple IDs found for 'path': 395, 36289 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36362 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36399 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36498 - using 524

WARN: multiple IDs found for 'inode': 245, 36784 - using 245

WARN: multiple IDs found for 'file': 390, 36868 - using 390

WARN: multiple IDs found for 'path': 395, 36869 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36910 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36920 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36951 - using 524

SORTTAB vmlinux

SYSMAP System.map

LTO [M] crypto/crypto_engine.lto.o

LTO [M] drivers/crypto/virtio/virtio_crypto.lto.o

$ clang --version
clang version 11.1.0 (https://github.com/llvm/llvm-project.git
1fdec59bffc11ae37eb51a1b9869f0696bfd5312)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin

clang12 is okay:

LTO vmlinux.o
OBJTOOL vmlinux.o
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
$ clang --version
clang version 12.0.0 (https://github.com/llvm/llvm-project.git
31001be371e8f2c74470e727e54503fb2aabec8b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin
Quoted 3 lines

I think we do not need to fix pahole for llvm11.
When linus tree 5.12 is out. clang 12 should have been released or very close, we can just recommend clang 12 and later.

Quoted 7 lines

Agreed, and it is just for _thin_ LTO, those warnings don't pop up when building for full LTO with clang 11, the one in Fedora 33.

And Fedora 34 beta has clang/llvm 12.0, so we're good.

/me goes back to building clang/llvm HEAD, reducing the number of linker instances to 1 as I have just 32GB of ram in this Ryzen machine... ;-)

32GB should be enough for a lot of parallel links - unless you're using ld.bfd - highly advised to switch to gold or lld & then you shuold be able to do a fair number of parallel links without loads of thrashing.

https://llvm.org/docs/GettingStarted.html#common-problems discusses a
few things to try

- Dave

#28 lore raw

Em Tue, Apr 06, 2021 at 11:17:56AM -0700, David Blaikie escreveu:

Quoted 198 lines
Quoted 188 lines · Arnaldo Carvalho de Melo

Em Tue, Apr 06, 2021 at 10:48:22AM -0700, Yonghong Song escreveu:

Quoted 182 lines · Yonghong Song
Quoted 87 lines · Arnaldo Carvalho de Melo
Quoted 83 lines

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 81 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 79 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 77 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo
Carvalho de Melo escreveu:

Quoted 75 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu* [acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux
make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
   CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen2()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen4()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen4()+0x12: unsupported
stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

Quoted 7 lines

This could be due to the compiler. The clang 11 is used here. Sedat is using clang 12 and didn't see warnings and I am using clang dev branch (clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Yes, I can see it with llvm11:

LD vmlinux

BTFIDS vmlinux

WARN: multiple IDs found for 'inode': 245, 36255 - using 245

WARN: multiple IDs found for 'file': 390, 36288 - using 390

WARN: multiple IDs found for 'path': 395, 36289 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36362 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36399 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36498 - using 524

WARN: multiple IDs found for 'inode': 245, 36784 - using 245

WARN: multiple IDs found for 'file': 390, 36868 - using 390

WARN: multiple IDs found for 'path': 395, 36869 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36910 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36920 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36951 - using 524

SORTTAB vmlinux

SYSMAP System.map

LTO [M] crypto/crypto_engine.lto.o

LTO [M] drivers/crypto/virtio/virtio_crypto.lto.o

$ clang --version
clang version 11.1.0 (https://github.com/llvm/llvm-project.git
1fdec59bffc11ae37eb51a1b9869f0696bfd5312)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin

clang12 is okay:

LTO vmlinux.o
OBJTOOL vmlinux.o
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
$ clang --version
clang version 12.0.0 (https://github.com/llvm/llvm-project.git
31001be371e8f2c74470e727e54503fb2aabec8b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin
Quoted 3 lines

I think we do not need to fix pahole for llvm11.
When linus tree 5.12 is out. clang 12 should have been released or very close, we can just recommend clang 12 and later.

Quoted 7 lines

Agreed, and it is just for _thin_ LTO, those warnings don't pop up when building for full LTO with clang 11, the one in Fedora 33.

And Fedora 34 beta has clang/llvm 12.0, so we're good.

/me goes back to building clang/llvm HEAD, reducing the number of linker instances to 1 as I have just 32GB of ram in this Ryzen machine... ;-)

Quoted 7 lines

32GB should be enough for a lot of parallel links - unless you're using ld.bfd - highly advised to switch to gold or lld & then you shuold be able to do a fair number of parallel links without loads of thrashing.

https://llvm.org/docs/GettingStarted.html#common-problems discusses a
few things to try

Thanks for the pointer!

- Arnaldo

#29 lore raw
In reply to David Blaikie
Quoted 204 lines · David Blaikie
Quoted 188 lines · Arnaldo Carvalho de Melo

Em Tue, Apr 06, 2021 at 10:48:22AM -0700, Yonghong Song escreveu:

Quoted 182 lines · Yonghong Song
Quoted 87 lines · Arnaldo Carvalho de Melo
Quoted 83 lines

Em Tue, Apr 06, 2021 at 01:59:30PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 81 lines

Em Tue, Apr 06, 2021 at 01:55:54PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 79 lines

Em Tue, Apr 06, 2021 at 01:40:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 77 lines

Em Tue, Apr 06, 2021 at 10:22:37AM -0300, Arnaldo
Carvalho de Melo escreveu:

Quoted 75 lines

I'm seeing these here:

[acme@five bpf]$ rm -f ../build/bpf_clang_thin_lto/*vmlinu* [acme@five bpf]$ time make -j28 LLVM=1 LLVM_IAS=1 O=../build/bpf_clang_thin_lto/ vmlinux
make[1]: Entering directory '/home/acme/git/build/bpf_clang_thin_lto'
GEN Makefile
DESCEND objtool
DESCEND bpf/resolve_btfids

   CALL    /home/acme/git/bpf/scripts/atomic/check-atomics.sh
   CALL    /home/acme/git/bpf/scripts/checksyscalls.sh
   CHK     include/generated/compile.h
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      init/version.o
   AR      init/built-in.a
   GEN     .tmp_initcalls.lds
   LTO     vmlinux.o

OBJTOOL vmlinux.o
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen2()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen2()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_init_avx_gen4()+0x12: unsupported stack
pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_enc_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_dec_update_avx_gen4()+0x12: unsupported
stack pointer realignment
vmlinux.o: warning: objtool:
aesni_gcm_finalize_avx_gen4()+0x12: unsupported
stack pointer realignment
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

   GEN     modules.builtin
   LD      .tmp_vmlinux.btf
   BTF     .btf.vmlinux.bin.o
   LD      .tmp_vmlinux.kallsyms1
   KSYMS   .tmp_vmlinux.kallsyms1.S
   AS      .tmp_vmlinux.kallsyms1.S
   LD      .tmp_vmlinux.kallsyms2
   KSYMS   .tmp_vmlinux.kallsyms2.S
   AS      .tmp_vmlinux.kallsyms2.S
   LD      vmlinux
   BTFIDS  vmlinux

WARN: multiple IDs found for 'inode': 232, 28822 - using 232
WARN: multiple IDs found for 'file': 374, 28855 - using 374
WARN: multiple IDs found for 'path': 379, 28856 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 28929 - using 177
WARN: multiple IDs found for 'task_struct': 97, 28966 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29059 - using 510
WARN: multiple IDs found for 'inode': 232, 29345 - using 232
WARN: multiple IDs found for 'file': 374, 29429 - using 374
WARN: multiple IDs found for 'path': 379, 29430 - using 379
WARN: multiple IDs found for 'vm_area_struct': 177, 29471 - using 177
WARN: multiple IDs found for 'task_struct': 97, 29481 - using 97
WARN: multiple IDs found for 'seq_file': 510, 29512 - using 510
SORTTAB vmlinux
SYSMAP System.map
make[1]: Leaving directory '/home/acme/git/build/bpf_clang_thin_lto'

[acme@five pahole]$ clang -v
clang version 11.0.0 (Fedora 11.0.0-2.fc33)

Quoted 7 lines

This could be due to the compiler. The clang 11 is used here. Sedat is using clang 12 and didn't see warnings and I am using clang dev branch (clang 13) and didn't see warnings either. clang 11 could generate some debuginfo where pahole didn't handle it properly.

I tried to build locally with clang 11 but it crashed as I enabled assert during compiler build. Will try a little bit more.

Yes, I can see it with llvm11:

LD vmlinux

BTFIDS vmlinux

WARN: multiple IDs found for 'inode': 245, 36255 - using 245

WARN: multiple IDs found for 'file': 390, 36288 - using 390

WARN: multiple IDs found for 'path': 395, 36289 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36362 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36399 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36498 - using 524

WARN: multiple IDs found for 'inode': 245, 36784 - using 245

WARN: multiple IDs found for 'file': 390, 36868 - using 390

WARN: multiple IDs found for 'path': 395, 36869 - using 395

WARN: multiple IDs found for 'vm_area_struct': 190, 36910 - using 190

WARN: multiple IDs found for 'task_struct': 93, 36920 - using 93

WARN: multiple IDs found for 'seq_file': 524, 36951 - using 524

SORTTAB vmlinux

SYSMAP System.map

LTO [M] crypto/crypto_engine.lto.o

LTO [M] drivers/crypto/virtio/virtio_crypto.lto.o

$ clang --version
clang version 11.1.0 (https://github.com/llvm/llvm-project.git
1fdec59bffc11ae37eb51a1b9869f0696bfd5312)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin

clang12 is okay:

LTO vmlinux.o
OBJTOOL vmlinux.o
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo

  GEN     modules.builtin
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
$ clang --version
clang version 12.0.0 (https://github.com/llvm/llvm-project.git
31001be371e8f2c74470e727e54503fb2aabec8b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/yhs/work/llvm-project/llvm/build/install/bin
Quoted 3 lines

I think we do not need to fix pahole for llvm11.
When linus tree 5.12 is out. clang 12 should have been released or very close, we can just recommend clang 12 and later.

Quoted 7 lines

Agreed, and it is just for _thin_ LTO, those warnings don't pop up when building for full LTO with clang 11, the one in Fedora 33.

And Fedora 34 beta has clang/llvm 12.0, so we're good.

/me goes back to building clang/llvm HEAD, reducing the number of linker instances to 1 as I have just 32GB of ram in this Ryzen machine... ;-)

32GB should be enough for a lot of parallel links - unless you're using ld.bfd - highly advised to switch to gold or lld & then you shuold be able to do a fair number of parallel links without loads of thrashing.

Quoted 4 lines

https://llvm.org/docs/GettingStarted.html#common-problems discusses a
few things to try

- Dave

In the Linux kernel land I remember ld.lld can use more memory than GNU ld. There are some lkp bot reports related to this.
It may be related to GNU ld not mmaping(?) input files. In any case no deep analysis has been performed tracking down the potential issues.

#30 lore raw
In reply to Bill Wendling
Quoted 30 lines · Bill Wendling
Quoted 21 lines · Bill Wendling
Quoted 19 lines · Yonghong Song

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 3 lines
 dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

Quoted 7 lines

I did notice these warnings following the "pahole -J .tmp_vmlinux.btf" command. I don't know the severity of them, but it might be good to investigate.

$ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
  BTFIDS  vmlinux
WARN: multiple IDs found for 'inode': 355, 8746 - using 355

the message means that there are multiple instances of 'inode' structs in the BTF data with ids 355 and 8746

that can happen when some object/struct name is duplicated in multiple objects and it's causing two distinct type hierarchies
in BTF data.. like the one we fixes in here:
f7f2b43eaf6b crypto: bcm - Rename struct device_private to bcm_device_private

if you provide the .config, I can check if I reproduce

jirka

Quoted 18 lines

WARN: multiple IDs found for 'file': 588, 8779 - using 588
WARN: multiple IDs found for 'path': 411, 8780 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 8836 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 8873 - using 538
WARN: multiple IDs found for 'task_struct': 28, 8880 - using 28
WARN: multiple IDs found for 'inode': 355, 9484 - using 355
WARN: multiple IDs found for 'file': 588, 9517 - using 588
WARN: multiple IDs found for 'path': 411, 9518 - using 411
WARN: multiple IDs found for 'seq_file': 1414, 9578 - using 1414
WARN: multiple IDs found for 'vm_area_struct': 538, 9615 - using 538
WARN: multiple IDs found for 'task_struct': 28, 9622 - using 28
WARN: multiple IDs found for 'seq_file': 1414, 12223 - using 1414
WARN: multiple IDs found for 'file': 588, 12237 - using 588
WARN: multiple IDs found for 'path': 411, 12238 - using 411
...

-bw

Tested-by Tested-by Bill Wendling
#31 lore raw
In reply to Bill Wendling

Em Thu, Apr 01, 2021 at 12:35:45PM -0700, Bill Wendling escreveu:

Quoted 20 lines · Yonghong Song
Quoted 18 lines

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 2 lines
 dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

Thanks, added, and also this "Committer testing" section:

Committer testing:

Using a thin-LTO built vmlinux that doesn't have the LINUX_ELFNOTE_BUILD_LTO note:

  $ readelf --notes vmlinux.clang.thin.LTO

Displaying notes found in: .notes

    Owner                Data size 	Description
    Xen                  0x00000006	Unknown note type: (0x00000006)
     description data: 6c 69 6e 75 78 00
    Xen                  0x00000004	Unknown note type: (0x00000007)
     description data: 32 2e 36 00
    Xen                  0x00000008	Unknown note type: (0x00000005)
     description data: 78 65 6e 2d 33 2e 30 00
    Xen                  0x00000008	Unknown note type: (0x00000003)
     description data: 00 00 00 80 ff ff ff ff
    Xen                  0x00000008	Unknown note type: (0x0000000f)
     description data: 00 00 00 00 80 00 00 00
    Xen                  0x00000008	NT_VERSION (version)
     description data: c0 e1 33 83 ff ff ff ff
    Xen                  0x00000008	NT_ARCH (architecture)
     description data: 00 20 00 81 ff ff ff ff
    Xen                  0x00000029	Unknown note type: (0x0000000a)
     description data: 21 77 72 69 74 61 62 6c 65 5f 70 61 67 65 5f 74 61 62 6c 65 73 7c 70 61 65 5f 70 67 64 69 72 5f 61 62 6f 76 65 5f 34 67 62
    Xen                  0x00000004	Unknown note type: (0x00000011)
     description data: 01 88 00 00
    Xen                  0x00000004	Unknown note type: (0x00000009)
     description data: 79 65 73 00
    Xen                  0x00000008	Unknown note type: (0x00000008)
     description data: 67 65 6e 65 72 69 63 00
    Xen                  0x00000010	Unknown note type: (0x0000000d)
     description data: 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
    Xen                  0x00000004	Unknown note type: (0x0000000e)
     description data: 01 00 00 00
    Xen                  0x00000004	Unknown note type: (0x00000010)
     description data: 01 00 00 00
    Xen                  0x00000008	Unknown note type: (0x0000000c)
     description data: 00 00 00 00 00 80 ff ff
    Xen                  0x00000008	Unknown note type: (0x00000004)
     description data: 00 00 00 00 00 00 00 00
    Xen                  0x00000008	Unknown note type: (0x00000012)
     description data: e0 02 00 01 00 00 00 00
    Linux                0x00000017	OPEN
     description data: 34 2e 31 39 2e 34 2d 33 30 30 2e 66 63 32 39 2e 78 38 36 5f 36 34 00
    GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
      Build ID: 354f81317b1b3c35f3f81f8d9f04d0c8caccb09a

$

Then with one with the new ELF note stating that this binary was built with LTO:

  [acme@five pahole]$ readelf --notes vmlinux.clang.thin.LTO+ELF_note 

Displaying notes found in: .notes

    Owner                Data size 	Description
    Xen                  0x00000006	Unknown note type: (0x00000006)
     description data: 6c 69 6e 75 78 00 
    Xen                  0x00000004	Unknown note type: (0x00000007)
     description data: 32 2e 36 00 
    Xen                  0x00000008	Unknown note type: (0x00000005)
     description data: 78 65 6e 2d 33 2e 30 00 
    Xen                  0x00000008	Unknown note type: (0x00000003)
     description data: 00 00 00 80 ff ff ff ff 
    Xen                  0x00000008	Unknown note type: (0x0000000f)
     description data: 00 00 00 00 80 00 00 00 
    Xen                  0x00000008	NT_VERSION (version)
     description data: c0 e1 33 83 ff ff ff ff 
    Xen                  0x00000008	NT_ARCH (architecture)
     description data: 00 20 00 81 ff ff ff ff 
    Xen                  0x00000029	Unknown note type: (0x0000000a)
     description data: 21 77 72 69 74 61 62 6c 65 5f 70 61 67 65 5f 74 61 62 6c 65 73 7c 70 61 65 5f 70 67 64 69 72 5f 61 62 6f 76 65 5f 34 67 62 
    Xen                  0x00000004	Unknown note type: (0x00000011)
     description data: 01 88 00 00 
    Xen                  0x00000004	Unknown note type: (0x00000009)
     description data: 79 65 73 00 
    Xen                  0x00000008	Unknown note type: (0x00000008)
     description data: 67 65 6e 65 72 69 63 00 
    Xen                  0x00000010	Unknown note type: (0x0000000d)
     description data: 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 
    Xen                  0x00000004	Unknown note type: (0x0000000e)
     description data: 01 00 00 00 
    Xen                  0x00000004	Unknown note type: (0x00000010)
     description data: 01 00 00 00 
    Xen                  0x00000008	Unknown note type: (0x0000000c)
     description data: 00 00 00 00 00 80 ff ff 
    Xen                  0x00000008	Unknown note type: (0x00000004)
     description data: 00 00 00 00 00 00 00 00 
    Xen                  0x00000008	Unknown note type: (0x00000012)
     description data: e0 02 00 01 00 00 00 00 
    Linux                0x00000017	OPEN
     description data: 34 2e 31 39 2e 34 2d 33 30 30 2e 66 63 32 39 2e 78 38 36 5f 36 34 00 
    Linux                0x00000004	func
     description data: 01 00 00 00 
    GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
      Build ID: aeba9ffc929acd3cd573b4d1afc8df9af4f3694d

$

Now to see the diff:

  $ readelf --notes vmlinux.clang.thin.LTO+ELF_note > with-note
  $ readelf --notes vmlinux.clang.thin.LTO > without-note
  $ diff -u without-note with-note 
  --- without-note	2021-04-02 10:23:57.545349084 -0300
  +++ with-note	2021-04-02 10:23:50.690196102 -0300
  @@ -37,5 +37,7 @@
      description data: e0 02 00 01 00 00 00 00 
     Linux                0x00000017	OPEN
      description data: 34 2e 31 39 2e 34 2d 33 30 30 2e 66 63 32 39 2e 78 38 36 5f 36 34 00 
  +  Linux                0x00000004	func
  +   description data: 01 00 00 00 
     GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
  -    Build ID: 354f81317b1b3c35f3f81f8d9f04d0c8caccb09a
  +    Build ID: aeba9ffc929acd3cd573b4d1afc8df9af4f3694d
  $
#32 lore raw
Quoted 147 lines · Arnaldo Carvalho de Melo

Em Thu, Apr 01, 2021 at 12:35:45PM -0700, Bill Wendling escreveu:

Quoted 20 lines · Yonghong Song
Quoted 18 lines

Function cus__merging_cu() is introduced in Commit 39227909db3c ("dwarf_loader: Permit merging all DWARF CU's for clang LTO built binary") to test whether cross-cu references may happen. The original implementation anticipates compilation flags in dwarf, but later some concerns about binary size surfaced and the decision is to scan .debug_abbrev as a faster way to check cross-cu references. Also putting a note in vmlinux to indicate whether lto is enabled for built or not can provide a much faster way.

This patch set implemented this two approaches, first checking the note (in Patch #2), if not found, then check .debug_abbrev (in Patch #1).

Yonghong Song (2):
dwarf_loader: check .debug_abbrev for cross-cu references
dwarf_loader: check .notes section for lto build info

Quoted 7 lines
Quoted 2 lines
  dwarf_loader.c | 76 ++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 55 insertions(+), 21 deletions(-)

With this series of patches, the compilation passes for me with ThinLTO. You may add this if you like:

Thanks, added, and also this "Committer testing" section:

Committer testing:

Using a thin-LTO built vmlinux that doesn't have the LINUX_ELFNOTE_BUILD_LTO note:

   $ readelf --notes vmlinux.clang.thin.LTO
   Displaying notes found in: .notes
     Owner                Data size 	Description
     Xen                  0x00000006	Unknown note type: (0x00000006)
      description data: 6c 69 6e 75 78 00
     Xen                  0x00000004	Unknown note type: (0x00000007)
      description data: 32 2e 36 00
     Xen                  0x00000008	Unknown note type: (0x00000005)
      description data: 78 65 6e 2d 33 2e 30 00
     Xen                  0x00000008	Unknown note type: (0x00000003)
      description data: 00 00 00 80 ff ff ff ff
     Xen                  0x00000008	Unknown note type: (0x0000000f)
      description data: 00 00 00 00 80 00 00 00
     Xen                  0x00000008	NT_VERSION (version)
      description data: c0 e1 33 83 ff ff ff ff
     Xen                  0x00000008	NT_ARCH (architecture)
      description data: 00 20 00 81 ff ff ff ff
     Xen                  0x00000029	Unknown note type: (0x0000000a)
      description data: 21 77 72 69 74 61 62 6c 65 5f 70 61 67 65 5f 74 61 62 6c 65 73 7c 70 61 65 5f 70 67 64 69 72 5f 61 62 6f 76 65 5f 34 67 62
     Xen                  0x00000004	Unknown note type: (0x00000011)
      description data: 01 88 00 00
     Xen                  0x00000004	Unknown note type: (0x00000009)
      description data: 79 65 73 00
     Xen                  0x00000008	Unknown note type: (0x00000008)
      description data: 67 65 6e 65 72 69 63 00
     Xen                  0x00000010	Unknown note type: (0x0000000d)
      description data: 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
     Xen                  0x00000004	Unknown note type: (0x0000000e)
      description data: 01 00 00 00
     Xen                  0x00000004	Unknown note type: (0x00000010)
      description data: 01 00 00 00
     Xen                  0x00000008	Unknown note type: (0x0000000c)
      description data: 00 00 00 00 00 80 ff ff
     Xen                  0x00000008	Unknown note type: (0x00000004)
      description data: 00 00 00 00 00 00 00 00
     Xen                  0x00000008	Unknown note type: (0x00000012)
      description data: e0 02 00 01 00 00 00 00
     Linux                0x00000017	OPEN
      description data: 34 2e 31 39 2e 34 2d 33 30 30 2e 66 63 32 39 2e 78 38 36 5f 36 34 00
     GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
       Build ID: 354f81317b1b3c35f3f81f8d9f04d0c8caccb09a
   $

Then with one with the new ELF note stating that this binary was built with LTO:

   [acme@five pahole]$ readelf --notes vmlinux.clang.thin.LTO+ELF_note
   Displaying notes found in: .notes
     Owner                Data size 	Description
     Xen                  0x00000006	Unknown note type: (0x00000006)
      description data: 6c 69 6e 75 78 00
     Xen                  0x00000004	Unknown note type: (0x00000007)
      description data: 32 2e 36 00
     Xen                  0x00000008	Unknown note type: (0x00000005)
      description data: 78 65 6e 2d 33 2e 30 00
     Xen                  0x00000008	Unknown note type: (0x00000003)
      description data: 00 00 00 80 ff ff ff ff
     Xen                  0x00000008	Unknown note type: (0x0000000f)
      description data: 00 00 00 00 80 00 00 00
     Xen                  0x00000008	NT_VERSION (version)
      description data: c0 e1 33 83 ff ff ff ff
     Xen                  0x00000008	NT_ARCH (architecture)
      description data: 00 20 00 81 ff ff ff ff
     Xen                  0x00000029	Unknown note type: (0x0000000a)
      description data: 21 77 72 69 74 61 62 6c 65 5f 70 61 67 65 5f 74 61 62 6c 65 73 7c 70 61 65 5f 70 67 64 69 72 5f 61 62 6f 76 65 5f 34 67 62
     Xen                  0x00000004	Unknown note type: (0x00000011)
      description data: 01 88 00 00
     Xen                  0x00000004	Unknown note type: (0x00000009)
      description data: 79 65 73 00
     Xen                  0x00000008	Unknown note type: (0x00000008)
      description data: 67 65 6e 65 72 69 63 00
     Xen                  0x00000010	Unknown note type: (0x0000000d)
      description data: 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
     Xen                  0x00000004	Unknown note type: (0x0000000e)
      description data: 01 00 00 00
     Xen                  0x00000004	Unknown note type: (0x00000010)
      description data: 01 00 00 00
     Xen                  0x00000008	Unknown note type: (0x0000000c)
      description data: 00 00 00 00 00 80 ff ff
     Xen                  0x00000008	Unknown note type: (0x00000004)
      description data: 00 00 00 00 00 00 00 00
     Xen                  0x00000008	Unknown note type: (0x00000012)
      description data: e0 02 00 01 00 00 00 00
     Linux                0x00000017	OPEN
      description data: 34 2e 31 39 2e 34 2d 33 30 30 2e 66 63 32 39 2e 78 38 36 5f 36 34 00
     Linux                0x00000004	func
      description data: 01 00 00 00
     GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
       Build ID: aeba9ffc929acd3cd573b4d1afc8df9af4f3694d
   $

Now to see the diff:

   $ readelf --notes vmlinux.clang.thin.LTO+ELF_note > with-note
   $ readelf --notes vmlinux.clang.thin.LTO > without-note
   $ diff -u without-note with-note
   --- without-note	2021-04-02 10:23:57.545349084 -0300
   +++ with-note	2021-04-02 10:23:50.690196102 -0300
   @@ -37,5 +37,7 @@
       description data: e0 02 00 01 00 00 00 00
      Linux                0x00000017	OPEN
       description data: 34 2e 31 39 2e 34 2d 33 30 30 2e 66 63 32 39 2e 78 38 36 5f 36 34 00
   +  Linux                0x00000004	func
   +   description data: 01 00 00 00
      GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
   -    Build ID: 354f81317b1b3c35f3f81f8d9f04d0c8caccb09a
   +    Build ID: aeba9ffc929acd3cd573b4d1afc8df9af4f3694d
   $
Quoted 13 lines

Looks great! Thanks a lot for detailed testing log and Suggested-by/Tested-by/Cc list!