Skip to contentSkip to conversation

kallsyms: fix nonconverging kallsyms table with lld

PATCH In review Arnd Bergmann 3 participants 11 messages Feb 4, 2021 – Jun 21, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kallsyms: fix nonconverging kallsyms table with lld

From: Arnd Bergmann <[email protected]>

ARM randconfig builds with lld sometimes show a build failure from kallsyms:

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround

The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.

This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.

I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.

---
 scripts/kallsyms.c | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/kallsyms.c +6 -0
index 7ecd2ccba531..54ad86d13784 100644
@@ -112,6 +112,12 @@ static bool is_ignored_symbol(const char *name, char type)
 		"__crc_",		/* modversions */
 		"__efistub_",		/* arm64 EFI stub namespace */
 		"__kvm_nvhe_",		/* arm64 non-VHE KVM namespace */
+		"__AArch64ADRPThunk_",	/* arm64 lld */
+		"__ARMV5PILongThunk_",	/* arm lld */
+		"__ARMV7PILongThunk_",
+		"__ThumbV7PILongThunk_",
+		"__LA25Thunk_",		/* mips lld */
+		"__microLA25Thunk_",
 		NULL
 	};
 
Signature · 1 line
2.29.2
#1 lore raw
Quoted 9 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

ARM randconfig builds with lld sometimes show a build failure from kallsyms:

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround

Quoted 7 lines

The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.

This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.

Yes, this is a whack-a-mole.

I fixed the typo "idenitifiers" -> "identifiers"
and applied to linux-kbuild.

Thanks.

Quoted 31 lines

I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.

---
 scripts/kallsyms.c | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/kallsyms.c +6 -1
index 7ecd2ccba531..54ad86d13784 100644
@@ -112,6 +112,12 @@ static bool is_ignored_symbol(const char *name, char type)
                "__crc_",               /* modversions */
                "__efistub_",           /* arm64 EFI stub namespace */
                "__kvm_nvhe_",          /* arm64 non-VHE KVM namespace */
+               "__AArch64ADRPThunk_",  /* arm64 lld */
+               "__ARMV5PILongThunk_",  /* arm lld */
+               "__ARMV7PILongThunk_",
+               "__ThumbV7PILongThunk_",
+               "__LA25Thunk_",         /* mips lld */
+               "__microLA25Thunk_",
                NULL
        };
 
--

2.29.2

Signature · 2 lines
Best Regards
Masahiro Yamada
#2 lore raw
In reply to Arnd Bergmann

Hi Arnd,

Quoted 16 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

ARM randconfig builds with lld sometimes show a build failure from kallsyms:

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround

The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.

This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.

Quoted 6 lines

I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.

This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?

In this context .. is there a chance to apply [1] after all ? This is getting really time consuming and annoying, and I really dislike having to fix the same problem over and over again.

Thanks,
Guenter

---
[1] https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/

#3 lore raw
Quoted 24 lines · Guenter Roeck
Quoted 16 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

ARM randconfig builds with lld sometimes show a build failure from kallsyms:

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround

The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.

This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.

Quoted 6 lines

I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.

Quoted 3 lines

This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?

The way I did it previously was to hack Kbuild to not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.

I suppose we could improve the situation if scripts/link-vmlinux.sh was able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.

Arnd

#4 lore raw
Quoted 25 lines · Arnd Bergmann
Quoted 24 lines · Guenter Roeck
Quoted 22 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

ARM randconfig builds with lld sometimes show a build failure from kallsyms:

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround

The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.

This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.

I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.

Quoted 7 lines
Quoted 3 lines

This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?

The way I did it previously was to hack Kbuild to not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.

Problem with that is that we have a non-deterministic problem: The build fails for us on some build servers, but we are unable to reproduce the problem when building the same image manually on a development server. That is similar to what I had observed before, where powerpc builds would pass on one server, but the same kernel with the same configuration would fail to build on a second almost identical server. It would really be great if we can find a better solution.

Quoted 4 lines

I suppose we could improve the situation if scripts/link-vmlinux.sh was able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.

Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.

Thanks,
Guenter

#5 lore raw
Quoted 35 lines · Guenter Roeck
Quoted 25 lines · Arnd Bergmann
Quoted 24 lines · Guenter Roeck
Quoted 22 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

ARM randconfig builds with lld sometimes show a build failure from kallsyms:

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround

The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.

This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.

I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.

Quoted 7 lines
Quoted 3 lines

This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?

The way I did it previously was to hack Kbuild to not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.

Quoted 7 lines

Problem with that is that we have a non-deterministic problem: The build fails for us on some build servers, but we are unable to reproduce the problem when building the same image manually on a development server. That is similar to what I had observed before, where powerpc builds would pass on one server, but the same kernel with the same configuration would fail to build on a second almost identical server. It would really be great if we can find a better solution.

Right, that sucks. I suppose removing the ignore-lists from scripts/kallsyms.c would make it more easily reproducible after a few local randconfig builds, at least enough to add some form of scripting that is able to print the names of the generated symbols.

Arnd

#6 lore raw
In reply to Guenter Roeck
Quoted 43 lines · Guenter Roeck
Quoted 25 lines · Arnd Bergmann
Quoted 24 lines · Guenter Roeck
Quoted 22 lines · Arnd Bergmann

From: Arnd Bergmann <[email protected]>

ARM randconfig builds with lld sometimes show a build failure from kallsyms:

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround

The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.

This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.

I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.

Quoted 7 lines
Quoted 3 lines

This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?

The way I did it previously was to hack Kbuild to not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.

Problem with that is that we have a non-deterministic problem: The build fails for us on some build servers, but we are unable to reproduce the problem when building the same image manually on a development server. That is similar to what I had observed before, where powerpc builds would pass on one server, but the same kernel with the same configuration would fail to build on a second almost identical server. It would really be great if we can find a better solution.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

I suppose we could improve the situation if scripts/link-vmlinux.sh was able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.

Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.

Turns out it wasn't that useful.

chromeos-kernel-5_10-5.10.42-r406: Symbol file differences:
chromeos-kernel-5_10-5.10.42-r406: 7c7
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325c8 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c0 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: 9,13c9,13

chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d0 g       .rodata	0000000000000000 kallsyms_num_syms
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d8 g       .rodata	0000000000000000 kallsyms_names
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cd7f0 g       .rodata	0000000000000000 kallsyms_markers
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cdb18 g       .rodata	0000000000000000 kallsyms_token_table
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cde78 g       .rodata	0000000000000000 kallsyms_token_index
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c8 g       .rodata	0000000000000000 kallsyms_num_syms
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325d0 g       .rodata	0000000000000000 kallsyms_names
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cd7d8 g       .rodata	0000000000000000 kallsyms_markers
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cdb00 g       .rodata	0000000000000000 kallsyms_token_table
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cde60 g       .rodata	0000000000000000 kallsyms_token_index

I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.

What am I missing ?

Thanks,
Guenter

#7 lore raw
Quoted 30 lines · Guenter Roeck
Quoted 8 lines · Guenter Roeck
Quoted 3 lines

I suppose we could improve the situation if scripts/link-vmlinux.sh was able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.

Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.

Turns out it wasn't that useful.

chromeos-kernel-5_10-5.10.42-r406: Symbol file differences:
chromeos-kernel-5_10-5.10.42-r406: 7c7
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325c8 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c0 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: 9,13c9,13

chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d0 g       .rodata   0000000000000000 kallsyms_num_syms
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d8 g       .rodata   0000000000000000 kallsyms_names
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cd7f0 g       .rodata   0000000000000000 kallsyms_markers
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cdb18 g       .rodata   0000000000000000 kallsyms_token_table
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cde78 g       .rodata   0000000000000000 kallsyms_token_index
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c8 g       .rodata   0000000000000000 kallsyms_num_syms
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325d0 g       .rodata   0000000000000000 kallsyms_names
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cd7d8 g       .rodata   0000000000000000 kallsyms_markers
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cdb00 g       .rodata   0000000000000000 kallsyms_token_table
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cde60 g       .rodata   0000000000000000 kallsyms_token_index
Quoted 4 lines

I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.

What am I missing ?

I probably misremembered the part about 'objdump --syms' and there was something more to it.

Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I assume the first symbol that has a different size is 'kallsyms_offsets', which is generated by kallsyms.

I see that link-vmlinux.sh already compares the System.map files, using "cmp -s System.map .tmp_System.map", which is roughly the same as the objdump --syms diff you got, so comparing these files probably doesn't help either. However, comparing the .tmp_System.map file with the previous version might reveal the problem. This might need another step to filter out the address and only compare the symbol names.

Arnd

#8 lore raw
Quoted 34 lines · Arnd Bergmann
Quoted 33 lines · Guenter Roeck
Quoted 8 lines · Guenter Roeck
Quoted 3 lines

I suppose we could improve the situation if scripts/link-vmlinux.sh was able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.

Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.

Turns out it wasn't that useful.

chromeos-kernel-5_10-5.10.42-r406: Symbol file differences:
chromeos-kernel-5_10-5.10.42-r406: 7c7
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325c8 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c0 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: 9,13c9,13

chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d0 g       .rodata   0000000000000000 kallsyms_num_syms
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d8 g       .rodata   0000000000000000 kallsyms_names
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cd7f0 g       .rodata   0000000000000000 kallsyms_markers
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cdb18 g       .rodata   0000000000000000 kallsyms_token_table
chromeos-kernel-5_10-5.10.42-r406: < 00000000000cde78 g       .rodata   0000000000000000 kallsyms_token_index
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c8 g       .rodata   0000000000000000 kallsyms_num_syms
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325d0 g       .rodata   0000000000000000 kallsyms_names
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cd7d8 g       .rodata   0000000000000000 kallsyms_markers
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cdb00 g       .rodata   0000000000000000 kallsyms_token_table
chromeos-kernel-5_10-5.10.42-r406: > 00000000000cde60 g       .rodata   0000000000000000 kallsyms_token_index

I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.

Quoted 7 lines
Quoted 1 line

What am I missing ?

I probably misremembered the part about 'objdump --syms' and there was something more to it.

Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I

It is the difference between step 1 and 2. Why would diff on objdump not show the additional symbol ? Is it possible that the symbol is not added to the object file ?

Quoted 2 lines

assume the first symbol that has a different size is 'kallsyms_offsets', which is generated by kallsyms.

I'll give it another try and run diff -u.

Quoted 7 lines

I see that link-vmlinux.sh already compares the System.map files, using "cmp -s System.map .tmp_System.map", which is roughly the same as the objdump --syms diff you got, so comparing these files probably doesn't help either. However, comparing the .tmp_System.map file with the previous version might reveal the problem. This might need another step to filter out the address and only compare the symbol names.

I'll do that as well.

Thanks!
Guenter

#9 lore raw
Quoted 9 lines · Guenter Roeck
Quoted 8 lines · Arnd Bergmann
Quoted 4 lines

I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.

What am I missing ?

I probably misremembered the part about 'objdump --syms' and there was something more to it.

Quoted 6 lines
Quoted 2 lines

Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I

It is the difference between step 1 and 2. Why would diff on objdump not show the additional symbol ? Is it possible that the symbol is not added to the object file ?

Note sure. The symbol must be in the object file, but perhaps the 'objdump --syms' output skips a different set of symbols compared to the list that is used as input for kallsyms, which comes from '${NM}'.

Comparing the nm output might be another thing to try.

Arnd

#10 lore raw
Quoted 17 lines · Arnd Bergmann
Quoted 9 lines · Guenter Roeck
Quoted 8 lines · Arnd Bergmann
Quoted 4 lines

I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.

What am I missing ?

I probably misremembered the part about 'objdump --syms' and there was something more to it.

Quoted 6 lines
Quoted 2 lines

Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I

It is the difference between step 1 and 2. Why would diff on objdump not show the additional symbol ? Is it possible that the symbol is not added to the object file ?

Quoted 6 lines

Note sure. The symbol must be in the object file, but perhaps the 'objdump --syms' output skips a different set of symbols compared to the list that is used as input for kallsyms, which comes from '${NM}'.

Comparing the nm output might be another thing to try.

Just following up on this: As Murphy would have told us, the problem disappeared while trying to track it down. We'll add some instrumentation into the ChromeOS kernel build to get data once/if/when it shows up again. When that happens, we'll try to come up with a patch to show the symbol file differences in the kernel build and submit it upstream.

Thanks,
Guenter