kallsyms: fix nonconverging kallsyms table with lld
PATCH 1/1 kallsyms: fix nonconverging kallsyms table with lld
From: Arnd Bergmann <[email protected]>
ARM randconfig builds with lld sometimes show a build failure from kallsyms:
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround
The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.
This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.
I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "__long_branch_" and "__plt_" prefixes would have to get added as well.
--- scripts/kallsyms.c | 6 ++++++ 1 file changed, 6 insertions(+)
scripts/kallsyms.c +6 -0
@@ -112,6 +112,12 @@ static bool is_ignored_symbol(const char *name, char type)"__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ "__kvm_nvhe_" ,/* arm64 non-VHE KVM namespace */ +"__AArch64ADRPThunk_" ,/* arm64 lld */ +"__ARMV5PILongThunk_" ,/* arm lld */ +"__ARMV7PILongThunk_" , +"__ThumbV7PILongThunk_" , +"__LA25Thunk_" ,/* mips lld */ +"__microLA25Thunk_" ,NULL };
Signature · 1 line
2.29.2
Quoted 9 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
ARM randconfig builds with lld sometimes show a build failure from kallsyms:
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround
Quoted 7 lines
The problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.
This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.
Yes, this is a whack-a-mole.
I fixed the typo "idenitifiers" -> "identifiers"
and applied to linux-kbuild.
Thanks.
Quoted 31 lines
I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "
__long_branch_" and "__plt_" prefixes would have to get added as well.Signed-off-by Arnd Bergmann <[email protected]>--- scripts/kallsyms.c | 6 ++++++ 1 file changed, 6 insertions(+)scripts/kallsyms.c +6
-1@@ -112,6 +112,12 @@ static bool is_ignored_symbol(const char *name, char type)"__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ "__kvm_nvhe_" ,/* arm64 non-VHE KVM namespace */ +"__AArch64ADRPThunk_" ,/* arm64 lld */ +"__ARMV5PILongThunk_" ,/* arm lld */ +"__ARMV7PILongThunk_" , +"__ThumbV7PILongThunk_" , +"__LA25Thunk_" ,/* mips lld */ +"__microLA25Thunk_" ,NULL }; --2.29.2
Signature · 2 lines
Best Regards Masahiro Yamada
Hi Arnd,
Quoted 16 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
ARM randconfig builds with lld sometimes show a build failure from kallsyms:
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaroundThe problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.
This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.
Quoted 6 lines
I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "
__long_branch_" and "__plt_" prefixes would have to get added as well.
This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?
In this context .. is there a chance to apply [1] after all ? This is getting really time consuming and annoying, and I really dislike having to fix the same problem over and over again.
Thanks,
Guenter
---
[1] https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/
Quoted 24 lines · Guenter Roeck
Quoted 16 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
ARM randconfig builds with lld sometimes show a build failure from kallsyms:
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaroundThe problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.
This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.
Quoted 6 lines
I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "
__long_branch_" and "__plt_" prefixes would have to get added as well.
Quoted 3 lines
This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?
The way I did it previously was to hack Kbuild to not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.
I suppose we could improve the situation if scripts/link-vmlinux.sh was able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.
Arnd
Quoted 25 lines · Arnd Bergmann
Quoted 24 lines · Guenter Roeck
Quoted 22 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
ARM randconfig builds with lld sometimes show a build failure from kallsyms:
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaroundThe problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.
This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.
I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "
__long_branch_" and "__plt_" prefixes would have to get added as well.
Quoted 7 lines
Quoted 3 lines
This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?
The way I did it previously was to hack
Kbuildto not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.
Problem with that is that we have a non-deterministic problem: The build fails for us on some build servers, but we are unable to reproduce the problem when building the same image manually on a development server. That is similar to what I had observed before, where powerpc builds would pass on one server, but the same kernel with the same configuration would fail to build on a second almost identical server. It would really be great if we can find a better solution.
Quoted 4 lines
I suppose we could improve the situation if
scripts/link-vmlinux.shwas able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.
Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.
Thanks,
Guenter
Quoted 35 lines · Guenter Roeck
Quoted 25 lines · Arnd Bergmann
Quoted 24 lines · Guenter Roeck
Quoted 22 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
ARM randconfig builds with lld sometimes show a build failure from kallsyms:
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaroundThe problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.
This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.
I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "
__long_branch_" and "__plt_" prefixes would have to get added as well.Quoted 7 lines
Quoted 3 lines
This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?
The way I did it previously was to hack
Kbuildto not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.
Quoted 7 lines
Problem with that is that we have a non-deterministic problem: The build fails for us on some build servers, but we are unable to reproduce the problem when building the same image manually on a development server. That is similar to what I had observed before, where powerpc builds would pass on one server, but the same kernel with the same configuration would fail to build on a second almost identical server. It would really be great if we can find a better solution.
Right, that sucks. I suppose removing the ignore-lists from scripts/kallsyms.c would make it more easily reproducible after a few local randconfig builds, at least enough to add some form of scripting that is able to print the names of the generated symbols.
Arnd
Quoted 43 lines · Guenter Roeck
Quoted 25 lines · Arnd Bergmann
Quoted 24 lines · Guenter Roeck
Quoted 22 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
ARM randconfig builds with lld sometimes show a build failure from kallsyms:
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaroundThe problem is the veneers/thunks getting added by the linker extend the symbol table, which in turn leads to more veneers being needed, so it may take a few extra iterations to converge.
This bug has been fixed multiple times before, but comes back every time a new symbol name is used. lld uses a different set of idenitifiers from ld.bfd, so the additional ones need to be added as well.
I looked through the sources and found that arm64 and mips define similar prefixes, so I'm adding those as well, aside from the ones I observed. I'm not sure about powerpc64, which seems to already be handled through a section match, but if it comes back, the "
__long_branch_" and "__plt_" prefixes would have to get added as well.Quoted 7 lines
Quoted 3 lines
This is such a whack-a-mole. The problem is hitting us yet again. I suspect it may be due to a new version of lld using new symbols, but I didn't really try to track it down. Is there an easy way to search for missed symbols ?
The way I did it previously was to hack
Kbuildto not remove the temporary files after a failure, and then compare the "objdump --syms" output of the last two stages.Problem with that is that we have a non-deterministic problem: The build fails for us on some build servers, but we are unable to reproduce the problem when building the same image manually on a development server. That is similar to what I had observed before, where powerpc builds would pass on one server, but the same kernel with the same configuration would fail to build on a second almost identical server. It would really be great if we can find a better solution.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
I suppose we could improve the situation if
scripts/link-vmlinux.shwas able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.
Turns out it wasn't that useful.
chromeos-kernel-5_10-5.10.42-r406: Symbol file differences:
chromeos-kernel-5_10-5.10.42-r406: 7c7
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325c8 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c0 g .rodata 0000000000000000 kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: 9,13c9,13
chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d0 g .rodata 0000000000000000 kallsyms_num_syms chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d8 g .rodata 0000000000000000 kallsyms_names chromeos-kernel-5_10-5.10.42-r406: < 00000000000cd7f0 g .rodata 0000000000000000 kallsyms_markers chromeos-kernel-5_10-5.10.42-r406: < 00000000000cdb18 g .rodata 0000000000000000 kallsyms_token_table chromeos-kernel-5_10-5.10.42-r406: < 00000000000cde78 g .rodata 0000000000000000 kallsyms_token_index chromeos-kernel-5_10-5.10.42-r406: --- chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c8 g .rodata 0000000000000000 kallsyms_num_syms chromeos-kernel-5_10-5.10.42-r406: > 00000000000325d0 g .rodata 0000000000000000 kallsyms_names chromeos-kernel-5_10-5.10.42-r406: > 00000000000cd7d8 g .rodata 0000000000000000 kallsyms_markers chromeos-kernel-5_10-5.10.42-r406: > 00000000000cdb00 g .rodata 0000000000000000 kallsyms_token_table chromeos-kernel-5_10-5.10.42-r406: > 00000000000cde60 g .rodata 0000000000000000 kallsyms_token_index
I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.
What am I missing ?
Thanks,
Guenter
Quoted 30 lines · Guenter Roeck
Quoted 8 lines · Guenter Roeck
Quoted 3 lines
I suppose we could improve the situation if
scripts/link-vmlinux.shwas able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.
Turns out it wasn't that useful.
chromeos-kernel-5_10-5.10.42-r406: Symbol file differences:
chromeos-kernel-5_10-5.10.42-r406: 7c7
chromeos-kernel-5_10-5.10.42-r406: <00000000000325c8g .rodata 0000000000000000kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: >00000000000325c0g .rodata 0000000000000000kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: 9,13c9,13chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d0 g .rodata 0000000000000000 kallsyms_num_syms chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d8 g .rodata 0000000000000000 kallsyms_names chromeos-kernel-5_10-5.10.42-r406: < 00000000000cd7f0 g .rodata 0000000000000000 kallsyms_markers chromeos-kernel-5_10-5.10.42-r406: < 00000000000cdb18 g .rodata 0000000000000000 kallsyms_token_table chromeos-kernel-5_10-5.10.42-r406: < 00000000000cde78 g .rodata 0000000000000000 kallsyms_token_index chromeos-kernel-5_10-5.10.42-r406: --- chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c8 g .rodata 0000000000000000 kallsyms_num_syms chromeos-kernel-5_10-5.10.42-r406: > 00000000000325d0 g .rodata 0000000000000000 kallsyms_names chromeos-kernel-5_10-5.10.42-r406: > 00000000000cd7d8 g .rodata 0000000000000000 kallsyms_markers chromeos-kernel-5_10-5.10.42-r406: > 00000000000cdb00 g .rodata 0000000000000000 kallsyms_token_table chromeos-kernel-5_10-5.10.42-r406: > 00000000000cde60 g .rodata 0000000000000000 kallsyms_token_index
Quoted 4 lines
I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.
What am I missing ?
I probably misremembered the part about 'objdump --syms' and there was something more to it.
Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I assume the first symbol that has a different size is 'kallsyms_offsets', which is generated by kallsyms.
I see that link-vmlinux.sh already compares the System.map files, using "cmp -s System.map .tmp_System.map", which is roughly the same as the objdump --syms diff you got, so comparing these files probably doesn't help either. However, comparing the .tmp_System.map file with the previous version might reveal the problem. This might need another step to filter out the address and only compare the symbol names.
Arnd
Quoted 34 lines · Arnd Bergmann
Quoted 33 lines · Guenter Roeck
Quoted 8 lines · Guenter Roeck
Quoted 3 lines
I suppose we could improve the situation if
scripts/link-vmlinux.shwas able to do that automatically, and compare the kallsyms output .S file between steps 1 and 2.Comparing the .S files doesn't result in useful data; turns out there are always irrelevant differences. We'll try to run a diff on the output of "objdump --syms". Hopefully that will generate something useful.
Turns out it wasn't that useful.
chromeos-kernel-5_10-5.10.42-r406: Symbol file differences:
chromeos-kernel-5_10-5.10.42-r406: 7c7
chromeos-kernel-5_10-5.10.42-r406: <00000000000325c8g .rodata 0000000000000000kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: ---
chromeos-kernel-5_10-5.10.42-r406: >00000000000325c0g .rodata 0000000000000000kallsyms_relative_base
chromeos-kernel-5_10-5.10.42-r406: 9,13c9,13chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d0 g .rodata 0000000000000000 kallsyms_num_syms chromeos-kernel-5_10-5.10.42-r406: < 00000000000325d8 g .rodata 0000000000000000 kallsyms_names chromeos-kernel-5_10-5.10.42-r406: < 00000000000cd7f0 g .rodata 0000000000000000 kallsyms_markers chromeos-kernel-5_10-5.10.42-r406: < 00000000000cdb18 g .rodata 0000000000000000 kallsyms_token_table chromeos-kernel-5_10-5.10.42-r406: < 00000000000cde78 g .rodata 0000000000000000 kallsyms_token_index chromeos-kernel-5_10-5.10.42-r406: --- chromeos-kernel-5_10-5.10.42-r406: > 00000000000325c8 g .rodata 0000000000000000 kallsyms_num_syms chromeos-kernel-5_10-5.10.42-r406: > 00000000000325d0 g .rodata 0000000000000000 kallsyms_names chromeos-kernel-5_10-5.10.42-r406: > 00000000000cd7d8 g .rodata 0000000000000000 kallsyms_markers chromeos-kernel-5_10-5.10.42-r406: > 00000000000cdb00 g .rodata 0000000000000000 kallsyms_token_table chromeos-kernel-5_10-5.10.42-r406: > 00000000000cde60 g .rodata 0000000000000000 kallsyms_token_indexI thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.
Quoted 7 lines
Quoted 1 line
What am I missing ?
I probably misremembered the part about 'objdump --syms' and there was something more to it.
Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I
It is the difference between step 1 and 2. Why would diff on objdump not show the additional symbol ? Is it possible that the symbol is not added to the object file ?
Quoted 2 lines
assume the first symbol that has a different size is '
kallsyms_offsets', which is generated by kallsyms.
I'll give it another try and run diff -u.
Quoted 7 lines
I see that
link-vmlinux.shalready compares the System.map files, using "cmp -s System.map .tmp_System.map", which is roughly the same as the objdump --syms diff you got, so comparing these files probably doesn't help either. However, comparing the .tmp_System.map file with the previous version might reveal the problem. This might need another step to filter out the address and only compare the symbol names.
I'll do that as well.
Thanks!
Guenter
Quoted 9 lines · Guenter Roeck
Quoted 8 lines · Arnd Bergmann
Quoted 4 lines
I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.
What am I missing ?
I probably misremembered the part about 'objdump --syms' and there was something more to it.
Quoted 6 lines
Quoted 2 lines
Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I
It is the difference between step 1 and 2. Why would diff on objdump not show the additional symbol ? Is it possible that the symbol is not added to the object file ?
Note sure. The symbol must be in the object file, but perhaps the 'objdump --syms' output skips a different set of symbols compared to the list that is used as input for kallsyms, which comes from '${NM}'.
Comparing the nm output might be another thing to try.
Arnd
Quoted 17 lines · Arnd Bergmann
Quoted 9 lines · Guenter Roeck
Quoted 8 lines · Arnd Bergmann
Quoted 4 lines
I thought I'd see the added symbols, but it looks like the only difference between the two files is the addresses.
What am I missing ?
I probably misremembered the part about 'objdump --syms' and there was something more to it.
Quoted 6 lines
Quoted 2 lines
Maybe this was the last version before converging? It looks like the '<' version has one extra symbol ompared to the '>' version. The diff has no context, but I
It is the difference between step 1 and 2. Why would diff on objdump not show the additional symbol ? Is it possible that the symbol is not added to the object file ?
Quoted 6 lines
Note sure. The symbol must be in the object file, but perhaps the 'objdump --syms' output skips a different set of symbols compared to the list that is used as input for kallsyms, which comes from '${NM}'.
Comparing the nm output might be another thing to try.
Just following up on this: As Murphy would have told us, the problem disappeared while trying to track it down. We'll add some instrumentation into the ChromeOS kernel build to get data once/if/when it shows up again. When that happens, we'll try to come up with a patch to show the symbol file differences in the kernel build and submit it upstream.
Thanks,
Guenter