linux-yocto 5.15.48 reproducibility failure
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Are you able to look at that soon or should I file a bug?
Regards,
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 10 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/
and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...
Cheers,
Richard
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...
A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
So one of the build hosts is failing the test, and another isn't.
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Bruce
Quoted 4 lines
Cheers,
Richard
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 21 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...
Quoted 8 lines
A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
So one of the build hosts is failing the test, and another isn't.
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Was this issue ever resolved? I'm seeing it on kirkstone now too :-(
Steve
Quoted 27 lines · Steve Sakoman
Quoted 26 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
Quoted 5 lines
Quoted 3 lines
So one of the build hosts is failing the test, and another isn't.
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Was this issue ever resolved? I'm seeing it on kirkstone now too :-(
No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73
but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.
Cheers,
Richard
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
So one of the build hosts is failing the test, and another isn't.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Was this issue ever resolved? I'm seeing it on kirkstone now too :-(
No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:
That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.
Quoted 5 lines
https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73
but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.
That being said, merge_config uses the compiler like so:
# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols with Kconfig default # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET
And we call it like this:
CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH} merge_config.sh -O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1
So for the merging of fragments, we are already covered.
and kernel_do_configure calls: KERNEL_CONFIG_COMMAND as the last thing it does, which is:
KERNEL_CONFIG_COMMAND ?= "oe_runmake_call -C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig || oe_runmake -C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"
Which already does set CC to KERNEL_CC.
So unless I'm missing how KCONFIG_CONFIG_COMMAND is sneaking in, I expect the behaviour will still be the same.
I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.
Bruce
Quoted 6 lines
Cheers,
Richard
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
So one of the build hosts is failing the test, and another isn't.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Was this issue ever resolved? I'm seeing it on kirkstone now too :-(
No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:
That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.
Quoted 5 lines
https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73
but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.
That being said,
merge_configuses the compiler like so:# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols withKconfigdefault # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGETAnd we call it like this:
CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH}
merge_config.sh-O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1So for the merging of fragments, we are already covered.
and
kernel_do_configurecalls:KERNEL_CONFIG_COMMANDas the last thing it does, which is:
KERNEL_CONFIG_COMMAND?= "oe_runmake_call-C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig ||oe_runmake-C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"
Quoted 8 lines
Which already does set CC to
KERNEL_CC.So unless I'm missing how
KCONFIG_CONFIG_COMMANDis sneaking in, I expect the behaviour will still be the same.I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.
You're not missing anything, it didn't help.
We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use /bin/echo instead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:
<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input
It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 102 lines · Richard Purdie via lists.openembedded.org
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
So one of the build hosts is failing the test, and another isn't.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Was this issue ever resolved? I'm seeing it on kirkstone now too :-(
No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:
That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.
Quoted 5 lines
https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73
but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.
That being said,
merge_configuses the compiler like so:# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols withKconfigdefault # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGETAnd we call it like this:
CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH}
merge_config.sh-O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1So for the merging of fragments, we are already covered.
and
kernel_do_configurecalls:KERNEL_CONFIG_COMMANDas the last thing it does, which is:
KERNEL_CONFIG_COMMAND?= "oe_runmake_call-C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig ||oe_runmake-C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"Quoted 8 lines
Which already does set CC to
KERNEL_CC.So unless I'm missing how
KCONFIG_CONFIG_COMMANDis sneaking in, I expect the behaviour will still be the same.I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.
You're not missing anything, it didn't help.
We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use
/bin/echoinstead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input
Quoted 2 lines
It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.
pokybuild@ubuntu1804-ty-3:~$ /bin/dash
$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character
<stdin>:1:29: error: missing terminating " character
<stdin>:2:5: error: expected ':' before '+' token
<stdin>:2:7: warning: missing terminating " character
<stdin>:2:7: error: missing terminating " character
<stdin>:2:1: error: expected declaration or statement at end of input
1
$ exit
pokybuild@ubuntu1804-ty-3:~$ bash
pokybuild@ubuntu1804-ty-3:~$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
0I think the command gets executed by the code in scripts/kconfig/preprocess.c:do_shell() which passes it to popen() which uses /bin/sh. Since this command has characters with interesting quoting requirements, I suspect our easiest solution is to submit a patch upstream which moves the content to a file which we reference in the command instead of using echo. cc-can-link.sh and similar kind of give some prior art for it.
Cheers,
Richard
Quoted 123 lines · Richard Purdie
Quoted 102 lines · Richard Purdie via lists.openembedded.org
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
So one of the build hosts is failing the test, and another isn't.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Was this issue ever resolved? I'm seeing it on kirkstone now too :-(
No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:
That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.
Quoted 5 lines
https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73
but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.
That being said,
merge_configuses the compiler like so:# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols withKconfigdefault # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGETAnd we call it like this:
CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH}
merge_config.sh-O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1So for the merging of fragments, we are already covered.
and
kernel_do_configurecalls:KERNEL_CONFIG_COMMANDas the last thing it does, which is:
KERNEL_CONFIG_COMMAND?= "oe_runmake_call-C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig ||oe_runmake-C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"Quoted 8 lines
Which already does set CC to
KERNEL_CC.So unless I'm missing how
KCONFIG_CONFIG_COMMANDis sneaking in, I expect the behaviour will still be the same.I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.
You're not missing anything, it didn't help.
We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use
/bin/echoinstead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of inputQuoted 2 lines
It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.
pokybuild@ubuntu1804-ty-3:~$ /bin/dash $ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $? <stdin>: In function 'foo': <stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input 1 $ exit pokybuild@ubuntu1804-ty-3:~$ bash pokybuild@ubuntu1804-ty-3:~$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $? 0
Quoted 8 lines
I think the command gets executed by the code in scripts/kconfig/preprocess.c:do_shell() which passes it to
popen()which uses/bin/sh. Since this command has characters with interesting quoting requirements, I suspect our easiest solution is to submit a patch upstream which moves the content to a file which we reference in the command instead of using echo.cc-can-link.shand similar kind of give some prior art for it.
Interesting. Since you can change the shell for Make, I could have sworn we could change the shell in that environment as well.
Either way, it looks like this is another kernel build bash host side requirement. Which is ok, since the kernel build has never claimed multi-shell support.
We can obviously test against linux-yocto. Once I get a few more 5.19 builds passing, I can have a go at it (unless you've already tried something on that front).
Bruce
Quoted 4 lines
Cheers,
Richard
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 137 lines · Bruce Ashfield
Quoted 123 lines · Richard Purdie
Quoted 102 lines · Richard Purdie via lists.openembedded.org
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Quoted 6 lines
Quoted 1 line
Are you able to look at that soon or should I file a bug?
FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/and
CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=ychanged value...A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.
So one of the build hosts is failing the test, and another isn't.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
I'm not sure how to fix that, unless we revert the change. I'll ponder it more.
Was this issue ever resolved? I'm seeing it on kirkstone now too :-(
No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:
That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.
Quoted 5 lines
https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73
but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.
That being said,
merge_configuses the compiler like so:# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols withKconfigdefault # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGETAnd we call it like this:
CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH}
merge_config.sh-O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1So for the merging of fragments, we are already covered.
and
kernel_do_configurecalls:KERNEL_CONFIG_COMMANDas the last thing it does, which is:
KERNEL_CONFIG_COMMAND?= "oe_runmake_call-C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig ||oe_runmake-C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"Quoted 8 lines
Which already does set CC to
KERNEL_CC.So unless I'm missing how
KCONFIG_CONFIG_COMMANDis sneaking in, I expect the behaviour will still be the same.I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.
You're not missing anything, it didn't help.
We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use
/bin/echoinstead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of inputQuoted 2 lines
It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.
pokybuild@ubuntu1804-ty-3:~$ /bin/dash $ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $? <stdin>: In function 'foo': <stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input 1 $ exit pokybuild@ubuntu1804-ty-3:~$ bash pokybuild@ubuntu1804-ty-3:~$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $? 0Quoted 8 lines
I think the command gets executed by the code in scripts/kconfig/preprocess.c:do_shell() which passes it to
popen()which uses/bin/sh. Since this command has characters with interesting quoting requirements, I suspect our easiest solution is to submit a patch upstream which moves the content to a file which we reference in the command instead of using echo.cc-can-link.shand similar kind of give some prior art for it.Interesting. Since you can change the shell for Make, I could have sworn we could change the shell in that environment as well.
Quoted 8 lines
Either way, it looks like this is another kernel build bash host side requirement. Which is ok, since the kernel build has never claimed multi-shell support.
We can obviously test against linux-yocto. Once I get a few more 5.19 builds passing, I can have a go at it (unless you've already tried something on that front).
I submitted a patch yesterday, I didn't get any reply yet but we are in the middle of the merge window:
https://lore.kernel.org/all/[email protected]/
I also sent that to the AB.
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 11 lines · Alexandre Belloni
Hello Bruce,
I got the following reproducible build failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio
To me, it looks like the in-kernel config changed, did the compression alg change?
Are you able to look at that soon or should I file a bug?
I won't be doing any more reproducibility work until the end of next week at this point.
FWIW, I have more -stable changes queued already, but the can stay pending until we sort these out.
Bruce
Quoted 7 lines
Regards,
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II