Skip to contentSkip to conversation

linux-yocto 5.15.48 reproducibility failure

DISCUSSION Alexandre Belloni 4 participants 11 messages Jul 6, 2022 – Aug 5, 2022 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
Quoted 10 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

Cheers,

Richard

#2 lore raw
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

So one of the build hosts is failing the test, and another isn't.

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Bruce

Quoted 4 lines

Cheers,

Richard

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#3 lore raw
Quoted 21 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

Quoted 8 lines

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

So one of the build hosts is failing the test, and another isn't.

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Was this issue ever resolved? I'm seeing it on kirkstone now too :-(

Steve

#4 lore raw
Quoted 27 lines · Steve Sakoman
Quoted 26 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

Quoted 5 lines
Quoted 3 lines

So one of the build hosts is failing the test, and another isn't.

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Was this issue ever resolved? I'm seeing it on kirkstone now too :-(

No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73

but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.

Cheers,

Richard

#5 lore raw
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

So one of the build hosts is failing the test, and another isn't.

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Was this issue ever resolved? I'm seeing it on kirkstone now too :-(

No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:

That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.

Quoted 5 lines

https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73

but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.

That being said, merge_config uses the compiler like so:

# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols with Kconfig default # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET

And we call it like this:

CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH} merge_config.sh -O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1

So for the merging of fragments, we are already covered.

and kernel_do_configure calls: KERNEL_CONFIG_COMMAND as the last thing it does, which is:

KERNEL_CONFIG_COMMAND ?= "oe_runmake_call -C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig || oe_runmake -C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"

Which already does set CC to KERNEL_CC.

So unless I'm missing how KCONFIG_CONFIG_COMMAND is sneaking in, I expect the behaviour will still be the same.

I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.

Bruce

Quoted 6 lines

Cheers,

Richard

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

#6 lore raw
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

So one of the build hosts is failing the test, and another isn't.

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Was this issue ever resolved? I'm seeing it on kirkstone now too :-(

No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:

That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.

Quoted 5 lines

https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73

but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.

That being said, merge_config uses the compiler like so:

# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols with Kconfig default # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET

And we call it like this:

CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH} merge_config.sh -O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1

So for the merging of fragments, we are already covered.

and kernel_do_configure calls: KERNEL_CONFIG_COMMAND as the last thing it does, which is:

KERNEL_CONFIG_COMMAND ?= "oe_runmake_call -C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig || oe_runmake -C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"

Quoted 8 lines

Which already does set CC to KERNEL_CC.

So unless I'm missing how KCONFIG_CONFIG_COMMAND is sneaking in, I expect the behaviour will still be the same.

I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.

You're not missing anything, it didn't help.

We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use /bin/echo instead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:

<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input

It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 102 lines · Richard Purdie via lists.openembedded.org
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

So one of the build hosts is failing the test, and another isn't.

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Was this issue ever resolved? I'm seeing it on kirkstone now too :-(

No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:

That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.

Quoted 5 lines

https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73

but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.

That being said, merge_config uses the compiler like so:

# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols with Kconfig default # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET

And we call it like this:

CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH} merge_config.sh -O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1

So for the merging of fragments, we are already covered.

and kernel_do_configure calls: KERNEL_CONFIG_COMMAND as the last thing it does, which is:

KERNEL_CONFIG_COMMAND ?= "oe_runmake_call -C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig || oe_runmake -C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"

Quoted 8 lines

Which already does set CC to KERNEL_CC.

So unless I'm missing how KCONFIG_CONFIG_COMMAND is sneaking in, I expect the behaviour will still be the same.

I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.

You're not missing anything, it didn't help.

We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use /bin/echo instead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:

<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input

Quoted 2 lines

It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.

pokybuild@ubuntu1804-ty-3:~$ /bin/dash
$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character
<stdin>:1:29: error: missing terminating " character
<stdin>:2:5: error: expected ':' before '+' token
<stdin>:2:7: warning: missing terminating " character
<stdin>:2:7: error: missing terminating " character
<stdin>:2:1: error: expected declaration or statement at end of input
1
$ exit
pokybuild@ubuntu1804-ty-3:~$ bash
pokybuild@ubuntu1804-ty-3:~$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
0

I think the command gets executed by the code in scripts/kconfig/preprocess.c:do_shell() which passes it to popen() which uses /bin/sh. Since this command has characters with interesting quoting requirements, I suspect our easiest solution is to submit a patch upstream which moves the content to a file which we reference in the command instead of using echo. cc-can-link.sh and similar kind of give some prior art for it.

Cheers,

Richard

#9 lore raw
Quoted 123 lines · Richard Purdie
Quoted 102 lines · Richard Purdie via lists.openembedded.org
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

So one of the build hosts is failing the test, and another isn't.

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Was this issue ever resolved? I'm seeing it on kirkstone now too :-(

No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:

That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.

Quoted 5 lines

https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73

but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.

That being said, merge_config uses the compiler like so:

# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols with Kconfig default # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET

And we call it like this:

CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH} merge_config.sh -O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1

So for the merging of fragments, we are already covered.

and kernel_do_configure calls: KERNEL_CONFIG_COMMAND as the last thing it does, which is:

KERNEL_CONFIG_COMMAND ?= "oe_runmake_call -C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig || oe_runmake -C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"

Quoted 8 lines

Which already does set CC to KERNEL_CC.

So unless I'm missing how KCONFIG_CONFIG_COMMAND is sneaking in, I expect the behaviour will still be the same.

I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.

You're not missing anything, it didn't help.

We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use /bin/echo instead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:

<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input

Quoted 2 lines

It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.

pokybuild@ubuntu1804-ty-3:~$ /bin/dash
$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character
<stdin>:1:29: error: missing terminating " character
<stdin>:2:5: error: expected ':' before '+' token
<stdin>:2:7: warning: missing terminating " character
<stdin>:2:7: error: missing terminating " character
<stdin>:2:1: error: expected declaration or statement at end of input
1
$ exit
pokybuild@ubuntu1804-ty-3:~$ bash
pokybuild@ubuntu1804-ty-3:~$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
0
Quoted 8 lines

I think the command gets executed by the code in scripts/kconfig/preprocess.c:do_shell() which passes it to popen() which uses /bin/sh. Since this command has characters with interesting quoting requirements, I suspect our easiest solution is to submit a patch upstream which moves the content to a file which we reference in the command instead of using echo. cc-can-link.sh and similar kind of give some prior art for it.

Interesting. Since you can change the shell for Make, I could have sworn we could change the shell in that environment as well.

Either way, it looks like this is another kernel build bash host side requirement. Which is ok, since the kernel build has never claimed multi-shell support.

We can obviously test against linux-yocto. Once I get a few more 5.19 builds passing, I can have a go at it (unless you've already tried something on that front).

Bruce

Quoted 4 lines

Cheers,

Richard

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#10 lore raw
Quoted 137 lines · Bruce Ashfield
Quoted 123 lines · Richard Purdie
Quoted 102 lines · Richard Purdie via lists.openembedded.org
Quoted 77 lines · Bruce Ashfield
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Steve Sakoman
Quoted 28 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
Quoted 9 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Quoted 6 lines
Quoted 1 line

Are you able to look at that soon or should I file a bug?

FWIW,
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20220705-g8ltxczy/packages/diff-html/

and CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y changed value...

A quick look at the -stable queue to init/Kconfig does indeed show that option arriving .. from my scan of the change, it looks like there are different compilers on the hosts, and hence that check is detecting a buggy gcc (or not) and changing the value.

So one of the build hosts is failing the test, and another isn't.

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

I'm not sure how to fix that, unless we revert the change. I'll ponder it more.

Was this issue ever resolved? I'm seeing it on kirkstone now too :-(

No, it wasn't. I took a quick look and my best guess is that we now have the kernel wanting to do $CC tests in kConfig and we don't set CC properly for all our make config calls. I chucked this in master-next as a quick guess at a possible fix:

That's what we've had to do with the linker in the past, so that is generally speaking the right place to force the compiler.

Quoted 5 lines

https://git.yoctoproject.org/poky/commit/?h=master-next&id=944cc04d1ed316a5652f3d058b966d4cfb9a6b73

but others might know better where we might be missing the CC settings. I suspect we need to clean up our parameters a bit looking at the different call site differences.

That being said, merge_config uses the compiler like so:

# Use the merged file as the starting point for:
# alldefconfig: Fills in any missing symbols with Kconfig default # allnoconfig: Fills in any missing symbols with # CONFIG_* is not set make HOSTCC="$HOSTCC" HOSTLD="$HOSTLD" CC="$CC" LD="$LD" \
KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET

And we call it like this:

CFLAGS="${CFLAGS} ${TOOLCHAIN_OPTIONS}" HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}" CC="${KERNEL_CC}" LD="${KERNEL_LD}" ARCH=${ARCH} merge_config.sh -O ${B} ${config_flags} ${configs} > ${meta_dir}/cfg/merge_config_build.log 2>&1

So for the merging of fragments, we are already covered.

and kernel_do_configure calls: KERNEL_CONFIG_COMMAND as the last thing it does, which is:

KERNEL_CONFIG_COMMAND ?= "oe_runmake_call -C ${S} CC="${KERNEL_CC}" LD="${KERNEL_LD}" O=${B} olddefconfig || oe_runmake -C ${S} O=${B} CC="${KERNEL_CC}" LD="${KERNEL_LD}" oldnoconfig"

Quoted 8 lines

Which already does set CC to KERNEL_CC.

So unless I'm missing how KCONFIG_CONFIG_COMMAND is sneaking in, I expect the behaviour will still be the same.

I spent the day digging through 5.19 kernel issues, so only had a few minutes to look at this tonight .. and it is late, so I very easily could have missed something.

You're not missing anything, it didn't help.

We're seeing this on ubuntu1804 so I logged in there and experimented. CC is set correctly. What isn't working well is the "echo". If I change the command to use /bin/echo instead of plain "echo" (which is probably a shell builtin), it works. Otherwise running manually I see:

<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character <stdin>:1:29: error: missing terminating " character <stdin>:2:5: error: expected ':' before '+' token <stdin>:2:7: warning: missing terminating " character <stdin>:2:7: error: missing terminating " character <stdin>:2:1: error: expected declaration or statement at end of input

Quoted 2 lines

It is looking like this is a dash vs. bash issue. Not sure how we fix it yet but it points to where the issue may be.

pokybuild@ubuntu1804-ty-3:~$ /bin/dash
$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
<stdin>: In function 'foo':
<stdin>:1:29: warning: missing terminating " character
<stdin>:1:29: error: missing terminating " character
<stdin>:2:5: error: expected ':' before '+' token
<stdin>:2:7: warning: missing terminating " character
<stdin>:2:7: error: missing terminating " character
<stdin>:2:1: error: expected declaration or statement at end of input
1
$ exit
pokybuild@ubuntu1804-ty-3:~$ bash
pokybuild@ubuntu1804-ty-3:~$ echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | x86_64-poky-linux-gcc -x c - -c -o /dev/null; echo $?
0
Quoted 8 lines

I think the command gets executed by the code in scripts/kconfig/preprocess.c:do_shell() which passes it to popen() which uses /bin/sh. Since this command has characters with interesting quoting requirements, I suspect our easiest solution is to submit a patch upstream which moves the content to a file which we reference in the command instead of using echo. cc-can-link.sh and similar kind of give some prior art for it.

Interesting. Since you can change the shell for Make, I could have sworn we could change the shell in that environment as well.

Quoted 8 lines

Either way, it looks like this is another kernel build bash host side requirement. Which is ok, since the kernel build has never claimed multi-shell support.

We can obviously test against linux-yocto. Once I get a few more 5.19 builds passing, I can have a go at it (unless you've already tried something on that front).

I submitted a patch yesterday, I didn't get any reply yet but we are in the middle of the merge window:
https://lore.kernel.org/all/[email protected]/

I also sent that to the AB.

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#11 lore raw
In reply to Alexandre Belloni
Quoted 11 lines · Alexandre Belloni

Hello Bruce,

I got the following reproducible build failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/1116/steps/12/logs/stdio

To me, it looks like the in-kernel config changed, did the compression alg change?

Are you able to look at that soon or should I file a bug?

I won't be doing any more reproducibility work until the end of next week at this point.

FWIW, I have more -stable changes queued already, but the can stay pending until we sort these out.

Bruce

Quoted 7 lines

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II