Skip to contentSkip to conversation

kernel.bbclass: explicitly depend on bison-native for deterministic builds

PATCH In review Denys Dmytriyenko 4 participants 10 messages Feb 19, 2018 – Feb 20, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kernel.bbclass: explicitly depend on bison-native for deterministic builds

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target 'scripts/kconfig/zconf.tab.c' failed

In most cases, this dependency comes indirectly via toolchain dependencies, specifically binutils-cross, which pulls bison-native. Different setups, such as with external toolchain, would expose this problem, since correct dependency is not marked explicitly.

---
I'm seeing this now on 4.16-rc2 with external toolchain.

 meta/classes/kernel.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/kernel.bbclass +1 -1
index 943938d..321c0a4 100644
@@ -4,7 +4,7 @@ KERNEL_PACKAGE_NAME ??= "kernel"
 KERNEL_DEPLOYSUBDIR ??= "${@ "" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else d.getVar("KERNEL_PACKAGE_NAME") }"
 
 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
-DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native lzop-native"
+DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native lzop-native bison-native"
 PACKAGE_WRITE_DEPS += "depmodwrapper-cross"
 
 do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot"
Signature · 1 line
2.7.4
#1 lore raw
Quoted 12 lines · Denys Dmytriyenko

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target 'scripts/kconfig/zconf.tab.c' failed
Quoted 8 lines

In most cases, this dependency comes indirectly via toolchain dependencies, specifically binutils-cross, which pulls bison-native. Different setups, such as with external toolchain, would expose this problem, since correct dependency is not marked explicitly.

---
I'm seeing this now on 4.16-rc2 with external toolchain.

I am not against the addition but in fact, it is because 4.16 stopped of using pre-generated files and now requires bison to be available. I am unsure it ought to be on kernel.bbclass at this moment.

Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
#2 lore raw
Quoted 18 lines · Otavio Salvador
Quoted 17 lines · Denys Dmytriyenko

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target 'scripts/kconfig/zconf.tab.c' failed

In most cases, this dependency comes indirectly via toolchain dependencies, specifically binutils-cross, which pulls bison-native. Different setups, such as with external toolchain, would expose this problem, since correct dependency is not marked explicitly.

Quoted 6 lines
Quoted 3 lines

---
I'm seeing this now on 4.16-rc2 with external toolchain.

I am not against the addition but in fact, it is because 4.16 stopped of using pre-generated files and now requires bison to be available.

Ok, good, I was wondering why I didn't see it before 4.16...

Quoted 1 line

I am unsure it ought to be on kernel.bbclass at this moment.

Why not? Because it's 4.16+ specific?

By default bison-native dependency gets added implicitly when built with the toolchain from oe-core, so this doesn't add any extra dependencies, only makes it explicit.

Signature · 1 line
Denys
#3 lore raw
Quoted 29 lines · Denys Dmytriyenko
Quoted 18 lines · Otavio Salvador
Quoted 17 lines · Denys Dmytriyenko

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target 'scripts/kconfig/zconf.tab.c' failed

In most cases, this dependency comes indirectly via toolchain dependencies, specifically binutils-cross, which pulls bison-native. Different setups, such as with external toolchain, would expose this problem, since correct dependency is not marked explicitly.

Quoted 6 lines
Quoted 3 lines

---
I'm seeing this now on 4.16-rc2 with external toolchain.

I am not against the addition but in fact, it is because 4.16 stopped of using pre-generated files and now requires bison to be available.

Ok, good, I was wondering why I didn't see it before 4.16...

Quoted 8 lines
Quoted 1 line

I am unsure it ought to be on kernel.bbclass at this moment.

Why not? Because it's 4.16+ specific?

By default bison-native dependency gets added implicitly when built with the toolchain from oe-core, so this doesn't add any extra dependencies, only makes it explicit.

it will be overcompensating for < 4.16 kernel recipes where this dependency is not needed. it might be a legit dependency for another component, that doesn't stage this package in kernel recipe-specific sysroot but this change will do that. So its not completely innocent change.

#4 lore raw
Quoted 51 lines · Khem Raj
Quoted 42 lines · Denys Dmytriyenko
Quoted 25 lines · Otavio Salvador
Quoted 22 lines · Denys Dmytriyenko

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required
for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target

'scripts/kconfig/zconf.tab.c' failed

In most cases, this dependency comes indirectly via toolchain dependencies,
specifically binutils-cross, which pulls bison-native. Different setups, such as with external toolchain, would expose this problem, since correct
dependency is not marked explicitly.

Quoted 7 lines
Quoted 3 lines

---
I'm seeing this now on 4.16-rc2 with external toolchain.

I am not against the addition but in fact, it is because 4.16 stopped of using pre-generated files and now requires bison to be available.

Ok, good, I was wondering why I didn't see it before 4.16...

Quoted 1 line

I am unsure it ought to be on kernel.bbclass at this moment.

Quoted 8 lines

Why not? Because it's 4.16+ specific?

By default bison-native dependency gets added implicitly when built with the
toolchain from oe-core, so this doesn't add any extra dependencies, only makes
it explicit.

Quoted 4 lines

it will be overcompensating for < 4.16 kernel recipes where this dependency is not needed. it might be a legit dependency for another component, that doesn't stage this package in kernel recipe-specific sysroot but this change will do that. So its not completely innocent change.

I'd be in favor of matching the version and adding it if it is a 4.16 or newer kernel. Adding for prior versions seems wrong for me.

Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
#5 lore raw

I'm OK with adding bison-native for all versions.

When we added bc-native 2+ years ago:
commit c067e52cffe002de3b39aa1bced308dd532859c1

Author: Alejandro Hernandez <[email protected]>
Date:   Thu Oct 2 15:33:23 2014 -0500

kernel: Added bc-native as DEPENDS

The makefile checks for bc during for compilation

it was also needed only for the latest kernel and we didn't make it conditional on the kernel version. Now with RSS it's even less dangerous as the bison-native will appear only in kernel's RSS and as Denys said it's quite common dependency so many components pull it into RSS anyway.

Quoted 73 lines · Otavio Salvador
Quoted 21 lines · Khem Raj
Quoted 20 lines · Denys Dmytriyenko
Quoted 18 lines · Otavio Salvador
Quoted 15 lines · Denys Dmytriyenko

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required
for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target

'scripts/kconfig/zconf.tab.c' failed

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In most cases, this dependency comes indirectly via toolchain dependencies,
specifically binutils-cross, which pulls bison-native. Different

setups,

Quoted 21 lines
Quoted 21 lines
Quoted 11 lines
Quoted 7 lines

such as with external toolchain, would expose this problem, since correct
dependency is not marked explicitly.

---
I'm seeing this now on 4.16-rc2 with external toolchain.

I am not against the addition but in fact, it is because 4.16 stopped of using pre-generated files and now requires bison to be available.

Ok, good, I was wondering why I didn't see it before 4.16...

Quoted 1 line

I am unsure it ought to be on kernel.bbclass at this moment.

Why not? Because it's 4.16+ specific?

Quoted 7 lines
Quoted 6 lines

By default bison-native dependency gets added implicitly when built with the
toolchain from oe-core, so this doesn't add any extra dependencies, only makes
it explicit.

it will be overcompensating for < 4.16 kernel recipes where this

dependency

Quoted 2 lines

is not needed. it might be a legit dependency for another component, that doesn't stage this package in kernel recipe-specific sysroot but this

change

Quoted 1 line

will do that. So its not completely innocent change.

I'd be in favor of matching the version and adding it if it is a 4.16 or newer kernel. Adding for prior versions seems wrong for me.

--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#6 lore raw
Quoted 2 lines · Martin Jansa

I'm OK with adding bison-native for all versions.

As long we know that there are no side effects of this dep in kernel builds

Quoted 96 lines

When we added bc-native 2+ years ago:
commit c067e52cffe002de3b39aa1bced308dd532859c1

Author: Alejandro Hernandez <[email protected]>
Date:   Thu Oct 2 15:33:23 2014 -0500

kernel: Added bc-native as DEPENDS

The makefile checks for bc during for compilation

it was also needed only for the latest kernel and we didn't make it conditional on the kernel version. Now with RSS it's even less dangerous as the bison-native will appear only in kernel's RSS and as Denys said it's quite common dependency so many components pull it into RSS anyway.

Quoted 63 lines · Otavio Salvador
Quoted 21 lines · Khem Raj
Quoted 20 lines · Denys Dmytriyenko
Quoted 18 lines · Otavio Salvador
Quoted 15 lines · Denys Dmytriyenko

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required
for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target

'scripts/kconfig/zconf.tab.c' failed

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In most cases, this dependency comes indirectly via toolchain dependencies,
specifically binutils-cross, which pulls bison-native. Different

setups,

Quoted 16 lines
Quoted 16 lines
Quoted 11 lines
Quoted 7 lines

such as with external toolchain, would expose this problem, since correct
dependency is not marked explicitly.

---
I'm seeing this now on 4.16-rc2 with external toolchain.

I am not against the addition but in fact, it is because 4.16 stopped of using pre-generated files and now requires bison to be available.

Ok, good, I was wondering why I didn't see it before 4.16...

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

I am unsure it ought to be on kernel.bbclass at this moment.

Why not? Because it's 4.16+ specific?

By default bison-native dependency gets added implicitly when built

with

Quoted 2 lines
Quoted 2 lines

the
toolchain from oe-core, so this doesn't add any extra dependencies,

only

Quoted 4 lines
Quoted 3 lines

makes
it explicit.

it will be overcompensating for < 4.16 kernel recipes where this

dependency

Quoted 1 line

is not needed. it might be a legit dependency for another component,

that

Quoted 1 line

doesn't stage this package in kernel recipe-specific sysroot but this

change

Quoted 1 line

will do that. So its not completely innocent change.

Quoted 8 lines

I'd be in favor of matching the version and adding it if it is a 4.16 or newer kernel. Adding for prior versions seems wrong for me.

--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750

--

Quoted 5 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#7 lore raw
Quoted 5 lines · Khem Raj
Quoted 1 line · Martin Jansa

I'm OK with adding bison-native for all versions.

As long we know that there are no side effects of this dep in kernel builds

You do realize that when you build with oe-core toolchain, every package gets bison-native dependency automatically? Regardless whether it was requested or not. If there were side effects, we would have seen it long time ago...

Quoted 101 lines
Quoted 95 lines

When we added bc-native 2+ years ago:
commit c067e52cffe002de3b39aa1bced308dd532859c1

Author: Alejandro Hernandez <[email protected]>
Date:   Thu Oct 2 15:33:23 2014 -0500

kernel: Added bc-native as DEPENDS

The makefile checks for bc during for compilation

it was also needed only for the latest kernel and we didn't make it conditional on the kernel version. Now with RSS it's even less dangerous as the bison-native will appear only in kernel's RSS and as Denys said it's quite common dependency so many components pull it into RSS anyway.

Quoted 63 lines · Otavio Salvador
Quoted 21 lines · Khem Raj
Quoted 20 lines · Denys Dmytriyenko
Quoted 18 lines · Otavio Salvador
Quoted 15 lines · Denys Dmytriyenko

From: Denys Dmytriyenko <[email protected]>

Explicitly depend on bison-native for deterministic builds, as it is required
for the build:

|   HOSTCC  scripts/basic/fixdep
|   GEN     ./Makefile
|   HOSTCC  scripts/kconfig/conf.o
|   YACC    scripts/kconfig/zconf.tab.c
| /bin/sh: bison: command not found
| scripts/Makefile.lib:217: recipe for target

'scripts/kconfig/zconf.tab.c' failed

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In most cases, this dependency comes indirectly via toolchain dependencies,
specifically binutils-cross, which pulls bison-native. Different

setups,

Quoted 16 lines
Quoted 16 lines
Quoted 11 lines
Quoted 7 lines

such as with external toolchain, would expose this problem, since correct
dependency is not marked explicitly.

---
I'm seeing this now on 4.16-rc2 with external toolchain.

I am not against the addition but in fact, it is because 4.16 stopped of using pre-generated files and now requires bison to be available.

Ok, good, I was wondering why I didn't see it before 4.16...

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

I am unsure it ought to be on kernel.bbclass at this moment.

Why not? Because it's 4.16+ specific?

By default bison-native dependency gets added implicitly when built

with

Quoted 2 lines
Quoted 2 lines

the
toolchain from oe-core, so this doesn't add any extra dependencies,

only

Quoted 4 lines
Quoted 3 lines

makes
it explicit.

it will be overcompensating for < 4.16 kernel recipes where this

dependency

Quoted 1 line

is not needed. it might be a legit dependency for another component,

that

Quoted 1 line

doesn't stage this package in kernel recipe-specific sysroot but this

change

Quoted 1 line

will do that. So its not completely innocent change.

Quoted 8 lines

I'd be in favor of matching the version and adding it if it is a 4.16 or newer kernel. Adding for prior versions seems wrong for me.

--
Otavio Salvador O.S. Systems
http://www.ossystems.com.br http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750

--

Quoted 5 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
#8 lore raw
Quoted 10 lines · Denys Dmytriyenko
Quoted 5 lines · Khem Raj
Quoted 1 line · Martin Jansa

I'm OK with adding bison-native for all versions.

As long we know that there are no side effects of this dep in kernel builds

You do realize that when you build with oe-core toolchain, every package gets bison-native dependency automatically? Regardless whether it was requested or not. If there were side effects, we would have seen it long time ago...

Yes I see it seems to be pulled in by toolchain regardless, so for this patch we might just be adding to determinism. but I think it raises question on why toolchain is adding this everywhere, I think we should remove it from toolchain default deps. It will make metadata more consistent across internal/external toolchains

#9 lore raw
Quoted 12 lines · Khem Raj
Quoted 10 lines · Denys Dmytriyenko
Quoted 5 lines · Khem Raj
Quoted 1 line · Martin Jansa

I'm OK with adding bison-native for all versions.

As long we know that there are no side effects of this dep in kernel builds

You do realize that when you build with oe-core toolchain, every package gets bison-native dependency automatically? Regardless whether it was requested or not. If there were side effects, we would have seen it long time ago...

Quoted 5 lines

Yes I see it seems to be pulled in by toolchain regardless, so for this patch we might just be adding to determinism. but I think it raises question on why toolchain is adding this everywhere, I think we should remove it from toolchain default deps. It will make metadata more consistent across internal/external toolchains

There were several similar patches I submitted last month for different components to add flex-native and/or bison-native. Most of them got accepted and we had a follow up discussion with Richard in gcc thread:
http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146295.html

He also submitted this patch to prevent indirect flex/bison-native dependencies, but I don't believe it got merged yet:
http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html

Richard, do you have any comments?

Signature · 1 line
Denys