core/pkg-kconfig: ensure we have necessary tools to run configurators

Some users of kconfig need some packages to be built before their kconfig infra be used.

For example, the linux kernel, starting with 4.16, needs flex and bison to generate the parser code. Furthermore, starting with 4.18, it will also need the cross-compiler before parsing the kconfig stuff, because that calls the compiler to check its features.

Currently, this is broken, even the flex/bison ones, even though they are listed, because there is no way to define dependencie that are guaranteed before the (visual) configurators. For example:

    $ make distclean
    $ make menuconfig
      --> enable the linux kernel, choose a defconfig, save, exit
    $ make linux-menuconfig
    [...]
      HOSTCC  scripts/basic/fixdep
      HOSTCC  scripts/kconfig/conf.o
      YACC    scripts/kconfig/zconf.tab.c
    /bin/sh: bison: command not found
      LEX     scripts/kconfig/zconf.lex.c
    scripts/Makefile.lib:196: recipe for target 
'scripts/kconfig/zconf.tab.c' failed
    make[3]: *** [scripts/kconfig/zconf.tab.c] Error 127
    make[3]: *** Waiting for unfinished jobs....
    /bin/sh: flex: command not found
    scripts/Makefile.lib:188: recipe for target 
'scripts/kconfig/zconf.lex.c' failed
    make[3]: *** [scripts/kconfig/zconf.lex.c] Error 127
    Makefile:528: recipe for target 'rpc_defconfig' failed
    make[2]: *** [rpc_defconfig] Error 2
    linux/linux.mk:511: recipe for target 
'/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config' 
failed
    make[1]: *** 
[/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config] 
Error 2
    Makefile:79: recipe for target '_all' failed
    make: *** [_all] Error 2

So, we introduce a new type of dependencies for kconfig-based packages, that are guaranteed to be built and installed before the (visual) configurators are called.

Since those dependencies are phony targets and therefore always out of date, a normal dependency would cause the .config target to be rebuilt on each invocation of make. So we use an order-only pre-requisite, like is done for the patch dependency.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 docs/manual/adding-packages-kconfig.txt | 4 ++++
 package/pkg-kconfig.mk                  | 9 +++++++++
 2 files changed, 13 insertions(+)

docs/manual/adding-packages-kconfig.txt +0 -0

index d4d8630a79..982c1229ea 100644
@@ -81,3 +81,7 @@ be set to suit the needs of the package under 

consideration:
be well suited for all packages that use the standard kconfig infrastructure as inherited from the Linux kernel; some packages use a derivative of kconfig that use a different location.

docs/manual/adding-packages-kconfig.txt +4 -0

+
+* +FOO_KCONFIG_DEPENDENCIES+: the list of packages (most probably, host
+  packages) that need to be built before this package's kconfig is
+  interpreted. Seldom used. By default, empty.

package/pkg-kconfig.mk +4 -0

index 81bba5220c..d6c95b897e 100644
@@ -41,6 +41,10 @@ endef
 
 define inner-kconfig-package
 
+# Register the kconfig dependencies as regular dependencies, so that
+# they are also accounted for in the generated graphs.
+$(2)_DEPENDENCIES += $$($(2)_KCONFIG_DEPENDENCIES)
+
 # Call the generic package infrastructure to generate the necessary
 # make targets.
 # Note: this must be done _before_ attempting to use $$($(2)_DIR) in a

@@ -121,6 +125,11 @@ $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): 

$$($(2)_KCONFIG_FILE) $$($(2)_KCONFIG_F

 # it explicitly. It doesn't hurt to always have it though.
 $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $(1)-patch

package/pkg-kconfig.mk +5 -0

+# Some packages may need additional tools to be present by the time their
+# kconfig structure is parsed (e.g. the linux kernel may need to call to
+# the compiler to test its features).
+$$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $$($(2)_KCONFIG_DEPENDENCIES)
+
 # In order to get a usable, consistent configuration, some 

fixup may be needed.
# The exact rules are specified by the package .mk file. define $(2)_FIXUP_DOT_CONFIG

Take this opportunity to also drop the append-assignment as the first assignment to LINUX_DEPENDENCIES.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
Changes v2 -> v3:
- also fix the append-assignment

---
 linux/linux.mk | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

linux/linux.mk +0 -0

index 7527b11673..60dad3a249 100644
@@ -62,7 +62,11 @@ LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
 LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
 

Take this opportunity to also drop the append-assignment as the first assignment to LINUX_DEPENDENCIES.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Quoted 6 lines

Take this opportunity to also drop the append-assignment as the first assignment to LINUX_DEPENDENCIES.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Thomas Petazzoni <[email protected]>

I am a bit confused. In this series, you suggest to make host-flex/host-bison dependencies of the configuration step of Linux. But in another series
(http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you suggest to make flex/bison hard requirements of Buildroot, so that we don't have to build them.

Could you clarify in which direction you would like to go ?

Also, if we
make bison/flex hard requirements of Buildroot, I'd like to have a clear rule on whether we keep or not our host-bison/host-flex packages, and if we keep them, when they should be used vs. when the system bison/flex should be used.

Quoted 1 line

Could you clarify in which direction you would like to go ?

This very series is about solving a technical problem, that kconfig-based packages may have dependencies required even before running the configurators.

On the other hand, that other series is about changing our requirements.

If we decide to require flex+bison, so in this series, only patch 2 can be dropped.

Quoted 5 lines

Also, if we
make bison/flex hard requirements of Buildroot, I'd like to have a clear rule on whether we keep or not our host-bison/host-flex packages, and if we keep them, when they should be used vs. when the system bison/flex should be used.

My position is that we will always want to have our host flex and bison to build packages that install things in host/, in staging/ or in target/.

For the linux kernel and other kconfig-based packages, we don't care which flex/bison are used, because the resulting binaries are not installed, unless those packages also generate code eventually installed in host/, staging/ or target/

Quoted 6 lines

Quoted 5 lines

Also, if we
make bison/flex hard requirements of Buildroot, I'd like to have a clear rule on whether we keep or not our host-bison/host-flex packages, and if we keep them, when they should be used vs. when the system bison/flex should be used.

Quoted 8 lines

My position is that we will always want to have our host flex and bison to build packages that install things in host/, in staging/ or in target/.

For the linux kernel and other kconfig-based packages, we don't care which flex/bison are used, because the resulting binaries are not installed, unless those packages also generate code eventually installed in host/, staging/ or target/

So the system-wide bison/flex would only be used for the kconfig programs built by Buildroot itself (if we move to a newer kconfig version that doesn't have the pre-generated files) or by Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use host-flex and host-bison. That's your proposal ?

Quoted 5 lines

So the system-wide bison/flex would only be used for the kconfig programs built by Buildroot itself (if we move to a newer kconfig version that doesn't have the pre-generated files) or by Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use host-flex and host-bison. That's your proposal ?

Yes, that is my proposal.

Thomas, All,

On 2018-08-14 16:21 +0200, Thomas Petazzoni spake thusly:

Quoted 8 lines · Yann E. MORIN

Quoted 6 lines

Take this opportunity to also drop the append-assignment as the first assignment to LINUX_DEPENDENCIES.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Quoted 8 lines

I am a bit confused. In this series, you suggest to make host-flex/host-bison dependencies of the configuration step of Linux. But in another series
(http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you suggest to make flex/bison hard requirements of Buildroot, so that we don't have to build them.

Could you clarify in which direction you would like to go ?

This very series is about solving a technical problem, that kconfig-based packages may have dependencies required even before running the configurators.

On the other hand, that other series is about changing our requirements.

If we decide to require flex+bison, so in this series, only patch 2 can be dropped.

Quoted 5 lines

Also, if we
make bison/flex hard requirements of Buildroot, I'd like to have a clear rule on whether we keep or not our host-bison/host-flex packages, and if we keep them, when they should be used vs. when the system bison/flex should be used.

My position is that we will always want to have our host flex and bison to build packages that install things in host/, in staging/ or in target/.

For the linux kernel and other kconfig-based packages, we don't care which flex/bison are used, because the resulting binaries are not installed, unless those packages also generate code eventually installed in host/, staging/ or target/

Regards,
Yann E. MORIN.

My position is that we do indeed need our host flex and bison for everything
installed in target/ (which implies everything installed in staging/), but not for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.

It would also be nice if dependencies.sh could do a version check and accept
the system flex/bison if it is the right version, like we do for cmake and tar.

But both of these wishes make things more complicated, so I'm OK with just
keeping all current host-flex/bison dependencies.

Quoted 4 lines

For the linux kernel and other kconfig-based packages, we don't care which flex/bison are used, because the resulting binaries are not installed, unless those packages also generate code eventually installed in host/, staging/ or target/

Note that this means you'll still need host-flex/bison for the kernel, because
the kernel may build dtc and that gets installed in host...

Quoted 7 lines

My position is that we do indeed need our host flex and bison for everything
installed in target/ (which implies everything installed in staging/), but not for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.

It would also be nice if dependencies.sh could do a version check and accept
the system flex/bison if it is the right version, like we do for cmake and tar.

Quoted 2 lines

But both of these wishes make things more complicated, so I'm OK with just
keeping all current host-flex/bison dependencies.

In fact, I think adding flex and bison as mandatory dependencies of the system is going to cause too much disruption. All the autobuilders would start complaining, all the Travis/Gitlab CI setups would have to be adjusted, and everyone building in containers/minimal environments will be affected.

So instead, I'd like to detect if flex/bison are available on the system. If they are available, then we use them for the Linux kernel configuration process. If they are not available, we build them.

Instead of:
LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
I'd prefer to see:
LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)

And the usual logic in support/dependencies/ to check for bison/flex. This will avoid disrupting all existing setups, and still provide the speed-up that we don't need to build flex/bison when doing "make linux-menuconfig".

Quoted 7 lines

Quoted 4 lines

For the linux kernel and other kconfig-based packages, we don't care which flex/bison are used, because the resulting binaries are not installed, unless those packages also generate code eventually installed in host/, staging/ or target/

Note that this means you'll still need host-flex/bison for the kernel, because
the kernel may build dtc and that gets installed in host...

Are you sure flex/bison are used when building the DTC copy in the Linux kernel ? Indeed we only added host-bison/host-flex as dependencies of linux in commit 1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping its pre-generated files. How could DTC be built before 1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Hello,

On Wed, 15 Aug 2018 00:50:42 +0200, Arnout Vandecappelle wrote:

I'd prefer to see:

  LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)

And the usual logic in support/dependencies/ to check for bison/flex. This will avoid disrupting all existing setups, and still provide the speed-up that we don't need to build flex/bison when doing "make linux-menuconfig".

Quoted 7 lines

Quoted 4 lines

For the linux kernel and other kconfig-based packages, we don't care which flex/bison are used, because the resulting binaries are not installed, unless those packages also generate code eventually installed in host/, staging/ or target/

Note that this means you'll still need host-flex/bison for the kernel, because
the kernel may build dtc and that gets installed in host...

Are you sure flex/bison are used when building the DTC copy in the Linux kernel ? Indeed we only added host-bison/host-flex as dependencies of linux in commit 1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping its pre-generated files. How could DTC be built before 1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?

Best regards,

Thomas

Starting with linux-4.18, the kconfig from the kernel can call to the compiler to test its capabilities; see:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/Kconfig.include

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

---
Changes v1 -> v2:
- take advantage of the kconfig-package infra (Thomas)

---
 linux/linux.mk | 5 +++++
 1 file changed, 5 insertions(+)

linux/linux.mk +0 -0

index 60dad3a249..48b2a16ce7 100644
@@ -68,6 +68,11 @@ LINUX_DEPENDENCIES = host-kmod
 # shipped with the kernel sources, so we need flex and bison.