core/pkg-kconfig: ensure we have necessary tools to run configurators

Hello All!

Recent version of the linux kernel introduced major changes in the kconfig infrastructure.

Starting with linux-4.16, the kconfig parser code is no longer shipped with the linux source tree, and mut be generated from the lex/yacc parser sources.

Similarly, starting with linux-4.18, the kconfig infra allows calling back to the shell to run various commands, especially calling to the compiler to test its features (and hide/show options accordingly).

So, we need to ensure the appropriate tools are available before we attempt to run any of the configurators. Since building host-flex and host-bison is not entirely free (they both have a few dependencies of their own), we want to be able to use the system ones if they are present.

This seven-patch series first documents the reasons for having host-flex and host-bison as dependencies of the linux package.

Then we add support for detecting if the system provides flex and/or bison; if either is missing, a conditional dependency is defined for either.

Third, the kconfig infra is enhanced to accept dependencies needed before the configurators are called.

The linux packages is then adapted, one step at a time, to use the conditional flex/bison dependencies for the configurators, and use the host-{flex,bison} dependencies only for the DTS support.

Then, the toolchain is added as a kconfig dependencies of the linux kernel.

Eventually, uboot is also fixed to conditionally require flex/bison.

Changes v3 -> v4:
- also fix uboot

(note: there was no v2.)

Changes v1 -> v3:
- introduce the KCONFIG_DEPENDENCIES in the kconfig-package infra (Thomas)
- change linux to take advantage of that (Thomas)

Regards,
Yann E. MORIN.

The following changes since commit f70b13a7cb6eb01e9d78fddcb695b84a22e5699b

dahdi-tools: unify dependencies comments (2018-08-17 17:07:11 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to ed63f1c812145907ef83c6bf245ad5765c67aded

boot/uboot: needs host-{flex,bison} (2018-08-17 17:40:03 +0200)

----------------------------------------------------------------
Yann E. MORIN (7):
linux: explain why we need host-{flex,bison} support/dependencies: check for system-provided bison and flex core/pkg-kconfig: allow dependencies before configurators
linux: kconfig may need host-{flex,bison} to build the configurators
linux: host-{flex,bison} only needed for DTS
linux: kconfig needs the toolchain
boot/uboot: needs host-{flex,bison}

 boot/uboot/uboot.mk                           |  3 +++
 docs/manual/adding-packages-kconfig.txt       |  4 ++++
 linux/linux.mk                                | 19 ++++++++++++++++++-
 package/pkg-kconfig.mk                        |  9 +++++++++
 support/dependencies/check-host-bison-flex.mk | 10 ++++++++++
 5 files changed, 44 insertions(+), 1 deletion(-)
 create mode 100644 support/dependencies/check-host-bison-flex.mk

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

The commit that added the dependency on host-{bison,flex} did so because the pre-generated kconfig parser source files were removed from the kernel tree, in linux-4.16.

But then, in linux-4.17, the pre-generated dtc parser source files were in turn removed as well.

So, document the two reasons why they are needed, so we don't accidentally remove them when we (soon) introduce the kconfig dependencies.

(Also fix the first assignment to LINUX_DEPENDENCIES to be a simple assignement, not an append-assignment.)

---
 linux/linux.mk | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

index e0d437427d..d307970065 100644
@@ -62,7 +62,13 @@ LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
 LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
 
 LINUX_INSTALL_IMAGES = YES
-LINUX_DEPENDENCIES += host-bison host-flex host-kmod
+LINUX_DEPENDENCIES = host-kmod
+
+# Starting with 4.16, the generated kconfig paser code is no longer
+# shipped with the kernel sources, so we need flex and bison.
+# Starting with 4.17, the generated dtc parser code is no longer
+# shipped with the kernel sources, so we need flex and bison.
+LINUX_DEPENDENCIES += host-bison host-flex
 
 # host tools needed for kernel compression
 ifeq ($(BR2_LINUX_KERNEL_LZ4),y)

2.14.1

Hello,

Applied to master, thanks.

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

---
 support/dependencies/check-host-bison-flex.mk | 10 ++++++++++
 1 file changed, 10 insertions(+)
 create mode 100644 support/dependencies/check-host-bison-flex.mk

new file mode 100644
index 0000000000..233b6c51cc
@@ -0,0 +1,10 @@
+# If the system lacks bison or flex, add
+# dependencies to suitable host packages
+
+ifeq ($(shell which bison 2>/dev/null),)
+BR2_BISON_HOST_DEPENDENCY = host-bison
+endif
+
+ifeq ($(shell which flex 2>/dev/null),)
+BR2_FLEX_HOST_DEPENDENCY = host-flex
+endif

2.14.1

Hello,

$ make
which: no bison in (/usr/local/bin:/usr/bin:/bin:/home/thomas/.rvm/bin:/usr/local/sbin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/home/thomas/.rvm/bin)

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Hello,

Wait, wait, scratch that. Even without your patch it does that. This error is displayed because of:

BISON := $(shell which bison || type -p bison)

in package/Makefile.in. I'm not even sure why we have this definition, we don't use it anywhere, ditto for the FLEX definition.

So basically your patch looks good, we probably need to remove the useless lines in package/Makefile.in, as a separate patch.

Best regards,

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Some users of kconfig need some packages to be built before their kconfig infra be used.

For example, the linux kernel, starting with 4.16, needs flex and bison to generate the parser code. Furthermore, starting with 4.18, it will also need the cross-compiler before parsing the kconfig stuff, because that calls the compiler to check its features.

Currently, this is broken, even the flex/bison ones, even though they are listed, because there is no way to define dependencie that are guaranteed before the (visual) configurators. For example:

    $ make distclean
    $ make menuconfig
      --> enable the linux kernel, choose a defconfig, save, exit
    $ make linux-menuconfig
    [...]
      HOSTCC  scripts/basic/fixdep
      HOSTCC  scripts/kconfig/conf.o
      YACC    scripts/kconfig/zconf.tab.c
    /bin/sh: bison: command not found
      LEX     scripts/kconfig/zconf.lex.c
    scripts/Makefile.lib:196: recipe for target 'scripts/kconfig/zconf.tab.c' failed
    make[3]: *** [scripts/kconfig/zconf.tab.c] Error 127
    make[3]: *** Waiting for unfinished jobs....
    /bin/sh: flex: command not found
    scripts/Makefile.lib:188: recipe for target 'scripts/kconfig/zconf.lex.c' failed
    make[3]: *** [scripts/kconfig/zconf.lex.c] Error 127
    Makefile:528: recipe for target 'rpc_defconfig' failed
    make[2]: *** [rpc_defconfig] Error 2
    linux/linux.mk:511: recipe for target '/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config' failed
    make[1]: *** [/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config] Error 2
    Makefile:79: recipe for target '_all' failed
    make: *** [_all] Error 2

So, we introduce a new type of dependencies for kconfig-based packages, that are guaranteed to be built and installed before the (visual) configurators are called.

Since those dependencies are phony targets and therefore always out of date, a normal dependency would cause the .config target to be rebuilt on each invocation of make. So we use an order-only pre-requisite, like is done for the patch dependency.

---
 docs/manual/adding-packages-kconfig.txt | 4 ++++
 package/pkg-kconfig.mk                  | 9 +++++++++
 2 files changed, 13 insertions(+)

index d4d8630a79..982c1229ea 100644
@@ -81,3 +81,7 @@ be set to suit the needs of the package under consideration:
   be well suited for all packages that use the standard kconfig
   infrastructure as inherited from the Linux kernel; some packages use
   a derivative of kconfig that use a different location.
+
+* +FOO_KCONFIG_DEPENDENCIES+: the list of packages (most probably, host
+  packages) that need to be built before this package's kconfig is
+  interpreted. Seldom used. By default, empty.

index 81bba5220c..d6c95b897e 100644
@@ -41,6 +41,10 @@ endef
 
 define inner-kconfig-package
 
+# Register the kconfig dependencies as regular dependencies, so that
+# they are also accounted for in the generated graphs.
+$(2)_DEPENDENCIES += $$($(2)_KCONFIG_DEPENDENCIES)
+
 # Call the generic package infrastructure to generate the necessary
 # make targets.
 # Note: this must be done _before_ attempting to use $$($(2)_DIR) in a

@@ -121,6 +125,11 @@ $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): $$($(2)_KCONFIG_FILE) $$($(2)_KCONFIG_F
 # it explicitly. It doesn't hurt to always have it though.
 $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $(1)-patch
 
+# Some packages may need additional tools to be present by the time their
+# kconfig structure is parsed (e.g. the linux kernel may need to call to
+# the compiler to test its features).
+$$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $$($(2)_KCONFIG_DEPENDENCIES)
+
 # In order to get a usable, consistent configuration, some fixup may be needed.
 # The exact rules are specified by the package .mk file.
 define $(2)_FIXUP_DOT_CONFIG

2.14.1

Hello,

Applied to master, thanks. If anybody wonders why this gets into
master: I believe people are going to hit this problem, either because of the bison or flex dependency, or because of the toolchain dependency, and this is something we should fix in our upcoming 2018.08 release.

Best regards,

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Rely on the system provided ones if avalable, and only resort to use our owns if the sytem does not provide them.

---
Notes: I did not carry Jan's Tested-by tag, because the patch is now using the conditional dependencies, so it would have been a bit presumptuous of me to carry that tag.

---
Changes v2 -> v3:
- use the newly-introduced conditional dependencies (Thomas, Arnout)

---
 linux/linux.mk | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

index d307970065..deb5a4ac57 100644
@@ -64,12 +64,18 @@ LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
 LINUX_INSTALL_IMAGES = YES
 LINUX_DEPENDENCIES = host-kmod
 
-# Starting with 4.16, the generated kconfig paser code is no longer
-# shipped with the kernel sources, so we need flex and bison.
 # Starting with 4.17, the generated dtc parser code is no longer
-# shipped with the kernel sources, so we need flex and bison.
+# shipped with the kernel sources, so we need flex and bison. For
+# reproducibility, we use our owns rather than the host ones.
 LINUX_DEPENDENCIES += host-bison host-flex
 
+# Starting with 4.16, the generated kconfig paser code is no longer
+# shipped with the kernel sources, so we need flex and bison, but
+# only if the host does not have them.
+LINUX_KCONFIG_DEPENDENCIES = \
+	$(BR2_BISON_HOST_DEPENDENCY) \
+	$(BR2_FLEX_HOST_DEPENDENCY)
+
 # host tools needed for kernel compression
 ifeq ($(BR2_LINUX_KERNEL_LZ4),y)
 LINUX_DEPENDENCIES += host-lz4

2.14.1

host-{flex,bison} are only needed to generate the dtc parser, so we don't need them if the kernel does not have support for device tree.

---
 linux/linux.mk | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

index deb5a4ac57..d6b0853518 100644
@@ -64,11 +64,6 @@ LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
 LINUX_INSTALL_IMAGES = YES
 LINUX_DEPENDENCIES = host-kmod
 
-# Starting with 4.17, the generated dtc parser code is no longer
-# shipped with the kernel sources, so we need flex and bison. For
-# reproducibility, we use our owns rather than the host ones.
-LINUX_DEPENDENCIES += host-bison host-flex
-
 # Starting with 4.16, the generated kconfig paser code is no longer
 # shipped with the kernel sources, so we need flex and bison, but
 # only if the host does not have them.

@@ -352,6 +347,11 @@ define LINUX_KCONFIG_FIXUP_CMDS
 endef
 
 ifeq ($(BR2_LINUX_KERNEL_DTS_SUPPORT),y)
+# Starting with 4.17, the generated dtc parser code is no longer
+# shipped with the kernel sources, so we need flex and bison. For
+# reproducibility, we use our owns rather than the host ones.
+LINUX_DEPENDENCIES += host-bison host-flex
+
 ifeq ($(BR2_LINUX_KERNEL_DTB_IS_SELF_BUILT),)
 define LINUX_BUILD_DTB
 	$(LINUX_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) $(LINUX_DTBS)

2.14.1

Starting with linux-4.18, the kconfig from the kernel can call to the compiler to test its capabilities; see:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/Kconfig.include

---
Changes v1 -> v2:
- take advantage of the kconfig-package infra (Thomas)

---
 linux/linux.mk | 5 +++++
 1 file changed, 5 insertions(+)

index d6b0853518..675e7906a8 100644
@@ -71,6 +71,11 @@ LINUX_KCONFIG_DEPENDENCIES = \
 	$(BR2_BISON_HOST_DEPENDENCY) \
 	$(BR2_FLEX_HOST_DEPENDENCY)
 
+# Starting with 4.18, the kconfig in the kernel calls the
+# cross-compiler to check its capabilities. So we need the
+# toolchain before we can call the configurators.
+LINUX_KCONFIG_DEPENDENCIES += toolchain
+
 # host tools needed for kernel compression
 ifeq ($(BR2_LINUX_KERNEL_LZ4),y)
 LINUX_DEPENDENCIES += host-lz4

2.14.1

Recent U-Boot no longer ship the flex/bison generated kconfig parser, as of commit e91610da7c8a9fe42f3e5a75f06c3d1a0cb5f815 (kconfig: re-sync with Linux 4.17-rc4).

So, add the conditional kconfig dependencies, as we just did for the kernel.

---
 boot/uboot/uboot.mk | 3 +++
 1 file changed, 3 insertions(+)

index 04c759be34..bddafe234d 100644
@@ -449,5 +449,8 @@ endif # BR2_TARGET_UBOOT && BR_BUILDING
 ifeq ($(BR2_TARGET_UBOOT_BUILD_SYSTEM_LEGACY),y)
 $(eval $(generic-package))
 else ifeq ($(BR2_TARGET_UBOOT_BUILD_SYSTEM_KCONFIG),y)
+UBOOT_KCONFIG_DEPENDENCIES = \
+	$(BR2_BISON_HOST_DEPENDENCY) \
+	$(BR2_FLEX_HOST_DEPENDENCY)
 $(eval $(kconfig-package))
 endif # BR2_TARGET_UBOOT_BUILD_SYSTEM_LEGACY

2.14.1

Hello,

[...]

  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  YACC    scripts/kconfig/zconf.tab.c

/bin/sh: 1: bison: not found
make[3]: *** [scripts/kconfig/zconf.tab.c] Error 127
make[3]: *** Waiting for unfinished jobs....
LEX scripts/kconfig/zconf.lex.c
/bin/sh: 1: flex: not found
make[3]: *** [scripts/kconfig/zconf.lex.c] Error 127
make[2]: *** [mvebu_mcbin-88f8040_config] Error 2
make[1]: *** [/home/test/outputs/mainline/build/uboot-2018.09/.stamp_configured] Error 2
make: *** [_all] Error 2

Thomas

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Hello,

Series applied to master. Thanks!

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com