Skip to contentSkip to conversation

Add a BR2_EXTERNAL mechanism

PATCH Superseded Thomas Petazzoni 6 participants 33 messages Sep 14, 2013 – Nov 26, 2013 Kbuild

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

Hello,

Here is the second version of the BR2_EXTERNAL patch series.

Changes since v1
================

* Store BR2_EXTERNAL in the Makefile generated in the output directory, so that BR2_EXTERNAL doesn't have to be passed over and over again. Suggested by Ryan Barnett.

* Patch set splitted in more fine-grained changes, as suggested by Arnout.

* Instead of having the top-level Config.in file generated at run-time, use instead a trick proposed by Arnout: make BR2_EXTERNAL point to support/dummy-external/, which contains an empty Config.in. This way, regardless of whether BR2_EXTERNAL is specified by the user or not, kconfig is happy as 'source "$BR2_EXTERNAL/Config.in"' will always point to an existing file. If BR2_EXTERNAL is not used, it will point to the dummy file, if BR2_EXTERNAL is used, it will point to Config.in file in the provided BR2_EXTERNAL directory.

With this, BR2_EXTERNAL always has a value, so testing whether BR2_EXTERNAL is empty no longer tells us if the user has provided a value or not. Therefore, we have another variable, BR2_EXTERNAL_USED, which tells if the mechanism is used by the user or not. This variable avoids trying to use defconfigs from the dummy directory, or to encode the BR2_EXTERNAL value in the Makefile wrapper.

* As a consequence of the previous change, it is no longer needed to refactor the *config dependencies using a variable, so I've dropped what was previously the first patch of the series.

* BR2_EXTERNAL is turned into an absolute path, in order to avoid any problem. Suggested by Arnout.

* Fixed the usage of <some board> vs. <boardname> in the documentation, as noted by Arnout.

Remaning questions
==================

* Interaction with the 'local.mk' mechanism. Should 'external.mk' replace the 'local.mk' mechanism? Should they co-exist?

Introduction to the patch series
================================

Following the e-mail from Ryan Barnett entitled "[RFC] New feature to handle a custom board folder", I finally took a bit of time to implement how I think the support for externally stored Buildroot customization should be done.

I really think the approach of having board-related configuration options within Buildroot is wrong: it is the Buildroot configuration as a whole that defines what the configuration of the user system is:
a combination of a kernel, a root filesystem with various packages, a selection of a root filesystem image format, some system configuration tuning, a bootloader, etc.

In Ryan e-mail, I believe the most important thing is the wish to store the board-specific details outside of the Buildroot tree. In fact, this is already possible: nothing prevents anyone from using $(TOPDIR)/../company/kernel.config as the kernel configuration file for example, where $(TOPDIR)/../company/ is outside of the Buildroot tree, and versioned in a completely different way.

However, it seems like this possibility is not 'visible' enough, and admittedly only usable for some Buildroot configuration elements, while an user may also be interested in storing Buildroot package recipes and Buildroot deconfig files outside of the main Buildroot tree.

So, this patch set adds the BR2_EXTERNAL mechanism. It is very lightweight, and doesn't change anything to the Buildroot overall logic. Basically, BR2_EXTERNAL is an environment variable that one can point to a directory that will contain Buildroot customizations. It is then usable for three things:

* It is available as $(BR2_EXTERNAL) within all of the Buildroot configuration options, to define the location of the kernel configuration file, the root filesystem overlay, etc.

* It allows to store additional Config.in options and makefile logic in $(BR2_EXTERNAL), since Buildroot automatically includes $(BR2_EXTERNAL)/Config.in into the top-level menu, and $(BR2_EXTERNAL)/external.mk in the makefile logic. The main purpose of this is to be able to store custom packages outside of the Buildroot tree.

* It allows to store defconfigs in $(BR2_EXTERNAL)/configs/. They appear automatically in 'make help' and can be loaded like any other defconfig using 'make <something>_defconfig'.

Best regards,

Thomas

Thomas Petazzoni (4):
core: introduce the BR2_EXTERNAL variable
core: allow external Config.in/makefile code to be integrated
core: allow external defconfigs to be used docs/manual: add explanations about BR2_EXTERNAL

 Config.in                            |   6 ++
 Makefile                             |  25 +++++++-
 docs/manual/customize-outside-br.txt | 112 +++++++++++++++++++++++++++++++++++
 docs/manual/customize.txt            |   2 +
 support/dummy-external/Config.in     |   0
 support/scripts/mkmakefile           |   3 +-
 6 files changed, 144 insertions(+), 4 deletions(-)
 create mode 100644 docs/manual/customize-outside-br.txt
 create mode 100644 support/dummy-external/Config.in
Signature · 1 line
1.8.1.2
#1 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

This commit introduces the BR2_EXTERNAL environment variable, which will allow to keep Buildroot customization (board-specific configuration files or root filesystem overlays, package Config.in and makefiles, as well as defconfigs) outside of the Buildroot tree.

This commit only introduces the variable itself, and ensures that it is available within Config.in options, so that string options used to specify paths to directories or files can use $BR2_EXTERNAL as a reference. For example, one can use $BR2_EXTERNAL/board/<someboard>/kernel.config as the BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.

Following patches extend the usage of BR2_EXTERNAL to other areas (packages and defconfigs).

In details, this commit:

* Introduces the BR2_EXTERNAL Kconfig option. This option has no prompt, and is therefore not visible to the user and also not stored in the .config file. It is automatically set to the value of the BR2_EXTERNAL environment variable. The only purpose of this BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be properly expanded when used inside Kconfig option values.

* Ensures that the path given in BR2_EXTERNAL is an absolute path, or if not path is given, ensures that BR2_EXTERNAL points to a dummy implementation in $(TOPDIR)/support/dummy-external/. This last part is not directly useful in this commit, but is needed to properly support Config.in options declared in BR2_EXTERNAL (next commit).

* Passes the BR2_EXTERNAL into the *config environment, so that its value is found when parsing/evaluating Config.in files and .config values.

* Encodes the BR2_EXTERNAL value into the Makefile wrapper (only if BR2_EXTERNAL is specified by the user, not if the dummy internal value is used).

---
 Config.in                  |  4 ++++
 Makefile                   | 13 +++++++++++--
 support/scripts/mkmakefile |  3 ++-
 3 files changed, 17 insertions(+), 3 deletions(-)
Config.in +4 -0
index 65c29b4..522eaf8 100644
@@ -14,6 +14,10 @@ config BR2_HOSTARCH
 	string
 	option env="HOSTARCH"
 
+config BR2_EXTERNAL
+	string
+	option env="BR2_EXTERNAL"
+
 # Hidden boolean selected by pre-built packages for x86, when they
 # need to run on x86-64 machines (example: pre-built external
 # toolchains, binary tools like SAM-BA, etc.).
Makefile +11 -2
index 8610592..13ad342 100644
@@ -51,6 +51,14 @@ CONFIG_CONFIG_IN=Config.in
 CONFIG=support/kconfig
 DATE:=$(shell date +%Y%m%d)
 
+# Turn BR2_EXTERNAL into an absolute path
+ifneq ($(BR2_EXTERNAL),)
+override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))
+BR2_EXTERNAL_USED = y
+else
+BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
+endif
+
 # Compute the full local version string so packages can use it as-is
 # Need to export it, so it can be got from environment in children (eg. mconf)
 export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
@@ -632,7 +640,8 @@ COMMON_CONFIG_ENV = \
 	KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
 	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
 	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
-	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
+	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
+	BR2_EXTERNAL=$(BR2_EXTERNAL)
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
@@ -739,7 +748,7 @@ source-check:
 # output directory.
 outputmakefile:
 ifeq ($(NEED_WRAPPER),y)
-	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
+	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O) $(if $(BR2_EXTERNAL_USED),$(BR2_EXTERNAL))
 endif
 
 # printvars prints all the variables currently defined in our Makefiles
support/scripts/mkmakefile +2 -1
index cef2ec7..e7c1bd0 100755
@@ -6,7 +6,7 @@
 # Usage
 # $1 - Kernel src directory
 # $2 - Output directory
-
+# $3 - BR2_EXTERNAL directory
 
 test ! -r $2/Makefile -o -O $2/Makefile || exit 0
 # Only overwrite automatically generated Makefiles
@@ -27,6 +27,7 @@ makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
 
 MAKEARGS := -C $1
 MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$(patsubst %/,%,\$(makedir))
+MAKEARGS += BR2_EXTERNAL=$3
 
 MAKEFLAGS += --no-print-directory
 
Signature · 1 line
1.8.1.2
#2 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Thomas,

Per our chat on IRC, putting BR2_EXTERNAL into the generated makefile isn't quite working out like I expected. I'm losing the BR2_EXTERNAL variable definition.

Here are the steps I'm taking after applying your v2 of this patch to a git clone of the mainline of buildroot. The buildroot clone is not in the directory of ~/tmp but elsewhere.

make O=~/tmp/build_xm BR2_EXTERNAL=~/tmp/buildroot rc_beagle_xm_min_defconfig
cd ~/tmp/build_xm
make

Here is the output from the make:

make -C /home/rjbarnet/projects/buildroot/buildroot-git O=/home/rjbarnet/projects/muos/build_xm/. BR2_EXTERNAL=/home/rjbarnet/projects/muos/muos-svn/buildroot
/usr/bin/make -j8 O=/home/rjbarnet/projects/muos/build_xm/. HOSTCC="/usr/bin/gcc" HOSTCXX="/usr/bin/g++" silentoldconfig
GEN /home/rjbarnet/projects/muos/build_xm/./Makefile
BR2_DEFCONFIG='' KCONFIG_AUTOCONFIG=/home/rjbarnet/projects/muos/build_xm/build/buildroot-config/auto.conf KCONFIG_AUTOHEADER=/home/rjbarnet/projects/muos/build_xm/build/buildroot-config/autoconf.h KCONFIG_TRISTATE=/home/rjbarnet/projects/muos/build_xm/build/buildroot-config/tristate.config BUILDROOT_CONFIG=/home/rjbarnet/projects/muos/build_xm/./.config BR2_EXTERNAL=/home/rjbarnet/projects/buildroot/buildroot-git/support/dummy-external/ /home/rjbarnet/projects/muos/build_xm/build/buildroot-config/conf --silentoldconfig Config.in
mkdir -p /home/rjbarnet/projects/muos/build_xm/target

rsync -a \
                --exclude .empty --exclude .svn --exclude .git \
                --exclude .hg --exclude=CVS --exclude '*~' \

/home/rjbarnet/projects/buildroot/buildroot-git/system/skeleton/
/home/rjbarnet/projects/muos/build_xm/target/
cp support/misc/target-dir-warning.txt /home/rjbarnet/projects/muos/build_xm/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM touch /home/rjbarnet/projects/muos/build_xm/build/.root
--2013-09-16 10:50:52--
<Stuff saying unable to downloading toolchain...>
--2013-09-16 10:50:55--
http://sources.buildroot.net//gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux.tar.xz
<Stuff saying can't download toolcahin....>
make[1]: *** [/home/apps/opt/buildroot-2011.05/dl/gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux.tar.xz] Error 1
make: *** [all] Error 2

Note: I know the toolchain can't be downloaded, but the point is that I'm losing BR2_EXTERNAL path.

Here is the config that I'm using:

BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_EABIHF=y
BR2_DL_DIR="/home/apps/opt/buildroot-2011.05/dl"
BR2_JLEVEL=8
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_LINARO_2013_04=y BR2_TOOLCHAIN_EXTERNAL_GDB_SERVER_COPY=y BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_DEVTMPFS=y BR2_TARGET_GENERIC_GETTY_PORT="ttyO2" BR2_ROOTFS_OVERLAY="$(BR2_EXTERNAL)/board/common-omap/local-skeleton" BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_VERSION=y BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.8.13" BR2_LINUX_KERNEL_PATCH="$(BR2_EXTERNAL)/board/common-omap/kernel-patches-3.8.13" BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="$(BR2_EXTERNAL)/board/common-omap/linux-3.8.13.config" BR2_LINUX_KERNEL_ZIMAGE=y
BR2_TARGET_ROOTFS_TAR_BZIP2=y
BR2_TARGET_UBOOT=y
BR2_TARGET_UBOOT_BOARDNAME="omap3_beagle" BR2_TARGET_UBOOT_CUSTOM_PATCH_DIR="$(BR2_EXTERNAL)/board/common-omap/uboot-patches-2013.07" BR2_TARGET_UBOOT_FORMAT_IMG=y
BR2_TARGET_UBOOT_SPL=y
BR2_TARGET_UBOOT_SPL_NAME="MLO"
BR2_PACKAGE_HOST_E2FSPROGS=y
BR2_PACKAGE_HOST_GENEXT2FS=y

Thanks,
-Ryan

-----------------------------------------------------------------
Ryan J Barnett / Software Engineer / Platform SW MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US
Phone: 319-263-3880 / VPN: 263-3880
rjbarnet at rockwellcollins.com www.rockwellcollins.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130916/398d2c22/attachment.html>

#3 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Dear Ryan Barnett,

Quoted 4 lines · Ryan Barnett

Per our chat on IRC, putting BR2_EXTERNAL into the generated makefile isn't quite working out like I expected. I'm losing the BR2_EXTERNAL variable definition.

Quoted 8 lines

Here are the steps I'm taking after applying your v2 of this patch to a git clone of the mainline of buildroot. The buildroot clone is not in the directory of ~/tmp but elsewhere.

make O=~/tmp/build_xm BR2_EXTERNAL=~/tmp/buildroot rc_beagle_xm_min_defconfig
cd ~/tmp/build_xm
make

Ah, right. The below patch fixes that:

Makefile +3 -1
index 799c529..5d942bd 100644
@@ -59,6 +59,8 @@ else
 BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
 endif
 
+EXTRAMAKEARGS += BR2_EXTERNAL="$(BR2_EXTERNAL)"
+
 # Compute the full local version string so packages can use it as-is
 # Need to export it, so it can be got from environment in children (eg. mconf)
 export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
@@ -95,7 +97,7 @@ MAKEOVERRIDES =
 override O:=$(O)
 CONFIG_DIR:=$(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
-EXTRAMAKEARGS = O=$(O)
+EXTRAMAKEARGS += O=$(O)
 NEED_WRAPPER=y
 endif
 
Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#4 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

In reply to Thomas Petazzoni
Quoted 8 lines · Thomas Petazzoni

This commit introduces the BR2_EXTERNAL environment variable, which will allow to keep Buildroot customization (board-specific configuration files or root filesystem overlays, package Config.in and makefiles, as well as defconfigs) outside of the Buildroot tree.

This commit only introduces the variable itself, and ensures that it is available within Config.in options, so that string options used to specify paths to directories or files can use $BR2_EXTERNAL as a

$(BR2_EXTERNAL). The $ is expanded by make. Same comment applies to
other places in the commit message as well. You did put it correctly in the manual.

Quoted 16 lines

reference. For example, one can use $BR2_EXTERNAL/board/<someboard>/kernel.config as the BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.

Following patches extend the usage of BR2_EXTERNAL to other areas (packages and defconfigs).

In details, this commit:

* Introduces the BR2_EXTERNAL Kconfig option. This option has no prompt, and is therefore not visible to the user and also not stored in the .config file. It is automatically set to the value of the BR2_EXTERNAL environment variable. The only purpose of this BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be properly expanded when used inside Kconfig option values.

Quoted 5 lines

* Ensures that the path given in BR2_EXTERNAL is an absolute path, or if not path is given, ensures that BR2_EXTERNAL points to a dummy implementation in $(TOPDIR)/support/dummy-external/. This last part is not directly useful in this commit, but is needed to properly support Config.in options declared in BR2_EXTERNAL (next commit).

It would make more sense to do it in that commit then, but that's just
nitpicking.

Quoted 4 lines

* Passes the BR2_EXTERNAL into the *config environment, so that its value is found when parsing/evaluating Config.in files and .config values.

That's pretty useless in this commit, since it anyway doesn't end up in
.config and isn't used in Config.in.

Quoted 17 lines

* Encodes the BR2_EXTERNAL value into the Makefile wrapper (only if BR2_EXTERNAL is specified by the user, not if the dummy internal value is used).

---
  Config.in                  |  4 ++++
  Makefile                   | 13 +++++++++++--
  support/scripts/mkmakefile |  3 ++-
  3 files changed, 17 insertions(+), 3 deletions(-)
Config.in +0 -0
index 65c29b4..522eaf8 100644

Config.in config BR2_HOSTARCH · line 20

Quoted 7 lines
Config.in +4 -0
@@ -14,6 +14,10 @@ config BR2_HOSTARCH
  	string
  	option env="HOSTARCH"
 
+config BR2_EXTERNAL
+	string
+	option env="BR2_EXTERNAL"
+

If you agree to store the value in the .config file, it could look
something like this.

-------

config BR2_EXTERNAL_FROM_ENV
string
option env="BR2_EXTERNAL"

# This condition forces the user to set BR2_EXTERNAL from the

# command line the first time.
if BR2_EXTERNAL_FROM_ENV != ""
config BR2_EXTERNAL
         string "External buildroot add-ons"
         default BR2_EXTERNAL_FROM_ENV
         help
           List of directories with buildroot add-ons. ...
endif

if BR2_EXTERNAL != BR2_EXTERNAL_FROM_ENV
comment "You need to re-run the config to see the new packages" endif

-------

It doesn't really work, though, because currently the .config is not
read in when doing 'make menuconfig', so that BR2_EXTERNAL_FROM_ENV will always be "". But I'm sure we'll be able to find something that works with a little more effort.

Quoted 55 lines
Config.in +0 -0
  # Hidden boolean selected by pre-built packages for x86, when they
  # need to run on x86-64 machines (example: pre-built external
  # toolchains, binary tools like SAM-BA, etc.).
Makefile +11 -2
index 8610592..13ad342 100644
@@ -51,6 +51,14 @@ CONFIG_CONFIG_IN=Config.in
  CONFIG=support/kconfig
  DATE:=$(shell date +%Y%m%d)
 
+# Turn BR2_EXTERNAL into an absolute path
+ifneq ($(BR2_EXTERNAL),)
+override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))
+BR2_EXTERNAL_USED = y
+else
+BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
+endif
+
  # Compute the full local version string so packages can use it as-is
  # Need to export it, so it can be got from environment in children (eg. mconf)
  export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
@@ -632,7 +640,8 @@ COMMON_CONFIG_ENV = \
  	KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
  	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
  	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
-	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
+	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
+	BR2_EXTERNAL=$(BR2_EXTERNAL)
 
  xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
  	@mkdir -p $(BUILD_DIR)/buildroot-config
@@ -739,7 +748,7 @@ source-check:
  # output directory.
  outputmakefile:
  ifeq ($(NEED_WRAPPER),y)
-	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
+	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O) $(if $(BR2_EXTERNAL_USED),$(BR2_EXTERNAL))
  endif
 
  # printvars prints all the variables currently defined in our Makefiles
support/scripts/mkmakefile +1 -1
index cef2ec7..e7c1bd0 100755
@@ -6,7 +6,7 @@
  # Usage
  # $1 - Kernel src directory
  # $2 - Output directory
-
+# $3 - BR2_EXTERNAL directory
 
  test ! -r $2/Makefile -o -O $2/Makefile || exit 0
  # Only overwrite automatically generated Makefiles

support/scripts/mkmakefile makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST))) · line 30

Quoted 4 lines
support/scripts/mkmakefile +1 -0
@@ -27,6 +27,7 @@ makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
 
  MAKEARGS := -C $1
  MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$(patsubst %/,%,\$(makedir))
+MAKEARGS += BR2_EXTERNAL=$3

Obviously, this won't be needed anymore if the EXTERNAL is stored in
the .config.

Regards,
Arnout

Quoted 4 lines
support/scripts/mkmakefile +0 -0
  MAKEFLAGS += --no-print-directory
 
 
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#5 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Dear Arnout Vandecappelle,

Quoted 6 lines · Arnout Vandecappelle
Quoted 5 lines · Thomas Petazzoni

This commit introduces the BR2_EXTERNAL environment variable, which will allow to keep Buildroot customization (board-specific configuration files or root filesystem overlays, package Config.in and makefiles, as well as defconfigs) outside of the Buildroot tree.

Quoted 7 lines
Quoted 3 lines

This commit only introduces the variable itself, and ensures that it is available within Config.in options, so that string options used to specify paths to directories or files can use $BR2_EXTERNAL as a

$(BR2_EXTERNAL). The $ is expanded by make. Same comment applies to
other places in the commit message as well. You did put it correctly in the manual.

Right, true.

Quoted 9 lines
Quoted 6 lines

* Ensures that the path given in BR2_EXTERNAL is an absolute
path, or if not path is given, ensures that BR2_EXTERNAL points to a dummy implementation in $(TOPDIR)/support/dummy-external/. This last part is not directly useful in this commit, but is needed to properly support Config.in options declared in BR2_EXTERNAL (next commit).

It would make more sense to do it in that commit then, but that's
just nitpicking.

Yeah, but since it was really part of the same chunk of code, it was annoying to split further. But I can certainly split that up further.

Quoted 6 lines
Quoted 3 lines

* Passes the BR2_EXTERNAL into the *config environment, so that
its value is found when parsing/evaluating Config.in files and .config values.

That's pretty useless in this commit, since it anyway doesn't end
up in .config and isn't used in Config.in.

Right, could be moved to the next commit then (but to me the separation of the commits is relatively artificial, since the patch set really only makes sense if everything is applied).

Quoted 19 lines

If you agree to store the value in the .config file, it could look
something like this.

-------

config BR2_EXTERNAL_FROM_ENV
string
option env="BR2_EXTERNAL"

# This condition forces the user to set BR2_EXTERNAL from the

# command line the first time.
if BR2_EXTERNAL_FROM_ENV != ""
config BR2_EXTERNAL
         string "External buildroot add-ons"
         default BR2_EXTERNAL_FROM_ENV
         help
           List of directories with buildroot add-ons. ...
endif
Quoted 6 lines

if BR2_EXTERNAL != BR2_EXTERNAL_FROM_ENV
comment "You need to re-run the config to see the new packages" endif

-------

I don't know. Do we really want to do that? Isn't it simpler to always pass it as an environment variable?

I mean, if an user changes the value of BR2_EXTERNAL in menuconfig, he has to exit / re-run menuconfig to anyway see the updated set of configuration options after the BR2_EXTERNAL value change. Does it really makes sense to set in menuconfig a configuration option that by itself changes the available configuration options, but which requires exiting/re-runnning menuconfig?

Quoted 4 lines

It doesn't really work, though, because currently the .config is
not read in when doing 'make menuconfig', so that BR2_EXTERNAL_FROM_ENV will always be "". But I'm sure we'll be able to find something that works with a little more effort.

Hum, the .config is not read when doing 'make menuconfig'? Then how can 'make menuconfig' show the current state of enabled options?

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#6 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Quoted 36 lines · Thomas Petazzoni

Dear Arnout Vandecappelle,

Quoted 6 lines · Arnout Vandecappelle
Quoted 5 lines · Thomas Petazzoni

This commit introduces the BR2_EXTERNAL environment variable, which will allow to keep Buildroot customization (board-specific configuration files or root filesystem overlays, package Config.in and makefiles, as well as defconfigs) outside of the Buildroot tree.

Quoted 7 lines
Quoted 3 lines

This commit only introduces the variable itself, and ensures that it is available within Config.in options, so that string options used to specify paths to directories or files can use $BR2_EXTERNAL as a

$(BR2_EXTERNAL). The $ is expanded by make. Same comment applies to
other places in the commit message as well. You did put it correctly in the manual.

Right, true.

Quoted 9 lines
Quoted 6 lines

* Ensures that the path given in BR2_EXTERNAL is an absolute
path, or if not path is given, ensures that BR2_EXTERNAL points to a dummy implementation in $(TOPDIR)/support/dummy-external/. This last part is not directly useful in this commit, but is needed to properly support Config.in options declared in BR2_EXTERNAL (next commit).

It would make more sense to do it in that commit then, but that's
just nitpicking.

Yeah, but since it was really part of the same chunk of code, it was annoying to split further. But I can certainly split that up further.

Quoted 4 lines
Quoted 3 lines

* Passes the BR2_EXTERNAL into the *config environment, so that
its value is found when parsing/evaluating Config.in files and .config values.

Quoted 6 lines
Quoted 2 lines

That's pretty useless in this commit, since it anyway doesn't end
up in .config and isn't used in Config.in.

Right, could be moved to the next commit then (but to me the separation of the commits is relatively artificial, since the patch set really only makes sense if everything is applied).

Fair enough.

Quoted 29 lines
Quoted 19 lines

If you agree to store the value in the .config file, it could look
something like this.

-------

config BR2_EXTERNAL_FROM_ENV
string
option env="BR2_EXTERNAL"

# This condition forces the user to set BR2_EXTERNAL from the

# command line the first time.
if BR2_EXTERNAL_FROM_ENV != ""
config BR2_EXTERNAL
          string "External buildroot add-ons"
          default BR2_EXTERNAL_FROM_ENV
          help
            List of directories with buildroot add-ons. ...
endif
Quoted 6 lines

if BR2_EXTERNAL != BR2_EXTERNAL_FROM_ENV
comment "You need to re-run the config to see the new packages" endif

-------

I don't know. Do we really want to do that? Isn't it simpler to always pass it as an environment variable?

Quoted 6 lines

I mean, if an user changes the value of BR2_EXTERNAL in menuconfig, he has to exit / re-run menuconfig to anyway see the updated set of configuration options after the BR2_EXTERNAL value change. Does it really makes sense to set in menuconfig a configuration option that by itself changes the available configuration options, but which requires exiting/re-runnning menuconfig?

The primary way of setting BR2_EXTERNAL would still be through the
environment - in fact, the code above forces you to set it through the environment the first time, otherwise the option won't be visible.

Quoted 7 lines
Quoted 4 lines

It doesn't really work, though, because currently the .config is
not read in when doing 'make menuconfig', so that BR2_EXTERNAL_FROM_ENV will always be "". But I'm sure we'll be able to find something that works with a little more effort.

Hum, the .config is not read when doing 'make menuconfig'? Then how can 'make menuconfig' show the current state of enabled options?

I mena that it isn't read by the Makefile.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#7 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Dear Arnout Vandecappelle,

Quoted 17 lines · Arnout Vandecappelle
Quoted 6 lines

I mean, if an user changes the value of BR2_EXTERNAL in menuconfig, he has to exit / re-run menuconfig to anyway see the updated set of configuration options after the BR2_EXTERNAL value change. Does it really makes sense to set in menuconfig a configuration option that by itself changes the available configuration options, but which requires exiting/re-runnning menuconfig?

The primary way of setting BR2_EXTERNAL would still be through the
environment - in fact, the code above forces you to set it through the environment the first time, otherwise the option won't be visible.

Quoted 5 lines
Quoted 4 lines

It doesn't really work, though, because currently the .config is
not read in when doing 'make menuconfig', so that BR2_EXTERNAL_FROM_ENV will always be "". But I'm sure we'll be able to find something that works with a little more effort.

Quoted 4 lines
Quoted 2 lines

Hum, the .config is not read when doing 'make menuconfig'? Then how can 'make menuconfig' show the current state of enabled options?

I mena that it isn't read by the Makefile.

Ok, so what do you suggest exactly? Because you proposed an implementation, but concluded with 'it doesn't work'. So I'm a little bit confused here :-)

Thanks!

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#8 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

In reply to Thomas Petazzoni

Thomas,

I've been using this patch for around a week now and today I tired to generate the manual with this patch applied. However, "make manual" fails pretty bad with this patch set applied. Have you tired generating the manual at all?

Here is what I've done:
* Applied all four patches for v2 of BR2_EXTERNAL
* make clean
* make manual

I'm not exactly sure the root cause of the problem but my guess is the top-level Config.in. To fix the problem, I just revert the 4 patches and I can build the manual fine. When I do make manual I get the following error output:

/home/rjbarnet/projects/buildroot/buildroot-git/Config.in:19:warning:
The symbol BR2_EXTERNAL references the non-existent environment variable BR2_EXTERNAL and will
get the empty string as its value.

If you're using kconfiglib via 'make (i)scriptconfig' it should have set up the
environment correctly for you. If you still got this message, that might be an
error, and you should e-mail kconfiglib at gmail.com.
.
Traceback (most recent call last):
File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py", line 376, in <module>
buildroot = Buildroot()
File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py", line 217, in __init__
self.root_config))
File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", line 214, in __init__
self.top_block = self._parse_file(filename, None, None, None)
File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", line 919, in _parse_file
return self._parse_block(line_feeder, None, parent, deps,
visible_if_deps, res)
File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", line 1114, in _parse_block

    self.base_dir))

IOError: /home/rjbarnet/projects/buildroot/buildroot-git/Config.in:490:
sourced file "$BR2_EXTERNAL/Config.in" (expands to "2_EXTERNAL/Config.in") not found. Perhaps base_dir (argument to Config.__init__(), currently "/home/rjbarnet/projects/buildroot/buildroot-git") is set to the wrong value.
make: *** [manual-update-lists] Error 1

Thanks,
-Ryan

Thomas Petazzoni <[email protected]> wrote on 09/14/2013 02:03:50 PM:

Quoted 68 lines

This commit introduces the BR2_EXTERNAL environment variable, which will allow to keep Buildroot customization (board-specific configuration files or root filesystem overlays, package Config.in and makefiles, as well as defconfigs) outside of the Buildroot tree.

This commit only introduces the variable itself, and ensures that it is available within Config.in options, so that string options used to specify paths to directories or files can use $BR2_EXTERNAL as a reference. For example, one can use $BR2_EXTERNAL/board/<someboard>/kernel.config as the BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.

Following patches extend the usage of BR2_EXTERNAL to other areas (packages and defconfigs).

In details, this commit:

* Introduces the BR2_EXTERNAL Kconfig option. This option has no prompt, and is therefore not visible to the user and also not stored in the .config file. It is automatically set to the value of the BR2_EXTERNAL environment variable. The only purpose of this BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be properly expanded when used inside Kconfig option values.

* Ensures that the path given in BR2_EXTERNAL is an absolute path, or if not path is given, ensures that BR2_EXTERNAL points to a dummy implementation in $(TOPDIR)/support/dummy-external/. This last part is not directly useful in this commit, but is needed to properly support Config.in options declared in BR2_EXTERNAL (next commit).

* Passes the BR2_EXTERNAL into the *config environment, so that its value is found when parsing/evaluating Config.in files and .config values.

* Encodes the BR2_EXTERNAL value into the Makefile wrapper (only if BR2_EXTERNAL is specified by the user, not if the dummy internal value is used).

---
 Config.in                  |  4 ++++
 Makefile                   | 13 +++++++++++--
 support/scripts/mkmakefile |  3 ++-
 3 files changed, 17 insertions(+), 3 deletions(-)
Config.in +4 -0
index 65c29b4..522eaf8 100644
@@ -14,6 +14,10 @@ config BR2_HOSTARCH
    string
    option env="HOSTARCH"
 
+config BR2_EXTERNAL
+   string
+   option env="BR2_EXTERNAL"
+
 # Hidden boolean selected by pre-built packages for x86, when they
 # need to run on x86-64 machines (example: pre-built external
 # toolchains, binary tools like SAM-BA, etc.).
Makefile +0 -0
index 8610592..13ad342 100644
@@ -51,6 +51,14 @@ CONFIG_CONFIG_IN=Config.in
 CONFIG=support/kconfig
 DATE:=$(shell date +%Y%m%d)
 

Makefile CONFIG_CONFIG_IN=Config.in · line 63

Quoted 10 lines
Makefile +8 -0
+# Turn BR2_EXTERNAL into an absolute path
+ifneq ($(BR2_EXTERNAL),)
+override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))
+BR2_EXTERNAL_USED = y
+else
+BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
+endif
+
 # Compute the full local version string so packages can use it as-is
 # Need to export it, so it can be got from environment in children (eg. 

mconf)

Quoted 51 lines
Makefile +0 -0
 export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/

scripts/setlocalversion)

Makefile +3 -2
@@ -632,7 +640,8 @@ COMMON_CONFIG_ENV = \
    KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
    KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
    KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
-   BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
+   BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
+   BR2_EXTERNAL=$(BR2_EXTERNAL)
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
    @mkdir -p $(BUILD_DIR)/buildroot-config
@@ -739,7 +748,7 @@ source-check:
 # output directory.
 outputmakefile:
 ifeq ($(NEED_WRAPPER),y)
-   $(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
+   $(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O) $(if $
 (BR2_EXTERNAL_USED),$(BR2_EXTERNAL))
 endif
 
 # printvars prints all the variables currently defined in our Makefiles
support/scripts/mkmakefile +2 -1
index cef2ec7..e7c1bd0 100755
@@ -6,7 +6,7 @@
 # Usage
 # $1 - Kernel src directory
 # $2 - Output directory
-
+# $3 - BR2_EXTERNAL directory
 
 test ! -r $2/Makefile -o -O $2/Makefile || exit 0
 # Only overwrite automatically generated Makefiles
@@ -27,6 +27,7 @@ makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
 
 MAKEARGS := -C $1
 MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$
 (patsubst %/,%,\$(makedir))
+MAKEARGS += BR2_EXTERNAL=$3
 
 MAKEFLAGS += --no-print-directory
Signature · 7 lines
-- 
1.8.1.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130923/7857176c/attachment-0001.html>

#9 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Hi Ryan, Thomas,

2013/9/23 Ryan Barnett <[email protected]>

Quoted 13 lines

Thomas,

I've been using this patch for around a week now and today I tired to generate the manual with this patch applied. However, "make manual" fails pretty bad with this patch set applied. Have you tired generating the manual at all?

Here is what I've done:
* Applied all four patches for v2 of BR2_EXTERNAL
* make clean
* make manual

Quoted 7 lines

I'm not exactly sure the root cause of the problem but my guess is the top-level Config.in. To fix the problem, I just revert the 4 patches and I can build the manual fine. When I do make manual I get the following error output:

/home/rjbarnet/projects/buildroot/buildroot-git/Config.in:19:warning:
The symbol BR2_EXTERNAL references the non-existent environment variable

BR2_EXTERNAL and will

Quoted 3 lines

get the empty string as its value.

If you're using kconfiglib via 'make (i)scriptconfig' it should have set

up the

Quoted 1 line

environment correctly for you. If you still got this message, that might

be an

Quoted 4 lines

error, and you should e-mail kconfiglib at gmail.com. .
Traceback (most recent call last):
File

"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py", line 376, in <module>

Quoted 2 lines

buildroot = Buildroot()
File

"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py", line 217, in __init__

Quoted 2 lines

self.root_config))
File

"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", line 214, in __init__

Quoted 2 lines

self.top_block = self._parse_file(filename, None, None, None)
File

"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", line 919, in _parse_file

Quoted 1 line

return self._parse_block(line_feeder, None, parent, deps,

visible_if_deps, res)

Quoted 1 line

File

"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", line 1114, in _parse_block

Quoted 2 lines
    self.base_dir))

IOError: /home/rjbarnet/projects/buildroot/buildroot-git/Config.in:490:

sourced file "$BR2_EXTERNAL/Config.in" (expands to

Quoted 3 lines

"2_EXTERNAL/Config.in") not found. Perhaps base_dir (argument to Config.__init__(), currently "/home/rjbarnet/projects/buildroot/buildroot-git") is set to the wrong

value.

Quoted 1 line

make: *** [manual-update-lists] Error 1

Good catch!
I missed it...

Could you try this patch: http://code.bulix.org/y27qmv-84542

Regards,

--
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130924/b6c49e3c/attachment.html>

#10 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Samuel,

Samuel Martin <[email protected]> wrote on
09/23/2013 05:17:08 PM:

Quoted 2 lines

Hi Ryan, Thomas,

[...]

Quoted 5 lines

Good catch!
I missed it...

Could you try this patch: http://code.bulix.org/y27qmv-84542

Could you send this in an email? I can't get to any pastebin type website from behind my work's firewall.

Quoted 6 lines

Regards,

--
Samuel

Thanks,
-Ryan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130923/2c2b9f31/attachment.html>

#11 lore raw

PATCH 1/1 core: introduce the BR2_EXTERNAL variable (additional patch to fix manual build)

---
 docs/manual/manual.mk | 1 +
 1 file changed, 1 insertion(+)
docs/manual/manual.mk +1 -0
index 157d13c..8cd732a 100644
@@ -1,6 +1,7 @@
 manual-update-lists: manual-check-dependencies-lists
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
+		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
 		$(TOPDIR)/support/scripts/gen-manual-lists.py
 
 manual-check-dependencies:
Signature · 1 line
1.8.4
#12 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

In reply to Ryan Barnett

Dear Ryan Barnett,

It's a problem I had already fixed in earlier versions, where I was passing the proper BR2_EXTERNAL value, but Arnout disagreed in that external packages should not be listed in the manual (which is certainly correct), so I removed the variable. But indeed, there needs to be a value, or the Kconfig logic is broken. The solution (below) from Samuel is correct.

Quoted 2 lines · Ryan Barnett

Could you send this in an email? I can't get to any pastebin type website from behind my work's firewall.

commit f160920f50898ee7fd8d96a4613af2e7a893db13

Author: Samuel Martin <[email protected]>
Date:   Tue Sep 24 00:07:09 2013 +0200

manual: fix BR2_EXTERNAL support when building the manual

docs/manual/manual.mk +1 -0
index 157d13c..8cd732a 100644
@@ -1,6 +1,7 @@
 manual-update-lists: manual-check-dependencies-lists
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
+		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
 		$(TOPDIR)/support/scripts/gen-manual-lists.py
 
 manual-check-dependencies:
Signature · 3 lines
Thomas Petazzoni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#13 lore raw

PATCH v2 1/4 core: introduce the BR2_EXTERNAL variable

Ryan, Thomas,

2013/9/24 Thomas Petazzoni <[email protected]>

Quoted 28 lines

Dear Ryan Barnett,

It's a problem I had already fixed in earlier versions, where I was passing the proper BR2_EXTERNAL value, but Arnout disagreed in that external packages should not be listed in the manual (which is certainly correct), so I removed the variable. But indeed, there needs to be a value, or the Kconfig logic is broken. The solution (below) from Samuel is correct.

Quoted 2 lines · Ryan Barnett

Could you send this in an email? I can't get to any pastebin type website from behind my work's firewall.

commit f160920f50898ee7fd8d96a4613af2e7a893db13

Author: Samuel Martin <[email protected]>
Date:   Tue Sep 24 00:07:09 2013 +0200

manual: fix BR2_EXTERNAL support when building the manual

docs/manual/manual.mk +0 -0
index 157d13c..8cd732a 100644

docs/manual/manual.mk line 7

Quoted 7 lines
docs/manual/manual.mk +1 -0
@@ -1,6 +1,7 @@
 manual-update-lists: manual-check-dependencies-lists
        $(Q)$(call MESSAGE,"Updating the manual lists...")
        $(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
+               BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
                $(TOPDIR)/support/scripts/gen-manual-lists.py
 
 manual-check-dependencies:

This patch fixes the undefined symbol error triggered by kconfiglib package, but not the wrong symbol expansion.
Renaming BR2_EXTERNAL to BR_EXTERNAL is enough to solve the later issue. This sounds like a kconfiglib bug... :(
Anyway, it does not explain why this was not triggered before...

Next step: understand and fix the expansion problem in kconfiglib

Regards,

Signature · 4 lines
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130924/84a91c54/attachment.html>
#14 lore raw

PATCH v2 2/4 core: allow external Config.in/makefile code to be integrated

In reply to Thomas Petazzoni

This commit allows the BR2_EXTERNAL directory to contain Config.in and Makefile code, which gets integrated into the Buildroot build logic:

- Buildroot automatically includes the $BR2_EXTERNAL/Config.in in the top-level configuration menu.

- Buildroot automatically includes the BR2_EXTERNAL/external.mk in the build logic, so it can for example be used to include other .mk files that define package recipes.

We also add a dummy Config.in file in support/dummy-external/ to ensure that the source "$BR2_EXTERNAL/Config.in" line will point to an existing file even when BR2_EXTERNAL is not used by the user.

---
 Config.in                        | 2 ++
 Makefile                         | 4 ++++
 support/dummy-external/Config.in | 0
 3 files changed, 6 insertions(+)
 create mode 100644 support/dummy-external/Config.in
Config.in +2 -0
index 522eaf8..9ef5466 100644
@@ -465,3 +465,5 @@ source "boot/Config.in"
 source "package/Config.in.host"
 
 source "Config.in.legacy"
+
+source "$BR2_EXTERNAL/Config.in"
Makefile +4 -0
index 13ad342..bea5ad5 100644
@@ -348,6 +348,10 @@ include boot/common.mk
 include linux/linux.mk
 include system/system.mk
 
+ifeq ($(BR2_EXTERNAL_USED),y)
+include $(BR2_EXTERNAL)/external.mk
+endif
+
 TARGETS+=target-finalize
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
support/dummy-external/Config.in new file +0 -0
new file mode 100644
index 0000000..e69de29
Signature · 1 line
1.8.1.2
#15 lore raw

PATCH v2 2/4 core: allow external Config.in/makefile code to be integrated

Quoted 27 lines · Thomas Petazzoni

This commit allows the BR2_EXTERNAL directory to contain Config.in and Makefile code, which gets integrated into the Buildroot build logic:

- Buildroot automatically includes the $BR2_EXTERNAL/Config.in in the top-level configuration menu.

- Buildroot automatically includes the BR2_EXTERNAL/external.mk in the build logic, so it can for example be used to include other .mk files that define package recipes.

We also add a dummy Config.in file in support/dummy-external/ to ensure that the source "$BR2_EXTERNAL/Config.in" line will point to an existing file even when BR2_EXTERNAL is not used by the user.

---
  Config.in                        | 2 ++
  Makefile                         | 4 ++++
  support/dummy-external/Config.in | 0
  3 files changed, 6 insertions(+)

create mode 100644 support/dummy-external/Config.in

Config.in +0 -0
index 522eaf8..9ef5466 100644

Config.in source "boot/Config.in" · line 469

Quoted 5 lines
Config.in +2 -0
@@ -465,3 +465,5 @@ source "boot/Config.in"
  source "package/Config.in.host"
 
  source "Config.in.legacy"
+
+source "$BR2_EXTERNAL/Config.in"

I would prefer this to be in package/Config.in, and source
$BR2_EXTERNAL/package/Config.in. I think 95% of the use cases will be for packages, and the other 5% are bootloaders which are the same as packages anyway. And even if it is not a proper generic package (e.g. it's a filesystem), putting it in the packages menu doesn't hurt that much. If you don't put it in the package subdirectory, then you either force users to create a oneline Config.in, or you end up with a messy looking directory mixing packages with the configs/ and board/ directories.

Also it is perhaps better to add to the Makefile:

ifeq ($(wildcard $(BR2_EXTERNAL)/package/Config.in,) EXTERNAL_PACKAGE_CONFIG_IN = support/dummy-external/Config.in else
EXTERNAL_PACKAGE_CONFIG_IN = $(BR2_EXTERNAL)/package/Config.in endif

(and pass that to the config env as well).

That way, you're not obliged to have a Config.in in your external tree,
and it is still possible to use it for nothing more than the defconfigs etc.

Makefile include boot/common.mk · line 352

Quoted 10 lines
Makefile +2 -0
index 13ad342..bea5ad5 100644
@@ -348,6 +348,10 @@ include boot/common.mk
  include linux/linux.mk
  include system/system.mk
 
+ifeq ($(BR2_EXTERNAL_USED),y)
+include $(BR2_EXTERNAL)/external.mk

Here I'd

-include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))

Regards,
Arnout

Quoted 9 lines
Makefile +2 -0
+endif
+
  TARGETS+=target-finalize
 
  ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
support/dummy-external/Config.in new file +0 -0
new file mode 100644
index 0000000..e69de29
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#16 lore raw

PATCH v2 2/4 core: allow external Config.in/makefile code to be integrated

Dear Arnout Vandecappelle,

Quoted 33 lines · Arnout Vandecappelle

I would prefer this to be in package/Config.in, and source
$BR2_EXTERNAL/package/Config.in. I think 95% of the use cases will be for packages, and the other 5% are bootloaders which are the same as packages anyway. And even if it is not a proper generic package (e.g. it's a filesystem), putting it in the packages menu doesn't hurt that much. If you don't put it in the package subdirectory, then you either force users to create a oneline Config.in, or you end up with a messy looking directory mixing packages with the configs/ and board/ directories.

Also it is perhaps better to add to the Makefile:

ifeq ($(wildcard $(BR2_EXTERNAL)/package/Config.in,) EXTERNAL_PACKAGE_CONFIG_IN = support/dummy-external/Config.in else
EXTERNAL_PACKAGE_CONFIG_IN = $(BR2_EXTERNAL)/package/Config.in endif

(and pass that to the config env as well).

That way, you're not obliged to have a Config.in in your external
tree, and it is still possible to use it for nothing more than the defconfigs etc.

Quoted 8 lines
Makefile +0 -0
index 13ad342..bea5ad5 100644
@@ -348,6 +348,10 @@ include boot/common.mk
  include linux/linux.mk
  include system/system.mk
 
Quoted 6 lines

Makefile include boot/common.mk · line 352

Quoted 2 lines
Makefile +2 -0
+ifeq ($(BR2_EXTERNAL_USED),y)
+include $(BR2_EXTERNAL)/external.mk

Here I'd

-include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))

Right, it's a question of vision on what BR2_EXTERNAL should allow. As I already stated, I originally had the same idea as you, but then thought that including the external Config.in at the top-level, and including an external.mk was offering more customization possibilities.

But I agree with you that 'enforcing' the usage of BR2_EXTERNAL only for packages might be a good idea. It provides a clear guideline on how BR2_EXTERNAL should be used/organized, which is probably good.

Unless there is opposition from others, I believe I'll move in the direction you're proposing.

Thanks!

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#17 lore raw

PATCH v2 3/4 core: allow external defconfigs to be used

In reply to Thomas Petazzoni

This commit allows the user to store defconfigs in $BR2_EXTERNAL/configs/. To achieve this:

* It adds a new %_defconfig that looks in $BR2_EXTERNAL/configs/ for the corresponding defconfig file.

* Updates the help target to also list external defconfigs.

---
 Makefile | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Makefile +7 -1
index bea5ad5..799c529 100644
@@ -729,6 +729,12 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
 	@$(COMMON_CONFIG_ENV) $< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
 
+ifeq ($(BR2_EXTERNAL_USED),y)
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
+	@mkdir -p $(BUILD_DIR)/buildroot-config
+	@$(COMMON_CONFIG_ENV) $< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
+endif
+
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
 	@$(COMMON_CONFIG_ENV) $< \
@@ -840,7 +846,7 @@ endif
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 	@echo '  make O=dir             - Locate all output files in "dir", including .config'
 	@echo
-	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig $(if $(BR2_EXTERNAL_USED),$(BR2_EXTERNAL)/configs/*_defconfig)))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 	@echo
 	@echo 'See docs/README, or generate the Buildroot manual for further details'
Signature · 1 line
1.8.1.2
#18 lore raw

PATCH v2 3/4 core: allow external defconfigs to be used

Quoted 9 lines · Thomas Petazzoni

This commit allows the user to store defconfigs in $BR2_EXTERNAL/configs/. To achieve this:

* It adds a new %_defconfig that looks in $BR2_EXTERNAL/configs/ for the corresponding defconfig file.

* Updates the help target to also list external defconfigs.

Looks good!

Regards,
Arnout

[snip]

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#19 lore raw

PATCH v2 4/4 docs/manual: add explanations about BR2_EXTERNAL

In reply to Thomas Petazzoni

This commit updates the manual to add details on how to use the BR2_EXTERNAL feature.

---
 docs/manual/customize-outside-br.txt | 112 +++++++++++++++++++++++++++++++++++
 docs/manual/customize.txt            |   2 +
 2 files changed, 114 insertions(+)
 create mode 100644 docs/manual/customize-outside-br.txt
docs/manual/customize-outside-br.txt new file +112 -0
new file mode 100644
index 0000000..74899af
@@ -0,0 +1,112 @@
+// -*- mode:doc -*- ;
+
+[[outside-br-custom]]
+Keeping customization outside Buildroot
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+While the Buildroot community recommends and encourages upstreaming to
+the official Buildroot version the packages and boards support that
+are written by developers, it is sometimes not possible or desirable,
+due to these packages or boards being highly specific or proprietary.
+
+In this case, Buildroot users are offered two choices:
+
+ * They can add their packages, board support and configuration files
+   directly within the Buildroot tree, and maintain them by using
+   branches in a version control system.
+
+ * They can use the +BR2_EXTERNAL+ mechanism, which allows to keep
+   package recipes, board support and configuration files outside of
+   the Buildroot tree, while still having them nicely integrated in
+   the build logic. The following paragraphs give details on how to
+   use +BR2_EXTERNAL+.
+
++BR2_EXTERNAL+ is an environment variable that one can use to point to
+a directory that contains Buildroot customizations. This environment
+variable must be passed everytime a Buildroot +make+ command is
+invoked, be it at configuration time (+make menuconfig+, +make
+xconfig+, etc.), at build time (+make+) or any other Buildroot command
+(+make source+, +make legal-info+, etc.). As an exception, when
+building out-of-tree with +O=+, the +BR2_EXTERNAL+ value is encoded
+into the Makefile wrapper generated by Buildroot, so that passing
++BR2_EXTERNAL+ is no longer needed.
+
+Examples:
+
+-----
+ buildroot/ $ make BR2_EXTERNAL=../company menuconfig
+ buildroot/ $ make BR2_EXTERNAL=../company
+-----
+
+This +BR2_EXTERNAL+ then allows three different things:
+
+ * One can store all the board-specific configuration files here, such
+   as the kernel configuration, the root filesystem overlay, or any
+   other configuration file for which Buildroot allows to set its
+   location. The +BR2_EXTERNAL+ value is available within the
+   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
+   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
+   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
+   filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
+   Buildroot option to +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+
+   (to specify the location of the kernel configuration file).  + To
+   achieve this, it is recommended but not mandatory, to store those
+   details in directories called +board/<boardname>/+ under
+   +BR2_EXTERNAL+.
+
+ * One can store package recipes (i.e +Config.in+ and
+   +<packagename>.mk+), or even custom configuration options and make
+   logic. Buildroot automatically includes +BR2_EXTERNAL/Config.in+ to
+   make it appear in the top-level configuration menu, and includes
+   +BR2_EXTERNAL/external.mk+ with the rest of the makefile logic.
++
+The main usage of this is to store package recipes. The recommended
+   way to do this is to write a +BR2_EXTERNAL/Config.in+ that looks
+   like:
++
+------
+menu "<somecompany> packages"
+
+source "$BR2_EXTERNAL/package/package1/Config.in"
+source "$BR2_EXTERNAL/package/package2/Config.in"
+
+endmenu
+------
++
+Then, have a +BR2_EXTERNAL/external.mk' file that looks like:
++
+------
+include $(BR2_EXTERNAL)/package/*/*.mk
+------
++
+And then in +BR2_EXTERNAL/package/package1+ and
+   +BR2_EXTERNAL/package/package2+ create normal Buildroot package
+   recipes, as explained in xref:adding-packages[].
+
+ * One can store Buildroot defconfigs in the +configs+ subdirectory of
+   +BR2_EXTERNAL+. Buildroot will automatically show them in the
+   output of +make help+ and allow them to be loaded with the normal
+   +make <name>_defconfig+ command.
+
+In the end, a typical +BR2_EXTERNAL+ directory organization would
+generally be:
+
+-----
+??? Config.in
+??? external.mk
+??? board/
+?   ??? <boardname>/
+?       ??? overlay/
+?           ??? etc/
+?               ??? <some file>
+??? configs/
+?   ??? <boardname>_defconfig
+??? package/
+    ??? Config.in
+    ??? package1/
+        ??? Config.in
+        ??? package1.mk
+    ??? package2/
+        ??? Config.in
+        ??? package2.mk
+------
docs/manual/customize.txt +2 -0
index 0456ef1..7e46fd8 100644
@@ -17,3 +17,5 @@ include::customize-toolchain.txt[]
 include::customize-store.txt[]
 
 include::customize-packages.txt[]
+
+include::customize-outside-br.txt[]
Signature · 1 line
1.8.1.2
#20 lore raw

PATCH v2 4/4 docs/manual: add explanations about BR2_EXTERNAL

Quoted 5 lines · Thomas Petazzoni
docs/manual/customize-outside-br.txt new file +5 -0
+This +BR2_EXTERNAL+ then allows three different things:
+
+ * One can store all the board-specific configuration files here, such
+   as the kernel configuration, the root filesystem overlay, or any
+   other configuration file for which Buildroot allows to set its

docs/manual/customize-outside-br.txt line 55

Quoted 10 lines
docs/manual/customize-outside-br.txt new file +10 -0
+   location. The +BR2_EXTERNAL+ value is available within the
+   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
+   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
+   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
+   filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
+   Buildroot option to +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+
+   (to specify the location of the kernel configuration file).  + To
+   achieve this, it is recommended but not mandatory, to store those
+   details in directories called +board/<boardname>/+ under
+   +BR2_EXTERNAL+.

A minor point: this is slightly inconsistent with the existing Buildroot manual (see customize-store.txt), where the root filesystem overlay subdirectory is named "fs-overlay", and the kernel configuration is named "linux.config".

docs/manual/customize-outside-br.txt line 74

Quoted 12 lines
docs/manual/customize-outside-br.txt new file +12 -0
+The main usage of this is to store package recipes. The recommended
+   way to do this is to write a +BR2_EXTERNAL/Config.in+ that looks
+   like:
++
+------
+menu "<somecompany> packages"
+
+source "$BR2_EXTERNAL/package/package1/Config.in"
+source "$BR2_EXTERNAL/package/package2/Config.in"
+
+endmenu
+------

Shouldn't this rather be in the $BR2_EXTERNAL/package/Config.in file (which appears in your example overlay directory layout tree diagram), which should itself be sourced from the $BR2_EXTERNAL/Config.in file?

Simon.

#21 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

In reply to Thomas Petazzoni
Quoted 3 lines · Thomas Petazzoni

Hello,

Here is the second version of the BR2_EXTERNAL patch series.

This is looking really good, and fits my typical workflow very nicely.

Quoted 5 lines

Remaning questions
==================

* Interaction with the 'local.mk' mechanism. Should 'external.mk' replace the 'local.mk' mechanism? Should they co-exist?

This is my only slight concern. I would personally prefer to see the local.mk mechanism replaced by the new external.mk mechanism; it just feels wrong to have both mechanisms in there.

Simon.

#22 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

Dear Simon Dawson,

Quoted 9 lines · Simon Dawson
Quoted 5 lines

Remaning questions
==================

* Interaction with the 'local.mk' mechanism. Should 'external.mk' replace the 'local.mk' mechanism? Should they co-exist?

This is my only slight concern. I would personally prefer to see the local.mk mechanism replaced by the new external.mk mechanism; it just feels wrong to have both mechanisms in there.

I tend to agree, but there are some differences between the two mechanisms:

(1) the location/name of local.mk is defined within the .config, while the BR2_EXTERNAL location must be passed as environment variable.

(2) with the BR2_EXTERNAL mechanism, you _have_ to create a BR2_EXTERNAL/Config.in file (even empty), which is not needed with local.mk when all what you want is to use the OVERRIDE_SRCDIR mechanism.

(3) The local.mk is included *before* the package .mk files from Buildroot (so that the <foo>_OVERRIDE_SRCDIR is defined before 'foo.mk' is included and calls its generic-package/autotools-package macro), while the external.mk file is included *after* all Buildroot package .mk files.

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#23 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

Quoted 8 lines · Thomas Petazzoni
Quoted 3 lines · Simon Dawson

This is my only slight concern. I would personally prefer to see the local.mk mechanism replaced by the new external.mk mechanism; it just feels wrong to have both mechanisms in there.

I tend to agree, but there are some differences between the two mechanisms:

Quoted 7 lines

(1) the location/name of local.mk is defined within the .config, while the BR2_EXTERNAL location must be passed as environment variable.

(2) with the BR2_EXTERNAL mechanism, you _have_ to create a BR2_EXTERNAL/Config.in file (even empty), which is not needed with local.mk when all what you want is to use the OVERRIDE_SRCDIR mechanism.

Hmm, yes. That does complicate the issue somewhat. That suggests to me that we should either:

* keep both local.mk and external.mk, and just live with the slight aesthetic niggle
* do away with the external.mk

Quoted 5 lines

(3) The local.mk is included *before* the package .mk files from Buildroot (so that the <foo>_OVERRIDE_SRCDIR is defined before 'foo.mk' is included and calls its generic-package/autotools-package macro), while the external.mk file is included *after* all Buildroot package .mk files.

Is there any particular reason to include external.mk after the internal Buildroot package .mk files? (Sorry if this has been explained already somewhere.)

Simon.

#24 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

In reply to Thomas Petazzoni

Thomas, All,

On 2013-09-14 21:03 +0200, Thomas Petazzoni spake thusly:

Quoted 3 lines

Here is the second version of the BR2_EXTERNAL patch series.

Changes since v1

[--SNIP--]

Quoted 2 lines

* BR2_EXTERNAL is turned into an absolute path, in order to avoid any problem. Suggested by Arnout.

I dsiagree on that one: just use whatever the user passed in BR2_EXTERNAL.

I see at least one case where it will help. Let's suppose I decide to distribute my Buildroot tree *and* my external tree. I decide to do so as thus:

    $ tar tf project.tar
    project/
    project/buildroot/
    project/buildroot/all-buildroot-files-and-dirs
    project/buildroot/.config
    project/external/
    project/external/all-external-files-and-dirs

and project/buildroot/.config would contain BR2_EXTERNAL="../external" so the recipient would just have to run make withour hurdles.

If .config would contain absolute paths, .config would be broken as soon as it gets distribute (chances the recipient uses the same directory layout as my build machine is near zero).

I think we should instead try to fix the cases were relative paths are broken, and use exactly what the user passes us, as-is, without mangling.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#25 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

Dear Yann E. MORIN,

Quoted 11 lines · Yann E. MORIN

I see at least one case where it will help. Let's suppose I decide to distribute my Buildroot tree *and* my external tree. I decide to do so as thus:

    $ tar tf project.tar
    project/
    project/buildroot/
    project/buildroot/all-buildroot-files-and-dirs
    project/buildroot/.config
    project/external/
    project/external/all-external-files-and-dirs
Quoted 2 lines

and project/buildroot/.config would contain BR2_EXTERNAL="../external" so the recipient would just have to run make withour hurdles.

The BR2_EXTERNAL value is *NOT* stored in .config. It must be passed as an environment variable, so in the case you describe above, you would have to distribute project.tar, with a small document that says:

""
To build the stuff:

1/ cd project/buildroot/
2/ make BR2_EXTERNAL=../external
""

The fact that BR2_EXTERNAL is *not* part of the .config has already been discussed in the thread of the previous version. The "config BR2_EXTERNAL" option is prompt-less and therefore the value of this variable is not stored in .config.

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#26 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

In reply to Thomas Petazzoni

Hi all,

Quoted 1 line · Thomas Petazzoni

Here is the second version of the BR2_EXTERNAL patch series.

I've taken a few days to let the dust settle on this patch series (and
it seems it still hasn't settled...). After reading the entire series this time (instead of commenting before I understood what was happening exactly), I have two overall comments still. More comments will follow in individual patches.

1. Peter nacked all previous attempts of having this type of feature, so I think it's best if he gives an indication if this series has a chance of survival before we all spend more time on it.

2. I really have a huge problem with the fact that the .config is no longer complete, and that instead you have to explicitly provide extra command-line arguments to be able to reproduce the build. And it doesn't help that it behaves differently when you have an out-of-tree output directory (where the BR2_EXTERNAL is stored) than when you're using the default output directory. I'm actually very surprised that nobody else seems to have a problem with this. So for me, this gets a big Nack unless the BR2_EXTERNAL is stored in the .config.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#27 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

Dear Arnout Vandecappelle,

Quoted 10 lines · Arnout Vandecappelle

I've taken a few days to let the dust settle on this patch series
(and it seems it still hasn't settled...). After reading the entire series this time (instead of commenting before I understood what was happening exactly), I have two overall comments still. More comments will follow in individual patches.

1. Peter nacked all previous attempts of having this type of feature, so I think it's best if he gives an indication if this series has a chance of survival before we all spend more time on it.

Adding Peter in Cc on this one. I know he is a bit behind in terms of reading the backlog of Buildroot e-mails (which has been enormous since a few days).

I have the pretension of thinking that the currently proposed implementation is simple enough to remain in the KISS spirit of Buildroot. I believe that what prevented earlier implementation from being merged was not so much that they allowed 'external packages', but more due to their complexity. But honestly, I don't really remember the implementation details of the previous proposals, so I might be wrong on that one.

Quoted 9 lines

2. I really have a huge problem with the fact that the .config is no longer complete, and that instead you have to explicitly provide extra command-line arguments to be able to reproduce the build. And it doesn't help that it behaves differently when you have an out-of-tree output directory (where the BR2_EXTERNAL is stored) than when you're using the default output directory. I'm actually very surprised that nobody else seems to have a problem with this. So for me, this gets a big Nack unless the BR2_EXTERNAL is stored in the .config.

On the wrapper Makefile that contains BR2_EXTERNAL, I am not convinced it is a good idea, since as you mention it creates a difference whether you're building out-of-tree from the output directory, or out-of-tree but from the source directory. I think this is not good, and therefore that BR2_EXTERNAL shouldn't be kept in the wrapper Makefile.

However, I still do not understand how storing BR2_EXTERNAL in .config will clarify things for the user. Changing BR2_EXTERNAL affects what 'menuconfig' shows. So there is no way for 'menuconfig' to update what it displays when the user changes the value of BR2_EXTERNAL without exiting/re-running menuconfig, which sounds really odd.

To me, BR2_EXTERNAL= is a bit like O=, it's a local build decision that you make to add some external configurations/external packages. Most likely, if you have external packages in BR2_EXTERNAL, the defconfig you have to use is also in BR2_EXTERNAL, so there's no way you can forget to specify BR2_EXTERNAL on the command, since otherwise you won't be able to even see your defconfig.

So while I do agree that being able to store BR2_EXTERNAL would be nice, I don't see how it can be achieved in a way that isn't confusing for the user.

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#28 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

Quoted 49 lines · Thomas Petazzoni

Dear Arnout Vandecappelle,

Quoted 10 lines · Arnout Vandecappelle

I've taken a few days to let the dust settle on this patch series
(and it seems it still hasn't settled...). After reading the entire series this time (instead of commenting before I understood what was happening exactly), I have two overall comments still. More comments will follow in individual patches.

1. Peter nacked all previous attempts of having this type of feature, so I think it's best if he gives an indication if this series has a chance of survival before we all spend more time on it.

Adding Peter in Cc on this one. I know he is a bit behind in terms of reading the backlog of Buildroot e-mails (which has been enormous since a few days).

I have the pretension of thinking that the currently proposed implementation is simple enough to remain in the KISS spirit of Buildroot. I believe that what prevented earlier implementation from being merged was not so much that they allowed 'external packages', but more due to their complexity. But honestly, I don't really remember the implementation details of the previous proposals, so I might be wrong on that one.

Quoted 9 lines

2. I really have a huge problem with the fact that the .config is no longer complete, and that instead you have to explicitly provide extra command-line arguments to be able to reproduce the build. And it doesn't help that it behaves differently when you have an out-of-tree output directory (where the BR2_EXTERNAL is stored) than when you're using the default output directory. I'm actually very surprised that nobody else seems to have a problem with this. So for me, this gets a big Nack unless the BR2_EXTERNAL is stored in the .config.

On the wrapper Makefile that contains BR2_EXTERNAL, I am not convinced it is a good idea, since as you mention it creates a difference whether you're building out-of-tree from the output directory, or out-of-tree but from the source directory. I think this is not good, and therefore that BR2_EXTERNAL shouldn't be kept in the wrapper Makefile.

However, I still do not understand how storing BR2_EXTERNAL in .config will clarify things for the user. Changing BR2_EXTERNAL affects what 'menuconfig' shows. So there is no way for 'menuconfig' to update what it displays when the user changes the value of BR2_EXTERNAL without exiting/re-running menuconfig, which sounds really odd.

Quoted 6 lines

To me, BR2_EXTERNAL= is a bit like O=, it's a local build decision that you make to add some external configurations/external packages. Most likely, if you have external packages in BR2_EXTERNAL, the defconfig you have to use is also in BR2_EXTERNAL, so there's no way you can forget to specify BR2_EXTERNAL on the command, since otherwise you won't be able to even see your defconfig.

But you usually don't run "make foo_defconfig; make". You usually run
either "make menuconfig; make" or "make foo-dirclean; make" or some other variant.

I just think it's a really bad idea to split the buildroot
configuration over the .config file and the environment. The concept just scares the hell out of me.

The comparison with O= is a good one: O= makes pretty damn sure that
things remain consistent, because the .config is stored in the O= directory rather than someplace else.

Quoted 3 lines

So while I do agree that being able to store BR2_EXTERNAL would be nice, I don't see how it can be achieved in a way that isn't confusing for the user.

Since the user probably is never going to change the value of
BR2_EXTERNAL, I think that's a relatively small price to pay and that it can be solved with documentation.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#29 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

In reply to Thomas Petazzoni

Thomas Petazzoni <[email protected]> wrote on 09/16/2013 11:37:17 PM:

Quoted 3 lines

Dear Arnout Vandecappelle,

On Mon, 16 Sep 2013 22:38:29 +0200, Arnout Vandecappelle wrote:

[...]

Quoted 10 lines
Quoted 9 lines

2. I really have a huge problem with the fact that the .config is no longer complete, and that instead you have to explicitly provide extra command-line arguments to be able to reproduce the build. And it doesn't help that it behaves differently when you have an out-of-tree output directory (where the BR2_EXTERNAL is stored) than when you're using the default output directory. I'm actually very surprised that nobody else seems to have a problem with this. So for me, this gets a big Nack unless the BR2_EXTERNAL is stored in the .config.

Quoted 5 lines

On the wrapper Makefile that contains BR2_EXTERNAL, I am not convinced it is a good idea, since as you mention it creates a difference whether you're building out-of-tree from the output directory, or out-of-tree but from the source directory. I think this is not good, and therefore that BR2_EXTERNAL shouldn't be kept in the wrapper Makefile.

I do think that concerns those are valid concerns but from my use with BR2_EXTERNAL (with both v1 and v2) here is my take away with the experience. Note: I've been using this almost exclusively the last week since I've had to put together some new board configuration.

Having to make sure that BR2_EXTERNAL is defined when you call make gets very annoying quickly. I understand that I can just export BR2_EXTERNAL
to your environment to not have to declare BR2_EXTERNAL every time. So to make my life easier, I've did just that but have run into one major gotcha with it. Exporting BR2_EXTERNAL to the environment is just as unclear as being written to the wrapper makefile and if I come back to shell and forget to check what BR2_EXTERNAL is, I can perform a build with BR2_EXTERNAL that isn't correct for that board.

Well beta testing this latest patchset, I've found that putting it the wrapper makefile has allowed to not always have to keep putting BR2_EXTERNAL
in but also not have BR2_EXTERNAL seek into other builds/configurations. I know in my external build folder that I'm using X defconfig the Y BR2_EXTERNAL and I usually don't have any reason to change Y because that folder has been dedicated to a specific project that share common BR2_EXTERNAL.

Maybe with BR2_EXTERNAL, we can encourage the use of O= since to quote Thomas from below: "it's a local build decision"? I don't know what is best do here since the way BR2_EXTERNAL has been implement here I feel is the best way and since BR2_EXTERNAL is completely optional I don't know if encouraging the use O= is a bad thing.

Quoted 6 lines

However, I still do not understand how storing BR2_EXTERNAL in .config will clarify things for the user. Changing BR2_EXTERNAL affects what 'menuconfig' shows. So there is no way for 'menuconfig' to update what it displays when the user changes the value of BR2_EXTERNAL without exiting/re-running menuconfig, which sounds really odd.

Quoted 6 lines

To me, BR2_EXTERNAL= is a bit like O=, it's a local build decision that you make to add some external configurations/external packages. Most likely, if you have external packages in BR2_EXTERNAL, the defconfig you have to use is also in BR2_EXTERNAL, so there's no way you can forget to specify BR2_EXTERNAL on the command, since otherwise you won't be able to even see your defconfig.

I completely agree with this as while using this patchset, I've forgotten to specify BR2_EXTERNAL so I don't get see the defconfig I'm looking for. I guess is a subtle reminder that I haven't specified BR2_EXTERNAL.

My usual workflow while using this patchset is follows:

make BR2_EXTERNAL=/some/path help
[find the defconfig I want to build for] make BR2_EXTERNAL=/some/path O=/build/path defconfig cd /build/path
[do buildroot menuconfig stuff if needed]] make

Here is my workflow when doing a new BR2_EXTERNAL build that I've found useful

make BR2_EXTERNAL=/some/path O=/build/path allnoconfig cd /build/path
make menuconfig
[make my buildroot configuration selections]
make savedefconfig
[NOTE: in buildroot menuconfig the save defconfig option has set to $(BR2_EXTERNAL)/configs/some_defconfig]

Quoted 3 lines

So while I do agree that being able to store BR2_EXTERNAL would be nice, I don't see how it can be achieved in a way that isn't confusing for the user.

I agree too, maybe

Quoted 12 lines

Best regards,

Thomas

Signature · 8 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130917/b019f883/attachment-0001.html>

#30 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

In reply to Thomas Petazzoni

Thomas/Yann/All,

Today on the IRC channel there was a discussion about BR2_EXTERNAL and the remaining issues with the patch. Thomas Petazonni said that he is waiting for the following two things:

1. The opinion of Jacmet (Peter Korsgaard), the Buildroot maintainer

2. A consensus with Arnout on whether BR2_EXTERNAL should be stored or not in the .config

For #1 - Thomas P. said that he was confident that he could convince Peter K. of including BR2_EXTERNAL into buildroot.

For #2 - there was some dicussion on how to store BR2_EXTERNAL between Thomas P., Yann E MORIN and myself about how best to accomplish this.

The discussion started with why BR2_EXTERNAL can't be a part of the ".config" file. As Thomas P. has stated many times before, there exists a chicken-and-egg problem with BR2_EXTERNAL in that we can't store it in the .config because menuconfig needs BR2_EXTERNAL to display the external packages in the menuconfig and there is no way to pull BR2_EXTERNAL's definition into menuconfig if it is in the config file.

The discussion then progressed with myself suggesting that we could put a check to search if BR2_EXTERNAL is in the ".config" and then prompt a warning when BR2_EXTERNAL exists in the ".config" but isn't defined in the environment. However, I quickly realized that this wouldn't cover the case where packages have been added to the .config from BR2_EXTERNAL and are now no longer defined.

The discussion then continued with Thomas P. and myself talking about my suggestion of having the wrapper makefile (out-of-tree (OOT) builds) store the value of BR2_EXTERNAL. Thomas P. said that storing BR2_EXTERNAL in the wrapper makefile is not good because non-OOT builds don't work the same OOT builds. I then made the point that having to ensure BR2_EXTERNAL is defined everytime you call make is annoying and that defining in your shell environment is dangerous. This is when you are managing multiple builds (OOT builds) with different BR2_EXTERNALs and using same buildroot source.

Yann then asked me to the sum up what this issue was with in-tree builds and BR2_EXTERNAL. Thomas P. then responded with "you have to pass the BR2_EXTERNAL value on the command line (or environment) at each Buildroot command"

I then suggested that BR2_EXTERNAL have a pre-requisite of being used with only out-of-tree builds. The documentation for BR2_EXTERNAL should then be updated to reflect this requirement and say something about this requirement.

Yann saw two options moving forward with this issue - 1) always require BR2_EXTERNAL (in env or on command line) or 2) only support BR2_EXTERNAL in OOT, stored in the wrapper.

Yann put the discussion of BR2_EXTERNAL on the agenda for the DevDays coming up at the end of October:

http://elinux.org/Buildroot:DeveloperDaysELCE2013

Thomas P. suggested that maybe the wrapper makefile could be genreated in the default output directory (O=output). Which, if a user wanted to have the BR2_EXTERNAL remembered, they would have to change their working directory to the default output directory (output) to use the wrapper.

Finally, Thomas P. suggested that make BR2_EXTERNAL should be contained in a file. I responded with asking how it would work with having different BR2_EXTERNAL for different builds while using the same buildroot source. To this question there was no response and that is where things stand today.

To expand on Thomas P.'s idea of having a file, I'm starting to warm to this idea as we could have buildroot default to look for a BR2_EXTERNAL in the top-level of the buildroot source. If present, it would use the directory that is listed there. However, we could override the location of this file by specifying a variable fo the file in the wrapper makefile.

I hope I did a decent job capturing this discussion becuase I rushed to put this together before I left for the day.

Thomas Petazzoni <[email protected]> wrote on 09/14/2013 02:03:49 PM:

[Thomas P.'s cover page]

Thanks,
-Ryan

------------------------------------------------------------------------------------------
Ryan J Barnett / Software Engineer / Platform SW MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US rjbarnet at rockwellcollins.com www.rockwellcollins.com

#31 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

In reply to Thomas Petazzoni

Hello All!

On 2013-09-14 21:03 +0200, Thomas Petazzoni spake thusly:

Quoted 1 line

Here is the second version of the BR2_EXTERNAL patch series.

I was wondering what the plan with BR2_EXTERNAL was, in the end?

From my recollection of the last DevDay, I understood we would go with it, with a few changes:
- add a generated, sub-makefile in $(O) with the definition of BR2_EXTERNAL, so that it is not needed to repeat it on the command line of future invocations of 'make'
- enforce the same directory layout, package/ board/ and configs/

Right?

If so, Thomas, since your the original submitter: do you have pending changes to that effect, or are you waiting for someone to kick in and continue the work on this patch series?

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#32 lore raw

v2 cover Add a BR2_EXTERNAL mechanism

Dear Yann E. MORIN,

Quoted 12 lines · Yann E. MORIN

On 2013-09-14 21:03 +0200, Thomas Petazzoni spake thusly:

Quoted 1 line

Here is the second version of the BR2_EXTERNAL patch series.

I was wondering what the plan with BR2_EXTERNAL was, in the end?

From my recollection of the last DevDay, I understood we would go with it, with a few changes:
- add a generated, sub-makefile in $(O) with the definition of BR2_EXTERNAL, so that it is not needed to repeat it on the command line of future invocations of 'make'
- enforce the same directory layout, package/ board/ and configs/

Quoted 5 lines

Right?

If so, Thomas, since your the original submitter: do you have pending changes to that effect, or are you waiting for someone to kick in and continue the work on this patch series?

I don't have pending changes right now. But since Ryan is in Toulouse this week (and we've been doing some Buildroot hacking together), I think we'll try to spend one evening on the BR2_EXTERNAL stuff and come up with an updated proposal that takes into account the ELCE discussions.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com