Skip to contentSkip to conversation

br2-external: support multiple trees at once (branch yem/multi-br2-external-2)

RFC PATCH Superseded Yann E. MORIN 2 participants 24 messages May 26, 2015 – May 27, 2015 Kbuild

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover br2-external: support multiple trees at once (branch yem/multi-br2-external-2)

Hello All!

This rather complex series introduces support for using multiple br2-external trees at once.

** This is an RFC !! ** Not ready to commit !! ** Technology preview !! **

Currently, Buildroot can use what we call a br2-external tree, that is a location where one can store customisations outisde of the Buildroot tree:
- defconfig files
- packages
- company-, project- and board-related configurations files, like a kernel config file, an rootfs overlay...
- or any other content that does not belong to the Buildroot tree.

This is very nice and very handy!

However, it is not possible to use more than one such br2-external tree at a time; Buildroot only supports using at most one br2-external tree.

Having support for using more than one br2-external tree at the same time would allow more flexibility organising those out-of-tree customisations. One could use a br2-external tree to store FLOSS packages not yet in Buildroot and a second br2-external tree for proprietary packages; such a separation would make compliance with FLOSS licenses easier (by allowing distributing only the br2-external tree with FLOSS packages, keeping the proprietary bits private.

It could also be used to commonalise br2-external trees from different internal projects, where each would provide a set of extensions (new packages, new boards defconfigs...) that a project could cherry-pick.

And countless other possibilities.

This series is an attempt at making it possible to use such setups.

The series is organised as such:

- patches 1-6 are generic enhancements and cleanups,

- patches 7-12 introduce the possibility to generate a kconfig snippet, which includes all the Config.in files from all the br2-external trees, to eventually get rid of our dummy br2-external tree,

- patches 13-17 are preparatory work to support multi br2-external trees, like abstracting the br2-external location into a variable specific to each br2-external tree (even if still only one is supported), and hiding BR2_EXTERNAL so it is not mis-used,

- patch 18 is the core of the change, and thanks to all the preparatory work, is pretty light (51+, 30-).

- patches 19-21 are the final touch to the manual, allowing to generate a manual with the list of packages in br2-external trees, and eventually documenting the multi br2-external support (that last part is by far incomplete, as the commit log suggests), but is still included so reviewers get at least a peek at it.

The pivotal change in this series is implemented sequentially in patches 13-17, and aims at providing br2-external trees with a way to find their own files, given that their locations at runtime is completely unkown, and can not even be deduced relative to Buildroot's own TOPDIR:

Kconfig does not allow us to redefine a variable at different moment of the parsing, because the values are only valid and acted upon when the complete Kconfig code has been parsed. So, we need a way to provide Config.in files from each br2-external with their actual location. For that, we need different variables, one for each br2-external tree.

So, we need a way to uniquely identify each br2-external tree, so we need a stable scheme. We can't use indexes, we can't use path-based hashing (because paths are not stable accross users/machines/time), we can't use content-based hashing (external.mk and/or Config.in may all be empty, giving the same hash), we can't use hashes altogether because they are not human-friendly.

The solution is to require br2-external trees to provide a unique ID that is used as a kind of 'name-space' for that tree: we eventually provide a single variable, BR2_EXTERNAL_$(ID) that points to the location of that tree; because $(ID) is known by the external tree, it also knows BR2_XTERNAL_$(ID) exists, so it can use it, and we export it both in Config.in and external.mk .

Please have a look at the various commit logs, they are usually pretty copious in details (sometimes even much bigger than the actual change they are commenting!).

And that's about all for this time. ;-)

Again, this series is just an RFC, a kind of peek-in-the-possibilities, for reference and early commenting. Do not expect this to land any time soon, or in the current shape, or even at all.

Regards,
Yann E. MORIN.

The following changes since commit f2888955ed21c54a3a887d5757b35ef18ba5e669:

janus-gateway: disable for static build (2015-05-26 23:58:42 +0200)

are available in the git repository at:

git://git.busybox.net/~ymorin/git/buildroot yem/multi-br2-external-2

for you to fetch changes up to 93cebd3df59e0168d04f2d3557909ad387684cc3:

docs/manual: document multi br2-external (2015-05-27 00:57:37 +0200)

----------------------------------------------------------------
Yann E. MORIN (21):
core: commonalise the bundled and br2-external %_defconfig rules pkg-infra/pkg-generic: define PKGDIR for all rules docs/manual: slightly re-organise br2-external sections
core: remove .br-external on distclean
docs/manual: do not override BR2_EXTERNAL
doc/asciidoc: add possibility to define document dependencies
core: introduce an intermediate rule before the configurators
core: move rule to create basic directories
core: introduce a generated kconfig snippet
docs/manual: prepare-config can be used as a dependency of documents
core: do not hard-code inclusion of br2-external in Kconfig
core: get rid of our dummy br2-external tree
core: require br2-external trees define a unique ID docs/manual: document the br2-external ID
core: no longer export the raw BR2_EXTERNAL to users
core: handle .br-external in a make rule
core: move pkg-utils.mk to support/
core: add support for multiple br2-external trees support/scripts: teach gen-manual-lists about BR2_EXTERNAL docs/manual: include packages from BR2_EXTERNAL if set docs/manual: document multi br2-external

 Config.in                                     |  11 +-
 Makefile                                      | 167 ++++++++++++++++----------
 docs/manual/adding-packages-asciidoc.txt      |   5 +
 docs/manual/appendix.txt                      |   5 +
 docs/manual/customize-outside-br.txt          |  98 +++++++++------
 docs/manual/manual.mk                         |  24 +++-
 package/Makefile.in                           |   3 +-
 package/doc-asciidoc.mk                       |   2 +-
 package/pkg-generic.mk                        |   8 ++
 support/dummy-external/Config.in              |   0
 support/dummy-external/external.mk            |   0
 package/pkg-utils.mk => support/misc/utils.mk |  16 ++-
 support/scripts/gen-manual-lists.py           |  24 +++-
 13 files changed, 239 insertions(+), 124 deletions(-)
 delete mode 100644 support/dummy-external/Config.in
 delete mode 100644 support/dummy-external/external.mk
 rename package/pkg-utils.mk => support/misc/utils.mk (94%)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

RFC PATCH 1/21 core: commonalise the bundled and br2-external %_defconfig rules

The code for both cases is exactly the same, and only differs in the location where defconfig files are looked for.

We use an intermediate macro to generate the corresponding rules, because directly generating the rules is ugly and needs lots of escaping and double-dollar-ing for the $(eval ...) and $(foreach ...) calls to play nicely together.

Furthermore, that will be tremendously useful when we support multiple br2-external trees.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Makefile | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
Makefile +6 -7
index cf244d7..0038d11 100644
@@ -765,13 +765,12 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
 
 # Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/configs/$@ \
-		$< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
-
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
-		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
+define PERCENT_DEFCONFIG
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
+		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
+endef
+$(eval $(foreach d,$(TOPDIR) $(BR2_EXTERNAL),$(call PERCENT_DEFCONFIG,$(d))$(sep)))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
Signature · 1 line
1.9.1
#2 lore raw

RFC PATCH 2/21 pkg-infra/pkg-generic: define PKGDIR for all rules

In reply to Yann E. MORIN

When we eventually support more than one br2-external tree, packages won't be able to use $(BR2_EXTERNAL)/PACKAGE_PATH to reference files in the package's directory (the directory with the package's .mk file).

Define PKGDIR for all our rules, so packages can simply reference it to find their files.

This can also be used for our own packages, to avoid hard-coding the path to our own files as well.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 package/pkg-generic.mk | 8 ++++++++
 1 file changed, 8 insertions(+)
package/pkg-generic.mk +8 -0
index d5b29f0..2a5ad21 100644
@@ -614,17 +614,25 @@ $(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
+$$($(2)_TARGET_INSTALL_TARGET):		PKGDIR=$(pkgdir)
 $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
+$$($(2)_TARGET_INSTALL_STAGING):	PKGDIR=$(pkgdir)
 $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
+$$($(2)_TARGET_INSTALL_IMAGES):		PKGDIR=$(pkgdir)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
+$$($(2)_TARGET_INSTALL_HOST):           PKGDIR=$(pkgdir)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
+$$($(2)_TARGET_BUILD):			PKGDIR=$(pkgdir)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
+$$($(2)_TARGET_CONFIGURE):		PKGDIR=$(pkgdir)
 $$($(2)_TARGET_RSYNC):                  SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
 $$($(2)_TARGET_RSYNC):                  PKG=$(2)
+$$($(2)_TARGET_RSYNC):                  PKGDIR=$(pkgdir)
 $$($(2)_TARGET_PATCH):			PKG=$(2)
 $$($(2)_TARGET_PATCH):			RAWNAME=$$(patsubst host-%,%,$(1))
 $$($(2)_TARGET_PATCH):			PKGDIR=$(pkgdir)
 $$($(2)_TARGET_EXTRACT):		PKG=$(2)
+$$($(2)_TARGET_EXTRACT):		PKGDIR=$(pkgdir)
 $$($(2)_TARGET_SOURCE):			PKG=$(2)
 $$($(2)_TARGET_SOURCE):			PKGDIR=$(pkgdir)
 $$($(2)_TARGET_DIRCLEAN):		PKG=$(2)
Signature · 1 line
1.9.1
#3 lore raw

RFC PATCH 3/21 docs/manual: slightly re-organise br2-external sections

In reply to Yann E. MORIN

Move the defconfig section before the custom package recipes and before the custom configuration files.

It is the easiest and most simple way to use BR2_EXTERNAL, so it seems natural to make it first.

Cc Samuel Martin <[email protected]>
Cc Thomas Petazzoni <[email protected]>

---
Besides, when we introduce support for multiple br2-external trees, it will be the easiest section to update. ;-)

---
 docs/manual/customize-outside-br.txt | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
docs/manual/customize-outside-br.txt +6 -6
index 9ad177d..a3d6efd 100644
@@ -59,6 +59,12 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
 
 +BR2_EXTERNAL+ allows three different things:
 
+ * One can store Buildroot defconfigs in the +configs+ subdirectory of
+   +$(BR2_EXTERNAL)+. Buildroot will automatically show them in the
+   output of +make list-defconfigs+ and allow them to be loaded with the
+   normal +make <name>_defconfig+ command. They will be visible under the
+   +User-provided configs+' label in the 'make list-defconfigs' output.
+
  * One can store all the board-specific configuration files there,
    such as the kernel configuration, the root filesystem overlay, or
    any other configuration file for which Buildroot allows to set its
@@ -101,9 +107,3 @@ And then in +$(BR2_EXTERNAL)/package/package1+ and
    package recipes, as explained in xref:adding-packages[].
    If you prefer, you can also group the packages in subdirectories
    called <boardname> and adapt the above paths accordingly.
-
- * One can store Buildroot defconfigs in the +configs+ subdirectory of
-   +$(BR2_EXTERNAL)+. Buildroot will automatically show them in the
-   output of +make list-defconfigs+ and allow them to be loaded with the
-   normal +make <name>_defconfig+ command. They will be visible under the
-   +User-provided configs+' label in the 'make list-defconfigs' output.
Signature · 1 line
1.9.1
#4 lore raw

RFC PATCH 4/21 core: remove .br-external on distclean

In reply to Yann E. MORIN

distclean is supposed to return the current directory, whether in-tree or out-of-tree, into pristine conditions, which means we should also forget about any br2-external tree on distclean.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
index 0038d11..9901ec5 100644
@@ -814,7 +814,7 @@ ifeq ($(O),output)
 	rm -rf $(O)
 endif
 	rm -rf $(BR2_CONFIG) $(CONFIG_DIR)/.config.old $(CONFIG_DIR)/..config.tmp \
-		$(CONFIG_DIR)/.auto.deps
+		$(CONFIG_DIR)/.auto.deps $(BR2_EXTERNAL_FILE)
 
 help:
 	@echo 'Cleaning:'
Signature · 1 line
1.9.1
#5 lore raw

RFC PATCH 5/21 docs/manual: do not override BR2_EXTERNAL

In reply to Yann E. MORIN

Currently, we forcibly override BR2_EXTERNAL when building the manual, so as to avoid referring to packages therein from the manual.

However, when generating the lists of packages, we limit ourselves to scanning for packages in (Buildroot's) TOPDIR and never search in BR2_EXTERNAL. So, we do not really need to override BR2_EXTERNAL when generating the manual to achieve the same result.

Furthermore, we're only looking for packages that are defined in the sub-menu "Target packages", and never anywhere else, and especially not in the sub-menu "User-provided options".

Finally, we're soon to completely eliminate use of BR2_EXTERNAL in Kconfig altogether, so that would no longer have any impact anyway.

So, just leave BR2_EXTERNAL alone when generating the manual.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Samuel Martin <[email protected]>
---
 docs/manual/manual.mk | 3 ---
 1 file changed, 3 deletions(-)
docs/manual/manual.mk +0 -3
index ad9bd90..caf080a 100644
@@ -8,13 +8,10 @@ MANUAL_SOURCES = $(sort $(wildcard docs/manual/*.txt) $(wildcard docs/images/*))
 MANUAL_RESOURCES = $(TOPDIR)/docs/images
 
 # Our manual needs to generate lists
-# Packages included in BR2_EXTERNAL are not part of buildroot, so they
-# should not be included in the manual.
 define MANUAL_GEN_LISTS
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)$(COMMON_CONFIG_ENV) \
 		BR2_DEFCONFIG="" \
-		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
 		TOPDIR=$(TOPDIR) \
 		O=$(@D) \
 		python -B $(TOPDIR)/support/scripts/gen-manual-lists.py
Signature · 1 line
1.9.1
#6 lore raw

RFC PATCH 6/21 doc/asciidoc: add possibility to define document dependencies

In reply to Yann E. MORIN

Currently, a document can not have dependencies, except for the purely internal ones (like checking asciidoc version, and presence of dblatex).

For our own manual, this will come in handy when we introduce a generated kconfig snippet, so we can actually make the manual depend on that snippet being generated first.

For external documents, it can be used to depend on host-packages if need be (e.g. a custom host packages that generates specific media files included in the manual).

Cc Thomas Petazzoni <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 docs/manual/adding-packages-asciidoc.txt | 3 +++
 package/doc-asciidoc.mk                  | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
docs/manual/adding-packages-asciidoc.txt +3 -0
index 6e21786..0c10fd1 100644
@@ -63,6 +63,9 @@ information is (assuming the document name is +foo+) :
   to one or more directories containing so-called resources (like CSS or
   images). By default, empty.
 
+* +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
+  host-packages) that must be built before building this document.
+
 There are also additional hooks (see xref:hooks[] for general information
 on hooks), that a document may set to define extra actions to be done at
 various steps:
package/doc-asciidoc.mk +1 -1
index 6ab5ad2..ede6ed1 100644
@@ -140,7 +140,7 @@ endef
 ################################################################################
 define ASCIIDOC
 # Single line, because splitting a foreach is not easy...
-$(1)-check-dependencies: asciidoc-check-dependencies
+$(1)-check-dependencies: asciidoc-check-dependencies $$($(2)_DEPENDENCIES)
 	$$(Q)$$(foreach hook,$$($(2)_CHECK_DEPENDENCIES_HOOKS),$$(call $$(hook))$$(sep))
 
 # Single line, because splitting a foreach is not easy...
Signature · 1 line
1.9.1
#7 lore raw

RFC PATCH 7/21 core: introduce an intermediate rule before the configurators

In reply to Yann E. MORIN

Currently, all configurators depend on generating the out-of-tree Makefile wrapper.

In an upcoming patch, we'll need to also generate a kconfig fragment, so it will have to kick in before we run the configurators.

Introduce a new intermediate "prepare-kconfig" rule, so we can commonalise the dependencies of the configurators. Move the dependency on the Makefile wrapper to that new intermediate rule.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 Makefile | 36 +++++++++++++++++++-----------------
 1 file changed, 19 insertions(+), 17 deletions(-)
Makefile +19 -17
index 9901ec5..351341e 100644
@@ -678,6 +678,8 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
+prepare-kconfig: outputmakefile
+
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
 	PKG_CONFIG_PATH="$(HOST_PKG_CONFIG_PATH)" $(MAKE) CC="$(HOSTCC_NOCCACHE)" HOSTCC="$(HOSTCC_NOCCACHE)" \
@@ -696,19 +698,19 @@ COMMON_CONFIG_ENV = \
 	BR2_EXTERNAL=$(BR2_EXTERNAL) \
 	SKIP_LEGACY=
 
-xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
+xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-gconfig: $(BUILD_DIR)/buildroot-config/gconf outputmakefile
+gconfig: $(BUILD_DIR)/buildroot-config/gconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) srctree=$(TOPDIR) $< $(CONFIG_CONFIG_IN)
 
-menuconfig: $(BUILD_DIR)/buildroot-config/mconf outputmakefile
+menuconfig: $(BUILD_DIR)/buildroot-config/mconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-nconfig: $(BUILD_DIR)/buildroot-config/nconf outputmakefile
+nconfig: $(BUILD_DIR)/buildroot-config/nconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-config: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+config: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
 # For the config targets that automatically select options, we pass
@@ -716,22 +718,22 @@ config: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 # no values are set for the legacy options so a subsequent oldconfig
 # will query them. Therefore, run an additional olddefconfig.
 
-oldconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+oldconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< --oldconfig $(CONFIG_CONFIG_IN)
 
-randconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+randconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --randconfig $(CONFIG_CONFIG_IN)
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allyesconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allyesconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --allyesconfig $(CONFIG_CONFIG_IN)
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allnoconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allnoconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --allnoconfig $(CONFIG_CONFIG_IN)
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-randpackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+randpackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
 		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
@@ -739,7 +741,7 @@ randpackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@rm -f $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
 		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
@@ -747,7 +749,7 @@ allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@rm -f $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
 		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
@@ -755,24 +757,24 @@ allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@rm -f $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-silentoldconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+silentoldconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	$(COMMON_CONFIG_ENV) $< --silentoldconfig $(CONFIG_CONFIG_IN)
 
-olddefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+olddefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN)
 
-defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+defconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
 
 # Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
 define PERCENT_DEFCONFIG
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig prepare-kconfig
 	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
 		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
 endef
 $(eval $(foreach d,$(TOPDIR) $(BR2_EXTERNAL),$(call PERCENT_DEFCONFIG,$(d))$(sep)))
 
-savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+savedefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< \
 		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
 		$(CONFIG_CONFIG_IN)
Signature · 1 line
1.9.1
#8 lore raw

RFC PATCH 8/21 core: move rule to create basic directories

In reply to Yann E. MORIN

Some of those directories will be needed even during configuration, like BUILD_DIR, where we'll store the generated kconfig snippet.

So, move the rule to create them outside the HAVE_CONFIG block.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 Makefile | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
Makefile +9 -9
index 351341e..b2d8a5a 100644
@@ -193,6 +193,15 @@ LEGAL_LICENSES_TXT_HOST = $(LEGAL_INFO_DIR)/host-licenses.txt
 LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
 LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
 
+################################################################################
+#
+# staging and target directories do NOT list these as
+# dependencies anywhere else
+#
+################################################################################
+$(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
+	@mkdir -p $@
+
 BR2_CONFIG = $(CONFIG_DIR)/.config
 
 # Pull in the user's configuration file
@@ -421,15 +430,6 @@ world: target-post-image
 	legal-info legal-info-prepare legal-info-clean printvars help \
 	list-defconfigs target-finalize target-post-image source-check
 
-################################################################################
-#
-# staging and target directories do NOT list these as
-# dependencies anywhere else
-#
-################################################################################
-$(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
-	@mkdir -p $@
-
 # We make a symlink lib32->lib or lib64->lib as appropriate
 # MIPS64/n32 requires lib32 even though it's a 64-bit arch.
 ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)
Signature · 1 line
1.9.1
#9 lore raw

RFC PATCH 9/21 core: introduce a generated kconfig snippet

In reply to Yann E. MORIN

Add the infrastructure for adding generated kconfig snippet in the menuconfig.

For now, the kconfig snippet is generated empty, the recipe for filling it in will be introduced in sub-sequent patches.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Samuel Martin <[email protected]>
---
 Config.in             | 6 ++++++
 Makefile              | 7 ++++++-
 docs/manual/manual.mk | 3 +++
 3 files changed, 15 insertions(+), 1 deletion(-)
Config.in +6 -0
index 2b39d6a..7c2575e 100644
@@ -18,6 +18,10 @@ config BR2_EXTERNAL
 	string
 	option env="BR2_EXTERNAL"
 
+config BR2_BUILD_DIR
+	string
+	option env="BUILD_DIR"
+
 # Hidden boolean selected by packages in need of Java in order to build
 # (example: xbmc)
 config BR2_NEEDS_HOST_JAVA
@@ -637,3 +641,5 @@ menu "User-provided options"
 source "$BR2_EXTERNAL/Config.in"
 
 endmenu
+
+source "$BR2_BUILD_DIR/.br2-external.in"
Makefile +6 -1
index b2d8a5a..4ddef1b 100644
@@ -678,7 +678,7 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
-prepare-kconfig: outputmakefile
+prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
@@ -696,6 +696,7 @@ COMMON_CONFIG_ENV = \
 	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
 	BR2_CONFIG=$(BR2_CONFIG) \
 	BR2_EXTERNAL=$(BR2_EXTERNAL) \
+	BUILD_DIR=$(BUILD_DIR) \
 	SKIP_LEGACY=
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
@@ -795,6 +796,10 @@ ifeq ($(NEED_WRAPPER),y)
 	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
 endif
 
+.PHONY: $(BUILD_DIR)/.br2-external.in
+$(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
+	@touch $@
+
 # printvars prints all the variables currently defined in our Makefiles
 printvars:
 	@$(foreach V, \
docs/manual/manual.mk +3 -0
index caf080a..b26f912 100644
@@ -7,6 +7,9 @@
 MANUAL_SOURCES = $(sort $(wildcard docs/manual/*.txt) $(wildcard docs/images/*))
 MANUAL_RESOURCES = $(TOPDIR)/docs/images
 
+# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
+MANUAL_DEPENDENCIES += prepare-kconfig
+
 # Our manual needs to generate lists
 define MANUAL_GEN_LISTS
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
Signature · 1 line
1.9.1
#10 lore raw

RFC PATCH 10/21 docs/manual: prepare-config can be used as a dependency of documents

In reply to Yann E. MORIN
Cc Thomas Petazzoni <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 docs/manual/adding-packages-asciidoc.txt | 2 ++
 1 file changed, 2 insertions(+)
docs/manual/adding-packages-asciidoc.txt +2 -0
index 0c10fd1..a278d44 100644
@@ -65,6 +65,8 @@ information is (assuming the document name is +foo+) :
 
 * +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
   host-packages) that must be built before building this document.
+  If a hook of your document needs to access the _Kconfig_ structure,
+  you may add +prepare-kconfig+ to the list of dependencies.
 
 There are also additional hooks (see xref:hooks[] for general information
 on hooks), that a document may set to define extra actions to be done at
Signature · 1 line
1.9.1
#11 lore raw

RFC PATCH 10/21 docs/manual: prepare-config can be used as a dependency of documents

Hi Yann,

Quoted 13 lines · Yann E. MORIN
Cc Thomas Petazzoni <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 docs/manual/adding-packages-asciidoc.txt | 2 ++
 1 file changed, 2 insertions(+)
docs/manual/adding-packages-asciidoc.txt +0 -0
index 0c10fd1..a278d44 100644

docs/manual/adding-packages-asciidoc.txt information is (assuming the document name is +foo+) : · line 69

Quoted 5 lines
docs/manual/adding-packages-asciidoc.txt +2 -0
@@ -65,6 +65,8 @@ information is (assuming the document name is +foo+) :
 
 * +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
   host-packages) that must be built before building this document.
+  If a hook of your document needs to access the _Kconfig_ structure,
+  you may add +prepare-kconfig+ to the list of dependencies.

Why not move the 'doc/asciidoc: add possibility to define document dependencies' patch here, and merge it with this one?

baruch

Signature · 3 lines
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -
#12 lore raw

RFC PATCH 10/21 docs/manual: prepare-config can be used as a dependency of documents

Baruch, All,

On 2015-05-27 21:51 +0300, Baruch Siach spake thusly:

Quoted 15 lines · Yann E. MORIN
Quoted 14 lines
Cc Thomas Petazzoni <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 docs/manual/adding-packages-asciidoc.txt | 2 ++
 1 file changed, 2 insertions(+)
docs/manual/adding-packages-asciidoc.txt +0 -0
index 0c10fd1..a278d44 100644
@@ -65,6 +65,8 @@ information is (assuming the document name is +foo+) :
 
Quoted 7 lines

docs/manual/adding-packages-asciidoc.txt information is (assuming the document name is +foo+) : · line 69

Quoted 4 lines
docs/manual/adding-packages-asciidoc.txt +2 -0
 * +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
   host-packages) that must be built before building this document.
+  If a hook of your document needs to access the _Kconfig_ structure,
+  you may add +prepare-kconfig+ to the list of dependencies.

Why not move the 'doc/asciidoc: add possibility to define document dependencies' patch here, and merge it with this one?

Because those two changes are semantically separate; the first one still completly makes sense without this one.

That is, even if we do not accept support for multi br2-external trees, or don;t accept it as-is, adding the possibilities for documetns to define dependencies is still useful (IMHO).

That's why they are separate changes.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#13 lore raw

RFC PATCH 11/21 core: do not hard-code inclusion of br2-external in Kconfig

In reply to Yann E. MORIN

Move the inclusion of br2-external's Config.in to the generated kconfig snippet.

This will ultimately allow us to use more than one br2-external tree.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Config.in | 7 -------
 Makefile  | 8 +++++++-
 2 files changed, 7 insertions(+), 8 deletions(-)
Config.in +0 -7
index 7c2575e..491051e 100644
@@ -635,11 +635,4 @@ source "package/Config.in.host"
 
 source "Config.in.legacy"
 
-menu "User-provided options"
-	depends on BR2_EXTERNAL != "support/dummy-external"
-
-source "$BR2_EXTERNAL/Config.in"
-
-endmenu
-
 source "$BR2_BUILD_DIR/.br2-external.in"
Makefile +7 -1
index 4ddef1b..ad43972 100644
@@ -798,7 +798,13 @@ endif
 
 .PHONY: $(BUILD_DIR)/.br2-external.in
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
-	@touch $@
+	$(Q)( \
+		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
+		printf 'menu "User-provided options"\n'; \
+		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
+		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
+		printf 'endmenu\n'; \
+	) >$@
 
 # printvars prints all the variables currently defined in our Makefiles
 printvars:
Signature · 1 line
1.9.1
#14 lore raw

RFC PATCH 12/21 core: get rid of our dummy br2-external tree

In reply to Yann E. MORIN

Now that we generate a kconfig snippet, we can conditionally include the BR2_EXTERNAL's Config.in only when BR2_EXTERNAL is supplied by the user, which means our empty/dummy Config.in is no longer usefull to us.

As for external.mk, we can also include it only when BR2_EXTERNAL is supplied by the user, which means our empty/dummy external.mk is no longer of any use to us.

Ditch both of those files, and:

- only generate actual content in the Kconfig snippet when we actually do havea BR2_EXTERNAL provided by tje user (i.e. BR2_EXTERNAL is not empty);

- add a variable that contains the path to the external.mk provided by the user, or empty if none, and include the path set in that variable (make can 'include' nothing without any problem! ;-) )

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Makefile                           | 18 +++++++-----------
 support/dummy-external/Config.in   |  0
 support/dummy-external/external.mk |  0
 3 files changed, 7 insertions(+), 11 deletions(-)
 delete mode 100644 support/dummy-external/Config.in
 delete mode 100644 support/dummy-external/external.mk
Makefile +7 -11
index ad43972..c7b719a 100644
@@ -142,16 +142,11 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 # on the command line, therefore the file is re-created every time make is run.
 #
 # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
-# line), the .br-external file is removed and we point to
-# support/dummy-external. This makes sure we can unconditionally include the
-# Config.in and external.mk from the BR2_EXTERNAL directory. In this case,
-# override is necessary so the user can clear BR2_EXTERNAL from the command
-# line, but the dummy path is still used internally.
+# line), the .br-external file is removed.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
 ifeq ($(BR2_EXTERNAL),)
-  override BR2_EXTERNAL = support/dummy-external
   $(shell rm -f $(BR2_EXTERNAL_FILE))
 else
   _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
@@ -160,6 +155,7 @@ else
   endif
   override BR2_EXTERNAL := $(_BR2_EXTERNAL)
   $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
+  BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
 endif
 
 # To make sure that the environment variable overrides the .config option,
@@ -414,7 +410,8 @@ include linux/linux.mk
 include system/system.mk
 include fs/common.mk
 
-include $(BR2_EXTERNAL)/external.mk
+# Nothing to include if no BR2_EXTERNAL tree in use
+include $(BR2_EXTERNAL_MK)
 
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR)
@@ -798,13 +795,12 @@ endif
 
 .PHONY: $(BUILD_DIR)/.br2-external.in
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
-	$(Q)( \
+	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
 		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
-		printf 'menu "User-provided options"\n'; \
-		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
+		printf 'menu "User-provided options"\n\n'; \
 		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
 		printf 'endmenu\n'; \
-	) >$@
+	fi >$@
 
 # printvars prints all the variables currently defined in our Makefiles
 printvars:
support/dummy-external/Config.in deleted +0 -0
deleted file mode 100644
index e69de29..0000000
support/dummy-external/external.mk deleted +0 -0
deleted file mode 100644
index e69de29..0000000
Signature · 1 line
1.9.1
#15 lore raw

RFC PATCH 13/21 core: require br2-external trees define a unique ID

In reply to Yann E. MORIN

This unique ID is used to construct a BR2_EXTERNAL-specific variable, BR2_EXTERNAL_$(ID), which contains the path to the br2-external tree.

This variable is available both from Kconfig (set in the Kconfig snippet) and from the .mk files.

Also display the BR2_EXTERNAL_$(ID) and its path as a comment in the menuconfig.

This will ultimately allow us to support multiple br2-external trees at once, with that ID (and thus BR2_EXTERNAL_$(ID)) uniquely defining which br2-external tree is being used.

Note: since the variables in the Makefile and in Kconfig are named the same, the one we computed early on (lines 167, second hunk) will be overridden by the one in .config when we have it. Thus, even though they are set to the same raw value, the one from .config is quoted and, being included later in the Makefile, will take precedence, so we must un-quote it before we include the br2-external's makefile. That's unfortunate, but there is no easy way around that as we want the two variables to be named the same in Makefile and Kconfig (and we can't ask the user to un-quote that variable himself either), hence the little dirty trick on line 425 (third hunk).

---
 Makefile | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
Makefile +17 -1
index c7b719a..37e552d 100644
@@ -143,6 +143,8 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 #
 # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
 # line), the .br-external file is removed.
+#
+# Ensure the BR2_EXTERNAL tree defines its ID, and set its top-level path.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
@@ -155,6 +157,14 @@ else
   endif
   override BR2_EXTERNAL := $(_BR2_EXTERNAL)
   $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
+  ifeq ($(wildcard $(BR2_EXTERNAL)/external.id),)
+  $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' has no ID file 'external.id')
+  endif
+  BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
+  ifeq ($(BR2_EXTERNAL_ID),)
+  $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
+  endif
+  BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
   BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
 endif
 
@@ -410,6 +420,9 @@ include linux/linux.mk
 include system/system.mk
 include fs/common.mk
 
+# The BR2_EXTERNAL_$(ID) variable is also present in .config, so it is
+# quoted. We must unquote it before feeding it to the br2-external makefile.
+BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
 # Nothing to include if no BR2_EXTERNAL tree in use
 include $(BR2_EXTERNAL_MK)
 
@@ -798,7 +811,10 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
 		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
 		printf 'menu "User-provided options"\n\n'; \
-		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
+		printf 'comment "BR2_EXTERNAL_%s (%s)"\n\n' $(BR2_EXTERNAL_ID) $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
+		printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
+		printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
+		printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
 		printf 'endmenu\n'; \
 	fi >$@
 
Signature · 1 line
1.9.1
#16 lore raw

RFC PATCH 14/21 docs/manual: document the br2-external ID

In reply to Yann E. MORIN

Update the manual with the new external.id mandatory file.

Take the opportunity to add a section listing all mandatory files, Config.in, external.mk and the new external.id, instead of just hinting about them in the external package recipes section.

Change the example to use the ID-suffixed variable instead of the raw BR2_EXTERNAL variable, which won;t be useable once we introduce support for multiple br2-external trees, as it would then contain more that one path.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Samuel Martin <[email protected]>
---
 docs/manual/customize-outside-br.txt | 38 +++++++++++++++++++++++-------------
 1 file changed, 24 insertions(+), 14 deletions(-)
docs/manual/customize-outside-br.txt +24 -14
index a3d6efd..744666d 100644
@@ -57,7 +57,20 @@ Or disable the usage of external definitions:
 buildroot/ $ make BR2_EXTERNAL= xconfig
 -----
 
-+BR2_EXTERNAL+ allows three different things:
+A +BR2_EXTERNAL+ tree must contain at least those three files:
+
+ * +external.id+, which should contain the ID for the BR2_EXTERNAL tree.
+   That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
+   available in +Config.in+ and +external.mk+ (see below), so that ID
+   must be a valid make and Kconfig variable.
++
+Examples: +FOO+, +BAR_42+...
+
+ * +Config.in+ and +external.mk+ (see below), which can be used to define
+   package recipes, like for packages bundled in Buildroot itself, or
+   other custom configuration options. Those two files can be empty.
+
++BR2_EXTERNAL+ then allows three different things:
 
  * One can store Buildroot defconfigs in the +configs+ subdirectory of
    +$(BR2_EXTERNAL)+. Buildroot will automatically show them in the
@@ -68,14 +81,15 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
  * One can store all the board-specific configuration files there,
    such as the kernel configuration, the root filesystem overlay, or
    any other configuration file for which Buildroot allows to set its
-   location. The +BR2_EXTERNAL+ value is available within the
-   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
-   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
-   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
+   location. The location of the +BR2_EXTERNAL+ tree is available in the
+   Buildroot configuration using the +$(BR2_EXTERNAL_$(ID))+ variable. As
+   an example, (considering the ID +BAR_42+), one could set the
+   +BR2_ROOTFS_OVERLAY+ Buildroot option to
+   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/overlay/+ (to specify a root
    filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
    Buildroot option to
-   +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+ (to specify the
-   location of the kernel configuration file).
+   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/kernel.config+ (to specify
+   the location of the kernel configuration file).
 
  * One can store package recipes (i.e. +Config.in+ and
    +<packagename>.mk+), or even custom configuration options and make
@@ -83,23 +97,19 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
    make it appear in the top-level configuration menu, and includes
    +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
 +
-.Note
-Providing +Config.in+ and +external.mk+ is mandatory, but they can be
-   empty.
-+
 The main usage of this is to store package recipes. The recommended
    way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
    looks like:
 +
 ------
-source "$BR2_EXTERNAL/package/package1/Config.in"
-source "$BR2_EXTERNAL/package/package2/Config.in"
+source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
+source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
 ------
 +
 Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:
 +
 ------
-include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
+include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
 ------
 +
 And then in +$(BR2_EXTERNAL)/package/package1+ and
Signature · 1 line
1.9.1
#17 lore raw

RFC PATCH 15/21 core: no longer export the raw BR2_EXTERNAL to users

In reply to Yann E. MORIN

When we support multiple br2-external trees, BR2_EXTERNAL will contain the paths to more than one directory, so it will not be possible to use it to refer to files in those br2-external trees.

However, a br2-external tree should use the newly-introduced BR2_EXTERNAL_ID variable, that is specific and unique to the corresponding br2-external tree.

So, do not export the raw BR2_EXTERNAL variable to br2-external trees, neither for consumption in external.mk nor in Config.in, by overriding it to an empty string.

We rename non-user-facing variables to be prefixed with BR_ instead of BR2_, and only keep the user-facing variables prefixed with BR2_ ('users' being files in a br2-external tree):

- BR2_EXTERNAL gets renamed for itnernal use to BR_EXTERNAL; however, we still save it as BR2_EXTERNAL in the cache file .br_external, so we can re-use it on future invocations;

- BR2_EXTERNAL_ID is renamed to BR_EXTERNAL_ID because it should not

    be exposed to users;

- BR2_EXTERNAL_MK is also renamed, to BR_EXTERNAL_MK, as it too is an internal variable not expoed to users.

The underlying reason is that BR2_EXTERNAL, as well as a few other BR2_-prefixed variables, will no longer be meaningful when we support multiple br2-external trees.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Config.in           |  4 ----
 Makefile            | 43 ++++++++++++++++++++++---------------------
 package/Makefile.in |  2 +-
 3 files changed, 23 insertions(+), 26 deletions(-)
Config.in +0 -4
index 491051e..5950da3 100644
@@ -14,10 +14,6 @@ config BR2_HOSTARCH
 	string
 	option env="HOSTARCH"
 
-config BR2_EXTERNAL
-	string
-	option env="BR2_EXTERNAL"
-
 config BR2_BUILD_DIR
 	string
 	option env="BUILD_DIR"
Makefile +22 -21

97 lines not on this page · raw message

package/Makefile.in +1 -1
index a0c5a7a..2dfc25f 100644
@@ -323,7 +323,7 @@ EXTRA_ENV = \
 	PATH=$(BR_PATH) \
 	BR2_DL_DIR=$(BR2_DL_DIR) \
 	BUILD_DIR=$(BUILD_DIR) \
-	BR2_EXTERNAL=$(BR2_EXTERNAL)
+	BR2_EXTERNAL=$(BR_EXTERNAL)
 
 ################################################################################
 # settings we need to pass to configure
Signature · 1 line
1.9.1
#18 lore raw

RFC PATCH 16/21 core: handle .br-external in a make rule

In reply to Yann E. MORIN

Currently, we treat the case where we have no br2-external tree (BR2_EXTERNAL is empty) differently from the case we do have one (BR2_EXTERNAL is not empty).

There is now no reason to treat those two cases differently:

- the kconfig snippet is always generated appropriately (i.e. it would

    include the br2-external tree if set, or include nothing otherwise);

- BR2_EXTERNAL (and its internal variant BR_EXTERNAL) is no longer

     useable by user;

- we no longer have a dummy br-external tree either.

As such, generate the .br-external cache file in both cases.

This will make it much easy to handle when we introduce support for multiple br2-external trees.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Makefile | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
Makefile +6 -7
index 1042192..b3f515b 100644
@@ -141,21 +141,15 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 # On subsequent invocations of make, it is read in. It can still be overridden
 # on the command line, therefore the file is re-created every time make is run.
 #
-# When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
-# line), the .br-external file is removed.
-#
 # Ensure the BR2_EXTERNAL tree defines its ID, and set its top-level path.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
-ifeq ($(BR2_EXTERNAL),)
-  $(shell rm -f $(BR2_EXTERNAL_FILE))
-else
+ifneq ($(BR2_EXTERNAL),)
   BR_EXTERNAL := $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
   ifeq ($(BR_EXTERNAL),)
     $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
   endif
-  $(shell echo BR2_EXTERNAL ?= $(BR_EXTERNAL) > $(BR2_EXTERNAL_FILE))
   ifeq ($(wildcard $(BR_EXTERNAL)/external.id),)
   $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' has no ID file 'external.id')
   endif
@@ -167,6 +161,11 @@ else
   BR_EXTERNAL_MK = $(BR2_EXTERNAL_$(BR_EXTERNAL_ID))/external.mk
 endif
 
+# This needs to be *after* we compute BR_EXTERNAL, above.
+.PHONY: $(BR2_EXTERNAL_FILE)
+$(BR2_EXTERNAL_FILE):
+	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
+
 # Don't let users use that variable, they must use the ID-suffixed variant
 override BR2_EXTERNAL :=
 
Signature · 1 line
1.9.1
#19 lore raw

RFC PATCH 17/21 core: move pkg-utils.mk to support/

In reply to Yann E. MORIN

pkg-utils.mk contains various definitions that are used in the package infrastructures and packages themselves.

However, those definitions can be useful in other parts of Buildroot, and are already used in a few places that are not related to the package infrastructure. Also, $(sep) will be needed early in the Makefile when we eventually support multiple br2-external trees.

Since this file only contains definitions, we can include it anytime.

So, consider that file to no longer be specific to the package infras:
- move it to support and rename it,
- move a few similar definitions from the main Makefile to that file.

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 Makefile                                      | 11 ++---------
 package/pkg-utils.mk => support/misc/utils.mk | 16 ++++++++++++++--
 2 files changed, 16 insertions(+), 11 deletions(-)
 rename package/pkg-utils.mk => support/misc/utils.mk (94%)
Makefile +2 -9
index b3f515b..ca87e57 100644
@@ -95,14 +95,8 @@ else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
 BR_BUILDING = y
 endif
 
-# Strip quotes and then whitespaces
-qstrip = $(strip $(subst ",,$(1)))
-#"))
-
-# Variables for use in Make constructs
-comma := ,
-empty :=
-space := $(empty) $(empty)
+# Include some helper macros and variables
+include support/misc/utils.mk
 
 ifneq ("$(origin O)", "command line")
 O := output
@@ -305,7 +299,6 @@ unexport DESTDIR
 # Causes breakage with packages that needs host-ruby
 unexport RUBYOPT
 
-include package/pkg-utils.mk
 include package/doc-asciidoc.mk
 
 ifeq ($(BR2_HAVE_DOT_CONFIG),y)
support/misc/utils.mk renamed from package/pkg-utils.mk +14 -2
similarity index 94%
rename from package/pkg-utils.mk
rename to support/misc/utils.mk
index 99a3c1e..1ed0a01 100644
@@ -1,10 +1,22 @@
 ################################################################################
 #
-# This file contains various utility functions used by the package
-# infrastructure, or by the packages themselves.
+# This file contains various utility macros and variables used about
+# everywhere in make constructs:
+#  - the core of Buildroot
+#  - package infrastructures
+#  - packages themselves
 #
 ################################################################################
 
+# Strip quotes and then whitespaces
+qstrip = $(strip $(subst ",,$(1)))
+#"))
+
+# Variables for use in Make constructs
+comma := ,
+empty :=
+space := $(empty) $(empty)
+
 # Case conversion macros. This is inspired by the 'up' macro from gmsl
 # (http://gmsl.sf.net). It is optimised very heavily because these macros
 # are used a lot. It is about 5 times faster than forking a shell and tr.
Signature · 1 line
1.9.1
#20 lore raw

RFC PATCH 18/21 core: add support for multiple br2-external trees

In reply to Yann E. MORIN

Being able to use more than one br2-external tree can be very useful.

The main use-case is to be able to add FLOSS packages in a br2-external tree, and proprietary packages in another. This allows to not touch the Buildroot tree at all, and still be able to get in compliance by providing only that br2-external tree(s) that contains FLOSS packages, leaving aside the br2-external tree(s) with the proprietary bits.

Of course, upstreaming those packages is the best course ultimately, but it is sometimes not possible to get those packages upstreamed quickly enough and keep up with the project's planning.

Another use-case would be for third-parties to provide their own Buildroot packaging in a br2-external tree, along-side the archives for their stuff.

What we do is to treat BR2_EXTERNAL as a space-separated list of paths, which we iterate to construct:

- the list of all br2-external IDs, BR2_EXTERNAL_IDS

- the per-br2-external tree BR2_EXTERNAL_$(ID) variables, which points to the actual location of the corresponding tree,

- the internal BR_EXTERNAL variable containing the paths made canonical (e.g not relative),

- EXTRA_ENV which now needs to contain all BR2_EXTERNAL_$(ID) variables is thus set early.

Once we have all those variables, we replace references to BR_EXTERNAL with either;

- a $(patsubst ...) to include the external.mk files (and docs), or

- a $(foreach ...) iteration to generate other Makefile code.

To be noted: we also use $(foreach ...) in the kconfig-snippet generating rule, because it is cleaner than doing so in the shell

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Makefile            | 78 ++++++++++++++++++++++++++++++++++-------------------
 package/Makefile.in |  3 +--
 2 files changed, 51 insertions(+), 30 deletions(-)
Makefile +50 -28
index ca87e57..2d01110 100644
@@ -134,26 +134,46 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 # The value of BR2_EXTERNAL is stored in .br-external in the output directory.
 # On subsequent invocations of make, it is read in. It can still be overridden
 # on the command line, therefore the file is re-created every time make is run.
-#
-# Ensure the BR2_EXTERNAL tree defines its ID, and set its top-level path.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
-ifneq ($(BR2_EXTERNAL),)
-  BR_EXTERNAL := $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
-  ifeq ($(BR_EXTERNAL),)
-    $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
-  endif
-  ifeq ($(wildcard $(BR_EXTERNAL)/external.id),)
-  $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' has no ID file 'external.id')
-  endif
-  BR_EXTERNAL_ID := $(shell cat $(BR_EXTERNAL)/external.id 2>/dev/null)
-  ifeq ($(BR_EXTERNAL_ID),)
-  $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
-  endif
-  BR2_EXTERNAL_$(BR_EXTERNAL_ID) = $(BR_EXTERNAL)
-  BR_EXTERNAL_MK = $(BR2_EXTERNAL_$(BR_EXTERNAL_ID))/external.mk
+
+# Those three variables need to be defined as simply-expanded variables, they
+# can't be recursively-expanded, because the values they are assigned change
+# with each iteration of the foreach, below.
+BR_EXTERNAL_IDS :=
+BR_EXTERNAL :=
+EXTRA_ENV :=
+
+# Validate the br2-external tree passed as $(1):
+# - check the directory actually exists
+# - check there is a non-empty ID file
+# - check the ID is not a duplicate
+# - set variables for later use
+define BR2_EXTERNAL_VALIDATE
+_BR_EXT_DIR := $$(shell cd $(1) >/dev/null 2>&1 && pwd)
+ifeq ($$(_BR_EXT_DIR),)
+$$(error BR2_EXTERNAL='$(1)' does not exist, relative to $$(TOPDIR))
+endif
+
+ifeq ($$(wildcard $$(_BR_EXT_DIR)/external.id),)
+$$(error BR2_EXTERNAL='$(1)' has no ID file 'external.id')
 endif
+_BR_EXT_ID := $$(shell cat $$(_BR_EXT_DIR)/external.id 2>/dev/null)
+ifeq ($$(_BR_EXT_ID),)
+$$(error BR2_EXTERNAL='$(1)' has no ID (in file 'external.id'))
+endif
+ifneq ($$(filter $$(_BR_EXT_ID),$$(BR_EXTERNAL_IDS)),)
+$$(error Duplicate ID '$$(_BR_EXT_ID)' in '$(1)', previously defined in '$$(BR2_EXTERNAL_$$(_BR_EXT_ID))')
+endif
+
+BR2_EXTERNAL_$$(_BR_EXT_ID) := $$(_BR_EXT_DIR)
+BR_EXTERNAL_IDS += $$(_BR_EXT_ID)
+BR_EXTERNAL += $$(_BR_EXT_DIR)
+EXTRA_ENV += BR2_EXTERNAL_$$(_BR_EXT_ID)=$$(_BR_EXT_DIR)
+endef
+
+$(eval $(foreach d,$(BR2_EXTERNAL),$(call BR2_EXTERNAL_VALIDATE,$(d))$(sep)))
 
 # This needs to be *after* we compute BR_EXTERNAL, above.
 .PHONY: $(BR2_EXTERNAL_FILE)
@@ -414,11 +434,11 @@ include linux/linux.mk
 include system/system.mk
 include fs/common.mk
 
-# The BR2_EXTERNAL_$(ID) variable is also present in .config, so it is
-# quoted. We must unquote it before feeding it to the br2-external makefile.
-BR2_EXTERNAL_$(BR_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR_EXTERNAL_ID)))
-# Nothing to include if no BR2_EXTERNAL tree in use
-include $(BR_EXTERNAL_MK)
+# The BR2_EXTERNAL_$(ID) variables are also present in .config, so they
+# are quoted. We must unquote them before feeding them to the br2-external
+# makefiles.
+$(eval $(foreach id,$(BR_EXTERNAL_IDS),BR2_EXTERNAL_$(id) := $(call qstrip,$(BR2_EXTERNAL_$(id)))$(sep)))
+include $(patsubst %,%/external.mk,$(BR_EXTERNAL))
 
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR)
@@ -804,10 +824,12 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)if [ -n '$(BR_EXTERNAL)' ]; then \
 		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
 		printf 'menu "User-provided options"\n\n'; \
-		printf 'comment "BR2_EXTERNAL_%s (%s)"\n\n' $(BR_EXTERNAL_ID) $(BR2_EXTERNAL_$(BR_EXTERNAL_ID)); \
-		printf 'config BR2_EXTERNAL_%s\n' $(BR_EXTERNAL_ID); \
-		printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR_EXTERNAL_ID)); \
-		printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR_EXTERNAL_ID); \
+		$(foreach id,$(BR_EXTERNAL_IDS),\
+			for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
+			printf 'comment "BR2_EXTERNAL_%s (%s)"\n\n' $(id) $(BR2_EXTERNAL_$(id)); \
+			printf 'config BR2_EXTERNAL_%s\n' $(id); \
+			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
+			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(id);) \
 		printf 'endmenu\n'; \
 	fi >$@
 
@@ -919,10 +941,10 @@ list-defconfigs:
 	@echo 'Built-in configs:'
 	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
-ifneq ($(wildcard $(BR_EXTERNAL)/configs/*_defconfig),)
+ifneq ($(wildcard $(patsubst %,%/configs/*_defconfig,$(BR_EXTERNAL))),)
 	@echo
 	@echo 'User-provided configs:'
-	@$(foreach b, $(sort $(notdir $(wildcard $(BR_EXTERNAL)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(wildcard $(patsubst %,%/configs/*_defconfig,$(BR_EXTERNAL))))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 endif
 	@echo
@@ -943,6 +965,6 @@ print-version:
 	@echo $(BR2_VERSION_FULL)
 
 include docs/manual/manual.mk
--include $(BR_EXTERNAL)/docs/*/*.mk
+-include $(patsubst %,%/docs/*/*.mk,$(BR_EXTERNAL))
 
 .PHONY: $(noconfig_targets)
package/Makefile.in +1 -2
index 2dfc25f..5e65dd9 100644
@@ -319,11 +319,10 @@ HOST_MAKE_ENV = \
 # packages use that variable internally, eg. uboot), so we have to
 # explicitly pass it to user-supplied external hooks (eg. post-build,
 # post-images)
-EXTRA_ENV = \
+EXTRA_ENV += \
 	PATH=$(BR_PATH) \
 	BR2_DL_DIR=$(BR2_DL_DIR) \
 	BUILD_DIR=$(BUILD_DIR) \
-	BR2_EXTERNAL=$(BR_EXTERNAL)
 
 ################################################################################
 # settings we need to pass to configure
Signature · 1 line
1.9.1
#21 lore raw

RFC PATCH 19/21 support/scripts: teach gen-manual-lists about BR2_EXTERNAL

In reply to Yann E. MORIN

When using BR2_EXTERNAL, generate the list of packages from there.

This is currently effectively unused for now, as BR2_EXTERNAL is not available (not exported in the environment); that will be updated in a later patch.

Cc Samuel Martin <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 support/scripts/gen-manual-lists.py | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)
support/scripts/gen-manual-lists.py +19 -5
index ec59030..7d9356c 100644
@@ -146,7 +146,6 @@ class Buildroot:
 
     """
     root_config = "Config.in"
-    package_dirname = "package"
     package_prefixes = ["BR2_PACKAGE_", "BR2_PACKAGE_HOST_"]
     re_pkg_prefix = re.compile(r"^(" + "|".join(package_prefixes) + ").*")
     deprecated_symbol = "BR2_DEPRECATED"
@@ -187,12 +186,22 @@ class Buildroot:
             'format': "_format_symbol_prompt_location",
             'sorted': False,
         },
+        'external': {
+            'filename': "br2-external-package-list",
+            'root_menu': "User-provided options",
+            'filter': "_is_real_package",
+            'format': "_format_symbol_prompt_location",
+            'sorted': True,
+        },
     }
 
     def __init__(self):
         self.base_dir = os.environ.get("TOPDIR")
         self.output_dir = os.environ.get("O")
-        self.package_dir = os.path.join(self.base_dir, self.package_dirname)
+        self.package_dirs = []
+        self.package_dirs.append(os.path.join(self.base_dir, "package"))
+        if os.getenv('BR2_EXTERNAL'):
+            self.package_dirs.extend(os.environ["BR2_EXTERNAL"].split())
         # The kconfiglib requires an environment variable named "srctree" to
         # load the configuration, so set it.
         os.environ.update({'srctree': self.base_dir})
@@ -266,9 +275,10 @@ class Buildroot:
         #   symbol.
         if not hasattr(self, "_package_list"):
             pkg_list = []
-            for _, _, files in os.walk(self.package_dir):
-                for file_ in (f for f in files if f.endswith(".mk")):
-                    pkg_list.append(re.sub(r"(.*?)\.mk", r"\1", file_))
+            for d in self.package_dirs:
+                for _, _, files in os.walk(d):
+                    for file_ in (f for f in files if f.endswith(".mk")):
+                        pkg_list.append(re.sub(r"(.*?)\.mk", r"\1", file_))
             setattr(self, "_package_list", pkg_list)
         for pkg in getattr(self, "_package_list"):
             if type == 'real':
@@ -495,6 +505,8 @@ class Buildroot:
 
 if __name__ == '__main__':
     list_types = ['target-packages', 'host-packages', 'virtual-packages', 'deprecated']
+    if os.getenv('BR2_EXTERNAL'):
+        list_types.append('external')
     parser = ArgumentParser()
     parser.add_argument("list_type", nargs="?", choices=list_types,
                         help="""\
@@ -509,6 +521,8 @@ Generate the given list (generate all lists if unspecified)""")
                         help="Output virtual package file")
     parser.add_argument("--output-deprecated", dest="output_deprecated",
                         help="Output deprecated file")
+    parser.add_argument("--output-external", dest="output_external",
+                        help="Output br2-external file")
     args = parser.parse_args()
     lists = [args.list_type] if args.list_type else list_types
     buildroot = Buildroot()
Signature · 1 line
1.9.1
#22 lore raw

RFC PATCH 20/21 docs/manual: include packages from BR2_EXTERNAL if set

In reply to Yann E. MORIN

If the user is using one or more br2-external trees, add the list of packages therein to the manual.

Note that we do re-introduce BR2_EXTERNAL (when it was removed in [-SHA1-] (docs/manual: do not override BR2_EXTERNAL). The reason for the removal is that BR2_EXTERNAL is no longer used as a Kconfig option.

However, BR2_EXTERNAL is now used by the gen-manual-lists.py script, which needs it to find packages' .mk files in the br2-external trees, not as a Kconfig option (which it is not).

So, even though that looks like a revert, that's not: the purpose is different, and we only pass it when not empty.

In case we have no br2-external, we want to create an empty file, so we can't just touch it, in case it was previously generated with br2-external trees; instead, we printf an empty string in there to ensure the file exists and is empty.

Cc Samuel Martin <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 docs/manual/appendix.txt |  5 +++++
 docs/manual/manual.mk    | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+)
docs/manual/appendix.txt +5 -0
index 87a20bd..7ef8bbc 100644
@@ -40,3 +40,8 @@ year following deprecation. For example, a symbol depending on
 +BR2_DEPRECATED_SINCE_2013_05+ can be removed from 2014.05 onwards.
 
 include::deprecated-list.txt[]
+
+// This file is generated:
+// - with the list of br2-external packages if BR2_EXTERNAL is set
+// - empty otherwise
+include::br2-external.txt[]
docs/manual/manual.mk +18 -0
index b26f912..3b4cef4 100644
@@ -10,14 +10,32 @@ MANUAL_RESOURCES = $(TOPDIR)/docs/images
 # Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
 MANUAL_DEPENDENCIES += prepare-kconfig
 
+# If using one or more BR2_EXTERNAL trees, include the list of
+# packages therein, otherwise, generate an empty file
+ifeq ($(BR_EXTERNAL),)
+define MANUAL_GEN_LISTS_BR2_EXTERNAL
+	$(Q)printf "" >$(@D)/br2-external.txt
+endef
+else
+define MANUAL_GEN_LISTS_BR2_EXTERNAL
+	$(Q)( \
+		printf "== List of BR2_EXTERNAL packages\n\n"; \
+		printf "include::br2-external-package-list.txt[]\n"; \
+	) >$(@D)/br2-external.txt
+endef
+MANUAL_GEN_LISTS_ENV = BR2_EXTERNAL="$(BR_EXTERNAL)"
+endif
+
 # Our manual needs to generate lists
 define MANUAL_GEN_LISTS
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)$(COMMON_CONFIG_ENV) \
 		BR2_DEFCONFIG="" \
+		$(MANUAL_GEN_LISTS_ENV) \
 		TOPDIR=$(TOPDIR) \
 		O=$(@D) \
 		python -B $(TOPDIR)/support/scripts/gen-manual-lists.py
+	$(MANUAL_GEN_LISTS_BR2_EXTERNAL)
 endef
 MANUAL_POST_RSYNC_HOOKS += MANUAL_GEN_LISTS
 
Signature · 1 line
1.9.1
#23 lore raw

RFC PATCH 21/21 docs/manual: document multi br2-external

In reply to Yann E. MORIN

[TO BE COMPLETED AND FIXED]

---
 docs/manual/customize-outside-br.txt | 50 +++++++++++++++++++++---------------
 1 file changed, 30 insertions(+), 20 deletions(-)
docs/manual/customize-outside-br.txt +30 -20
index 744666d..e188131 100644
@@ -17,19 +17,18 @@ place project-specific customizations in two locations:
    having them nicely integrated in the build logic. This section
    explains how to use +BR2_EXTERNAL+.
 
-+BR2_EXTERNAL+ is an environment variable that can be used to point to
-a directory that contains Buildroot customizations. It can be passed
-to any Buildroot +make+ invocation. It is automatically saved in the
-hidden +.br-external+ file in the output directory. Thanks to this,
-there is no need to pass +BR2_EXTERNAL+ at every +make+ invocation. It
-can however be changed at any time by passing a new value, and can be
++BR2_EXTERNAL+ is an environment variable that can be set to contain the
+paths to one or more directories that contains Buildroot customizations.
+It can be passed to any Buildroot +make+ invocation. It is automatically
+saved in the hidden +.br-external+ file in the output directory. Thanks
+to this, there is no need to pass +BR2_EXTERNAL+ at every +make+ invocation.
+It can however be changed at any time by passing a new value, and can be
 removed by passing an empty value.
 
 .Note
-The +BR2_EXTERNAL+ path can be either an absolute or a relative path,
-but if it's passed as a relative path, it is important to note that it
-is interpreted relative to the main Buildroot source directory, *not*
-to the Buildroot output directory.
+Paths in +BR2_EXTERNAL+ can be either absolute or relative paths, but it
+is important to note that relative paths are interpreted relative to the
+main Buildroot source directory, *not* to the Buildroot output directory.
 
 Some examples:
 
@@ -51,6 +50,12 @@ We can switch to another external definitions directory at any time:
 buildroot/ $ make BR2_EXTERNAL=/where/we/have/barfoo xconfig
 -----
 
+We can also use multiple +BR2_EXTERNAL+ locations:
+
+----
+buildroot/ $ make BR2_EXTERNAL="/path/to/foobar /where/we/have/barfoo" menuconfig
+----
+
 Or disable the usage of external definitions:
 
 -----
@@ -73,10 +78,15 @@ Examples: +FOO+, +BAR_42+...
 +BR2_EXTERNAL+ then allows three different things:
 
  * One can store Buildroot defconfigs in the +configs+ subdirectory of
-   +$(BR2_EXTERNAL)+. Buildroot will automatically show them in the
+   a +BR2_EXTERNAL+ tree. Buildroot will automatically show them in the
    output of +make list-defconfigs+ and allow them to be loaded with the
    normal +make <name>_defconfig+ command. They will be visible under the
-   +User-provided configs+' label in the 'make list-defconfigs' output.
+   +User-provided configs+ label in the 'make list-defconfigs' output.
++
+.Note
+If a defconfig file is present in more than one +BR2_EXTERNAL+ location,
+   then the last one is used. It is also possible to override a defconfig
+   bundled in Buildroot.
 
  * One can store all the board-specific configuration files there,
    such as the kernel configuration, the root filesystem overlay, or
@@ -91,22 +101,22 @@ Examples: +FOO+, +BAR_42+...
    +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/kernel.config+ (to specify
    the location of the kernel configuration file).
 
- * One can store package recipes (i.e. +Config.in+ and
-   +<packagename>.mk+), or even custom configuration options and make
-   logic. Buildroot automatically includes +$(BR2_EXTERNAL)/Config.in+ to
-   make it appear in the top-level configuration menu, and includes
-   +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
+ * One can store package recipes (i.e. +Config.in+ and +<packagename>.mk+),
+   or even custom configuration options and make logic. Buildroot
+   automatically includes the +Config.in+ from each paths in +BR2_EXTERNAL+
+   to make it appear in a sub-menu labelled _User-provided options_ at the
+   root of the main configuration menu, and includes the +external.mk+ from
+   each paths in +BR2_EXTERNAL+ with the rest of the makefile logic.
 +
 The main usage of this is to store package recipes. The recommended
-   way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
-   looks like:
+   way to do this is to write +Config.in+ files that look like:
 +
 ------
 source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
 source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
 ------
 +
-Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:
+Then, write +/external.mk+ files that look like:
 +
 ------
 include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
Signature · 1 line
1.9.1