Skip to contentSkip to conversation

Makefile: Support merged and nested defconfigs.

PATCH New Sam Bobroff 7 participants 18 messages Jun 29, 2016 – Jul 6, 2016 Kbuild

lore
Share & tools
lore
#0 lore raw

v3 cover Makefile: Support merged and nested defconfigs.

This is a combination and cleanup of the previous efforts by Sam Bobroff and Elizabeth Liner to introduce merged defconfigs and nested defconfig directories respectively.

Merged defconfigs allow us to create defconfig snippets for aspects that are common across multiple machines and create a combined defconfig from the snippets. The support scripts for merged kernel defconfigs is reused.

Nested defconfig directories allow us to organize the defconfig directory into sub-directories. A likely use for this is to place defconfigs into architecture-specific subdirectories.

Changes v2 -> v3:

* Added missing Signed-off-bys.
* Split documentation into separate patch.

Patch 4/6: Makefile: Add nested config dirs to list-defconfigs

* Removed unnecessary "/" ("wildcard" adds a trailing "/" to directories).

Changes v1 -> v2:

This is a complete rework of the config merging system, discarding the .merge files and instead using Makefile fragments included from the configuration directories that invoke a Make function. This allows much better error handling as well as a cleaner looking system (somewhat similar to the package .mk files).

I've also added some documentation to explain what is supported and an example to show how it's used. I haven't touched the documentation before and I'm not familiar with the formatting so I hope I've done it the right way and that it's useful (feedback would be great). "make manual-html" seems to work at least.

Elizabeth Liner (1):
Makefile: Support nested config directories

Patrick Williams (3):
Makefile: Generate %_defconfig recipes from macro.
Makefile: Add nested config dirs to list-defconfigs
Makefile: Add merged defconfigs to list-defconfigs.

Sam Bobroff (2):
Add support for merged defconfigs
Add documentation for merged defconfigs

 Makefile                             | 58 +++++++++++++++++++++++++++++-----
 docs/manual/adding-board-support.txt |  3 ++
 docs/manual/appendix.txt             |  1 +
 docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
 4 files changed, 114 insertions(+), 8 deletions(-)
 create mode 100644 docs/manual/merged-defconfigs.txt
Signature · 1 line
2.1.0
#1 lore raw

PATCH v3 1/6 Add support for merged defconfigs

---
v3:

* Documentation moved to separate patch.
* Fixed "make savedefconfig".

 Makefile | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
Makefile +38 -0
index 78b44c5..9f3fc92 100644
@@ -858,6 +858,44 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 
 .PHONY: defconfig savedefconfig
 
+# Find the path of a config file that may be either in TOPDIR or BR2_EXTERNAL.
+# Missing or duplicate matches must be checked for elsewhere.
+define FIND_CONFIG_FILE =
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(wildcard $(d)/$(1)))
+endef
+
+define CONFIG_FILE_LIST =
+$(foreach f,$1,$(call FIND_CONFIG_FILE,$f))
+endef
+
+# Check that each config file is found once and only once.
+define CHECK_CONFIG_FILE_LIST =
+$(foreach f,$(1),
+ifeq "$(words $(call FIND_CONFIG_FILE,$(f)))" "0"
+$$(error ERROR: Missing input file: $(f))
+else
+ifneq "$(words $(call FIND_CONFIG_FILE,$(f)))" "1"
+$$(error ERROR: Duplicate input file: $(f))
+endif
+endif
+)
+endef
+
+# To be called by configuration fragments (*.mk) to set up defconfigs built
+# by merge_config.sh, via the include that follows:
+define merge_config =
+$(call CHECK_CONFIG_FILE_LIST,$2 $3)
+$(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputmakefile
+	@mkdir $$(CONFIG_DIR)/.merge_config
+	@$$(TOPDIR)/support/kconfig/merge_config.sh -m -O $$(CONFIG_DIR)/.merge_config \
+		$(call CONFIG_FILE_LIST,$2 $3)
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
+		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)
+	@rm -rf $$(CONFIG_DIR)/.merge_config
+endef
+
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
+
 ################################################################################
 #
 # Cleanup and misc junk
Signature · 1 line
2.1.0
#4 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

In reply to Sam Bobroff
---
 docs/manual/adding-board-support.txt |  3 ++
 docs/manual/appendix.txt             |  1 +
 docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100644 docs/manual/merged-defconfigs.txt
docs/manual/adding-board-support.txt +3 -0
index f6d74ae..086a67e 100644
@@ -24,6 +24,9 @@ savedefconfig+. This will generate a minimal +defconfig+ file at the
 root of the Buildroot source tree. Move this file into the +configs/+
 directory, and rename it +<boardname>_defconfig+.
 
+It is also possible to create configurations by specifying differences from
+an existing configuration. See xref:merged-defconfigs[].
+
 It is recommended to use as much as possible upstream versions of the
 Linux kernel and bootloaders, and to use as much as possible default
 kernel and bootloader configurations. If they are incorrect for your
docs/manual/appendix.txt +1 -0
index 87a20bd..9abc37e 100644
@@ -3,6 +3,7 @@
 
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
+include::merged-defconfigs.txt[]
 
 
 // Automatically generated lists:
docs/manual/merged-defconfigs.txt new file +60 -0

60 lines not on this page · raw message

Signature · 1 line
2.1.0
#14 lore raw

PATCH v3 3/6 Makefile: Generate %_defconfig recipes from macro.

In reply to Sam Bobroff

From: Patrick Williams <[email protected]>

To reduce duplication in the %_defconfig recipes with $(TOPDIR) and $(BR2_EXTERNAL) versions, generate these from a macro. The macro is now called on a list of directories containing the appropriate ones.

---
 Makefile | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
Makefile +8 -7
index 9f3fc92..e78b9e2 100644
@@ -842,13 +842,14 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
 
 # Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/configs/$@ \
-		$< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
+define CREATE_DEFCONFIG_RECIPES
+%_defconfig: $$(BUILD_DIR)/buildroot-config/conf $1/%_defconfig outputmakefile
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(TOPDIR)/configs/$$@ \
+		$$< --defconfig=$1/$$@ $$(CONFIG_CONFIG_IN)
+endef
 
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
-		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
+BR2_DEFCONFIG_PATHS=$(TOPDIR)/configs $(BR2_EXTERNAL)/configs
+$(foreach path,$(BR2_DEFCONFIG_PATHS),$(eval $(call CREATE_DEFCONFIG_RECIPES,$(path))))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
@@ -894,7 +895,7 @@ $(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputm
 	@rm -rf $$(CONFIG_DIR)/.merge_config
 endef
 
-$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
+$(foreach d,$(BR2_DEFCONFIG_PATHS),$(eval -include $(d)/*.mk))
 
 ################################################################################
 #
Signature · 1 line
2.1.0
#15 lore raw

PATCH v3 4/6 Makefile: Support nested config directories

In reply to Sam Bobroff

From: Elizabeth Liner <[email protected]>

Extend the BR2_DEFCONFIG_PATH to support nested config directories. This allows us to place config files in both .../config and .../config/<arch>.

Both $(TOPDIR) and $(BR2_EXTERNAL) are supported.

---
 Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Makefile +2 -1
index e78b9e2..fd07f8c 100644
@@ -848,7 +848,8 @@ define CREATE_DEFCONFIG_RECIPES
 		$$< --defconfig=$1/$$@ $$(CONFIG_CONFIG_IN)
 endef
 
-BR2_DEFCONFIG_PATHS=$(TOPDIR)/configs $(BR2_EXTERNAL)/configs
+BR2_DEFCONFIG_PATHS=$(sort $(dir $(wildcard $(TOPDIR)/configs/*/))) \
+		$(sort $(dir $(wildcard $(BR2_EXTERNAL)/configs/*/)))
 $(foreach path,$(BR2_DEFCONFIG_PATHS),$(eval $(call CREATE_DEFCONFIG_RECIPES,$(path))))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
Signature · 1 line
2.1.0
#16 lore raw

PATCH v3 5/6 Makefile: Add nested config dirs to list-defconfigs

In reply to Sam Bobroff

From: Patrick Williams <[email protected]>

---
v3:

* Removed unnecessary "/" ("wildcard" adds a trailing "/" to directories).

 Makefile | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
Makefile +6 -4
index fd07f8c..da6008e 100644
@@ -206,6 +206,10 @@ LEGAL_LICENSES_TXT_HOST = $(LEGAL_INFO_DIR)/host-licenses.txt
 LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
 LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
 
+BR2_DEFCONFIG_PATHS_LOCAL=$(sort $(dir $(wildcard $(TOPDIR)/configs/*)))
+BR2_DEFCONFIG_PATHS_USER=$(sort $(dir $(wildcard $(BR2_EXTERNAL)/configs/*)))
+BR2_DEFCONFIG_PATHS=$(BR2_DEFCONFIG_PATHS_LOCAL) $(BR2_DEFCONFIG_PATHS_USER)
+
 BR2_CONFIG = $(CONFIG_DIR)/.config
 
 # Pull in the user's configuration file
@@ -848,8 +852,6 @@ define CREATE_DEFCONFIG_RECIPES
 		$$< --defconfig=$1/$$@ $$(CONFIG_CONFIG_IN)
 endef
 
-BR2_DEFCONFIG_PATHS=$(sort $(dir $(wildcard $(TOPDIR)/configs/*/))) \
-		$(sort $(dir $(wildcard $(BR2_EXTERNAL)/configs/*/)))
 $(foreach path,$(BR2_DEFCONFIG_PATHS),$(eval $(call CREATE_DEFCONFIG_RECIPES,$(path))))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
@@ -1009,12 +1011,12 @@ help:
 
 list-defconfigs:
 	@echo 'Built-in configs:'
-	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_LOCAL),$(wildcard $(path)/*_defconfig)))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
 	@echo
 	@echo 'User-provided configs:'
-	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_USER),$(wildcard $(path)/*_defconfig)))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 endif
 	@echo
Signature · 1 line
2.1.0
#17 lore raw

PATCH v3 6/6 Makefile: Add merged defconfigs to list-defconfigs.

In reply to Sam Bobroff

From: Patrick Williams <[email protected]>

---
 Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Makefile +2 -2
index da6008e..329db79 100644
@@ -1011,12 +1011,12 @@ help:
 
 list-defconfigs:
 	@echo 'Built-in configs:'
-	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_LOCAL),$(wildcard $(path)/*_defconfig)))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_LOCAL),$(basename $(wildcard $(path)/*_defconfig $(path)/*_defconfig.mk))))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
 	@echo
 	@echo 'User-provided configs:'
-	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_USER),$(wildcard $(path)/*_defconfig)))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_USER),$(basename $(wildcard $(path)/*_defconfig $(path)/*_defconfig.mk))))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 endif
 	@echo
Signature · 1 line
2.1.0
#2 lore raw

PATCH v3 1/6 Add support for merged defconfigs

In reply to Sam Bobroff

Hi Sam,

Le 29/06/2016 ? 07:06, Sam Bobroff a ?crit :

Quoted 6 lines

---
v3:

* Documentation moved to separate patch.
* Fixed "make savedefconfig".

Sorry, I don't see how "make savedefconfig" has been fixed in this patch. The patch seems the same as v2.

With the series applied I get the same error.

Best regards,
Romain

Quoted 54 lines
 Makefile | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
Makefile +38 -0
index 78b44c5..9f3fc92 100644
@@ -858,6 +858,44 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 
 .PHONY: defconfig savedefconfig
 
+# Find the path of a config file that may be either in TOPDIR or BR2_EXTERNAL.
+# Missing or duplicate matches must be checked for elsewhere.
+define FIND_CONFIG_FILE =
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(wildcard $(d)/$(1)))
+endef
+
+define CONFIG_FILE_LIST =
+$(foreach f,$1,$(call FIND_CONFIG_FILE,$f))
+endef
+
+# Check that each config file is found once and only once.
+define CHECK_CONFIG_FILE_LIST =
+$(foreach f,$(1),
+ifeq "$(words $(call FIND_CONFIG_FILE,$(f)))" "0"
+$$(error ERROR: Missing input file: $(f))
+else
+ifneq "$(words $(call FIND_CONFIG_FILE,$(f)))" "1"
+$$(error ERROR: Duplicate input file: $(f))
+endif
+endif
+)
+endef
+
+# To be called by configuration fragments (*.mk) to set up defconfigs built
+# by merge_config.sh, via the include that follows:
+define merge_config =
+$(call CHECK_CONFIG_FILE_LIST,$2 $3)
+$(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputmakefile
+	@mkdir $$(CONFIG_DIR)/.merge_config
+	@$$(TOPDIR)/support/kconfig/merge_config.sh -m -O $$(CONFIG_DIR)/.merge_config \
+		$(call CONFIG_FILE_LIST,$2 $3)
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
+		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)
+	@rm -rf $$(CONFIG_DIR)/.merge_config
+endef
+
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
+
 ################################################################################
 #
 # Cleanup and misc junk
#5 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

In reply to Sam Bobroff

Hi Sam,

Le 29/06/2016 ? 07:06, Sam Bobroff a ?crit :

Quoted 51 lines
---
 docs/manual/adding-board-support.txt |  3 ++
 docs/manual/appendix.txt             |  1 +
 docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100644 docs/manual/merged-defconfigs.txt
docs/manual/adding-board-support.txt +3 -0
index f6d74ae..086a67e 100644
@@ -24,6 +24,9 @@ savedefconfig+. This will generate a minimal +defconfig+ file at the
 root of the Buildroot source tree. Move this file into the +configs/+
 directory, and rename it +<boardname>_defconfig+.
 
+It is also possible to create configurations by specifying differences from
+an existing configuration. See xref:merged-defconfigs[].
+
 It is recommended to use as much as possible upstream versions of the
 Linux kernel and bootloaders, and to use as much as possible default
 kernel and bootloader configurations. If they are incorrect for your
docs/manual/appendix.txt +1 -0
index 87a20bd..9abc37e 100644
@@ -3,6 +3,7 @@
 
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
+include::merged-defconfigs.txt[]
 
 
 // Automatically generated lists:
docs/manual/merged-defconfigs.txt new file +11 -0
new file mode 100644
index 0000000..e1f50fd
@@ -0,0 +1,60 @@
+// -*- mode:doc -*- ;
+// vim: set syntax=asciidoc:
+
+[[merged-defconfigs]]
+== Merged configurations
+
+Buildroot supports the use of the Linux kernel's +merge_config.sh+ script to
+create configurations by merging a set of partial configurations (or overlays)
+on top of a base configuration. This may be useful when there are several
+configurations that are specializations of a single generic configuration.
+

docs/manual/merged-defconfigs.txt line 21

Quoted 10 lines
docs/manual/merged-defconfigs.txt new file +10 -0
+Files in the +configs/+ directory with the extension
++.mk+ will be read by the top level Makefile and the Make function
++merge_config+ can be used within them to define new merged configurations.
+
+The merge_config function takes three parameters: the first is the name of the
+new configuration (which should end with +_defconfig+), the second is the name
+of the base configuration file and the third is a space separated list of the
+partial configuration files to merge from the +configs/+ directory. All files
+should be specified relative to the +configs/+ directory and the name of the
+Make fragment file should match the name of the configuration it creates.

This series has been discussed with other developers and we concluded that this feature is not the responsibility of Buildroot to generate a defconfig from partial configuration files.

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

Best regards,
Romain

Quoted 40 lines
docs/manual/merged-defconfigs.txt new file +39 -0
+
+The new configuration can be selected as if it were a normal configuration, i.e. with
++make foo_defconfig+.
+
+Partial configurations have the same format as full configuration
+files except that they contain only a subset of values. They can be created by
+starting with a normal configuration file and deleting lines as appropriate.
+
+External configuration files from the +$(BR2_EXTERNAL)/configs+ directory are fully
+supported, and merge configurations may freely refer to files from either
+location.
+
+Example:
+
+If configs/foo_defconfig contains:
+-----
+BR2_TARGET_GENERIC_HOSTNAME="buildroot"
+-----
+
+And configs/bar_defconfig.mk contains:
+-----
+$(eval $(call merge_config,bar_defconfig,foo_defconfig,b.config))
+-----
+
+And configs/b.config contains:
+-----
+BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
+-----
+
+After running the command:
+-----
+make bar_defconfig
+-----
+
+The resulting .config file would contain:
+-----
+BR2_TARGET_GENERIC_HOSTNAME="buildroot"
+BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
+-----
#3 lore raw

PATCH v3 1/6 Add support for merged defconfigs

In reply to Romain Naour
Quoted 7 lines · Romain Naour

Hi Sam,

Le 29/06/2016 ? 07:06, Sam Bobroff a ?crit :

Quoted 4 lines

---
v3:

Quoted 7 lines
Quoted 2 lines

* Documentation moved to separate patch.
* Fixed "make savedefconfig".

Sorry, I don't see how "make savedefconfig" has been fixed in this patch. The patch seems the same as v2.

With the series applied I get the same error.

Ah, sorry, I managed to lose that change somehow, this part:

+ @$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
+ $$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)

Should have been:

+ @$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(dir $(call FIND_CONFIG_FILE,$2))$1 \
+ $$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)

Which would place the saved defconfig in a file with the name of your merged defconfig in which ever directory contained the "base" part of the config. I'm not entirely sure what "make safedefconfig" should really mean for a merged config but this seemed OK.

But, given that you've decided not to take the set, I won't post a new version. ;-)

Cheers,
Sam.

Quoted 57 lines

Best regards,
Romain

Quoted 54 lines
 Makefile | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
Makefile +38 -0
index 78b44c5..9f3fc92 100644
@@ -858,6 +858,44 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 
 .PHONY: defconfig savedefconfig
 
+# Find the path of a config file that may be either in TOPDIR or BR2_EXTERNAL.
+# Missing or duplicate matches must be checked for elsewhere.
+define FIND_CONFIG_FILE =
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(wildcard $(d)/$(1)))
+endef
+
+define CONFIG_FILE_LIST =
+$(foreach f,$1,$(call FIND_CONFIG_FILE,$f))
+endef
+
+# Check that each config file is found once and only once.
+define CHECK_CONFIG_FILE_LIST =
+$(foreach f,$(1),
+ifeq "$(words $(call FIND_CONFIG_FILE,$(f)))" "0"
+$$(error ERROR: Missing input file: $(f))
+else
+ifneq "$(words $(call FIND_CONFIG_FILE,$(f)))" "1"
+$$(error ERROR: Duplicate input file: $(f))
+endif
+endif
+)
+endef
+
+# To be called by configuration fragments (*.mk) to set up defconfigs built
+# by merge_config.sh, via the include that follows:
+define merge_config =
+$(call CHECK_CONFIG_FILE_LIST,$2 $3)
+$(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputmakefile
+	@mkdir $$(CONFIG_DIR)/.merge_config
+	@$$(TOPDIR)/support/kconfig/merge_config.sh -m -O $$(CONFIG_DIR)/.merge_config \
+		$(call CONFIG_FILE_LIST,$2 $3)
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
+		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)
+	@rm -rf $$(CONFIG_DIR)/.merge_config
+endef
+
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
+
 ################################################################################
 #
 # Cleanup and misc junk
#6 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

In reply to Romain Naour
Quoted 2 lines · Romain Naour

Hi Sam,

[snip]

Quoted 4 lines

This series has been discussed with other developers and we concluded that this feature is not the responsibility of Buildroot to generate a defconfig from partial configuration files.

Fair enough.

Quoted 3 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

That seems fine, it might even be possible to add something to $(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.

Could you elaborate a bit on what you mean by using merge_config.sh outside of buildroot? Would the fragment files still be checked into buildroot somewhere (but just not hooked into the build system) or would they be maintained outside it?

Quoted 2 lines

Best regards,
Romain

Cheers,
Sam.

#7 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

Hello Sam,

Quoted 11 lines · Sam Bobroff
Quoted 3 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

That seems fine, it might even be possible to add something to $(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.

Could you elaborate a bit on what you mean by using merge_config.sh outside of buildroot? Would the fragment files still be checked into buildroot somewhere (but just not hooked into the build system) or would they be maintained outside it?

The idea is that Buildroot just takes as input a configuration file. How this configuration file is provided/generated is not Buildroot's responsibility.

For example, you could have in your BR2_EXTERNAL tree a bunch of defconfig fragments, and then a "genconfig" shell script to generate the final configuration you want to build using the defconfig fragments. This "genconfig" script relies on merge_config.sh to actually merge the relevant fragments together.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#9 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

In reply to Romain Naour
Quoted 6 lines · Romain Naour

Hi Sam,

This series has been discussed with other developers and we concluded that this feature is not the responsibility of Buildroot to generate a defconfig from partial configuration files.

Quoted 7 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

Best regards,
Romain

Earlier we had two independent patch sets that we were asked to merge.

1) Merged defconfigs (now this patch set).
2) Subdirs for defconfigs.

If #1 has been rejected as undesireable, should we now prepare an indepenent patch set for #2 or is that also not wanted now?

There is currently no way for us to manage this behavior in a BR2_EXTERNAL layer alone because there is no Makefile from the BR2_EXTERNAL path sourced in the defconfig case. Is there any issue with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

I realize we could have a new script, but we already have users that are use to 'make <system>_defconfig && make' so we would like to be able to preserve that behavior even though we are moving to merged defconfig fragments.

Signature · 8 lines
Patrick Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160705/3485a4b1/attachment.asc>
#10 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

Quoted 20 lines · Patrick Williams
Quoted 6 lines · Romain Naour

Hi Sam,

This series has been discussed with other developers and we concluded that this feature is not the responsibility of Buildroot to generate a defconfig from partial configuration files.

Quoted 7 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

Best regards,
Romain

Earlier we had two independent patch sets that we were asked to merge.

1) Merged defconfigs (now this patch set).
2) Subdirs for defconfigs.

Quoted 7 lines

If #1 has been rejected as undesireable, should we now prepare an indepenent patch set for #2 or is that also not wanted now?

There is currently no way for us to manage this behavior in a BR2_EXTERNAL layer alone because there is no Makefile from the BR2_EXTERNAL path sourced in the defconfig case. Is there any issue with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

Actually there might be - Sam was telling me yesterday that the main Makefile includes?$(BR2_EXTERNAL)/external.mk, which is something we may use to get mergeconfigs in op-build. That might work for nested configs as well.

Quoted 6 lines

I realize we could have a new script, but we already have users that are use to 'make <system>_defconfig && make' so we would like to be able to preserve that behavior even though we are moving to merged defconfig fragments.

#11 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

Quoted 29 lines · Samuel Mendoza-Jonas
Quoted 20 lines · Patrick Williams
Quoted 6 lines · Romain Naour

Hi Sam,

This series has been discussed with other developers and we concluded that this feature is not the responsibility of Buildroot to generate a defconfig from partial configuration files.

Quoted 7 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

Best regards,
Romain

Earlier we had two independent patch sets that we were asked to merge.

1) Merged defconfigs (now this patch set).
2) Subdirs for defconfigs.

Quoted 7 lines

If #1 has been rejected as undesireable, should we now prepare an indepenent patch set for #2 or is that also not wanted now?

There is currently no way for us to manage this behavior in a BR2_EXTERNAL layer alone because there is no Makefile from the BR2_EXTERNAL path sourced in the defconfig case. Is there any issue with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

Quoted 5 lines

Actually there might be - Sam was telling me yesterday that the main Makefile includes?$(BR2_EXTERNAL)/external.mk, which is something we may use to get mergeconfigs in op-build. That might work for nested configs as well.

I've already been down this route. The 'include $(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq ($(BR2_HAVE_DOT_CONFIG),y)' on line 336. That effectively means if you don't already have a .config (which comes about from running a 'defconfig' recipe) then you don't get a source to external.mk. That is why I'm asking about a new 'external-defconfig.mk'.

Quoted 7 lines
Quoted 6 lines

I realize we could have a new script, but we already have users that are use to 'make <system>_defconfig && make' so we would like to be able to preserve that behavior even though we are moving to merged defconfig fragments.

Signature · 8 lines
Patrick Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160705/d32d6c33/attachment.asc>
#8 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

In reply to Thomas Petazzoni
Quoted 20 lines · Thomas Petazzoni

Hello Sam,

Quoted 11 lines · Sam Bobroff
Quoted 3 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

That seems fine, it might even be possible to add something to $(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.

Could you elaborate a bit on what you mean by using merge_config.sh outside of buildroot? Would the fragment files still be checked into buildroot somewhere (but just not hooked into the build system) or would they be maintained outside it?

The idea is that Buildroot just takes as input a configuration file. How this configuration file is provided/generated is not Buildroot's responsibility.

Quoted 6 lines

For example, you could have in your BR2_EXTERNAL tree a bunch of defconfig fragments, and then a "genconfig" shell script to generate the final configuration you want to build using the defconfig fragments. This "genconfig" script relies on merge_config.sh to actually merge the relevant fragments together.

Ah right, yes I'm sure something like that will work for work that stays external to buildroot, but the case I'm concerned about is where BR2_EXTERNAL isn't being used. We want to get our defconfigs out there for everyone to use and contribute to :-) It looks like existing defconfigs could make use of this
too: from a bit of diff'ing in the configs directory there are already configs that share a lot of values and would be easier to maintain using merging.

So how would you suggest handling merged configs that should be contributed back to the community? Would you expect that we maintain our own merge-config system and fragments and then contribute the full configs back to buildroot?

That seems like the simplest way to do it but I can't see a good way to handle patches to them: users who have downloaded buildroot would have the defconfigs but not the input fragments that were used to create them, so it would be difficult to handle changes they need to make.

Would an alternative approach be to include the fragments and genconfig system in buildroot but make it a separate component, so it wouldn't add complexity to the build system? (e.g. you "make configs" to update your defconfigs.)

Cheers,
Sam.

#12 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

In reply to Patrick Williams

Hi Patrick, Samuel, All,

Le 06/07/2016 ? 00:41, Patrick Williams a ?crit :

Quoted 36 lines · Samuel Mendoza-Jonas
Quoted 29 lines · Patrick Williams
Quoted 20 lines · Romain Naour
Quoted 6 lines

Hi Sam,

This series has been discussed with other developers and we concluded that this feature is not the responsibility of Buildroot to generate a defconfig from partial configuration files.

Quoted 7 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

Best regards,
Romain

Earlier we had two independent patch sets that we were asked to merge.

1) Merged defconfigs (now this patch set).
2) Subdirs for defconfigs.

Quoted 7 lines

If #1 has been rejected as undesireable, should we now prepare an indepenent patch set for #2 or is that also not wanted now?

There is currently no way for us to manage this behavior in a BR2_EXTERNAL layer alone because there is no Makefile from the BR2_EXTERNAL path sourced in the defconfig case. Is there any issue with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

Quoted 5 lines

Actually there might be - Sam was telling me yesterday that the main Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may use to get mergeconfigs in op-build. That might work for nested configs as well.

Quoted 6 lines

I've already been down this route. The 'include $(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq ($(BR2_HAVE_DOT_CONFIG),y)' on line 336. That effectively means if you don't already have a .config (which comes about from running a 'defconfig' recipe) then you don't get a source to external.mk. That is why I'm asking about a new 'external-defconfig.mk'.

Patrick, you're right. It's currently no way to include an external makefile when BR2_HAVE_DOT_CONFIG is not set. Neither from local.mk or external.mk.

Indeed, adding $(BR2_EXTERNAL)/external-defconfig.mk should work in this case.

I tried with this patch [1] and used Samuel's patch [2] applied to external-defconfig.mk instead of the Buildroot Makefile.

Lets see what other Buidroot developers think about this, especially Yann which is currently working on multi-BR2_EXTERNAL.

About the nested configs directory, I don't see any particular problem...

Best regards,
Romain

[1]
https://github.com/RomainNaour/buildroot/commit/396744b032567b0d7fcc6e3d56c37143efde368e
[2] http://patchwork.ozlabs.org/patch/641838/

Quoted 16 lines
Quoted 8 lines
Quoted 7 lines

I realize we could have a new script, but we already have users that are use to 'make <system>_defconfig && make' so we would like to be able to preserve that behavior even though we are moving to merged defconfig fragments.

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#13 lore raw

PATCH v3 2/6 Add documentation for merged defconfigs

Romain, Samuel, Patrick, All,

On 2016-07-06 21:08 +0200, Romain Naour spake thusly:

Quoted 30 lines

Le 06/07/2016 ? 00:41, Patrick Williams a ?crit :

Quoted 29 lines · Samuel Mendoza-Jonas
Quoted 28 lines · Patrick Williams
Quoted 27 lines · Romain Naour
Quoted 6 lines

Hi Sam,

This series has been discussed with other developers and we concluded that this feature is not the responsibility of Buildroot to generate a defconfig from partial configuration files.

Quoted 7 lines

Instead we recommend to use merge_config.sh from outside of Buildroot to generate the merged defconfig before calling make. Maybe we should add some documentation in the manual to explain how to use it.

Best regards,
Romain

Earlier we had two independent patch sets that we were asked to merge.

1) Merged defconfigs (now this patch set).
2) Subdirs for defconfigs.

If #1 has been rejected as undesireable, should we now prepare an indepenent patch set for #2 or is that also not wanted now?

There is currently no way for us to manage this behavior in a BR2_EXTERNAL layer alone because there is no Makefile from the BR2_EXTERNAL path sourced in the defconfig case. Is there any issue with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

I can understand the frustration. I am sorry that we came to that conclusion and that you feel disapointed.

We did reject this because, as Romain explained, we believe it is much more flexible to handle such a case outside of Buildroot, as Romain explained.

Quoted 13 lines
Quoted 6 lines
Quoted 5 lines

Actually there might be - Sam was telling me yesterday that the main Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may use to get mergeconfigs in op-build. That might work for nested configs as well.

Quoted 6 lines

I've already been down this route. The 'include $(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq ($(BR2_HAVE_DOT_CONFIG),y)' on line 336. That effectively means if you don't already have a .config (which comes about from running a 'defconfig' recipe) then you don't get a source to external.mk. That is why I'm asking about a new 'external-defconfig.mk'.

Quoted 7 lines

Patrick, you're right. It's currently no way to include an external makefile when BR2_HAVE_DOT_CONFIG is not set. Neither from local.mk or external.mk.

Indeed, adding $(BR2_EXTERNAL)/external-defconfig.mk should work in this case.

I tried with this patch [1] and used Samuel's patch [2] applied to external-defconfig.mk instead of the Buildroot Makefile.

[--SNIP--]

Quoted 1 line

[1] https://github.com/RomainNaour/buildroot/commit/396744b032567b0d7fcc6e3d56c37143efde368e

No, we already rejected a similar proposal (by me!) to source custom help from br2-external:

    http://lists.busybox.net/pipermail/buildroot/2016-March/thread.html#156705
    http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#158710

The correct solution would be that we source external.mk unconditionally. The patch is actually trivial (I had it locally some time ago, when working on the aforementioned custom help).

For this, we should ensure it is safe to do so, but in my (very little) epxerimentation, it seemed OK to do so. However, we must be really carefull in doing so.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'