Skip to contentSkip to conversation

Makefile: Support merged defconfigs

PATCH Superseded Sam Bobroff 5 participants 6 messages May 9, 2016 – May 13, 2016 Kbuild

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Makefile: Support merged defconfigs

Within the Linux kernel, and several other packages, it's currently possible to maintain defconfigs as diffs (fragments) against other configs but this is not possible with buildroot itself.

This patch adds the capability, although using a slightly different implementation. Files may be added in the normal config directories ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format "xxx_defconfig.merge" that contain, one per line, the files to be passed to merge_config.sh to create the matching xxx_defconfig file (the first line should contain the base config followed by the fragments). The generated defconfig file is then handled as it would normally be.

---
For a discussion about this from the OP build project, see
https://github.com/open-power/op-build/issues/457

A couple of things about the implementation:

* merge_config.sh will only output to a file called ".config" so I've set it up to output directly to $TOPDIR/.config, since that's about to be overwritten anyway, but I don't feel this is particularly nice.
* It seems strange to pass --defconfig=.config when running conf but I couldn't quickly tell if there was something better to do. Using --oldconfig would pick up .config by default but I don't know if it would then process the file correctly so I went with what seemed safe.
* I realise that there are now four very similar rules that handle defconfigs. I could look at factoring them but it didn't seem worth the effort and complexity at this point.

 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)
Makefile +10 -0
index 71735b5..eb80152 100644
@@ -845,6 +845,16 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
 		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
 
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig.merge outputmakefile
+	cd $(TOPDIR)/configs && xargs -a [email protected] -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig.merge outputmakefile
+	cd $(BR2_EXTERNAL)/configs && xargs -a [email protected] -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
 		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/1 Makefile: Support merged defconfigs

Quoted 13 lines · Sam Bobroff

Within the Linux kernel, and several other packages, it's currently possible to maintain defconfigs as diffs (fragments) against other configs but this is not possible with buildroot itself.

This patch adds the capability, although using a slightly different implementation. Files may be added in the normal config directories ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format "xxx_defconfig.merge" that contain, one per line, the files to be passed to merge_config.sh to create the matching xxx_defconfig file (the first line should contain the base config followed by the fragments). The generated defconfig file is then handled as it would normally be.

Quoted 1 line

This neatly solves some problems for us over in op-build land. Tested with several fragments and output is what I expect:

Quoted 48 lines

---
For a discussion about this from the OP build project, see
https://github.com/open-power/op-build/issues/457

A couple of things about the implementation:

* merge_config.sh will only output to a file called ".config" so I've set it up to output directly to $TOPDIR/.config, since that's about to be overwritten anyway, but I don't feel this is particularly nice.
* It seems strange to pass --defconfig=.config when running conf but I couldn't quickly tell if there was something better to do. Using --oldconfig would pick up .config by default but I don't know if it would then process the file correctly so I went with what seemed safe.
* I realise that there are now four very similar rules that handle defconfigs. I could look at factoring them but it didn't seem worth the effort and complexity at this point.

 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)
Makefile +10 -0
index 71735b5..eb80152 100644
@@ -845,6 +845,16 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
 		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
 
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig.merge outputmakefile
+	cd $(TOPDIR)/configs && xargs -a [email protected] -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig.merge outputmakefile
+	cd $(BR2_EXTERNAL)/configs && xargs -a [email protected] -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
 		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
Signature · 6 lines
2.1.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
#2 lore raw

PATCH 1/1 Makefile: Support merged defconfigs

In reply to Sam Bobroff
Quoted 13 lines · On Mon, 9 May 2016 15:22:32 +1000 Sam Bobroff <[email protected]> wrote:

Within the Linux kernel, and several other packages, it's currently possible to maintain defconfigs as diffs (fragments) against other configs but this is not possible with buildroot itself.

This patch adds the capability, although using a slightly different implementation. Files may be added in the normal config directories ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format "xxx_defconfig.merge" that contain, one per line, the files to be passed to merge_config.sh to create the matching xxx_defconfig file (the first line should contain the base config followed by the fragments). The generated defconfig file is then handled as it would normally be.

Quoted 5 lines

---
For a discussion about this from the OP build project, see
https://github.com/open-power/op-build/issues/457

Hi Sam,

Quoted 9 lines

A couple of things about the implementation:

* merge_config.sh will only output to a file called ".config" so I've set it up to output directly to $TOPDIR/.config, since that's about to be overwritten anyway, but I don't feel this is particularly nice.
* It seems strange to pass --defconfig=.config when running conf but I couldn't quickly tell if there was something better to do. Using --oldconfig would pick up .config by default but I don't know if it would then process the file correctly so I went with what seemed safe.

Looks like what you've done will work. I've come to the same conclusion as you that --oldconfig might actually be a better fit, and by extension --olddefconfig may be even better moving forward as the fragments age and new symbols get added.

Then it occurred to me I've had this question before and I had to go look what I did for linux in arch/powerpc/Makefile and turns out, olddefconfig.

Quoted 4 lines

* I realise that there are now four very similar rules that handle defconfigs. I could look at factoring them but it didn't seem worth the effort and complexity at this point.

I have no doubt there may be a refactored solution in there somewhere but I don't believe it will be as easy to follow as what currently exists. Currently it is easy to understand why there are four, what they are trying to achieve, I'd bet that with any refactoring that clarity will be dulled.

Quoted 24 lines
 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)
Makefile +10 -0
index 71735b5..eb80152 100644
@@ -845,6 +845,16 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
 		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
 
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig.merge outputmakefile
+	cd $(TOPDIR)/configs && xargs -a [email protected] -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig.merge outputmakefile
+	cd $(BR2_EXTERNAL)/configs && xargs -a [email protected] -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
 		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
#3 lore raw

PATCH 1/1 Makefile: Support merged defconfigs

In reply to Sam Bobroff

Hello,

Quoted 13 lines · Sam Bobroff

Within the Linux kernel, and several other packages, it's currently possible to maintain defconfigs as diffs (fragments) against other configs but this is not possible with buildroot itself.

This patch adds the capability, although using a slightly different implementation. Files may be added in the normal config directories ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format "xxx_defconfig.merge" that contain, one per line, the files to be passed to merge_config.sh to create the matching xxx_defconfig file (the first line should contain the base config followed by the fragments). The generated defconfig file is then handled as it would normally be.

Quoted 1 line

Warning: the below is definitely not a final opinion or decision. It is merely the start of a discussion.

I do like the usage of fragments. Your Github link at
https://github.com/open-power/op-build/issues/457 even points to Buildroot training materials that I wrote, which explains how to use fragments.

However, I am not sure whether we want in Buildroot a mechanism to build a defconfig from fragments. To me, Buildroot should remain a simple tool that takes as input a complete configuration, and generates a Linux system from it.

Whether this "complete configuration" comes from a full .config, a minimal defconfig, a defconfig generated from fragments, a defconfig generated by a separate tool, by a webpage, or anything else, is not Buildroot's matter. Buildroot's job starts when the configuration is ready.

Essentially, the part that's missing in your commit is the update to the documentation. And when you'll start updating the manual to explain how this feature is working, you'll probably see that it's a bit tricky to explain.

So, I'm definitely not saying "no" to this, but my belief is that it's on the edges of Buildroot features. On the one hand, it brings a well-defined and correct way of storing fragments and generated a defconfig from fragments, that everyone can use and rely on. On the other hand, it adds one more slightly "tricky" feature to Buildroot, increasing its learning curve.

For a customer project, Gustavo has written a "genconfig" shell script, that takes as input various parameters that define the system to be built (which HW platform, which software stack, whether it should be a debug or release build) and produces the defconfig for Buildroot.

What do others think?

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#4 lore raw

PATCH 1/1 Makefile: Support merged defconfigs

Quoted 49 lines · Thomas Petazzoni

Hello,

Quoted 13 lines · Sam Bobroff

Within the Linux kernel, and several other packages, it's currently possible to maintain defconfigs as diffs (fragments) against other configs but this is not possible with buildroot itself.

This patch adds the capability, although using a slightly different implementation. Files may be added in the normal config directories ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format "xxx_defconfig.merge" that contain, one per line, the files to be passed to merge_config.sh to create the matching xxx_defconfig file (the first line should contain the base config followed by the fragments). The generated defconfig file is then handled as it would normally be.

Quoted 1 line

Warning: the below is definitely not a final opinion or decision. It is merely the start of a discussion.

I do like the usage of fragments. Your Github link at
https://github.com/open-power/op-build/issues/457 even points to Buildroot training materials that I wrote, which explains how to use fragments.

However, I am not sure whether we want in Buildroot a mechanism to build a defconfig from fragments. To me, Buildroot should remain a simple tool that takes as input a complete configuration, and generates a Linux system from it.

Whether this "complete configuration" comes from a full .config, a minimal defconfig, a defconfig generated from fragments, a defconfig generated by a separate tool, by a webpage, or anything else, is not Buildroot's matter. Buildroot's job starts when the configuration is ready.

Essentially, the part that's missing in your commit is the update to the documentation. And when you'll start updating the manual to explain how this feature is working, you'll probably see that it's a bit tricky to explain.

So, I'm definitely not saying "no" to this, but my belief is that it's on the edges of Buildroot features. On the one hand, it brings a well-defined and correct way of storing fragments and generated a defconfig from fragments, that everyone can use and rely on. On the other hand, it adds one more slightly "tricky" feature to Buildroot, increasing its learning curve.

Quoted 4 lines

For a customer project, Gustavo has written a "genconfig" shell script, that takes as input various parameters that define the system to be built (which HW platform, which software stack, whether it should be a debug or release build) and produces the defconfig for Buildroot.

I tend to agree with Thomas's reasoning. However, I think it would be good if
there was a clearer canonical way of using buildroot. Gustavoz' genconfig script could be part of that. I, for example, have a local.mk fragment that autodetects packages you have checked out in $(BR2_EXTERNAL)/src and adds OVERRIDE_SRCDIR for them. I remember there was this template br2-external github repo at some point that you could use as a basis to start your own br2-external project (but I can't find it back).

I do think it is good if such supporting functionality is carried in the core
buildroot repo. But it's better if it's a bit out of the way, not in the main Makefile, and probably called through some helper script.

I have a similar feeling about the genimage template script BTW.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#5 lore raw

PATCH 1/1 Makefile: Support merged defconfigs

Hello,

Quoted 7 lines · Arnout Vandecappelle

I tend to agree with Thomas's reasoning. However, I think it would be good if
there was a clearer canonical way of using buildroot. Gustavoz' genconfig script could be part of that. I, for example, have a local.mk fragment that autodetects packages you have checked out in $(BR2_EXTERNAL)/src and adds OVERRIDE_SRCDIR for them. I remember there was this template br2-external github repo at some point that you could use as a basis to start your own br2-external project (but I can't find it back).

You're probably referring to
https://github.com/Openwide-Ingenierie/buildroot-submodule.

Quoted 5 lines

I do think it is good if such supporting functionality is carried in the core
buildroot repo. But it's better if it's a bit out of the way, not in the main Makefile, and probably called through some helper script.

I have a similar feeling about the genimage template script BTW.

Should we create a section in the manual with some kind of "best practices" or "going further" ideas?

Regarding Gustavo's 'genconfig' script, I am not sure it makes a lot of sense to have something like that in the tree. All what it does is very project-specific, and down the road, it simply calls the merge_config.sh script, which is already in our tree.

So, for the specific situation of creating a defconfig by combining fragments, isn't the problem more a documentation problem than anything else?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com