Introduction of kconfig-package infrastructure

This patch series introduces a kconfig-package infrastructure and already converts the uclibc and busybox packages.
The series is based on the to-be-applied uclibc patch series that improves the behavior of the kconfig parts.

The final patch in this series is a bit an outlier: it is a preparation for the conversion of the linux package to the kconfig-package infrastructure, and I already included it to see if it's controversial or not.

I believe this series is complete in the sense that it could be applied as one whole, not yet including the conversion of linux and barebox.
Two sets of changes are still needed:
- any changes based on your comments
- documentation and comments still missing in kconfig-package. I'm awaiting your feedback first. I can either add this as part of the series or use a followup patch to fix this.

I'm shooting for 2014.08 still, if you can provide feedback in time.

In a subsequent series, linux and barebox can be converted to kconfig-package too, but there will need to be some changes to kconfig-package to support them. One particular difference with uclibc/busybox is that linux/barebox use a different method to copy the original config file: instead of simply copying it to build_dir/.config, it is first copied to arch/.../buildroot_defconfig and then 'make ... buildroot_defconfig' is called. I'm considering in adding a FOO_KCONFIG_INSTALL_CONFIG_CMDS variable to support this, which defaults to the simple copy used by busybox/uclibc, but can be overwritten appropriately by linux and barebox.
Anyway, this part is not yet ready, and is likely not ready in time for 2014.08.

Thanks for your feedback,
Thomas

---
 fs/initramfs/initramfs.mk  |   2 +-
 linux/linux.mk             |  20 ++++++++++----------
 package/Makefile.in        |   1 +
 package/busybox/busybox.mk |  25 ++++++++++++-------------
 package/pkg-kconfig.mk     |  33 +++++++++++++++++++++++++++++++++
 package/uclibc/uclibc.mk   |  23 +++++++----------------
 6 files changed, 64 insertions(+), 40 deletions(-)

There are several packages that have a configuration file managed by
kconfig: uclibc, busybox, linux and barebox. All these packages need some make targets to handle the kconfig specificities: creating a configuration (menuconfig, ...) and saving it back (update-config, ...)

These targets should be the same for each of these packages, but unfortunately they are not. Especially with respect to saving back the configuration to the original config file, there are many differences.

A previous set of patches fixed these targets for the uclibc package. This patch extracts these targets into a common kconfig-package infrastructure, with the goals of:
- aligning the behavior of all kconfig-based packages
- removing code duplication

In order to use this infrastructure, a package should at a minimum specify FOO_KCONFIG_FILE and eval the kconfig-package macro. The supported configuration editors can be set with FOO_KCONFIG_EDITORS and defaults to menuconfig only.
Additionally, a package can specify FOO_KCONFIG_OPT for extra options to pass to the invocation of the kconfig editors, and FOO_KCONFIG_FIXUP_CMDS for a list of shell commands used to fixup the .config file after a configuration has been created/edited.

---
Notes:

- the issue with MAKE_OPT vs MAKE_OPTS has been worked around currently by not relying on any MAKE_OPT(S) but instead requiring the package to add the necessary variable to FOO_KCONFIG_OPT.
However, I still believe we should create a consistent set here, either MAKE_OPT or MAKE_OPTS, and the same for CONFIGURE_OPT(S), but this is now no longer related to the introduction of kconfig-package.

- the targets in this patch are sufficient to handle uclibc and busybox. For linux and barebox, some changes will be needed, but for clarity this will be added later.

- documentation and comments for this package should be added still, but I would like to await your feedback first.

 package/Makefile.in    |   1 +
 package/pkg-kconfig.mk |  33 +++++++++++++++++++++++++++++++++
 2 files changed, 34 insertions(+), 0 deletions(-)

diff -r 5916a58248be -r c9dcec875ddb package/Makefile.in
@@ -388,3 +388,4 @@
 include package/pkg-python.mk
 include package/pkg-virtual.mk
 include package/pkg-generic.mk
+include package/pkg-kconfig.mk

diff -r 5916a58248be -r c9dcec875ddb package/pkg-kconfig.mk
@@ -0,0 +1,33 @@
+
+define inner-kconfig-package
+
+$(2)_KCONFIG_EDITORS ?= menuconfig
+$(2)_KCONFIG_OPT ?=
+$(2)_KCONFIG_FIXUP_CMDS ?=
+
+ifndef $(2)_KCONFIG_FILE
+$$(error Internal error: no value specified for $(2)_KCONFIG_FILE)
+endif
+
+$$($(2)_DIR)/.config: $$($(2)_KCONFIG_FILE) | $(1)-patch
+	$$(INSTALL) -m 0644 $$($(2)_KCONFIG_FILE) $$($(2)_DIR)/.config
+
+$$($(2)_DIR)/.stamp_kconfig_fixup_done: $$($(2)_DIR)/.config
+	$$($(2)_KCONFIG_FIXUP_CMDS)
+	$$(Q)touch $$@
+
+$$($(2)_TARGET_CONFIGURE): $$($(2)_DIR)/.stamp_kconfig_fixup_done
+
+# configuration editors (menuconfig, ...)
+$$(addprefix $(1)-,$$($(2)_KCONFIG_EDITORS)): $$($(2)_DIR)/.config
+	$$($(2)_MAKE_ENV) $$(MAKE) -C $$($(2)_DIR) \
+		$$($(2)_KCONFIG_OPT) $$(subst $(1)-,,$$@)
+	rm -f $$($(2)_DIR)/.stamp_{kconfig_fixup_done,configured,built}
+	rm -f $$($(2)_DIR)/.stamp_{target,staging}_installed
+
+$(1)-update-config: $$($(2)_DIR)/.stamp_kconfig_fixup_done
+	cp -f $$($(2)_DIR)/.config $$($(2)_KCONFIG_FILE)
+
+endef # inner-kconfig-package
+
+kconfig-package = $(call inner-kconfig-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)))

There is no real reason to run uclibc-menuconfig in non-parallel mode, even though one can neither expect performance benefits from a parallel menuconfig.
Nevertheless, $(MAKE) is the default, so this patch removes the unnecessary non-default $(MAKE1) usage for uclibc-menuconfig.

This is a simplification introduced in preparation of the kconfig-package infrastructure.

---
 package/uclibc/uclibc.mk |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff -r c9dcec875ddb -r 7c00a2ad3724 package/uclibc/uclibc.mk
@@ -543,7 +543,7 @@
 $(UCLIBC_TARGET_CONFIGURE): $(UCLIBC_DIR)/.stamp_config_fixup_done
 
 uclibc-menuconfig: $(UCLIBC_DIR)/.config
-	$(MAKE1) -C $(UCLIBC_DIR) \
+	$(MAKE) -C $(UCLIBC_DIR) \
 		$(UCLIBC_MAKE_FLAGS) \
 		PREFIX=$(STAGING_DIR) \
 		DEVEL_PREFIX=/usr/ \

This patch converts the uclibc package to the new kconfig-package infrastructure, thus removing code duplication and ensuring a consistent behavior of kconfig packages.

---
 package/uclibc/uclibc.mk |  21 ++++-----------------
 1 files changed, 4 insertions(+), 17 deletions(-)

diff -r 7c00a2ad3724 -r ec98df5449b0 package/uclibc/uclibc.mk
@@ -393,7 +393,7 @@
 	UCLIBC_EXTRA_CFLAGS="$(UCLIBC_EXTRA_CFLAGS) $(TARGET_ABI)" \
 	HOSTCC="$(HOSTCC)"
 
-define UCLIBC_FIXUP_DOT_CONFIG
+define UCLIBC_KCONFIG_FIXUP_CMDS
 	$(call KCONFIG_SET_OPT,CROSS_COMPILER_PREFIX,"$(TARGET_CROSS)",$(@D)/.config)
 	$(call KCONFIG_ENABLE_OPT,TARGET_$(UCLIBC_TARGET_ARCH),$(@D)/.config)
 	$(call KCONFIG_SET_OPT,TARGET_ARCH,"$(UCLIBC_TARGET_ARCH)",$(@D)/.config)

@@ -533,27 +533,14 @@
 
 $(eval $(generic-package))
 
-$(UCLIBC_DIR)/.config: $(UCLIBC_CONFIG_FILE) | uclibc-patch
-	$(INSTALL) -m 0644 $(UCLIBC_CONFIG_FILE) $(UCLIBC_DIR)/.config
-
-$(UCLIBC_DIR)/.stamp_config_fixup_done: $(UCLIBC_DIR)/.config
-	$(UCLIBC_FIXUP_DOT_CONFIG)
-	$(Q)touch $@
-
-$(UCLIBC_TARGET_CONFIGURE): $(UCLIBC_DIR)/.stamp_config_fixup_done
-
-uclibc-menuconfig: $(UCLIBC_DIR)/.config
-	$(MAKE) -C $(UCLIBC_DIR) \
+UCLIBC_KCONFIG_FILE = $(UCLIBC_CONFIG_FILE)
+UCLIBC_KCONFIG_OPT = \
 		$(UCLIBC_MAKE_FLAGS) \
 		PREFIX=$(STAGING_DIR) \
 		DEVEL_PREFIX=/usr/ \
 		RUNTIME_PREFIX=$(STAGING_DIR)/ \
-		menuconfig
-	rm -f $(UCLIBC_DIR)/.stamp_{config_fixup_done,configured,built}
-	rm -f $(UCLIBC_DIR)/.stamp_{target,staging}_installed
 
-uclibc-update-config: $(UCLIBC_DIR)/.stamp_config_fixup_done
-	cp -f $(UCLIBC_DIR)/.config $(UCLIBC_CONFIG_FILE)
+$(eval $(kconfig-package))
 
 # Before uClibc is built, we must have the second stage cross-compiler
 $(UCLIBC_TARGET_BUILD): | host-gcc-intermediate

In analogy of linux.mk, only enable the kconfig targets (menuconfig, update-config, ...) when the uclibc package is actually enabled.

---
 package/uclibc/uclibc.mk |  4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff -r ec98df5449b0 -r 80c907c0dfca package/uclibc/uclibc.mk
@@ -533,6 +533,8 @@
 
 $(eval $(generic-package))
 
+ifeq ($(BR2_TOOLCHAIN_BUILDROOT_UCLIBC),y)
+
 UCLIBC_KCONFIG_FILE = $(UCLIBC_CONFIG_FILE)
 UCLIBC_KCONFIG_OPT = \
 		$(UCLIBC_MAKE_FLAGS) \

@@ -542,5 +544,7 @@
 
 $(eval $(kconfig-package))
 
+endif
+
 # Before uClibc is built, we must have the second stage cross-compiler
 $(UCLIBC_TARGET_BUILD): | host-gcc-intermediate

This patch converts the busybox package to the new kconfig-package infrastructure, thus removing code duplication and ensuring a consistent behavior of kconfig packages.

---
 package/busybox/busybox.mk |  21 ++++++++-------------
 1 files changed, 8 insertions(+), 13 deletions(-)

diff -r 80c907c0dfca -r a850aa264c3d package/busybox/busybox.mk
@@ -136,10 +136,6 @@
 endef
 endif
 
-define BUSYBOX_COPY_CONFIG
-	$(INSTALL) -D -m 0644 $(BUSYBOX_CONFIG_FILE) $(BUSYBOX_BUILD_CONFIG)
-endef
-
 # Disable shadow passwords support if unsupported by the C library
 ifeq ($(BR2_TOOLCHAIN_HAS_SHADOW_PASSWORDS),)
 define BUSYBOX_INTERNAL_SHADOW_PASSWORDS

@@ -190,8 +186,7 @@
 	$(SED) 's/^noclobber="0"$$/noclobber="1"/' $(@D)/applets/install.sh
 endef
 
-define BUSYBOX_CONFIGURE_CMDS
-	$(BUSYBOX_COPY_CONFIG)
+define BUSYBOX_KCONFIG_FIXUP_CMDS
 	$(BUSYBOX_SET_MMU)
 	$(BUSYBOX_SET_LARGEFILE)
 	$(BUSYBOX_SET_IPV6)

@@ -203,6 +198,9 @@
 	$(BUSYBOX_INTERNAL_SHADOW_PASSWORDS)
 	$(BUSYBOX_SET_INIT)
 	$(BUSYBOX_SET_WATCHDOG)
+endef
+
+define BUSYBOX_CONFIGURE_CMDS
 	@yes "" | $(MAKE) ARCH=$(KERNEL_ARCH) CROSS_COMPILE="$(TARGET_CROSS)" \
 		-C $(@D) oldconfig
 	$(BUSYBOX_NOCLOBBER_INSTALL)

@@ -226,11 +224,8 @@
 
 $(eval $(generic-package))
 
-busybox-menuconfig busybox-xconfig busybox-gconfig: busybox-patch
-	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(BUSYBOX_DIR) \
-		$(subst busybox-,,$@)
-	rm -f $(BUSYBOX_DIR)/.stamp_built
-	rm -f $(BUSYBOX_DIR)/.stamp_target_installed
+BUSYBOX_KCONFIG_FILE = $(BUSYBOX_CONFIG_FILE)
+BUSYBOX_KCONFIG_EDITORS = menuconfig xconfig gconfig
+BUSYBOX_KCONFIG_OPT = $(BUSYBOX_MAKE_OPTS)
 
-busybox-update-config: busybox-configure
-	cp -f $(BUSYBOX_BUILD_CONFIG) $(BUSYBOX_CONFIG_FILE)
+$(eval $(kconfig-package))

In analogy of linux.mk, only enable the kconfig targets (menuconfig, update-config, ...) when the busybox package is actually enabled.

---
 package/busybox/busybox.mk |  4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff -r a850aa264c3d -r 01c5523b5777 package/busybox/busybox.mk
@@ -224,8 +224,12 @@
 
 $(eval $(generic-package))
 
+ifeq ($(BR2_PACKAGE_BUSYBOX),y)
+
 BUSYBOX_KCONFIG_FILE = $(BUSYBOX_CONFIG_FILE)
 BUSYBOX_KCONFIG_EDITORS = menuconfig xconfig gconfig
 BUSYBOX_KCONFIG_OPT = $(BUSYBOX_MAKE_OPTS)
 
 $(eval $(kconfig-package))
+
+endif

The linux-* mirror targets of linux26-* have been added a very long time ago (2010) and linux 2.6 is now considered 'old' anyway. It no longer makes sense to support these linux26-* targets, so this patch removes them.

This is a simplification introduced in preparation of the kconfig-package infrastructure.

---
 fs/initramfs/initramfs.mk |   2 +-
 linux/linux.mk            |  20 ++++++++++----------
 2 files changed, 11 insertions(+), 11 deletions(-)

diff -r 01c5523b5777 -r 4dc6eccbfc41 fs/initramfs/initramfs.mk
@@ -7,7 +7,7 @@
 
 ROOTFS_INITRAMFS_DEPENDENCIES += rootfs-cpio
 
-ROOTFS_INITRAMFS_POST_TARGETS += linux26-rebuild-with-initramfs
+ROOTFS_INITRAMFS_POST_TARGETS += linux-rebuild-with-initramfs
 
 
 # The generic fs infrastructure isn't very useful here.

diff -r 01c5523b5777 -r 4dc6eccbfc41 linux/linux.mk
@@ -183,7 +183,7 @@
 	# As the kernel gets compiled before root filesystems are
 	# built, we create a fake cpio file. It'll be
 	# replaced later by the real cpio archive, and the kernel will be
-	# rebuilt using the linux26-rebuild-with-initramfs target.
+	# rebuilt using the linux-rebuild-with-initramfs target.
 	$(if $(BR2_TARGET_ROOTFS_INITRAMFS),
 		touch $(BINARIES_DIR)/rootfs.cpio
 		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_SOURCE,"$(BINARIES_DIR)/rootfs.cpio",$(@D)/.config)

@@ -312,24 +312,24 @@
 $(eval $(generic-package))
 
 ifeq ($(BR2_LINUX_KERNEL),y)
-linux-menuconfig linux-xconfig linux-gconfig linux-nconfig linux26-menuconfig linux26-xconfig linux26-gconfig linux26-nconfig: linux-configure
+linux-menuconfig linux-xconfig linux-gconfig linux-nconfig: linux-configure
 	$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) \
-		$(subst linux-,,$(subst linux26-,,$@))
+		$(subst linux-,,$@)
 	rm -f $(LINUX_DIR)/.stamp_{built,target_installed,images_installed}
 
-linux-savedefconfig linux26-savedefconfig: linux-configure
+linux-savedefconfig: linux-configure
 	$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) \
-		$(subst linux-,,$(subst linux26-,,$@))
+		$(subst linux-,,$@)
 
 ifeq ($(BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG),y)
-linux-update-config linux26-update-config: linux-configure $(LINUX_DIR)/.config
+linux-update-config: linux-configure $(LINUX_DIR)/.config
 	cp -f $(LINUX_DIR)/.config $(BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE)
 
-linux-update-defconfig linux26-update-defconfig: linux-savedefconfig
+linux-update-defconfig: linux-savedefconfig
 	cp -f $(LINUX_DIR)/defconfig $(BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE)
 else
-linux-update-config linux26-update-config: ;
-linux-update-defconfig linux26-update-defconfig: ;
+linux-update-config: ;
+linux-update-defconfig: ;
 endif
 endif
 

@@ -348,7 +348,7 @@
 
 # The initramfs building code must make sure this target gets called
 # after it generated the initramfs list of files.
-linux-rebuild-with-initramfs linux26-rebuild-with-initramfs: $(LINUX_DIR)/.stamp_initramfs_rebuilt
+linux-rebuild-with-initramfs: $(LINUX_DIR)/.stamp_initramfs_rebuilt
 
 # Checks to give errors that the user can understand
 ifeq ($(filter source,$(MAKECMDGOALS)),)

Dear Thomas De Schampheleire,

Weird, you don't have the summary of patches in the cover letter. Don't you use 'git --cover format-patch' to generate the cover letter?

Anyway:

* I've applied two preparation patches of this series:
uclibc: use $(MAKE) iso $(MAKE1) for menuconfig target
linux: remove support of linux26-* targets

This way, you don't have to carry them anymore.

* Regarding the kconfig-package infra stuff, I'm mostly fine with it, except for one detail: it pretends to be a package infrastructure, but it is not. It is only a kind of "library" / "helper" to use next to a real package infrastructure. And I think this is pretty confusing considering the name that was chosen, and the way it works.

My proposal would therefore be to turn it into a real package infrastructure by simply making it inherit from generic-package, much like autotools-package, cmake-package and al. All kconfig-based packages are otherwise generic-package. I don't see the logic behind using Kconfig for an autotools-based or cmake-based package, since autotools and cmake are precisely here to provide configuration capabilities that overlap with what kconfig provides. And as of today, we have linux, barebox, uclibc and busybox, and all of these use the generic-package infra.

I think it's really a minor change in the patches, and with this change, I'm willing to merge this package infra.

Of course, comments/reviews from others are more than welcome!

Thanks,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Hi Thomas,

Is this a trick question? ;-)
I'm using mercurial to send out patches, unfortunately it does not have a ready-made option to get this summary of patches. I have wanted to do that in the past too, so I'll write a little script to mimic the output of git here.

Thanks!

I never thought of it this way, but what you say makes perfect sense to me.

Great! I'm on it, hope to send out the adapted series soon...

Best regards,
Thomas

Thomas,

Might be :-)

You know, git is not *that* dangerous!

Cool!

Thanks,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

I know :)
I actually followed some documentation online, and am forcing myself to use it for some smaller tasks, but for actual development in buildroot I'm still much faster with Mercurial. I also doubt I will ever favor git over Mercurial, but it's always better to have an informed judgement.

Best regards,
Thomas

----- Mail original -----

<...>

Quick question... is it possible to relocate the .config file elsewhere using env variables ? I am still on my quest to move all project specific files (i.e configuration files) out of the buildroot directory to be able to build a buildroot project without touching the main buildroot directory...

Thx

J?r?my

Hi Jeremy,

Do you really mean the .config file (the destination file) or do you mean the file that contains the starting configuration, and that may be updated with foo-update-config ?

For the latter, individual packages provide config options for this, for example uclibc and busybox do. You can specify UCLIBC_CONFIG_FILE on the command line to overwrite it, similarly for BUSYBOX_CONFIG_FILE.

Best regards,
Thomas

----- Mail original -----

I don't like the way you have to separately save your kernel configuration after editing it (update after menuconfig) i'd rather not have to do that, but that's the general buildroot philosophy, so i'll follow it

so yes, what I want is the config file that will eventually be used by buildroot.

If these are overloadable from the command line, I should be good, thx

Hi Jeremy,

Another option might be to use the BR2_EXTERNAL concept to config manage your customization(s) outside of the main buildroot directory.

http://buildroot.uclibc.org/downloads/manual/manual.html#outside-br-custom

Matthew L Weber / Sr Software Engineer / Platform Software
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
Phone: 319-295-7349
matthew.weber at rockwellcollins.com
www.rockwellcollins.com

Note: Any Export License Required Information and License Restricted
Third Party Intellectual Property (TPIP) content must be encrypted and
sent to matthew.weber at corp.rockwellcollins.com.