br2-external support existing package customization
PATCH 1/1 br2-external support existing package customization
Tested on the pkg-generic.
BR2_PACKAGE_OVERLAY_DIR may be set as $(BR2_EXTERNAL)/package_overlay
pkgname1
pkgname1.mk
pkgname2
pkgname2.mk
Tested on the nginx package
nginx.mk
# Test overlay package
$(waning start overlay)
define NGINX_POST_PATCH_FIXUP
git clone https:/github.com/arut/nginx-rtmp-module.git $(@D)/rtmp
endefNGINX_CONF_OPT += --add-module=$(@D)/rtmp
$(warning NGINX_CONF_OPTS)
# -= analog
# _CONF_OPTS := $(filter-out option.$(_CONF_OPTS))
--- Config.in | 5 +++++ package/pkg-generic.mk | 4 ++++ package/pkg-utils.mk | 18 ++++++++++++++++++ 3 files changed, 27 insertions(+)
Config.in +5 -0
@@ -603,6 +603,11 @@ config BR2_GLOBAL_PATCH_DIR Otherwise, if the directory <global-patch-dir>/<packagename> exists, then all *.patch files in the directory will be applied. +config BR2_PACKAGE_OVERLAY_DIR + string "package overlay directory" + help + Dirictory with package overlay directory structure. + menu "Advanced" config BR2_COMPILER_PARANOID_UNSAFE_PATH
package/pkg-generic.mk +4 -0
@@ -304,6 +304,7 @@ be selected at a time. Please fix your configuration) endif endef + ################################################################################ # inner-generic-package -- generates the make targets needed to build a # generic package
@@ -341,6 +342,8 @@ endef define inner-generic-package +$(eval $(call package-overlay,$(call qstrip,$(BR2_PACKAGE_OVERLAY_DIR)),$(1)/$(1).mk)) + # Define default values for various package-related variables, if not # already defined. For some variables (version, source, site and # subdir), if they are undefined, we try to see if a variable without
@@ -895,6 +898,7 @@ endif endif # $(2)_KCONFIG_VAR endef # inner-generic-package + ################################################################################ # generic-package -- the target generator macro for generic packages ################################################################################
package/pkg-utils.mk +18 -0
@@ -150,3 +150,21 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET} mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \ cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2) endef + +define newline + + +endef + +newline-stub = __NEWLINE_STUB__ + +# package-overlay -- load and eval makefile in place +# example: +# $(eval $(call package-overlay,$(call qstrip,$(BR2_PACKAGE_OVERLAY_DIR)),RELATIVE_PATH_TO_THE_FILE)) +define package-overlay +ifneq ("$(1)","") +ifneq ($$(wildcard $(1)/$(2)),) +$$(eval $$(subst $$(newline-stub),$$(newline),$$(shell perl -p -e 's/\n/$$(newline-stub)/' $(1)/$(2)))) +endif +endif +endef
Signature · 1 line
1.9.1
PATCH 1/1 br2-external support existing package customization
Volkov, All,
Thanks you for your submission. :-)
I have some concerns with it, please read on...
On 2015-10-23 19:25 +0300, Volkov Viacheslav spake thusly:
Quoted 7 lines
Tested on the pkg-generic.
BR2_PACKAGE_OVERLAY_DIRmay be set as $(BR2_EXTERNAL)/package_overlay
pkgname1
pkgname1.mk
pkgname2
pkgname2.mk
Even though I understand your reason for wanting this feature (but see below for suggestion to solve it differently), I have to admit that I am not sure we ultimately want that in Buildroot.
It allows for too many opportunities to actually break things.
For example, suppose that a package has a version in Buldroot:
FOO_VERSION = 1.2.3
and an overlay overrides that to another version:
FOO_VERSION = 4.5.6
which is totally different from the one known to Buildroot, like uses a different build system (cmake instead of autotools).
Surely, this will break horribly.
And there are even worse sceanrios I can't even think of (well, I can, I have a wicked mind ;-) ).
Anyway, see below forsome comments nonetheless, about the implemenation...
Quoted 10 lines
Tested on the nginx package
nginx.mk
# Test overlay package
$(waning start overlay)
defineNGINX_POST_PATCH_FIXUPgit clone https:/github.com/arut/nginx-rtmp-module.git $(@D)/rtmpendef
NGINX_CONF_OPT+= --add-module=$(@D)/rtmp
$(warningNGINX_CONF_OPTS)
There are a few issues with how you are doing this.
First, the macro NGINX_POST_PATCH_FIXUP is never used; but I get that's an omission in the commit log and you assigned it to a post-patch hook in your tree.
Second, it looks like it is a post-patch hook (looking at the name of the macro). This is purely wrong, because we are not guaranteed to have network access at that time. Only download commands and their pre and post hooks are guaranteed to have network access. It's used by people that do not have a continuous internet connection, so they can connect, run 'make source' to get everything they need, and then disconnect.
Third, this does not provide a reproducible build, since you always end up using the lastest commit at the time of the clone, hence between two builds, there is no guarantee you'd get the same result.
Fourth, the licensing information is not updated to account for that new body of code being dropped (but fortunately, both are BSD-2c). Still, that code will not be part of the legal-info output.
However, the most obvious solution is to try and get something upstream in Buildroot. I would suggest that you update the existing nginx package, like so:
- in package/nginx/Config.in:
comment "external modules"
config BR2_PACKAGE_NGINX_RTMP_MODULE
bool "rtmp"
- in package/nginx/nginx.mk:
NGINX_RTMP_MODULE_VERSION = v1.1.7
NGINX_RTMP_MODULE_SOURCE = nginx-rtmp-module-$(NGINX_RTMP_MODULE_VERSION).tar.gz
NGINX_RTMP_MODULE_SITE = $(call github,arut,nginx-rtmp-module,$(NGINX_RTMP_MODULE_VERSION))
NGINX_EXTRA_DOWNLOADS = $(NGINX_RTMP_MODULE_SITE)/$(NGINX_RTMP_MODULE_SOURCE)
ifeq ($(BR2_PACKAGE_NGINX_RTMP_MODULE),y)
define NGINX_RTMP_MODULE_EXTRACT
mkdir -p $(@D)/rtmp
$(call suitable-extractor,$(NGINX_RTMP_MODULE_SOURCE)) \
$(DL_DIR)/$(NGINX_RTMP_MODULE_SOURCE) |\
$(TAR) --strip-components=1 -C $(@D)/rtmp $(TAR_OPTIONS) -
endef
NGINX_POST_EXTRACT_HOOKS += NGINX_RTMP_MODULE_EXTRACT
NGINX_CONF_OPTS += --add-module=$(@D)/rtmp
endif # BR2_PACKAGE_NGINX_RTMP_MODULEAlternatively, I guess the same can be achieved by putting that code directly in a br2-external tree and including the files as-is. Since the variables are expanded only at the moment they are needed, you can well complement existing variables from a br2-external tree (even though that is really ugly).
Quoted 4 lines
# -= analog
#_CONF_OPTS:= $(filter-out option.$(_CONF_OPTS))Signed-off-by sv99 <[email protected]>
You must use your real name here.
[--SNIP--]
package/pkg-generic.mk endef · line 345
Quoted 9 lines
package/pkg-generic.mk +1
-0@@ -341,6 +342,8 @@ endef define inner-generic-package +$(eval $(call package-overlay,$(call qstrip,$(BR2_PACKAGE_OVERLAY_DIR)),$(1)/$(1).mk))
Why not simply something like:
ifneq ($$(BR2_PACKAGE_OVERLAY_DIR),)
-include $$(BR2_PACKAGE_OVERLAY_DIR)/$(1)/$(1).mk
endifNotes:
- the include directive starts with a dash '-', so that make will not
complain if the included file does not exist;
- the variable is double-dollared, because we're inside the -inner macro; read the text above it for explanations.
This should be just enough, and would not need the weird macro with the weird and convoluted perl stuff below.
An alternative would be:
ifneq ($$(BR2_PACKAGE_OVERLAY_DIR),)
ifneq ($$(wildcard $$(BR2_PACKAGE_OVERLAY_DIR)/$(1)/$(1).mk),)
$$(warning Applying overlay for package $(1))
include $$(BR2_PACKAGE_OVERLAY_DIR)/$(1)/$(1).mk
endif
endifWhich still does not require the weird package-overlay macro and, as an added bonus, warns the user about which package is being overlaid.
package/pkg-generic.mk endif · line 901
Quoted 8 lines
package/pkg-generic.mk +1
-0# Define default values for various package-related variables, if not # already defined. For some variables (version, source, site and # subdir), if they are undefined, we try to see if a variable without@@ -895,6 +898,7 @@ endif endif # $(2)_KCONFIG_VAR endef # inner-generic-package +
Please avoid spurious empty lines like this (second occurence).
Quoted 8 lines
package/pkg-generic.mk +0
-0################################################################################ # generic-package -- the target generator macro for generic packages ################################################################################package/pkg-utils.mk +0
-0
package/pkg-utils.mk define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET} · line 157
Quoted 8 lines
package/pkg-utils.mk +5
-0@@ -150,3 +150,21 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET} mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \ cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2) endef + +define newline + + +endef
We already have $(newline); it called $(sep).
package/pkg-utils.mk define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET} · line 167
Quoted 9 lines
package/pkg-utils.mk +9
-0+newline-stub = __NEWLINE_STUB__ + +# package-overlay -- load and eval makefile in place +# example: +# $(eval $(call package-overlay,$(call qstrip,$(BR2_PACKAGE_OVERLAY_DIR)),RELATIVE_PATH_TO_THE_FILE)) +define package-overlay +ifneq ("$(1)","") +ifneq ($$(wildcard $(1)/$(2)),) +$$(eval $$(subst $$(newline-stub),$$(newline),$$(shell perl -p -e 's/\n/$$(newline-stub)/' $(1)/$(2))))
Eeek! :-(
Seriously? :-/
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 br2-external support existing package customization
This my first patch sended throu git send-mail. Excuse me my wary bad english.
I have customized clone old version buildroot. Most packets not changed. Some additional packages - good solution for BR2_EXTERNAL.
Two packet have changed CONF_OPTS.
This solution for simple customization evailable packages. Add or remove single CONF params.
Include not worked in the eval environment,? after $(eval $(generic-package)) in the make. Code in the included make will executed after eval fully executed - for customization purposes needs run code in the first lines generic-package macro!!
My solution executed additional make lines readed from file in the special directory.
Version with git clone rtmp worked in old version and not changed, I simple tested working for this code. You version with NGINX_POST_EXTRACT_HOOKS looked wery well.
In current version buildroot not allowed overload or tuning existing packages without patching original tree.
Thanks
Quoted 200 lines
???????, 24 ??????? 2015, 0:41 +02:00 ?? "Yann E. MORIN" <[email protected]>:
Volkov, All,
Thanks you for your submission. :-)
I have some concerns with it, please read on...
On 2015-10-23 19:25 +0300, Volkov Viacheslav spake thusly:
Quoted 7 lines
Tested on the pkg-generic.
BR2_PACKAGE_OVERLAY_DIRmay be set as $(BR2_EXTERNAL)/package_overlay
pkgname1
pkgname1.mk
pkgname2
pkgname2.mkEven though I understand your reason for wanting this feature (but see below for suggestion to solve it differently), I have to admit that I am not sure we ultimately want that in Buildroot.
It allows for too many opportunities to actually break things.
For example, suppose that a package has a version in Buldroot:
????FOO_VERSION = 1.2.3
and an overlay overrides that to another version:
????FOO_VERSION = 4.5.6
which is totally different from the one known to Buildroot, like uses a different build system (cmake instead of autotools).
Surely, this will break horribly.
And there are even worse sceanrios I can't even think of (well, I can, I have a wicked mind ;-) ).
Anyway, see below forsome comments nonetheless, about the implemenation...
Quoted 10 lines
Tested on the nginx package
nginx.mk
# Test overlay package
$(waning start overlay)
defineNGINX_POST_PATCH_FIXUPgit clone https:/github.com/arut/nginx-rtmp-module.git $(@D)/rtmpendef
NGINX_CONF_OPT+= --add-module=$(@D)/rtmp
$(warningNGINX_CONF_OPTS)There are a few issues with how you are doing this.
First, the macro
NGINX_POST_PATCH_FIXUPis never used; but I get that's an omission in the commit log and you assigned it to a post-patch hook in your tree.Second, it looks like it is a post-patch hook (looking at the name of the macro). This is purely wrong, because we are not guaranteed to have network access at that time. Only download commands and their pre and post hooks are guaranteed to have network access. It's used by people that do not have a continuous internet connection, so they can connect, run 'make source' to get everything they need, and then disconnect.
Third, this does not provide a reproducible build, since you always end up using the lastest commit at the time of the clone, hence between two builds, there is no guarantee you'd get the same result.
Fourth, the licensing information is not updated to account for that new body of code being dropped (but fortunately, both are BSD-2c). Still, that code will not be part of the legal-info output.
However, the most obvious solution is to try and get something upstream in Buildroot. I would suggest that you update the existing nginx package, like so:
??- in package/nginx/Config.in:
????comment "external modules"
????config
BR2_PACKAGE_NGINX_RTMP_MODULE
????????bool "rtmp"??- in
package/nginx/nginx.mk:????NGINX_RTMP_MODULE_VERSION = v1.1.7
????NGINX_RTMP_MODULE_SOURCE = nginx-rtmp-module-$(NGINX_RTMP_MODULE_VERSION).tar.gz ????NGINX_RTMP_MODULE_SITE = $(call github,arut,nginx-rtmp-module,$(NGINX_RTMP_MODULE_VERSION))????NGINX_EXTRA_DOWNLOADS = $(NGINX_RTMP_MODULE_SITE)/$(NGINX_RTMP_MODULE_SOURCE)
????ifeq ($(BR2_PACKAGE_NGINX_RTMP_MODULE),y)
????defineNGINX_RTMP_MODULE_EXTRACT
????????mkdir -p $(@D)/rtmp
????????$(call suitable-extractor,$(NGINX_RTMP_MODULE_SOURCE)) \
????????????$(DL_DIR)/$(NGINX_RTMP_MODULE_SOURCE) |\
????????????$(TAR) --strip-components=1 -C $(@D)/rtmp $(TAR_OPTIONS) - ????endef
????NGINX_POST_EXTRACT_HOOKS +=NGINX_RTMP_MODULE_EXTRACT????NGINX_CONF_OPTS += --add-module=$(@D)/rtmp
????endif #BR2_PACKAGE_NGINX_RTMP_MODULEAlternatively, I guess the same can be achieved by putting that code directly in a br2-external tree and including the files as-is. Since the variables are expanded only at the moment they are needed, you can well complement existing variables from a br2-external tree (even though that is really ugly).
Quoted 4 lines
# -= analog
#_CONF_OPTS:= $(filter-out option.$(_CONF_OPTS))Signed-off-by sv99 < [email protected] >You must use your real name here.
[--SNIP--]
package/pkg-generic.mk endef · line 345
Quoted 9 lines
package/pkg-generic.mk +1
-0@@ -341,6 +342,8 @@ endef define inner-generic-package +$(eval $(call package-overlay,$(call qstrip,$(BR2_PACKAGE_OVERLAY_DIR)),$(1)/$(1).mk))Why not simply something like:
????ifneq ($$(BR2_PACKAGE_OVERLAY_DIR),) ????-include $$(BR2_PACKAGE_OVERLAY_DIR)/$(1)/$(1).mk ????endif
Notes:
??- the include directive starts with a dash '-', so that make will not ????complain if the included file does not exist;
??- the variable is double-dollared, because we're inside the -inner ????macro; read the text above it for explanations.This should be just enough, and would not need the weird macro with the weird and convoluted perl stuff below.
An alternative would be:
????ifneq ($$(BR2_PACKAGE_OVERLAY_DIR),)
????ifneq ($$(wildcard $$(BR2_PACKAGE_OVERLAY_DIR)/$(1)/$(1).mk),) ????$$(warning Applying overlay for package $(1)) ????include $$(BR2_PACKAGE_OVERLAY_DIR)/$(1)/$(1).mk ????endif
????endifWhich still does not require the weird package-overlay macro and, as an added bonus, warns the user about which package is being overlaid.
package/pkg-generic.mk endif · line 901
Quoted 8 lines
package/pkg-generic.mk +1
-0# Define default values for various package-related variables, if not # already defined. For some variables (version, source, site and # subdir), if they are undefined, we try to see if a variable without@@ -895,6 +898,7 @@ endif endif # $(2)_KCONFIG_VAR endef # inner-generic-package +Please avoid spurious empty lines like this (second occurence).
Quoted 8 lines
package/pkg-generic.mk +0
-0################################################################################ # generic-package -- the target generator macro for generic packages ################################################################################package/pkg-utils.mk +0
-0package/pkg-utils.mk define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET} · line 157
Quoted 8 lines
package/pkg-utils.mk +5
-0@@ -150,3 +150,21 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET} mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \ cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2) endef + +define newline + + +endefWe already have $(newline); it called $(sep).
package/pkg-utils.mk define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET} · line 167
Quoted 9 lines
package/pkg-utils.mk +9
-0+newline-stub = __NEWLINE_STUB__ + +# package-overlay -- load and eval makefile in place +# example: +# $(eval $(call package-overlay,$(call qstrip,$(BR2_PACKAGE_OVERLAY_DIR)),RELATIVE_PATH_TO_THE_FILE)) +define package-overlay +ifneq ("$(1)","") +ifneq ($$(wildcard $(1)/$(2)),) +$$(eval $$(subst $$(newline-stub),$$(newline),$$(shell perl -p -e 's/\n/$$(newline-stub)/' $(1)/$(2))))Eeek! :-(
Seriously? :-/
Quoted 14 lines
Regards,
Yann E. MORIN.Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
? ?????????, ?????? ????????!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151024/55fe8853/attachment.html>
PATCH 1/1 br2-external support existing package customization
On 2015-10-24 17:38 +0300, ?????? ???????? spake thusly:
Quoted 2 lines
This my first patch sended throu git send-mail. Excuse me my wary bad english.
No problem! ;-) Not everyone is a native english speaker, don't worry.
But please:
- do not post HTML mail;
- wrap your lines to ~72 chars.
Quoted 1 line
I have customized clone old version buildroot.
Then you should probably update to the latest version.
Quoted 4 lines
Two packet have changed
CONF_OPTS.This solution for simple customization evailable packages. Add or remove single CONF params.
I understand that your solution matches your need to add additional module support in nginx.
But even though your use-case is mostly harmless, providing this feature would open too many possibilities to break the build in non-trivial ways, which will be hard to debug (know that we the Buildroot community tries to provide support for users and help them when something fails, and this feature would be too problematic to support).
Quoted 4 lines
Include not worked in the eval environment, after $(eval $(generic-package)) in the make. Code in the included make will executed after eval fully executed - for customization purposes needs run code in the first lines generic-package macro!!
I made it to work with a very slight change to what I already posted, see:
Config.in +3 -0
@@ -2,6 +2,9 @@ mainmenu "Buildroot $BR2_VERSION Configuration" +config BR2_OVERLAY_DIR + string "overlay dir" + config BR2_HAVE_DOT_CONFIG bool default y
Makefile +2 -0
@@ -218,6 +218,8 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),) -include $(BR2_CONFIG) endif +BR_OVERLAY_DIR := $(call qstrip,$(BR2_OVERLAY_DIR)) + # To put more focus on warnings, be less verbose as default # Use 'make V=1' to see the full commands ifeq ("$(origin V)", "command line")
package/pkg-generic.mk +4 -0
@@ -341,6 +341,10 @@ endef define inner-generic-package +ifneq ($$(BR_OVERLAY_DIR),) +-include $$(BR_OVERLAY_DIR)/$(1)/$(1).mk +endif + # Define default values for various package-related variables, if not # already defined. For some variables (version, source, site and # subdir), if they are undefined, we try to see if a variable without
See, the trick was to un-quote $(BR2_OVERLAY_DIR) with the 'qstrip' function. And then to use $$ to dereference the BR_OVERLAY_DIR variable, since it is in an inner macro.
Quoted 2 lines
Version with git clone rtmp worked in old version and not changed, I simple tested working for this code.
The problem is not that it worked in an old version of Buildroot, but as time passes, the rtmp module source tree will change, and the next time you want to rebuild your project, then you will not get the same source code for the rtmp module, since you clone the repository and use the latest commit from the master branch.
Hence, the build is not reproducible.
Quoted 1 line
You version with
NGINX_POST_EXTRACT_HOOKSlooked wery well.
I believe it is working (but it may need a few tweaks), and so completely removes the need for your overlay proposal. Just put that ion your BR2_EXTERNAL tree, and you should be set.
But better yet: send a patch so we can have that support directly in Buildroot and you will not need to patch Buildroot.
Quoted 2 lines
In current version buildroot not allowed overload or tuning existing packages without patching original tree.
No, that's indeed not supported. And as I said previosuly, I am not sure we would want to support that, even with my proposal above which is in my humble opinion much cleaner than your convoluted eval+call+perl solution.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'