Add find utility functions / exclude files from strip / reduce python
PATCH v2 1/1 Add find utility functions / exclude files from strip / reduce python
This patch series adds a few utility functions to Makefile.package.in that help in composing find commands, and uses these functions to:
* exclude user-configured files and directories from being stripped in target-finalize, and
* reduce a python installation to the bare minimum.
--- Config.in | 19 +++++++++++++++++++ Makefile | 9 +++++++-- package/Makefile.package.in | 11 +++++++++++ package/python/Config.in | 26 ++++++++++++++++++++++---- package/python/python.mk | 28 +++++++++++++++++++++++++++- 5 files changed, 86 insertions(+), 7 deletions(-)
PATCH v2 1/1 Makefile.package.in: add utility functions find*clauses and notfirstword
# HG changeset patch
# User Thomas De Schampheleire <[email protected]> # Date 1318518890 -7200
# Node ID b6579582d02dd802adca94e2b38972cc19cd8b76
# Parent baa870774b887fb01a9de0cc69161f250553f896 Makefile.package.in: add utility functions find*clauses and notfirstword
This patch adds a few utility functions to Makefile.package.in.
Functions finddirclauses and findfileclauses help in building a find command that skips a set of directories and performs operations on a set of files. This pattern can for example be used to keep certain files or directories from being stripped, or to remove certain files from a package installation.
The notfirstword function is the inverse of the 'firstword' function in make:
it returns all but the first word.
---
v2: integrate _cont variants in main find*clauses functions. Thanks, Arnout.
package/Makefile.package.in | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)
package/Makefile.package.in +11 -0
@@ -87,6 +87,17 @@ MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG) TERM_BOLD := $(shell tput smso) TERM_RESET := $(shell tput rmso) +# Utility functions for 'find' +# findfileclauses: creates "-name 'X' -o -name 'Y'" +# [1:namelist] +findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1))) +# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'" +# [1:basedir, 2:namelist] +finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2))) + +# Miscellaneous utility functions +notfirstword=$(wordlist 2,$(words $(1)),$(1)) + # Download method commands WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET) SVN:=$(call qstrip,$(BR2_SVN))
PATCH v2 1/1 stripping: use findfileclauses utility function
In reply to Thomas De Schampheleire# HG changeset patch
# User Thomas De Schampheleire <[email protected]> # Date 1318576710 -7200
# Node ID 26239e4561037c19ccfc7ac7ea216d36a6a8e87e
# Parent b6579582d02dd802adca94e2b38972cc19cd8b76
stripping: use findfileclauses utility function
--- Makefile | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
Makefile +1 -1
@@ -454,7 +454,7 @@ endif ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y) find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f endif - find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \ + find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \ xargs $(STRIPCMD) 2>/dev/null || true find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \ xargs -r $(KSTRIPCMD) || true
PATCH v2 1/1 build: add option to exclude executables/dirs from being stripped
In reply to Thomas De Schampheleire# HG changeset patch
# User Sven Neumann <[email protected]>
# Date 1314025139 -7200
# Node ID 76f1a5eeba5e79cb841338d5317e3c618bf7c93c
# Parent 26239e4561037c19ccfc7ac7ea216d36a6a8e87e
build: add option to exclude executables/dirs from being stripped
Sometimes it may be desirable to keep debug symbols for some binaries and libraries on the target. This commit introduces the config option BR2_STRIP_EXCLUDE_FILES, which is interpreted as a list of such binaries and libraries, and the option BR2_STRIP_EXCLUDE_DIRS, which indicates directories excluded from stripping entirely.
These exclusions are passed to the find command in the target-finalize step.
---
v2: add the possibility to exclude entire directories
Config.in | 19 +++++++++++++++++++ Makefile | 9 +++++++-- 2 files changed, 26 insertions(+), 2 deletions(-)
Config.in +19 -0
@@ -247,6 +247,25 @@ config BR2_STRIP_none none do not strip (only for debugging!) endchoice +config BR2_STRIP_EXCLUDE_FILES + string "executables that should not be stripped" + depends on !BR2_STRIP_none + default "" + help + You may specify a space-separated list of binaries and libraries + here that should not be stripped on the target. + +config BR2_STRIP_EXCLUDE_DIRS + string "directories that should be skipped when stripping" + depends on !BR2_STRIP_none + default "" + help + You may specify a space-separated list of directories that should + be skipped when stripping. Binaries and libraries in these + directories will not be touched. + The directories should be specified relative to the target directory, + without leading slash. + choice prompt "gcc optimization level" default BR2_OPTIMIZE_S
Makefile +7 -2
@@ -430,6 +430,12 @@ endif erase-fakeroots: rm -f $(BUILD_DIR)/.fakeroot* +STRIP_FIND_CMD=find $(TARGET_DIR) +ifneq (,$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) +STRIP_FIND_CMD+=\( $(call finddirclauses,$(TARGET_DIR),$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) \) -prune -o +endif +STRIP_FIND_CMD+=-type f -perm +111 -not \( $(call findfileclauses,libthread_db*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print + target-finalize: ifeq ($(BR2_HAVE_DEVFILES),y) ( support/scripts/copy.sh $(STAGING_DIR) $(TARGET_DIR) )
@@ -454,8 +460,7 @@ endif ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y) find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f endif - find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \ - xargs $(STRIPCMD) 2>/dev/null || true + $(STRIP_FIND_CMD) | xargs $(STRIPCMD) 2>/dev/null || true find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \ xargs -r $(KSTRIPCMD) || true
PATCH v2 1/1 python config: move configuration into menu
In reply to Thomas De Schampheleire# HG changeset patch
# User Thomas De Schampheleire <[email protected]> # Date 1318517686 -7200
# Node ID 97f99d539f586caa866009c9702596c63f3637fe
# Parent 76f1a5eeba5e79cb841338d5317e3c618bf7c93c
python config: move configuration into menu
Group all Python-related configuration in a separate menuconfig.
--- package/python/Config.in | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
package/python/Config.in +4 -4
@@ -1,4 +1,7 @@ -config BR2_PACKAGE_PYTHON +comment "python requires a toolchain with WCHAR support" + depends on !BR2_USE_WCHAR + +menuconfig BR2_PACKAGE_PYTHON bool "python" depends on BR2_USE_WCHAR select BR2_PACKAGE_LIBFFI
@@ -7,9 +10,6 @@ config BR2_PACKAGE_PYTHON http://www.python.org/ -comment "python requires a toolchain with WCHAR support" - depends on !BR2_USE_WCHAR - if BR2_PACKAGE_PYTHON choice
PATCH v2 1/1 python: allow reduced installation
In reply to Thomas De Schampheleire# HG changeset patch
# User Thomas De Schampheleire <[email protected]> # Date 1318517171 -7200
# Node ID 3fb04c960d0e58ea5bb10ecbed382fc1c41c39eb
# Parent 97f99d539f586caa866009c9702596c63f3637fe
python: allow reduced installation
A default python installation takes more than 13M of disk space on target. The python library takes about 4,5M, and the other 9M is occupied by various modules in /usr/lib/python2.7. A usable installation doesn't need such a large module directory. A minimal configuration with only those modules needed to start an interactive session requires only about 450K, plus the python library itself.
This patch adds an option to reduce the number of installed python modules to the bare minimum. Additional modules to install can be specified in the config file as a string.
---
As an example, to be able to run the 'dstat' tool, I had to add the following extra modules:
BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES="__future__.pyc sched.pyc heapq.pyc bisect.pyc collections.pyc keyword.pyc getopt.pyc fnmatch.pyc getpass.pyc glob.pyc time.so itertools.so operator.so resource.so _collections.so"
package/python/Config.in | 18 ++++++++++++++++++ package/python/python.mk | 28 +++++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 1 deletions(-)
package/python/Config.in +18 -0
@@ -95,4 +95,22 @@ config BR2_PACKAGE_PYTHON_ZLIB endmenu + +config BR2_PACKAGE_PYTHON_REDUCED + bool "reduced python installation" + help + Select this option to greatly reduce the number of python modules that are + installed on target (from approx. 9M to 400K). Only the modules needed to + start an interactive python session are installed by default. + You can install additional modules by specifying them in + BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES. + +config BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES + string "extra modules to include in reduced python installation" + depends on BR2_PACKAGE_PYTHON_REDUCED + help + To install additional modules in a reduced Python installation, + specify them here, space-separated. This list should only include the + file name, not the directory part. + endif
package/python/python.mk +27 -1
@@ -144,12 +144,14 @@ ifneq ($(BR2_HAVE_DEVFILES),y) PYTHON_POST_INSTALL_TARGET_HOOKS += PYTHON_REMOVE_DEVFILES endif +PYTHON_TARGET_DIR=$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) + # # Remove useless files. In the config/ directory, only the Makefile # and the pyconfig.h files are needed at runtime. # define PYTHON_REMOVE_USELESS_FILES - for i in `find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/config/ \ + for i in `find $(PYTHON_TARGET_DIR)/config/ \ -type f -not -name pyconfig.h -a -not -name Makefile` ; do \ rm -f $$i ; \ done
@@ -157,6 +159,30 @@ endef PYTHON_POST_INSTALL_TARGET_HOOKS += PYTHON_REMOVE_USELESS_FILES +# +# Reduced Python installation: remove unneeded modules +# +ifeq ($(BR2_PACKAGE_PYTHON_REDUCED),y) +PYTHON_REDUCED_MINIMAL_MODULES=$(patsubst %,%.pyc,\ + site os stat posixpath genericpath warnings linecache types UserDict \ + _abcoll abc _weakrefset copy_reg sysconfig re sre_compile sre_parse \ + sre_constants) \ + $(call qstrip,$(BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES)) +PYTHON_REDUCED_MINIMAL_DIRS=config + +PYTHON_REDUCED_FIND_CMD=find $(PYTHON_TARGET_DIR) +PYTHON_REDUCED_FIND_CMD+=\( $(call finddirclauses,$(PYTHON_TARGET_DIR),$(PYTHON_REDUCED_MINIMAL_DIRS)) \) -prune -o +PYTHON_REDUCED_FIND_CMD += -type f -not \( $(call findfileclauses,$(PYTHON_REDUCED_MINIMAL_MODULES)) \) -print + +define PYTHON_REDUCE_INSTALLATION + for i in `$(PYTHON_REDUCED_FIND_CMD)`; do \ + rm -f $$i ; \ + done +endef + +PYTHON_POST_INSTALL_TARGET_HOOKS += PYTHON_REDUCE_INSTALLATION +endif + PYTHON_AUTORECONF = YES $(eval $(call AUTOTARGETS))
PATCH v2 1/1 Makefile.package.in: add utility functions find*clauses and notfirstword
In reply to Thomas De SchampheleireLe Thu, 17 Nov 2011 20:41:48 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :
Quoted 5 lines
Functions finddirclauses and findfileclauses help in building a find command that skips a set of directories and performs operations on a set of files. This pattern can for example be used to keep certain files or directories from being stripped, or to remove certain files from a package installation.
I must say I am not a big fan of this. Those macros are horribly complicated to understand and they make the code really cryptic. I think I would really prefer to see the implementation of these "do not strip this file/directory" option to be done without those cryptic macros.
Regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 1/1 python: allow reduced installation
In reply to Thomas De SchampheleireLe Thu, 17 Nov 2011 20:42:04 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :
Quoted 10 lines
A default python installation takes more than 13M of disk space on target. The python library takes about 4,5M, and the other 9M is occupied by various modules in
/usr/lib/python2.7. A usable installation doesn't need such a large module directory. A minimal configuration with only those modules needed to start an interactive session requires only about 450K, plus the python library itself.This patch adds an option to reduce the number of installed python modules to the bare minimum. Additional modules to install can be specified in the config file as a string.
While I agree that the feature is nice, I am not a fan of the implementation. Currently, we already have a way of enabling/disabling certain Python modules with one Kconfig option for each module that can be enabled/disabled. The list of modules that can be enabled/disabled this way has been built taking into account two factors :
* We want to be able to configure out modules that have external dependencies (on other libraries)
* We want to be able to configure certain modules that take a huge amount of space (I'm thinking of all the japanese/korean handling) and may not be useful in most situations
So we already have a way of configuring which modules are built/installed with one Kconfig option per-module.
Now, you're adding a new option "I want a minimal version of Python" with an additional option "but I also want those additional modules". I think that taken together with the existing per-module Kconfig options, it makes the Python configuration very strange. I am not sure what to propose exactly (having one Kconfig option for each and every module would be horrible), but I don't like really much this proposal.
Regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 1/1 Makefile.package.in: add utility functions find*clauses and notfirstword
In reply to Thomas PetazzoniHi Thomas,
Quoted 9 lines · Thomas Petazzoni
Le Thu, 17 Nov 2011 20:41:48 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :Quoted 5 lines
Functions finddirclauses and findfileclauses help in building a find command that skips a set of directories and performs operations on a set of files. This pattern can for example be used to keep certain files or directories from being stripped, or to remove certain files from a package installation.
Quoted 5 lines
I must say I am not a big fan of this. Those macros are horribly complicated to understand and they make the code really cryptic. I think I would really prefer to see the implementation of these "do not strip this file/directory" option to be done without those cryptic macros.
Do you mean to add the contents of the macro directly in the target-finalize target? I fear this will not make target-finalize much clearer.
My reasoning of putting this in a macro, is that the complexity is just in one place. If a user can trust that the macro is right, he doesn't have to care about it.
Also, I don't immediately see a simpler way of adding such -name clauses to the find command, based on user input.
Thanks for your feedback,
Thomas
PATCH v2 1/1 stripping: use findfileclauses utility function
In reply to Thomas De SchampheleireQuoted 8 lines · On Thursday 17 November 2011 19:41:56 Thomas De Schampheleire wrote:
# HG changeset patch
# User Thomas De Schampheleire <[email protected]> # Date 1318576710 -7200
# Node ID26239e4561037c19ccfc7ac7ea216d36a6a8e87e
# Parentb6579582d02dd802adca94e2b38972cc19cd8b76
stripping: use findfileclauses utility functionSigned-off-by Thomas De Schampheleire <[email protected]>
I think this patch is redundant: it doesn't simplify anything, nobody would
want to apply or revert it individually.
Regards,
Arnout
Quoted 23 lines
--- Makefile | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)Makefile +1
-1@@ -454,7 +454,7 @@ endif ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y) find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f endif - find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \ + find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \ xargs $(STRIPCMD) 2>/dev/null || true find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \ xargs -r $(KSTRIPCMD) || true_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 31BB CF53 8660 6F88 345D 54CC A836 5879 20D7 CF43
PATCH v2 1/1 Makefile.package.in: add utility functions find*clauses and notfirstword
In reply to Thomas De SchampheleireQuoted 6 lines · On Thursday 17 November 2011 19:41:48 Thomas De Schampheleire wrote:
# HG changeset patch
# User Thomas De Schampheleire <[email protected]> # Date 1318518890 -7200
# Node IDb6579582d02dd802adca94e2b38972cc19cd8b76
# Parentbaa870774b887fb01a9de0cc69161f250553f896Makefile.package.in: add utility functions find*clauses and notfirstword
Is it possible to tell hg to remove these lines? git-am will include them
in the commit message...
Quoted 8 lines
This patch adds a few utility functions to Makefile.package.in.
Functions finddirclauses and findfileclauses help in building a find command that skips a set of directories and performs operations on a set of files. This pattern can for example be used to keep certain files or directories from being stripped, or to remove certain files from a package installation.
Quoted 4 lines
The notfirstword function is the inverse of the 'firstword' function in make:
it returns all but the first word.Signed-off-by Thomas De Schampheleire <[email protected]>
I think you could safely have added a
And to ThomasP: this helper does simplify patch 3/5 a lot. Even though
the macros look a bit ugly indeed.
Regards,
Arnout
Quoted 34 lines
---
v2: integrate_contvariants in main find*clauses functions. Thanks, Arnout.package/Makefile.package.in | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)package/Makefile.package.in +11
-0@@ -87,6 +87,17 @@ MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG) TERM_BOLD := $(shell tput smso) TERM_RESET := $(shell tput rmso) +# Utility functions for 'find' +# findfileclauses: creates "-name 'X' -o -name 'Y'" +# [1:namelist] +findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1))) +# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'" +# [1:basedir, 2:namelist] +finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2))) + +# Miscellaneous utility functions +notfirstword=$(wordlist 2,$(words $(1)),$(1)) + # Download method commands WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET) SVN:=$(call qstrip,$(BR2_SVN))_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 31BB CF53 8660 6F88 345D 54CC A836 5879 20D7 CF43
PATCH v2 1/1 python: allow reduced installation
In reply to Thomas PetazzoniQuoted 30 lines · On Thursday 17 November 2011 19:55:01 Thomas Petazzoni wrote:
Le Thu, 17 Nov 2011 20:42:04 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :Quoted 10 lines
A default python installation takes more than 13M of disk space on target. The python library takes about 4,5M, and the other 9M is occupied by various modules in
/usr/lib/python2.7. A usable installation doesn't need such a large module directory. A minimal configuration with only those modules needed to start an interactive session requires only about 450K, plus the python library itself.This patch adds an option to reduce the number of installed python modules to the bare minimum. Additional modules to install can be specified in the config file as a string.
While I agree that the feature is nice, I am not a fan of the implementation. Currently, we already have a way of enabling/disabling certain Python modules with one
Kconfigoption for each module that can be enabled/disabled. The list of modules that can be enabled/disabled this way has been built taking into account two factors :* We want to be able to configure out modules that have external dependencies (on other libraries)
* We want to be able to configure certain modules that take a huge amount of space (I'm thinking of all the japanese/korean handling) and may not be useful in most situations
So we already have a way of configuring which modules are built/installed with one
Kconfigoption per-module.
Quoted 6 lines
Now, you're adding a new option "I want a minimal version of Python" with an additional option "but I also want those additional modules". I think that taken together with the existing per-module
Kconfigoptions, it makes the Python configuration very strange. I am not sure what to propose exactly (having oneKconfigoption for each and every module would be horrible), but I don't like really much this proposal.
Maybe a middle way is to have the reduced setup by default, with aBR2_PACKAGE_PYTHON_EXTRA_MODULES option and another option to enable all 'usual' modules (which would disable the EXTRA_MODULES option). At a later stage, more options for 'usual' modules can be added.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 31BB CF53 8660 6F88 345D 54CC A836 5879 20D7 CF43
PATCH v2 1/1 stripping: use findfileclauses utility function
In reply to Thomas De SchampheleireMakefile endif · line 457
Quoted 9 lines · Thomas De Schampheleire
Makefile +0
-0@@ -454,7 +454,7 @@ endif ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y) find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -fendif
Makefile +1
-1- find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \ + find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \
This is less readable than what was there before. The findfileclauses macro only makes sense when it's expanding input from a variable. Literals should stay as they are...
Quoted 2 lines
xargs $(STRIPCMD) 2>/dev/null || true find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
... just like this one.
PATCH v2 1/1 stripping: use findfileclauses utility function
Quoted 16 lines · Cam Hutchison
Quoted 9 lines · Thomas De Schampheleire
Makefile +0
-0@@ -454,7 +454,7 @@ endif ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)? ? ? find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
endif
- ? ? ?find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \
+ ? ? ?find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \This is less readable than what was there before. The findfileclauses macro only makes sense when it's expanding input from a variable. Literals should stay as they are...
Quoted 4 lines
Quoted 2 lines
? ? ? ? ? ? ? xargs $(STRIPCMD) 2>/dev/null || true ? ? ? find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
... just like this one.
The problem arises when you combine such a literal with files from a
variable: the variable may be empty or not, and then you don't know whether or not to add '-o'.
If you do the following:
find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' -o $(call findfileclauses,$(BR2_EXCLUDE_FILES))
while BR2_EXCLUDE_FILES is empty, the syntax is incorrect. Changing the order doesn't help:
find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,$(BR2_EXCLUDE_FILES)) -o -name 'libthread_db*.so*'
I agree that this patch on its own is not good. However, I think it'll be necessary in combination with the BR2_EXCLUDE_FILES patch.
Best regards,
Thomas
PATCH v2 1/1 python: allow reduced installation
In reply to Thomas PetazzoniHi Thomas,
Quoted 30 lines · Thomas Petazzoni
Le Thu, 17 Nov 2011 20:42:04 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :Quoted 10 lines
A default python installation takes more than 13M of disk space on target. The python library takes about 4,5M, and the other 9M is occupied by various modules in
/usr/lib/python2.7. A usable installation doesn't need such a large module directory. A minimal configuration with only those modules needed to start an interactive session requires only about 450K, plus the python library itself.This patch adds an option to reduce the number of installed python modules to the bare minimum. Additional modules to install can be specified in the config file as a string.
While I agree that the feature is nice, I am not a fan of the implementation. Currently, we already have a way of enabling/disabling certain Python modules with one
Kconfigoption for each module that can be enabled/disabled. The list of modules that can be enabled/disabled this way has been built taking into account two factors :?* We want to be able to configure out modules that have external ? dependencies (on other libraries)
?* We want to be able to configure certain modules that take a huge ? amount of space (I'm thinking of all the japanese/korean handling) ? and may not be useful in most situations
So we already have a way of configuring which modules are built/installed with one
Kconfigoption per-module.
Quoted 6 lines
Now, you're adding a new option "I want a minimal version of Python" with an additional option "but I also want those additional modules". I think that taken together with the existing per-module
Kconfigoptions, it makes the Python configuration very strange. I am not sure what to propose exactly (having oneKconfigoption for each and every module would be horrible), but I don't like really much this proposal.
Ok, it's true that this makes the Python configuration pretty strange. We need to improve this.
Having a separate option for each module is indeed way overkill / undesirable. Having a single string option to specify what you want or what you don't want is also not very user-friendly, certainly not in case of dependencies.
The 'reduced python' option I proposed is really an expert option, as you have to determine which additional modules you need for a given script. But, in cases where space is limited, I think it is of value.
An alternative would be to identify a limited number of module 'groups', each having a Kconfig option, but I'm not sure whether this is at all possible.
What if we split the configuration in two ways?
* Either you start from the almost-full installation, with the option of deselecting certain packages (this corresponds to the current situation)
* Or you tick the 'expert' option, and then you start from the reduced installation, with the option of enabling certain packages with a string option.
The main difference between this and what is proposed in the patch, is that the two ways of configuring become mutually exclusive which makes it clearer.
Best regards,
Thomas
PATCH v2 1/1 Makefile.package.in: add utility functions find*clauses and notfirstword
In reply to Arnout VandecappelleQuoted 10 lines · Arnout Vandecappelle
Quoted 6 lines · On Thursday 17 November 2011 19:41:48 Thomas De Schampheleire wrote:
# HG changeset patch
# User Thomas De Schampheleire <[email protected]> # Date 1318518890 -7200
# Node IDb6579582d02dd802adca94e2b38972cc19cd8b76
# Parent ?baa870774b887fb01a9de0cc69161f250553f896 Makefile.package.in: add utility functions find*clauses and notfirstword?Is it possible to tell hg to remove these lines? ?git-am will include them in the commit message...
Yes, absolutely, the --plain option to 'hg email' does just that. I normally always give that option but seems I didn't this time, sorry!
Quoted 66 lines
Quoted 9 lines
This patch adds a few utility functions to Makefile.package.in.
Functions finddirclauses and findfileclauses help in building a find command that skips a set of directories and performs operations on a set of files. This pattern can for example be used to keep certain files or directories from being stripped, or to remove certain files from a package installation.
Quoted 4 lines
The notfirstword function is the inverse of the 'firstword' function in make:
it returns all but the first word.Signed-off-by Thomas De Schampheleire <[email protected]>?I think you could safely have added a
?And to ThomasP: this helper does simplify patch 3/5 a lot. ?Even though the macros look a bit ugly indeed.
?Regards,
?ArnoutQuoted 29 lines
---
v2: integrate_contvariants in main find*clauses functions. Thanks, Arnout.?package/Makefile.package.in | ?11 +++++++++++ ?1 files changed, 11 insertions(+), 0 deletions(-)
package/Makefile.package.in +11
-0@@ -87,6 +87,17 @@ MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG) ?TERM_BOLD := $(shell tput smso) ?TERM_RESET := $(shell tput rmso) +# Utility functions for 'find' +# findfileclauses: creates "-name 'X' -o -name 'Y'" +# [1:namelist] +findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1))) +# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'" +# [1:basedir, 2:namelist] +finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2))) + +# Miscellaneous utility functions +notfirstword=$(wordlist 2,$(words $(1)),$(1)) +?# Download method commands
?WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET)
?SVN:=$(call qstrip,$(BR2_SVN))Quoted 5 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot--
Arnout Vandecappelle ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? arnout at mind be Senior Embedded Software Architect ? ? ? ? ? ? ? ? +32-16-286540 Essensium/Mind ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium ? ? ? ? ? ? ? ?BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: ?31BB CF53 8660 6F88 345D ?54CC A836 5879 20D7 CF43
PATCH v2 1/1 Makefile.package.in: add utility functions find*clauses and notfirstword
In reply to Thomas De SchampheleireThomas De Schampheleire <[email protected]> writes:
Apologies if you have seen this, but I haven't seen my response to this come through (probably a gmane stuffup).
package/Makefile.package.in MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG) · line 96
Quoted 3 lines
package/Makefile.package.in +3
-0+# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'" +# [1:basedir, 2:namelist] +finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2)))
I think the first -wholename argument is meant to be
'$(1)/$(firstword $(2))'
according to the comment above it.
Also, my man page for find(1) says this about -wholename:
See -path. This alternative is less portable than -path.
Do you find -wholename to be more readable? It's not an option I've used much, so I need to look up the man page anyway, but -path is easier to look up because you do not get redirected to another option like you do with -wholename.
PATCH v2 1/1 Makefile.package.in: add utility functions find*clauses and notfirstword
Hello Cam,
Quoted 5 lines · Cam Hutchison
Thomas De Schampheleire <[email protected]> writes:
Apologies if you have seen this, but I haven't seen my response to this come through (probably a gmane stuffup).
Quoted 8 lines
package/Makefile.package.in MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG) · line 96
Quoted 3 lines
package/Makefile.package.in +3
-0+# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'" +# [1:basedir, 2:namelist] +finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2)))I think the first -wholename argument is meant to be ?'$(1)/$(firstword $(2))'
according to the comment above it.
You are absolutely right. I can't believe I missed that! Thanks for noticing.
Quoted 3 lines
Also, my man page for find(1) says this about -wholename:
?See -path. ? ?This alternative is less portable than -path.
That is funny. My man page redirects in the opposite way:
-path pattern
See -wholename. The predicate -path is also supported
by HP-UX find.
-wholename pattern
File name matches shell pattern pattern. The
metacharacters do not treat ?/? or ?.? specially; so, for example,
find . -wholename ?./sr*sc?
will print an entry for a directory called ?./src/misc?
(if one exists). To ignore a whole directory tree, use -prune
rather than checking every file in the tree. For
example, to skip the directory ?src/emacs? and all files and
directories under it, and print the names of the other
files found, do something like this:
find . -wholename ?./src/emacs? -prune -o -print
But beware, this is CentOS 5, and probably more ancient than your system.
Quoted 5 lines
Do you find -wholename to be more readable? It's not an option I've used much, so I need to look up the man page anyway, but -path is easier to look up because you do not get redirected to another option like you do with -wholename.
The main reason was because my man page uses -wholename when giving -prune information, and I was not aware of -path. But I'll change to -path if that's more portable.
Thanks,
Thomas