Add per-package staging feature

RFC Superseded Fabio Porcedda 6 participants 37 messages Jun 28, 2015 – Oct 5, 2015 Kbuild

Share & tools

Fabio Porcedda

author

2015-06-28 19:42 UTC #0 lore raw

v4 cover Add per-package staging feature

Hi all,
this patch set aims to improve build reproducibility by using a per-package staging directory.
Also this feature aims to enable the top-level parallel make.

This patch set is available at:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4

Every package instead of installing the files in the shared staging directory (STAGING_DIR) install all files in a subdirectory of the STAGINGPKG_DIR.

At the beginning of the configure stage, all the dependencies staging directory are copied in the package staging directory, e.g.:

zlib-install-staging: install files in /output/stagingpkg/zlib/

cramsfs-configure: copy all files from /output/stagingpkg/zlib/* to
/output/stagingpkg/cramfs/

The STAGING_DIR variable is redefined for every package in order to point to the per-package directory, so a package cannot access automatically to staging files of packages that aren't in the (PKG)_DENDENDENCIES variable.

To improve performance and reduce space utilization hard links are used.

To evaluate performance and space utilization I've done some tests.

HW-MED:
CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
RAM: 16GB
SSD: 512GB

HW-HIGH:

CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads

RAM: 32GB
SATA: RAID0 2x2TB 3.5" 7200rpm

defconfig-small, long chain dependency, unfavorable scenario for parallelization:

BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_LIBGTK3=y

defconfig-full:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
make allyespackageconfig
Also i removed few packages that do not built on my system. make show-targets | wc -w
1366

Test about output using hard links or note using the "defconfig-full" on HW-MED:

|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all

Test about performances of this patch set vs master branch:

| 199m | HW-MED | defconfig-full | master branch | no top-level make |
|  99m | HW-MED | defconfig-full | master branch | top-level make    | 
| 100m | HW-MED | defconfig-full | patch set     | top-level make    |

| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
|  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
|  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |

| 10m | HW-MED | defconfig-small | master branch | no top-level make |
|  5m | HW-MED | defconfig-small | master branch | top-level make    | 
|  5m | HW-MED | defconfig-samll | patch set     | top-level make    |

| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
|  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
|  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

TODO:
- use the toolchaing wrapper even for the internal toolchain
- rebase over master (this patch set is based on 2015.05-161-gca5f11e).
- more testing (tested only for arm target).

Best regads

Fabio Porcedda (16):
packages: use TARGET_MAKE_ENV to add pps support
packages: for staging stuff use backtick instead of the shell function packages/scons: make avaiable the GCC_SYSROOT environment variable packages/qt: read spec files from the per-package staging directory
pkg-perl: use TARGET_MAKE_ENV to add pps support
tooclahin-external: add support for GCC_SYSROOT
toolchain-external: move some code in a new function
toolchain-external: use the wrapper even for the linker "ld"
infra: add GCC_SYSROOT environment variable to support pps feature
pkg-cmake: add GCC_SYSROOT environment variable to support pps feature
pkg-python: add GCC_SYSROOT variable to add pps support pacakge/luarocks: add GCC_SYSROOT environment variable to support pps package/pkgconf: use GCC_SYSROOT to support pps feature
Makefile: add STAGINGNOPKG_DIR variable
pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages
infra: add per-package staging feature

 Makefile                                           | 16 ++--
 package/Makefile.in                                |  5 +-
 package/aiccu/aiccu.mk                             |  2 +-
 package/blktrace/blktrace.mk                       |  2 +-
 package/boost/boost.mk                             |  2 +-
 package/botan/botan.mk                             |  2 +-
 package/bsdiff/bsdiff.mk                           |  4 +-
 package/dbus-python/dbus-python.mk                 |  8 +-
 package/dhcpdump/dhcpdump.mk                       |  3 +-
 package/dtc/dtc.mk                                 |  3 +-
 package/exim/exim.mk                               |  5 +-
 package/fbv/fbv.mk                                 |  2 +-
 package/feh/feh.mk                                 |  4 +-
 package/gadgetfs-test/gadgetfs-test.mk             |  2 +-
 package/gdk-pixbuf/gdk-pixbuf.mk                   |  2 +-
 package/iodine/iodine.mk                           |  2 +-
 package/iw/iw.mk                                   |  2 +-
 package/jack2/jack2.mk                             |  3 +-
 package/libmicrohttpd/libmicrohttpd.mk             |  2 +-
 package/libnss/libnss.mk                           |  4 +-
 package/libupnpp/libupnpp.mk                       |  2 +-
 package/lmbench/lmbench.mk                         |  8 +-
 package/logrotate/logrotate.mk                     |  2 +-
 package/lua-msgpack-native/lua-msgpack-native.mk   |  3 +-
 ...ctions-to-the-global-environment-of-the-c.patch | 54 ++++++++++++
 package/luarocks/0002-fixed-typo.patch             | 27 ++++++
 ...e-environment-table-for-the-system-config.patch | 76 +++++++++++++++++
 ...is-not-being-integrated-bug-introduced-in.patch | 62 ++++++++++++++
 package/luarocks/luarocks.mk                       | 11 ++-
 package/madplay/madplay.mk                         |  2 +-
 package/moarvm/moarvm.mk                           |  2 +-
 package/mongoose/mongoose.mk                       |  8 +-
 package/mpg123/mpg123.mk                           |  2 +-
 package/mplayer/mplayer.mk                         |  2 +-
 package/ncurses/ncurses.mk                         |  4 +-
 package/neardal/neardal.mk                         |  2 +-
 package/nginx/nginx.mk                             |  1 +
 package/nmap/nmap.mk                               |  2 +-
 package/open2300/open2300.mk                       |  2 +-
 package/openldap/openldap.mk                       |  2 +-
 package/openssl/openssl.mk                         |  2 +-
 package/opentyrian/opentyrian.mk                   |  2 +-
 package/pax-utils/pax-utils.mk                     |  3 +-
 package/php/php.mk                                 |  4 +-
 package/pifmrds/pifmrds.mk                         |  3 +-
 package/pkg-cmake.mk                               |  1 +
 package/pkg-generic.mk                             | 43 +++++++++-
 package/pkg-perl.mk                                |  2 +-
 package/pkg-python.mk                              |  2 +
 package/pkgconf/pkg-config.in                      |  4 +-
 package/pkgconf/pkgconf.mk                         |  5 +-
 package/poco/poco.mk                               |  3 +-
 package/poppler/poppler.mk                         |  2 +-
 package/pppd/pppd.mk                               |  2 +-
 package/python-gobject/python-gobject.mk           |  4 +-
 package/python-pygame/python-pygame.mk             |  6 +-
 package/python-pyqt/python-pyqt.mk                 |  1 +
 package/python-sip/python-sip.mk                   |  2 +-
 package/qt/qt.mk                                   | 18 ++--
 package/qtuio/qtuio.mk                             |  3 +-
 package/rtmpdump/rtmpdump.mk                       |  2 +-
 package/scons/scons.mk                             |  3 +-
 package/scons/site_init.py                         | 19 +++++
 package/speex/speex.mk                             |  3 +-
 package/statserial/statserial.mk                   |  3 +-
 package/tcpreplay/tcpreplay.mk                     |  6 +-
 package/ti-utils/ti-utils.mk                       |  6 +-
 package/tvheadend/tvheadend.mk                     |  4 +-
 package/upmpdcli/upmpdcli.mk                       |  2 +-
 package/ushare/ushare.mk                           |  3 +-
 package/vsftpd/vsftpd.mk                           |  4 +-
 support/misc/toolchainfile.cmake.in                |  9 +-
 .../toolchain-external/ext-toolchain-wrapper.c     | 96 ++++++++++++++--------
 toolchain/toolchain-external/toolchain-external.mk |  2 +-
 74 files changed, 476 insertions(+), 147 deletions(-)
 create mode 100644 package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
 create mode 100644 package/luarocks/0002-fixed-typo.patch
 create mode 100644 package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
 create mode 100644 package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch
 create mode 100644 package/scons/site_init.py

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #1 lore raw

RFC v4 1/16 packages: use TARGET_MAKE_ENV to add pps support

In the TARGET_MAKE_ENV variable will be added the GCC_SYSROOT variable that is needed for per-package staging feature.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/blktrace/blktrace.mk                     | 2 +-
 package/boost/boost.mk                           | 2 +-
 package/botan/botan.mk                           | 2 +-
 package/bsdiff/bsdiff.mk                         | 4 ++--
 package/dhcpdump/dhcpdump.mk                     | 3 ++-
 package/dtc/dtc.mk                               | 3 ++-
 package/exim/exim.mk                             | 5 +++--
 package/feh/feh.mk                               | 4 ++--
 package/gadgetfs-test/gadgetfs-test.mk           | 2 +-
 package/iodine/iodine.mk                         | 2 +-
 package/iw/iw.mk                                 | 2 +-
 package/jack2/jack2.mk                           | 3 ++-
 package/libnss/libnss.mk                         | 4 ++--
 package/lmbench/lmbench.mk                       | 4 +++-
 package/lua-msgpack-native/lua-msgpack-native.mk | 3 ++-
 package/moarvm/moarvm.mk                         | 2 +-
 package/mongoose/mongoose.mk                     | 8 +++++---
 package/mplayer/mplayer.mk                       | 2 +-
 package/ncurses/ncurses.mk                       | 4 ++--
 package/nginx/nginx.mk                           | 1 +
 package/open2300/open2300.mk                     | 2 +-
 package/openssl/openssl.mk                       | 2 +-
 package/opentyrian/opentyrian.mk                 | 2 +-
 package/pax-utils/pax-utils.mk                   | 3 ++-
 package/pifmrds/pifmrds.mk                       | 3 ++-
 package/poco/poco.mk                             | 3 ++-
 package/pppd/pppd.mk                             | 2 +-
 package/qt/qt.mk                                 | 2 ++
 package/qtuio/qtuio.mk                           | 3 ++-
 package/rtmpdump/rtmpdump.mk                     | 2 +-
 package/speex/speex.mk                           | 3 ++-
 package/statserial/statserial.mk                 | 3 ++-
 package/ti-utils/ti-utils.mk                     | 2 +-
 package/tvheadend/tvheadend.mk                   | 4 ++--
 package/ushare/ushare.mk                         | 3 ++-
 package/vsftpd/vsftpd.mk                         | 2 +-
 36 files changed, 61 insertions(+), 42 deletions(-)

package/blktrace/blktrace.mk +1 -1

index c193375..8d5e52b 100644
@@ -11,7 +11,7 @@ BLKTRACE_LICENSE = GPLv2+
 BLKTRACE_LICENSE_FILES = COPYING
 
 define BLKTRACE_BUILD_CMDS
-	$(MAKE1) -C $(@D) $(TARGET_CONFIGURE_OPTS)
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(TARGET_CONFIGURE_OPTS)
 endef
 
 define BLKTRACE_INSTALL_TARGET_CMDS

package/boost/boost.mk +1 -1

index dbb5de5..24f46e2 100644
@@ -148,7 +148,7 @@ define HOST_BOOST_INSTALL_CMDS
 endef
 
 define BOOST_INSTALL_STAGING_CMDS
-	(cd $(@D) && ./bjam -j$(PARALLEL_JOBS) -q -d+1 \
+	(cd $(@D) && $(TARGET_MAKE_ENV) ./bjam -j$(PARALLEL_JOBS) -q -d+1 \
 	--user-config=$(@D)/user-config.jam \
 	$(BOOST_OPTS) \
 	--prefix=$(STAGING_DIR)/usr \

package/botan/botan.mk +1 -1

index 6ac3ce7..58456fb 100644
@@ -48,7 +48,7 @@ define BOTAN_CONFIGURE_CMDS
 endef
 
 define BOTAN_BUILD_CMDS
-	$(MAKE) -C $(@D) AR="$(TARGET_AR) crs"
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) AR="$(TARGET_AR) crs"
 endef
 
 define BOTAN_INSTALL_STAGING_CMDS

package/bsdiff/bsdiff.mk +2 -2

index 7d2b296..eb903e0 100644
@@ -11,9 +11,9 @@ BSDIFF_LICENSE = BSD-2c
 BSDIFF_LICENSE_FILES = bsdiff.c
 
 define BSDIFF_BUILD_CMDS
-	$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
+	$(TARGET_MAKE_ENV) $(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
 		$(@D)/bsdiff.c -lbz2 -o $(@D)/bsdiff
-	$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
+	$(TARGET_MAKE_ENV) $(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
 		$(@D)/bspatch.c -lbz2 -o $(@D)/bspatch
 endef

package/dhcpdump/dhcpdump.mk +2 -1

index 0756319..8df5135 100644
@@ -16,7 +16,8 @@ DHCPDUMP_LIBS += $(shell $(STAGING_DIR)/usr/bin/pcap-config --static --additiona
 endif
 
 define DHCPDUMP_BUILD_CMDS
-	$(MAKE) -C $(@D) CC="$(TARGET_CC) $(TARGET_CFLAGS)" LIBS="$(DHCPDUMP_LIBS)"
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) \
+		CC="$(TARGET_CC) $(TARGET_CFLAGS)" LIBS="$(DHCPDUMP_LIBS)"
 endef
 
 define DHCPDUMP_INSTALL_TARGET_CMDS

package/dtc/dtc.mk +2 -1

index e094f7b..0800fb0 100644
@@ -41,7 +41,8 @@ define DTC_INSTALL_STAGING_CMDS
 endef
 
 define DTC_INSTALL_TARGET_CMDS
-	$(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) PREFIX=/usr $(DTC_INSTALL_GOAL)
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		-C $(@D) DESTDIR=$(TARGET_DIR) PREFIX=/usr $(DTC_INSTALL_GOAL)
 endef
 
 $(eval $(generic-package))

package/exim/exim.mk +3 -2

index b852793..4f75e9a 100644
@@ -111,8 +111,9 @@ endef
 # Need to replicate the LFLAGS in install, as exim still wants to build
 # something when installing...
 define EXIM_INSTALL_TARGET_CMDS
-	DESTDIR=$(TARGET_DIR) INSTALL_ARG="-no_chown -no_symlink" build=br \
-	  $(MAKE1) -C $(@D) $(EXIM_STATIC_FLAGS) install
+	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) \
+		INSTALL_ARG="-no_chown -no_symlink" build=br \
+		$(MAKE1) -C $(@D) $(EXIM_STATIC_FLAGS) install
 	chmod u+s $(TARGET_DIR)/usr/sbin/exim
 endef

package/feh/feh.mk +2 -2

index 0c4c164..27ac53b 100644
@@ -12,8 +12,8 @@ FEH_LICENSE = MIT
 FEH_LICENSE_FILES = COPYING
 
 define FEH_BUILD_CMDS
-	$(MAKE1) CC="$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS)" \
-		-C $(@D) all
+	$(TARGET_MAKE_ENV) $(MAKE1) CC="$(TARGET_CC) $(TARGET_CFLAGS) \
+		$(TARGET_LDFLAGS)" -C $(@D) all
 endef
 
 define FEH_INSTALL_TARGET_CMDS

package/gadgetfs-test/gadgetfs-test.mk +1 -1

index 655f50a..bdeaede 100644
@@ -15,7 +15,7 @@ GADGETFS_TEST_MAKEOPTS += USE_AIO=y
 endif
 
 define GADGETFS_TEST_BUILD_CMDS
-	$(MAKE) -C $(@D) $(GADGETFS_TEST_MAKEOPTS)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) $(GADGETFS_TEST_MAKEOPTS)
 endef
 
 define GADGETFS_TEST_INSTALL_TARGET_CMDS

package/iodine/iodine.mk +1 -1

index 57deb9e..c3df831 100644
@@ -11,7 +11,7 @@ IODINE_LICENSE = MIT
 IODINE_LICENSE_FILES = README
 
 define IODINE_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" ARCH=$(BR2_ARCH) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" ARCH=$(BR2_ARCH) -C $(@D)
 endef
 
 define IODINE_INSTALL_TARGET_CMDS

package/iw/iw.mk +1 -1

index 2766850..3aa285b 100644
@@ -11,7 +11,7 @@ IW_LICENSE = iw license
 IW_LICENSE_FILES = COPYING
 IW_DEPENDENCIES = host-pkgconf libnl
 IW_MAKE_OPTS = CC="$(TARGET_CC)" LD="$(TARGET_LD)" LDFLAGS="$(TARGET_LDFLAGS)"
-IW_MAKE_ENV = PKG_CONFIG="$(HOST_DIR)/usr/bin/pkg-config" \
+IW_MAKE_ENV = $(TARGET_MAKE_ENV) PKG_CONFIG="$(HOST_DIR)/usr/bin/pkg-config" \
 	GIT_DIR=$(IW_DIR)
 
 define IW_BUILD_CMDS

package/jack2/jack2.mk +2 -1

index 73328d3..3db7f3d 100644
@@ -20,7 +20,8 @@ define JACK2_CONFIGURE_CMDS
 endef
 
 define JACK2_BUILD_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python2 ./waf build -j $(PARALLEL_JOBS))
+	(cd $(@D); $(TARGET_MAKE_ENV) \
+		$(HOST_DIR)/usr/bin/python2 ./waf build -j $(PARALLEL_JOBS))
 endef
 
 define JACK2_INSTALL_TARGET_CMDS

package/libnss/libnss.mk +2 -2

index db272b3..529b773 100644
@@ -38,12 +38,12 @@ endif
 endif
 
 define LIBNSS_BUILD_CMDS
-	$(MAKE1) -C $(@D)/nss coreconf \
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)/nss coreconf \
 		SOURCE_MD_DIR=$(@D)/$(LIBNSS_DISTDIR) \
 		DIST=$(@D)/$(LIBNSS_DISTDIR) \
 		CHECKLOC= \
 		$(LIBNSS_BUILD_VARS)
-	$(MAKE1) -C $(@D)/nss lib/dbm all \
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)/nss lib/dbm all \
 		SOURCE_MD_DIR=$(@D)/$(LIBNSS_DISTDIR) \
 		DIST=$(@D)/$(LIBNSS_DISTDIR) \
 		CHECKLOC= \

package/lmbench/lmbench.mk +3 -1

index f593428..d4be8d4 100644
@@ -31,7 +31,9 @@ endef
 # script. So the variables override below don't take direct effect in
 # src/Makefile.
 define LMBENCH_BUILD_CMDS
-	$(MAKE) CFLAGS="$(LMBENCH_CFLAGS)" LDLIBS="$(LMBENCH_LDLIBS)" OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		CFLAGS="$(LMBENCH_CFLAGS)" LDLIBS="$(LMBENCH_LDLIBS)" \
+		OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
 endef
 
 define LMBENCH_INSTALL_TARGET_CMDS

package/lua-msgpack-native/lua-msgpack-native.mk +2 -1

index acd3a0e..cc68b3b 100644
@@ -11,7 +11,8 @@ LUA_MSGPACK_NATIVE_LICENSE = Apache-2.0
 LUA_MSGPACK_NATIVE_LICENSE_FILES = LICENSE.txt
 
 define LUA_MSGPACK_NATIVE_BUILD_CMDS
-	$(TARGET_CC) $(TARGET_CFLAGS) -fPIC -shared -o $(@D)/msgpack.so $(@D)/mp.c
+	$(TARGET_MAKE_ENV) $(TARGET_CC) \
+		$(TARGET_CFLAGS) -fPIC -shared -o $(@D)/msgpack.so $(@D)/mp.c
 endef
 
 define LUA_MSGPACK_NATIVE_INSTALL_TARGET_CMDS

package/moarvm/moarvm.mk +1 -1

index 5889436..e00058b 100644
@@ -32,7 +32,7 @@ define MOARVM_CONFIGURE_CMDS
 endef
 
 define MOARVM_BUILD_CMDS
-	$(MAKE) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define MOARVM_INSTALL_STAGING_CMDS

package/mongoose/mongoose.mk +5 -3

index 6797e58..0961946 100644
@@ -19,11 +19,13 @@ MONGOOSE_CFLAGS += -DNS_ENABLE_SSL -lssl -lcrypto -lz
 endif
 
 define MONGOOSE_BUILD_CMDS
-	$(TARGET_CC) $(@D)/examples/web_server/web_server.c $(@D)/mongoose.c \
+	$(TARGET_MAKE_ENV) $(TARGET_CC) \
+		$(@D)/examples/web_server/web_server.c $(@D)/mongoose.c \
 		-I$(@D) -o $(@D)/examples/web_server/web_server \
 		$(MONGOOSE_CFLAGS) -pthread
-	$(TARGET_CC) -c $(@D)/mongoose.c $(MONGOOSE_CFLAGS) -o $(@D)/mongoose.o
-	$(TARGET_AR) rcs $(@D)/libmongoose.a $(@D)/mongoose.o
+	$(TARGET_MAKE_ENV) $(TARGET_CC) \
+		-c $(@D)/mongoose.c $(MONGOOSE_CFLAGS) -o $(@D)/mongoose.o
+	$(TARGET_MAKE_ENV) $(TARGET_AR) rcs $(@D)/libmongoose.a $(@D)/mongoose.o
 endef
 
 define MONGOOSE_INSTALL_TARGET_CMDS

package/mplayer/mplayer.mk +1 -1

index 3a61b8f..6ac91ab 100644
@@ -269,7 +269,7 @@ define MPLAYER_CONFIGURE_CMDS
 endef
 
 define MPLAYER_BUILD_CMDS
-	$(MAKE) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define MPLAYER_INSTALL_TARGET_CMDS

package/ncurses/ncurses.mk +2 -2

index 8368c69..6dcf496 100644
@@ -115,9 +115,9 @@ endif
 # ncurses breaks with parallel build, but takes quite a while to
 # build single threaded. Work around it similar to how Gentoo does
 define NCURSES_BUILD_CMDS
-	$(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) sources
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) sources
 	rm -rf $(@D)/misc/pc-files
-	$(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR)
 endef
 
 ifneq ($(BR2_STATIC_LIBS),y)

package/nginx/nginx.mk +1 -0

index 90ebbcc..3a4599b 100644
@@ -31,6 +31,7 @@ endef
 
 # disable external libatomic_ops because its detection fails.
 NGINX_CONF_ENV += \
+	$(TARGET_MAKE_ENV) \
 	ngx_force_c_compiler=yes \
 	ngx_force_c99_have_variadic_macros=yes \
 	ngx_force_gcc_have_variadic_macros=yes \

package/open2300/open2300.mk +1 -1

index 73db2a0..253a8cd 100644
@@ -24,7 +24,7 @@ OPEN2300_LDFLAGS += -L$(STAGING_DIR)/usr/lib/mysql -lmysqlclient
 endif
 
 define OPEN2300_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_LD)" \
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_LD)" \
 		CFLAGS="$(OPEN2300_CFLAGS)" CC_LDFLAGS="$(OPEN2300_LDFLAGS)" \
 		-C $(@D) $(OPEN2300_BINS)
 endef

package/openssl/openssl.mk +1 -1

index f90c811..0dc3750 100644
@@ -99,7 +99,7 @@ define HOST_OPENSSL_BUILD_CMDS
 endef
 
 define OPENSSL_BUILD_CMDS
-	$(MAKE1) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)
 endef
 
 define OPENSSL_INSTALL_STAGING_CMDS

package/opentyrian/opentyrian.mk +1 -1

index b74e4cf..6efbe4e 100644
@@ -20,7 +20,7 @@ OPENTYRIAN_NETWORK = false
 endif
 
 define OPENTYRIAN_BUILD_CMDS
-	$(MAKE) PLATFORM=UNIX \
+	$(TARGET_MAKE_ENV) $(MAKE) PLATFORM=UNIX \
 		CC="$(TARGET_CC)" \
 		STRIP="/bin/true" \
 		SDL_CONFIG="$(STAGING_DIR)/usr/bin/sdl-config" \

package/pax-utils/pax-utils.mk +2 -1

index fc0589f..462413d 100644
@@ -31,7 +31,8 @@ define HOST_PAX_UTILS_BUILD_CMDS
 endef
 
 define PAX_UTILS_BUILD_CMDS
-	$(MAKE) $(TARGET_CONFIGURE_OPTS) $(PAX_UTILS_USE_CAP) -C $(@D)
+	$(TARGET_MAKE_ENV) \
+		$(MAKE) $(TARGET_CONFIGURE_OPTS) $(PAX_UTILS_USE_CAP) -C $(@D)
 endef
 
 define HOST_PAX_UTILS_INSTALL_CMDS

package/pifmrds/pifmrds.mk +2 -1

index 5179715..9e71916 100644
@@ -11,7 +11,8 @@ PIFMRDS_LICENSE = GPLv3+
 PIFMRDS_LICENSE_FILES = LICENSE
 
 define PIFMRDS_BUILD_CMDS
-	$(MAKE) -C $(@D)/src CC="$(TARGET_CC)" LDFLAGS="$(TARGET_LDFLAGS)" \
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		-C $(@D)/src CC="$(TARGET_CC)" LDFLAGS="$(TARGET_LDFLAGS)" \
 		CFLAGS="$(TARGET_CFLAGS) -std=gnu99 -ffast-math -c" \
 		app rds_wav
 endef

package/poco/poco.mk +2 -1

index 7233d43..dff71e4 100644
@@ -51,7 +51,8 @@ define POCO_CONFIGURE_CMDS
 endef
 
 define POCO_BUILD_CMDS
-	$(MAKE1) POCO_TARGET_OSARCH=$(ARCH) CROSSENV=$(TARGET_CROSS) \
+	$(TARGET_MAKE_ENV) $(MAKE1) \
+		POCO_TARGET_OSARCH=$(ARCH) CROSSENV=$(TARGET_CROSS) \
 		MYSQL_LIBDIR=$(STAGING_DIR)/usr/lib/mysql \
 		MYSQL_INCDIR=$(STAGING_DIR)/usr/include/mysql -C $(@D)
 endef

package/pppd/pppd.mk +1 -1

index 0f9945f..e655457 100644
@@ -50,7 +50,7 @@ define PPPD_CONFIGURE_CMDS
 endef
 
 define PPPD_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" COPTS="$(TARGET_CFLAGS)" \
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" COPTS="$(TARGET_CFLAGS)" \
 		-C $(@D) $(PPPD_MAKE_OPTS)
 endef

package/qt/qt.mk +2 -0

index a7bbe5f..b2b9bbe 100644
@@ -482,6 +482,8 @@ define QT_CONFIGURE_CONFIG_FILE
 endef
 endif
 
+QT_CONFIGURE_ENV += $(TARGET_MAKE_ENV)
+
 define QT_CONFIGURE_CMDS
 	-[ -f $(@D)/Makefile ] && $(MAKE) -C $(@D) confclean
 	$(QT_CONFIGURE_IPV6)

package/qtuio/qtuio.mk +2 -1

index 5fa10a7..58008bb 100644
@@ -32,7 +32,8 @@ endef
 ifeq ($(BR2_QTUIO_EXAMPLES),y)
 define QTUIO_BUILD_EXAMPLES
 	for example in $(QTUIO_EXAMPLES) ; do \
-		$(MAKE) -C $(@D)/examples/$${example} || exit 1; \
+		$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/examples/$${example} || \
+			exit 1; \
 	done
 endef
 endif

package/rtmpdump/rtmpdump.mk +1 -1

index 613f1cd..aaa0071 100644
@@ -36,7 +36,7 @@ RTMPDUMP_SHARED = "SHARED="
 endif
 
 define RTMPDUMP_BUILD_CMDS
-	$(MAKE) CRYPTO=$(RTMPDUMP_CRYPTO) \
+	$(TARGET_MAKE_ENV) $(MAKE) CRYPTO=$(RTMPDUMP_CRYPTO) \
 		prefix=/usr \
 		XCFLAGS="$(RTMPDUMP_CFLAGS)" \
 		XLDFLAGS="$(TARGET_LDFLAGS)" \

package/speex/speex.mk +2 -1

index a34bfac..c7d77a0 100644
@@ -32,7 +32,8 @@ endef
 SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
 
 define SPEEX_BUILD_CMDS
-	$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_MAKE_OPTS) -C $(@D)/$($(PKG)_SUBDIR)
+	$(TARGET_MAKE_ENV) $($(PKG)_MAKE_ENV) $(MAKE) \
+		$($(PKG)_MAKE_OPTS) -C $(@D)/$($(PKG)_SUBDIR)
 endef
 
 $(eval $(autotools-package))

package/statserial/statserial.mk +2 -1

index f0e15bc..1d646d9 100644
@@ -11,7 +11,8 @@ STATSERIAL_LICENSE = GPLv2+
 STATSERIAL_LICENSE_FILES = COPYING
 
 define STATSERIAL_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" LDFLAGS="$(TARGET_LDFLAGS)" -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_CC)" \
+		CFLAGS="$(TARGET_CFLAGS)" LDFLAGS="$(TARGET_LDFLAGS)" -C $(@D)
 endef
 
 define STATSERIAL_INSTALL_TARGET_CMDS

package/ti-utils/ti-utils.mk +1 -1

index 3141219..35ecc19 100644
@@ -14,7 +14,7 @@ TI_UTILS_CFLAGS = $(shell $(PKG_CONFIG_HOST_BINARY) --cflags libnl-genl-3.0)
 TI_UTILS_LIBS = $(shell $(PKG_CONFIG_HOST_BINARY) --libs libnl-genl-3.0)
 
 define TI_UTILS_BUILD_CMDS
-	$(MAKE1) NFSROOT="$(STAGING_DIR)" \
+	$(TARGET_MAKE_ENV) $(MAKE1) NFSROOT="$(STAGING_DIR)" \
 		CC="$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) $(TI_UTILS_CFLAGS)" \
 		LIBS="$(TI_UTILS_LIBS)" -C $(@D) all
 endef

package/tvheadend/tvheadend.mk +2 -2

index 2a0019e..38a4ebe 100644
@@ -68,11 +68,11 @@ define TVHEADEND_CONFIGURE_CMDS
 endef
 
 define TVHEADEND_BUILD_CMDS
-	$(MAKE) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define TVHEADEND_INSTALL_TARGET_CMDS
-	$(MAKE) -C $(@D) DESTDIR="$(TARGET_DIR)" install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR="$(TARGET_DIR)" install
 endef
 
 # Remove documentation and source files that are not needed because we

package/ushare/ushare.mk +2 -1

index 25f880c..c9c6ae8 100644
@@ -26,7 +26,8 @@ define USHARE_CONFIGURE_CMDS
 endef
 
 define USHARE_BUILD_CMDS
-	$(MAKE) LDFLAGS="$(TARGET_LDFLAGS) $(USHARE_LDFLAGS)" -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		LDFLAGS="$(TARGET_LDFLAGS) $(USHARE_LDFLAGS)" -C $(@D)
 endef
 
 define USHARE_INSTALL_TARGET_CMDS

package/vsftpd/vsftpd.mk +1 -1

index 3091f06..db85015 100644
@@ -39,7 +39,7 @@ VSFTPD_LIBS += -lpam
 endif
 
 define VSFTPD_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
 		LDFLAGS="$(TARGET_LDFLAGS)" LIBS="$(VSFTPD_LIBS)" -C $(@D)
 endef

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #2 lore raw

RFC v4 2/16 packages: for staging stuff use backtick instead of the shell function

In reply to Fabio Porcedda

When the shell function execute the command before the per-package staging directory is created, use the backtick that is executed after the per-package staging directory is created.

This commit prevents build failures when the per-package staging feature is added.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/dbus-python/dbus-python.mk       |  8 ++++----
 package/python-gobject/python-gobject.mk |  4 ++--
 package/python-pygame/python-pygame.mk   |  6 +++---
 package/qt/qt.mk                         | 14 +++++++-------
 package/tcpreplay/tcpreplay.mk           |  6 +++---
 5 files changed, 19 insertions(+), 19 deletions(-)

package/dbus-python/dbus-python.mk +4 -4

index 9bda018..4079967 100644
@@ -17,15 +17,15 @@ DBUS_PYTHON_DEPENDENCIES += python host-python
 
 DBUS_PYTHON_CONF_ENV += \
 	PYTHON=$(HOST_DIR)/usr/bin/python2 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python2-config --includes)" \
-	PYTHON_LIBS="$(shell $(STAGING_DIR)/usr/bin/python2-config --ldflags)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python2-config --includes`" \
+	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python2-config --ldflags`"
 else
 DBUS_PYTHON_DEPENDENCIES += python3 host-python3
 
 DBUS_PYTHON_CONF_ENV += \
 	PYTHON=$(HOST_DIR)/usr/bin/python3 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python3-config --includes)" \
-	PYTHON_LIBS="$(shell $(STAGING_DIR)/usr/bin/python3-config --ldflags)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python3-config --includes`" \
+	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python3-config --ldflags`"
 endif
 
 $(eval $(autotools-package))

package/python-gobject/python-gobject.mk +2 -2

index c47954c..74cb054 100644
@@ -20,13 +20,13 @@ PYTHON_GOBJECT_DEPENDENCIES += python host-python
 
 PYTHON_GOBJECT_CONF_ENV = \
 	PYTHON=$(HOST_DIR)/usr/bin/python2 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python2-config --includes)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python2-config --includes`"
 else
 PYTHON_GOBJECT_DEPENDENCIES += python3 host-python3
 
 PYTHON_GOBJECT_CONF_ENV = \
 	PYTHON=$(HOST_DIR)/usr/bin/python3 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python3-config --includes)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python3-config --includes`"
 endif
 
 ifeq ($(BR2_PACKAGE_LIBFFI),y)

package/python-pygame/python-pygame.mk +3 -3

index 51f1c20..995f80a 100644
@@ -66,14 +66,14 @@ define PYTHON_PYGAME_UNCONFIGURE_SCRAP
 endef
 endif
 
-PYTHON_PYGAME_SDL_FLAGS = $(shell $(STAGING_DIR)/usr/bin/sdl-config --cflags)
-PYTHON_PYGAME_SDL_FLAGS += $(shell $(STAGING_DIR)/usr/bin/sdl-config --libs)
+PYTHON_PYGAME_SDL_FLAGS = `$(STAGING_DIR)/usr/bin/sdl-config --cflags`
+PYTHON_PYGAME_SDL_FLAGS += `$(STAGING_DIR)/usr/bin/sdl-config --libs`
 
 # Pygame needs a Setup file where options should be commented out if
 # dependencies are not available
 define PYTHON_PYGAME_CONFIGURE_CMDS
 	cp -f $(@D)/Setup.in $(@D)/Setup
-	$(SED) 's~^SDL = ~SDL = $(PYTHON_PYGAME_SDL_FLAGS) \n#~' $(@D)/Setup
+	$(SED) "s~^SDL = ~SDL = $(PYTHON_PYGAME_SDL_FLAGS) \n#~" $(@D)/Setup
 	$(SED) 's/^pypm/#pypm/' $(@D)/Setup
 	$(PYTHON_PYGAME_UNCONFIGURE_IMAGE)
 	$(PYTHON_PYGAME_UNCONFIGURE_FONT)

package/qt/qt.mk +7 -7

index b2b9bbe..a4a55ec 100644
@@ -230,9 +230,9 @@ QT_DEPENDENCIES += fontconfig xlib_libXi xlib_libX11 xlib_libXrender \
                 xlib_libXcursor xlib_libXrandr xlib_libXext xlib_libXv
 # Using pkg-config avoids us some logic to redefine and sed again mkspecs files
 # to add X11 include path and link options
-QT_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags x11)
-QT_CXXFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags x11)
-QT_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs x11 xext)
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs x11 xext`
 QT_CONFIGURE_OPTS += -arch $(QT_EMB_PLATFORM) \
 		-xplatform qws/linux-$(QT_EMB_PLATFORM)-g++ -x11 -no-gtkstyle -no-sm \
 		-no-openvg

@@ -329,9 +329,9 @@ endif
 ifeq ($(BR2_PACKAGE_QT_OPENGL_ES),y)
 QT_CONFIGURE_OPTS += -opengl es2 -egl
 QT_DEPENDENCIES += libgles libegl
-QT_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)
-QT_CXXFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)
-QT_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs egl)
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs egl`
 else
 QT_CONFIGURE_OPTS += -no-opengl
 endif

@@ -473,7 +473,7 @@ QT_QMAKE = $(HOST_DIR)/usr/bin/qmake -spec qws/linux-$(QT_EMB_PLATFORM)-g++
 ################################################################################
 define QT_QMAKE_SET
 	$(SED) '/$(1)/d' $(3)/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++/qmake.conf
-	$(SED) '/include.*qws.conf/a$(1) = $(2)' $(3)/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++/qmake.conf
+	$(SED) "/include.*qws.conf/a$(1) = $(2)" $(3)/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++/qmake.conf
 endef
 
 ifneq ($(QT_CONFIG_FILE),)

package/tcpreplay/tcpreplay.mk +3 -3

index 1dbe453..f6bd08f 100644
@@ -16,9 +16,9 @@ TCPREPLAY_CONF_OPTS = --with-libpcap=$(STAGING_DIR)/usr
 TCPREPLAY_DEPENDENCIES = libpcap
 
 # libpcap may depend on symbols in other libs
-TCPREPLAY_LIBS = $(shell $(STAGING_DIR)/usr/bin/pcap-config --static --libs)
-TCPREPLAY_CONF_ENV += ac_cv_search_pcap_close='$(TCPREPLAY_LIBS)' \
-	LIBS="$(TCPREPLAY_LIBS)"
+TCPREPLAY_LIBS = $(STAGING_DIR)/usr/bin/pcap-config --static --libs
+TCPREPLAY_CONF_ENV += ac_cv_search_pcap_close="`$(TCPREPLAY_LIBS)`" \
+	LIBS="`$(TCPREPLAY_LIBS)`"
 
 ifeq ($(BR2_PACKAGE_TCPDUMP),y)
 TCPREPLAY_CONF_ENV += ac_cv_path_tcpdump_path=/usr/sbin/tcpdump

Signature · 1 line

2.4.3

Arnout Vandecappelle

2015-07-12 14:39 UTC #3 lore raw

RFC v4 2/16 packages: for staging stuff use backtick instead of the shell function

Quoted 6 lines · Fabio Porcedda

When the shell function execute the command before the per-package staging directory is created, use the backtick that is executed after the per-package staging directory is created.
This commit prevents build failures when the per-package staging feature is added.

At first, I was pretty opposed to this change because I like make more than
shell :-)

However, it also fixes another issue: make printvars prints a lot of errors in
addition to the real thing, because it evaluates all those $(shell ...) variables while the programs they call or not defined yet.

But for that, a lot more needs to be changed. So I've taken this over and
posted a patch series [1] that removes all the $(shell ...) calls.

Regards,
Arnout

[1] http://patchwork.ozlabs.org/patch/494127/

Quoted 2 lines

Signed-off-by Fabio Porcedda <[email protected]>

[snip]

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Fabio Porcedda

author

2015-06-28 19:42 UTC #4 lore raw

RFC v4 3/16 packages/scons: make avaiable the GCC_SYSROOT environment variable

In reply to Fabio Porcedda

In order to support the per-package staging feature, make avaiable the GCC_SYSROOT environment variable to scons packages so the toolchain wrapper can use it.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/scons/scons.mk     |  3 ++-
 package/scons/site_init.py | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)
 create mode 100644 package/scons/site_init.py

package/scons/scons.mk +2 -1

index 39fb258..e8ae086 100644
@@ -18,4 +18,5 @@ HOST_SCONS_INSTALL_OPTS = \
 $(eval $(host-python-package))
 
 # variables used by other packages
-SCONS = $(HOST_DIR)/usr/bin/python2 $(HOST_DIR)/usr/bin/scons $(if $(QUIET),-s)
+SCONS = $(HOST_DIR)/usr/bin/python2 $(HOST_DIR)/usr/bin/scons \
+	$(if $(QUIET),-s) --site-dir=$(TOPDIR)/package/scons

package/scons/site_init.py new file +19 -0

new file mode 100644
index 0000000..f0d8b53
@@ -0,0 +1,19 @@
+"""
+Decorate SCons Environment constructor so that the "GCC_SYSROOT"
+variable is always included.
+"""
+
+from functools import wraps
+import SCons.Environment
+
+def add_gcc_sysroot(environment_init):
+    @wraps(environment_init)
+    def wrapper(self, *args, **kwargs):
+        environment_init(self, *args, **kwargs)
+
+        self.PrependENVPath('GCC_SYSROOT', os.environ['GCC_SYSROOT'])
+
+    return wrapper
+
+SCons.Environment.Environment.__init__ = \
+    add_gcc_sysroot(SCons.Environment.Environment.__init__)

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #5 lore raw

RFC v4 4/16 packages/qt: read spec files from the per-package staging directory

In reply to Fabio Porcedda

In order to support the per-package staging feature, read spec files from the subdirectory "mkspecs" that is in the per-package staging directory.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/python-pyqt/python-pyqt.mk | 1 +
 package/python-sip/python-sip.mk   | 2 +-
 package/qt/qt.mk                   | 2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)

package/python-pyqt/python-pyqt.mk +1 -0

index fa298c4..0ee9352 100644
@@ -73,6 +73,7 @@ PYTHON_PYQT_CONF_OPTS = \
 	--bindir $(TARGET_DIR)/usr/bin \
 	--destdir $(TARGET_DIR)/usr/lib/$(PYTHON_PYQT_PYTHON_DIR)/site-packages \
 	--qmake $(HOST_DIR)/usr/bin/qmake \
+	--spec $(STAGING_DIR)/usr/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++ \
 	--sysroot $(STAGING_DIR)/usr \
 	-w --confirm-license \
 	--no-designer-plugin \

package/python-sip/python-sip.mk +1 -1

index 4651186..4ac6413 100644
@@ -46,7 +46,7 @@ define PYTHON_SIP_CONFIGURE_CMDS
 			--sipdir $(TARGET_DIR)/usr/share/sip \
 			--sysroot $(STAGING_DIR)/usr \
 			--use-qmake && \
-		$(HOST_DIR)/usr/bin/qmake)
+		$(QT_QMAKE))
 endef
 
 define PYTHON_SIP_BUILD_CMDS

package/qt/qt.mk +1 -1

index a4a55ec..0644fbe 100644
@@ -457,7 +457,7 @@ endif
 # End of workaround.
 
 # Variable for other Qt applications to use
-QT_QMAKE = $(HOST_DIR)/usr/bin/qmake -spec qws/linux-$(QT_EMB_PLATFORM)-g++
+QT_QMAKE = $(HOST_DIR)/usr/bin/qmake -spec $(STAGING_DIR)/usr/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++
 
 ################################################################################
 # QT_QMAKE_SET -- helper macro to set <variable> = <value> in

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #6 lore raw

RFC v4 5/16 pkg-perl: use TARGET_MAKE_ENV to add pps support

In reply to Fabio Porcedda

In the TARGET_MAKE_ENV variable will be added the GCC_SYSROOT variable that is needed for per-package staging feature.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/pkg-perl.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

package/pkg-perl.mk +1 -1

index 6577588..d879771 100644
@@ -130,7 +130,7 @@ define $(2)_BUILD_CMDS
 	cd $$($$(PKG)_SRCDIR) && if [ -f Build.PL ] ; then \
 		$$(PERL_RUN) Build $$($(2)_BUILD_OPTS) build; \
 	else \
-		$$(MAKE1) \
+		$$(TARGET_MAKE_ENV) $$(MAKE1) \
 			PERL_INC=$$(STAGING_DIR)/usr/lib/perl5/$$(PERL_VERSION)/$$(PERL_ARCHNAME)/CORE \
 			$$($(2)_BUILD_OPTS) pure_all; \
 	fi

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #7 lore raw

RFC v4 6/16 tooclahin-external: add support for GCC_SYSROOT

In reply to Fabio Porcedda

To support the per-package staging feature, when the GCC_SYSROOT environament variable is defined pass it to gcc in the "--sysroot" option.

Signed-off-by Fabio Porcedda <[email protected]>

---
 toolchain/toolchain-external/ext-toolchain-wrapper.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

toolchain/toolchain-external/ext-toolchain-wrapper.c +8 -1

index e2d01dc..f262b3d 100644
@@ -98,6 +98,7 @@ int main(int argc, char **argv)
 	char *basename;
 	char *env_debug;
 	char *paranoid_wrapper;
+	char *gcc_sysroot;
 	int paranoid;
 	int ret, i, count = 0, debug;

@@ -145,7 +146,13 @@ int main(int argc, char **argv)
 		perror(__FILE__ ": overflow");
 		return 3;
 	}
-	ret = snprintf(sysroot, sizeof(sysroot), "%s/" BR_SYSROOT, absbasedir);
+	gcc_sysroot = getenv("GCC_SYSROOT");
+	if (gcc_sysroot) {
+		ret = snprintf(sysroot, sizeof(sysroot), "%s", gcc_sysroot);
+	} else {
+		ret = snprintf(sysroot, sizeof(sysroot), "%s/" BR_SYSROOT, \
+			       absbasedir);
+	}
 	if (ret >= sizeof(sysroot)) {
 		perror(__FILE__ ": overflow");
 		return 3;

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #8 lore raw

RFC v4 7/16 toolchain-external: move some code in a new function

In reply to Fabio Porcedda

To simplify the following commit, refactor the main() function moving some code in a new add_gcc_args() function.

Signed-off-by Fabio Porcedda <[email protected]>

---
 .../toolchain-external/ext-toolchain-wrapper.c     | 72 ++++++++++++----------
 1 file changed, 40 insertions(+), 32 deletions(-)

toolchain/toolchain-external/ext-toolchain-wrapper.c +40 ~~-32~~

index f262b3d..f8695a4 100644
@@ -71,6 +71,45 @@ static char *predef_args[] = {
 #endif
 };
 
+static char **add_gcc_args(int argc, char **argv, char **cur)
+{
+	int i;
+
+#ifdef BR_FLOAT_ABI
+	/* add float abi if not overridden in args */
+	for (i = 1; i < argc; i++) {
+		if (!strncmp(argv[i], "-mfloat-abi=", strlen("-mfloat-abi=")) ||
+		    !strcmp(argv[i], "-msoft-float") ||
+		    !strcmp(argv[i], "-mhard-float"))
+			break;
+	}
+
+	if (i == argc)
+		*cur++ = "-mfloat-abi=" BR_FLOAT_ABI;
+#endif
+
+#if defined(BR_ARCH) || \
+    defined(BR_CPU)
+	/* Add our -march/cpu/abi flags, but only if none are
+	 * already specified on the commandline
+	 */
+	for (i = 1; i < argc; i++) {
+		if (!strncmp(argv[i], "-march=", strlen("-march=")) ||
+		    !strncmp(argv[i], "-mcpu=",  strlen("-mcpu=" )))
+			break;
+	}
+	if (i == argc) {
+#ifdef BR_ARCH
+		*cur++ = "-march=" BR_ARCH;
+#endif
+#ifdef BR_CPU
+		*cur++ = "-mcpu=" BR_CPU;
+#endif
+	}
+#endif /* ARCH || CPU */
+	return cur;
+}
+
 static void check_unsafe_path(const char *path, int paranoid)
 {
 	char **c;

@@ -169,38 +208,7 @@ int main(int argc, char **argv)
 	memcpy(cur, predef_args, sizeof(predef_args));
 	cur += sizeof(predef_args) / sizeof(predef_args[0]);
 
-#ifdef BR_FLOAT_ABI
-	/* add float abi if not overridden in args */
-	for (i = 1; i < argc; i++) {
-		if (!strncmp(argv[i], "-mfloat-abi=", strlen("-mfloat-abi=")) ||
-		    !strcmp(argv[i], "-msoft-float") ||
-		    !strcmp(argv[i], "-mhard-float"))
-			break;
-	}
-
-	if (i == argc)
-		*cur++ = "-mfloat-abi=" BR_FLOAT_ABI;
-#endif
-
-#if defined(BR_ARCH) || \
-    defined(BR_CPU)
-	/* Add our -march/cpu/abi flags, but only if none are
-	 * already specified on the commandline
-	 */
-	for (i = 1; i < argc; i++) {
-		if (!strncmp(argv[i], "-march=", strlen("-march=")) ||
-		    !strncmp(argv[i], "-mcpu=",  strlen("-mcpu=" )))
-			break;
-	}
-	if (i == argc) {
-#ifdef BR_ARCH
-		*cur++ = "-march=" BR_ARCH;
-#endif
-#ifdef BR_CPU
-		*cur++ = "-mcpu=" BR_CPU;
-#endif
-	}
-#endif /* ARCH || CPU */
+	cur = add_gcc_args(argc, argv, cur);
 
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #9 lore raw

RFC v4 8/16 toolchain-external: use the wrapper even for the linker "ld"

In reply to Fabio Porcedda

In order to add the per-package staging feature we need to use the wrappper even for the linker "ld".

Signed-off-by Fabio Porcedda <[email protected]>

---
 .../toolchain-external/ext-toolchain-wrapper.c      | 21 +++++++++++++++------
 toolchain/toolchain-external/toolchain-external.mk  |  2 +-
 2 files changed, 16 insertions(+), 7 deletions(-)

toolchain/toolchain-external/ext-toolchain-wrapper.c +15 -6

index f8695a4..c839377 100644
@@ -39,9 +39,12 @@ static char sysroot[PATH_MAX];
  */
 #define EXCLUSIVE_ARGS	3
 
-static char *predef_args[] = {
+static char *predef_common_args[] = {
 	path,
-	"--sysroot", sysroot,
+	"--sysroot", sysroot
+};
+
+static char *predef_gcc_args[] = {
 #ifdef BR_ABI
 	"-mabi=" BR_ABI,
 #endif

@@ -197,7 +200,8 @@ int main(int argc, char **argv)
 		return 3;
 	}
 
-	cur = args = malloc(sizeof(predef_args) +
+	cur = args = malloc(sizeof(predef_common_args) +
+			    sizeof(predef_gcc_args) +
 			    (sizeof(char *) * (argc + EXCLUSIVE_ARGS)));
 	if (args == NULL) {
 		perror(__FILE__ ": malloc");

@@ -205,10 +209,15 @@ int main(int argc, char **argv)
 	}
 
 	/* start with predefined args */
-	memcpy(cur, predef_args, sizeof(predef_args));
-	cur += sizeof(predef_args) / sizeof(predef_args[0]);
+	memcpy(cur, predef_common_args, sizeof(predef_common_args));
+	cur += sizeof(predef_common_args) / sizeof(predef_common_args[0]);
 
-	cur = add_gcc_args(argc, argv, cur);
+	if (strcmp(path + strlen(path) - 2, "ld") !=  0) {
+		memcpy(cur, predef_gcc_args, sizeof(predef_gcc_args));
+		cur += sizeof(predef_gcc_args) / sizeof(predef_gcc_args[0]);
+
+		cur = add_gcc_args(argc, argv, cur);
+	}
 
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)

toolchain/toolchain-external/toolchain-external.mk +1 -1

index 22f8118..75d61a9 100644
@@ -673,7 +673,7 @@ define TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER
 		*-ar|*-ranlib|*-nm) \
 			ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%../..%') .; \
 			;; \
-		*cc|*cc-*|*++|*++-*|*cpp) \
+		*cc|*cc-*|*++|*++-*|*cpp|*ld) \
 			ln -sf ext-toolchain-wrapper $$base; \
 			;; \
 		*gdb|*gdbtui) \

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #10 lore raw

RFC v4 9/16 infra: add GCC_SYSROOT environment variable to support pps feature

In reply to Fabio Porcedda

In order to support the per-package staging feature, add the GCC_SYSROOT environment variable to the TARGET_CONFIGURE_OPTS and TARGET_MAKE_ENV variables.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/Makefile.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

package/Makefile.in +2 -1

index 2ed7cf7..924c808 100644
@@ -280,9 +280,10 @@ TARGET_CONFIGURE_OPTS = \
 	FCFLAGS="$(TARGET_FCFLAGS)" \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	STAGING_DIR="$(STAGING_DIR)" \
+	GCC_SYSROOT="$(STAGING_DIR)" \
 	INTLTOOL_PERL=$(PERL)
 
-TARGET_MAKE_ENV = PATH=$(BR_PATH)
+TARGET_MAKE_ENV = PATH=$(BR_PATH) GCC_SYSROOT="$(STAGING_DIR)"
 
 
 HOST_CONFIGURE_OPTS = \

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #11 lore raw

RFC v4 10/16 pkg-cmake: add GCC_SYSROOT environment variable to support pps feature

In reply to Fabio Porcedda

In order to support the per-package staging feature, add the GCC_SYSROOT environment variable and use it for building packages with cmake.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/pkg-cmake.mk                | 1 +
 support/misc/toolchainfile.cmake.in | 9 +++++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

package/pkg-cmake.mk +1 -0

index f17a883..8af1e48 100644
@@ -85,6 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
+	GCC_SYSROOT=$$(STAGING_DIR) \
 	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \

support/misc/toolchainfile.cmake.in +7 -2

index cd41254..fa82af4 100644
@@ -19,12 +19,17 @@ set(CMAKE_EXE_LINKER_FLAGS "@@TARGET_LDFLAGS@@ ${CMAKE_EXE_LINKER_FLAGS}" CACHE
 set(CMAKE_INSTALL_SO_NO_EXE 0)
 
 set(CMAKE_PROGRAM_PATH "${RELOCATED_HOST_DIR}/usr/bin")
-set(CMAKE_FIND_ROOT_PATH "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+if(DEFINED ENV{GCC_SYSROOT})
+	set(CMAKE_FIND_ROOT_PATH "$ENV{GCC_SYSROOT}")
+	set(ENV{PKG_CONFIG_SYSROOT_DIR} "$ENV{GCC_SYSROOT}")
+else()
+	set(CMAKE_FIND_ROOT_PATH "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+	set(ENV{PKG_CONFIG_SYSROOT_DIR} "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+endif()
 set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
 set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
 set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
 set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
-set(ENV{PKG_CONFIG_SYSROOT_DIR} "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
 
 # This toolchain file can be used both inside and outside Buildroot.
 # * When used inside Buildroot, ccache support is explicitly driven using the

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #12 lore raw

RFC v4 11/16 pkg-python: add GCC_SYSROOT variable to add pps support

In reply to Fabio Porcedda

In order to support the per-package staging feature, add the GCC_SYSROOT environment variable to the python packages.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/pkg-python.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-python.mk +2 -0

index 426301a..126846e 100644
@@ -27,6 +27,7 @@ PKG_PYTHON_DISTUTILS_ENV = \
 	CFLAGS="$(TARGET_CFLAGS)" \
 	LDFLAGS="$(TARGET_LDFLAGS)" \
 	LDSHARED="$(TARGET_CROSS)gcc -shared" \
+	GCC_SYSROOT="$(STAGING_DIR)" \
 	PYTHONPATH="$(if $(BR2_PACKAGE_PYTHON3),$(PYTHON3_PATH),$(PYTHON_PATH))" \
 	_python_sysroot=$(STAGING_DIR) \
 	_python_prefix=/usr \

@@ -51,6 +52,7 @@ HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS = \
 # Target setuptools-based packages
 PKG_PYTHON_SETUPTOOLS_ENV = \
 	PATH=$(BR_PATH) \
+	GCC_SYSROOT="$(STAGING_DIR)" \
 	PYTHONPATH="$(if $(BR2_PACKAGE_PYTHON3),$(PYTHON3_PATH),$(PYTHON_PATH))" \
 	_python_sysroot=$(STAGING_DIR) \
 	_python_prefix=/usr \

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #13 lore raw

RFC v4 12/16 pacakge/luarocks: add GCC_SYSROOT environment variable to support pps

In reply to Fabio Porcedda

In order to support the per-package staging feature, add the GCC_SYSROOT environment variable and use it for building luarocks pacakges.

Signed-off-by Fabio Porcedda <[email protected]>

---
 ...ctions-to-the-global-environment-of-the-c.patch | 54 +++++++++++++++
 package/luarocks/0002-fixed-typo.patch             | 27 ++++++++
 ...e-environment-table-for-the-system-config.patch | 76 ++++++++++++++++++++++
 ...is-not-being-integrated-bug-introduced-in.patch | 62 ++++++++++++++++++
 package/luarocks/luarocks.mk                       | 11 ++--
 5 files changed, 226 insertions(+), 4 deletions(-)
 create mode 100644 package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
 create mode 100644 package/luarocks/0002-fixed-typo.patch
 create mode 100644 package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
 create mode 100644 package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch

package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch new file +54 -0

new file mode 100644
index 0000000..f83c139
@@ -0,0 +1,54 @@
+From b3ea8783a4d73d437a27befc1d8ab47ef69d4ba1 Mon Sep 17 00:00:00 2001
+From: Thijs Schreijer <[email protected]>
+Date: Tue, 16 Jun 2015 19:31:01 +0200
+Subject: [PATCH 1/4] added 2 functions to the global environment of the
+ config-file loader sandbox; `os_getenv()` and `__dump_env()`
+
+(Upstream patch taken from master branch)
+(cherry picked from commit e1dd6375822d800984e1d3d821899a315bc1b222)
+---
+ src/luarocks/cfg.lua | 17 ++++++++++++++++-
+ 1 file changed, 16 insertions(+), 1 deletion(-)
+
+diff --git a/src/luarocks/cfg.lua b/src/luarocks/cfg.lua
+index 99b4077..cf0c0d3 100644
+--- a/src/luarocks/cfg.lua
++++ b/src/luarocks/cfg.lua
+@@ -145,6 +145,14 @@ end
+ cfg.variables = {}
+ cfg.rocks_trees = {}
+ 
++-- some extras for the global enviornment in the config files;
++cfg.os_getenv = os.getenv
++cfg.__dump_env = function()
++  -- debug function, calling it from a config file will show all 
++  -- available globals to that config file
++  print(util.show_table(cfg, "global environment"))
++end
++
+ sys_config_file = site_config.LUAROCKS_SYSCONFIG or sys_config_dir.."/config-"..cfg.lua_version..".lua"
+ local err, errcode
+ sys_config_ok, err, errcode = persist.load_into_table(sys_config_file, cfg)
+@@ -158,11 +166,18 @@ if (not sys_config_ok) and errcode ~= "open" then -- either "load" or "run"; bad
+    os.exit(cfg.errorcodes.CONFIGFILE)
+ end
+ 
+-local env_for_config_file = {
++local env_for_config_file
++env_for_config_file = {
+    home = cfg.home,
+    lua_version = cfg.lua_version,
+    platform = util.make_shallow_copy(detected),
+    processor = proc,
++   os_getenv = os.getenv, 
++   __dump_env = function()
++     -- debug function, calling it from a config file will show all 
++     -- available globals to that config file
++     print(util.show_table(env_for_config_file, "global environment"))
++   end,
+ }
+ 
+ if not site_config.LUAROCKS_FORCE_CONFIG then
+-- 
+2.4.3
+

package/luarocks/0002-fixed-typo.patch new file +27 -0

new file mode 100644
index 0000000..f4defef
@@ -0,0 +1,27 @@
+From 53c9599371b3e0815d62d3d34b1f1aef2b2877db Mon Sep 17 00:00:00 2001
+From: Thijs Schreijer <[email protected]>
+Date: Wed, 17 Jun 2015 11:01:48 +0200
+Subject: [PATCH 2/4] fixed typo
+
+(Upstream patch taken from master branch)
+(cherry picked from commit 4792618d8882eb36c9e9d05efac8b1ae336f00cb)
+---
+ src/luarocks/cfg.lua | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/src/luarocks/cfg.lua b/src/luarocks/cfg.lua
+index cf0c0d3..278be4a 100644
+--- a/src/luarocks/cfg.lua
++++ b/src/luarocks/cfg.lua
+@@ -145,7 +145,7 @@ end
+ cfg.variables = {}
+ cfg.rocks_trees = {}
+ 
+--- some extras for the global enviornment in the config files;
++-- some extras for the global environment in the config files;
+ cfg.os_getenv = os.getenv
+ cfg.__dump_env = function()
+   -- debug function, calling it from a config file will show all 
+-- 
+2.4.3
+

package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch new file +76 -0

76 lines not on this page · raw message

package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch new file +62 -0

62 lines not on this page · raw message

package/luarocks/luarocks.mk +7 -4

index 413e23d..042019f 100644
@@ -35,16 +35,17 @@ define HOST_LUAROCKS_INSTALL_CMDS
 	rm -f $(LUAROCKS_CONFIG_FILE)
 	$(MAKE1) -C $(@D) install \
 		PREFIX=$(HOST_DIR)/usr
+	echo "gcc_sysroot = os_getenv([[GCC_SYSROOT]])"         >> $(LUAROCKS_CONFIG_FILE)
 	echo "-- BR cross-compilation"                          >> $(LUAROCKS_CONFIG_FILE)
 	echo "variables = {"                                    >> $(LUAROCKS_CONFIG_FILE)
-	echo "   LUA_INCDIR = [[$(STAGING_DIR)/usr/include]],"  >> $(LUAROCKS_CONFIG_FILE)
-	echo "   LUA_LIBDIR = [[$(STAGING_DIR)/usr/lib]],"      >> $(LUAROCKS_CONFIG_FILE)
+	echo "   LUA_INCDIR = gcc_sysroot..[[/usr/include]],"   >> $(LUAROCKS_CONFIG_FILE)
+	echo "   LUA_LIBDIR = gcc_sysroot..[[/usr/lib]],"       >> $(LUAROCKS_CONFIG_FILE)
 	echo "   CC = [[$(TARGET_CC)]],"                        >> $(LUAROCKS_CONFIG_FILE)
 	echo "   LD = [[$(TARGET_CC)]],"                        >> $(LUAROCKS_CONFIG_FILE)
 	echo "   CFLAGS = [[$(LUAROCKS_CFLAGS)]],"              >> $(LUAROCKS_CONFIG_FILE)
 	echo "   LIBFLAG = [[-shared $(TARGET_LDFLAGS)]],"      >> $(LUAROCKS_CONFIG_FILE)
 	echo "}"                                                >> $(LUAROCKS_CONFIG_FILE)
-	echo "external_deps_dirs = { [[$(STAGING_DIR)/usr]] }"  >> $(LUAROCKS_CONFIG_FILE)
+	echo 'external_deps_dirs = { gcc_sysroot..[[/usr]] }'   >> $(LUAROCKS_CONFIG_FILE)
 	echo "gcc_rpath = false"                                >> $(LUAROCKS_CONFIG_FILE)
 	echo "rocks_trees = { [[$(TARGET_DIR)/usr]] }"          >> $(LUAROCKS_CONFIG_FILE)
 	echo "wrap_bin_scripts = false"                         >> $(LUAROCKS_CONFIG_FILE)

@@ -53,7 +54,9 @@ endef
 
 $(eval $(host-generic-package))
 
-LUAROCKS_RUN_ENV = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua"
+LUAROCKS_RUN_ENV = \
+	GCC_SYSROOT="$(STAGING_DIR)" \
+	LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua"
 LUAROCKS_RUN_CMD = $(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks
 
 LUAROCKS_RUN = $(LUAROCKS_RUN_ENV) $(LUAROCKS_RUN_CMD)

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #14 lore raw

RFC v4 13/16 package/pkgconf: use GCC_SYSROOT to support pps feature

In reply to Fabio Porcedda

In order to support the per-package staging feature, use the GCC_SYSROOT environment variable to find *.pc files in the per-package staging directory.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/aiccu/aiccu.mk                 |  2 +-
 package/fbv/fbv.mk                     |  2 +-
 package/gdk-pixbuf/gdk-pixbuf.mk       |  2 +-
 package/libmicrohttpd/libmicrohttpd.mk |  2 +-
 package/libupnpp/libupnpp.mk           |  2 +-
 package/lmbench/lmbench.mk             |  4 ++--
 package/logrotate/logrotate.mk         |  2 +-
 package/madplay/madplay.mk             |  2 +-
 package/mpg123/mpg123.mk               |  2 +-
 package/neardal/neardal.mk             |  2 +-
 package/nmap/nmap.mk                   |  2 +-
 package/openldap/openldap.mk           |  2 +-
 package/php/php.mk                     |  4 ++--
 package/pkgconf/pkg-config.in          |  4 +++-
 package/pkgconf/pkgconf.mk             |  5 ++---
 package/poppler/poppler.mk             |  2 +-
 package/qt/qt.mk                       | 12 ++++++------
 package/ti-utils/ti-utils.mk           |  4 ++--
 package/upmpdcli/upmpdcli.mk           |  2 +-
 package/vsftpd/vsftpd.mk               |  2 +-
 20 files changed, 31 insertions(+), 30 deletions(-)

package/aiccu/aiccu.mk +1 -1

index f2cd3de..b906b63 100644
@@ -16,7 +16,7 @@ AICCU_LFDLAGS = $(TARGET_LDFLAGS)
 # aiccu forgets to link with gnutls' dependencies breaking the build when
 # linking statically
 ifeq ($(BR2_STATIC_LIBS),y)
-AICCU_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --static --libs gnutls)
+AICCU_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --static --libs gnutls)
 endif
 
 # dummy RPM_OPT_FLAGS to disable stripping

package/fbv/fbv.mk +1 -1

index 8788b0e..993125d 100644
@@ -18,7 +18,7 @@ FBV_DEPENDENCIES += libpng
 
 # libpng in turn depends on other libraries
 ifeq ($(BR2_STATIC_LIBS),y)
-FBV_CONFIGURE_OPTS += "--libs=$(shell $(PKG_CONFIG_HOST_BINARY) --libs libpng --static)"
+FBV_CONFIGURE_OPTS += "--libs=$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libpng --static)"
 endif
 
 else

package/gdk-pixbuf/gdk-pixbuf.mk +1 -1

index eb30215..c408170 100644
@@ -35,7 +35,7 @@ GDK_PIXBUF_CONF_OPTS += --without-libtiff
 else
 GDK_PIXBUF_DEPENDENCIES += tiff host-pkgconf
 GDK_PIXBUF_CONF_ENV += \
-	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs libtiff-4)'
+	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libtiff-4)'
 endif
 
 ifeq ($(BR2_PACKAGE_XLIB_LIBX11),y)

package/libmicrohttpd/libmicrohttpd.mk +1 -1

index a6662a7..ebaf248 100644
@@ -13,7 +13,7 @@ LIBMICROHTTPD_CONF_OPTS = --disable-curl --disable-spdy --disable-examples
 ifeq ($(BR2_PACKAGE_LIBMICROHTTPD_SSL),y)
 LIBMICROHTTPD_LICENSE = LGPLv2.1+
 LIBMICROHTTPD_DEPENDENCIES += host-pkgconf gnutls libgcrypt
-LIBMICROHTTPD_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs gnutls)"
+LIBMICROHTTPD_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs gnutls)"
 LIBMICROHTTPD_CONF_OPTS += --enable-https --with-gnutls=$(STAGING_DIR)/usr \
 	--with-libgcrypt-prefix=$(STAGING_DIR)/usr
 else

package/libupnpp/libupnpp.mk +1 -1

index defc16a..3960c2b 100644
@@ -16,7 +16,7 @@ LIBUPNPP_DEPENDENCIES = expat libcurl libupnp
 ifeq ($(BR2_STATIC_LIBS),y)
 LIBUPNPP_DEPENDENCIES += host-pkgconf
 LIBUPNPP_CONF_ENV += \
-	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs libupnp libcurl)'
+	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libupnp libcurl)'
 endif
 
 $(eval $(autotools-package))

package/lmbench/lmbench.mk +2 -2

index d4be8d4..92ea257 100644
@@ -14,8 +14,8 @@ LMBENCH_CFLAGS = $(TARGET_CFLAGS)
 
 ifeq ($(BR2_PACKAGE_LIBTIRPC),y)
 LMBENCH_DEPENDENCIES += host-pkgconf libtirpc
-LMBENCH_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags libtirpc)
-LMBENCH_LDLIBS = $(shell $(PKG_CONFIG_HOST_BINARY) --libs libtirpc)
+LMBENCH_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags libtirpc)
+LMBENCH_LDLIBS = $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libtirpc)
 endif
 
 define LMBENCH_CONFIGURE_CMDS

package/logrotate/logrotate.mk +1 -1

index 0af7f4a..7cfb7a1 100644
@@ -11,7 +11,7 @@ LOGROTATE_LICENSE_FILES = COPYING
 LOGROTATE_DEPENDENCIES = popt host-pkgconf
 # tarball does not have a generated configure script
 LOGROTATE_AUTORECONF = YES
-LOGROTATE_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs popt)"
+LOGROTATE_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs popt)"
 LOGROTATE_CONF_OPTS = --without-selinux
 
 ifeq ($(BR2_PACKAGE_ACL),y)

package/madplay/madplay.mk +1 -1

index ff12aa1..c39d0e8 100644
@@ -16,7 +16,7 @@ MADPLAY_DEPENDENCIES = libmad libid3tag $(if $(BR2_PACKAGE_GETTEXT),gettext)
 ifeq ($(BR2_PACKAGE_MADPLAY_ALSA),y)
 MADPLAY_CONF_OPTS += --with-alsa
 MADPLAY_DEPENDENCIES += host-pkgconf alsa-lib
-MADPLAY_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs alsa)"
+MADPLAY_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs alsa)"
 endif
 
 $(eval $(autotools-package))

package/mpg123/mpg123.mk +1 -1

index a16e400..f02ba9c 100644
@@ -54,7 +54,7 @@ MPG123_CONF_OPTS += --with-default-audio=portaudio
 MPG123_DEPENDENCIES += portaudio
 # configure script does NOT use pkg-config to figure out how to link
 # with portaudio, breaking static linking as portaudio uses pthreads
-MPG123_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs portaudio-2.0)'
+MPG123_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs portaudio-2.0)'
 endif
 
 ifeq ($(BR2_PACKAGE_SDL),y)

package/neardal/neardal.mk +1 -1

index ddaecbe..1a7ec7f 100644
@@ -25,7 +25,7 @@ endif
 # forgets to take that into account, causing the detection to fail
 # when linking statically
 ifeq ($(BR2_STATIC_LIBS),y)
-NEARDAL_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs ncurses)'
+NEARDAL_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs ncurses)'
 endif
 
 define NEARDAL_INSTALL_NCL

package/nmap/nmap.mk +1 -1

index d46d58b..35c6a82 100644
@@ -30,7 +30,7 @@ NMAP_POST_PATCH_HOOKS += NMAP_WRAPPER_EXEC
 ifeq ($(BR2_PACKAGE_OPENSSL),y)
 NMAP_CONF_OPTS += --with-openssl="$(STAGING_DIR)/usr"
 NMAP_DEPENDENCIES += host-pkgconf openssl
-NMAP_LIBS_FOR_STATIC_LINK += $(shell $(PKG_CONFIG_HOST_BINARY) --libs --static openssl)
+NMAP_LIBS_FOR_STATIC_LINK += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs --static openssl)
 else
 NMAP_CONF_OPTS += --without-openssl
 endif

package/openldap/openldap.mk +1 -1

index 68464ab..93facb4 100644
@@ -31,7 +31,7 @@ endif
 ifeq ($(BR2_PACKAGE_OPENSSL),y)
 OPENLDAP_MP = bignum
 OPENLDAP_DEPENDENCIES += openssl
-OPENLDAP_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs libssl libcrypto)"
+OPENLDAP_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libssl libcrypto)"
 else ifeq ($(BR2_PACKAGE_GMP),y)
 OPENLDAP_MP = gmp
 OPENLDAP_DEPENDENCIES += gmp

package/php/php.mk +2 -2

index c9bcfe3..f7c064c 100644
@@ -116,7 +116,7 @@ PHP_CONF_OPTS += --with-openssl=$(STAGING_DIR)/usr
 PHP_DEPENDENCIES += openssl
 # openssl needs zlib, but the configure script forgets to link against
 # it causing detection failures with static linking
-PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs --static openssl)
+PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs --static openssl)
 endif
 
 ifeq ($(BR2_PACKAGE_PHP_EXT_LIBXML2),y)

@@ -187,7 +187,7 @@ endif
 ifeq ($(BR2_PACKAGE_PHP_EXT_SQLITE),y)
 PHP_CONF_OPTS += --with-sqlite3=$(STAGING_DIR)/usr
 PHP_DEPENDENCIES += sqlite
-PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs --static sqlite3)
+PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs --static sqlite3)
 endif
 
 ### PDO

package/pkgconf/pkg-config.in +3 -1

index 4dec487..9e5408f 100644
@@ -1,2 +1,4 @@
 #!/bin/sh
-PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:- at PKG_CONFIG_LIBDIR@} PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:- at STAGING_DIR@} $(dirname $0)/pkgconf @STATIC@ $@
+PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:-$GCC_SYSROOT/usr/lib/pkgconfig:$GCC_SYSROOT/usr/share/pkgconfig} \
+    PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:-$GCC_SYSROOT} \
+    $(dirname $0)/pkgconf @STATIC@ $@

package/pkgconf/pkgconf.mk +2 -3

index 3c93fd5..0c496be 100644
@@ -11,6 +11,8 @@ PKGCONF_LICENSE = pkgconf license
 PKGCONF_LICENSE_FILES = COPYING
 
 PKG_CONFIG_HOST_BINARY = $(HOST_DIR)/usr/bin/pkg-config
+PKG_CONFIG_HOST_BINARY_GCC_SYSROOT = \
+	GCC_SYSROOT=$(STAGING_DIR) $(HOST_DIR)/usr/bin/pkg-config
 
 define PKGCONF_LINK_PKGCONFIG
 	ln -sf pkgconf $(TARGET_DIR)/usr/bin/pkg-config

@@ -19,9 +21,6 @@ endef
 define HOST_PKGCONF_INSTALL_WRAPPER
 	$(INSTALL) -m 0755 -D package/pkgconf/pkg-config.in \
 		$(HOST_DIR)/usr/bin/pkg-config
-	$(SED) 's, at PKG_CONFIG_LIBDIR@,$(STAGING_DIR)/usr/lib/pkgconfig:$(STAGING_DIR)/usr/share/pkgconfig,' \
-		-e 's, at STAGING_DIR@,$(STAGING_DIR),' \
-		$(HOST_DIR)/usr/bin/pkg-config
 endef
 
 define HOST_PKGCONF_STATIC

package/poppler/poppler.mk +1 -1

index 234b7cc..c5ff161 100644
@@ -24,7 +24,7 @@ ifeq ($(BR2_PACKAGE_TIFF),y)
 POPPLER_CONF_OPTS += --enable-libtiff
 # Help poppler to find libtiff in static linking scenarios
 POPPLER_CONF_ENV += \
-	LIBTIFF_LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs libtiff-4)"
+	LIBTIFF_LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libtiff-4)"
 POPPLER_DEPENDENCIES += tiff
 else
 POPPLER_CONF_OPTS += --disable-libtiff

package/qt/qt.mk +6 -6

index 0644fbe..3a9eda6 100644
@@ -230,9 +230,9 @@ QT_DEPENDENCIES += fontconfig xlib_libXi xlib_libX11 xlib_libXrender \
                 xlib_libXcursor xlib_libXrandr xlib_libXext xlib_libXv
 # Using pkg-config avoids us some logic to redefine and sed again mkspecs files
 # to add X11 include path and link options
-QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
-QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
-QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs x11 xext`
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags x11`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags x11`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs x11 xext`
 QT_CONFIGURE_OPTS += -arch $(QT_EMB_PLATFORM) \
 		-xplatform qws/linux-$(QT_EMB_PLATFORM)-g++ -x11 -no-gtkstyle -no-sm \
 		-no-openvg

@@ -329,9 +329,9 @@ endif
 ifeq ($(BR2_PACKAGE_QT_OPENGL_ES),y)
 QT_CONFIGURE_OPTS += -opengl es2 -egl
 QT_DEPENDENCIES += libgles libegl
-QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
-QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
-QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs egl`
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags egl`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags egl`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs egl`
 else
 QT_CONFIGURE_OPTS += -no-opengl
 endif

package/ti-utils/ti-utils.mk +2 -2

index 35ecc19..19d53fd 100644
@@ -10,8 +10,8 @@ TI_UTILS_DEPENDENCIES = libnl host-pkgconf
 TI_UTILS_LICENSE = BSD-3c
 TI_UTILS_LICENSE_FILES = COPYING
 
-TI_UTILS_CFLAGS = $(shell $(PKG_CONFIG_HOST_BINARY) --cflags libnl-genl-3.0)
-TI_UTILS_LIBS = $(shell $(PKG_CONFIG_HOST_BINARY) --libs libnl-genl-3.0)
+TI_UTILS_CFLAGS = $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags libnl-genl-3.0)
+TI_UTILS_LIBS = $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libnl-genl-3.0)
 
 define TI_UTILS_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE1) NFSROOT="$(STAGING_DIR)" \

package/upmpdcli/upmpdcli.mk +1 -1

index a5c6370..1587a85 100644
@@ -19,7 +19,7 @@ ifeq ($(BR2_STATIC_LIBS),y)
 # manually pass it to make.
 UPMPDCLI_DEPENDENCIES += host-pkgconf
 UPMPDCLI_MAKE_OPTS = \
-	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs expat libcurl libupnp)'
+	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs expat libcurl libupnp)'
 endif
 
 # Upmpdcli only runs if user upmpdcli exists

package/vsftpd/vsftpd.mk +1 -1

index db85015..b6517d4 100644
@@ -24,7 +24,7 @@ endif
 
 ifeq ($(BR2_PACKAGE_OPENSSL),y)
 VSFTPD_DEPENDENCIES += openssl host-pkgconf
-VSFTPD_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs libssl libcrypto)
+VSFTPD_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libssl libcrypto)
 VSFTPD_POST_CONFIGURE_HOOKS += VSFTPD_ENABLE_SSL
 endif

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #15 lore raw

RFC v4 14/16 Makefile: add STAGINGNOPKG_DIR variable

In reply to Fabio Porcedda

When the per-package staging feature will be added, the STAGING_DIR variable will be used to refer to the per-package staging directory so to be able to refer to the original STAGING_DIR a STAGINGNOPKG_DIR is added.

Signed-off-by Fabio Porcedda <[email protected]>

---
 Makefile            | 14 +++++---------
 package/Makefile.in |  1 +
 2 files changed, 6 insertions(+), 9 deletions(-)

Makefile +5 -9

index a0d4abb..b4e5a10 100644
@@ -437,15 +437,11 @@ else
 LIB_SYMLINK = lib32
 endif
 
-$(STAGING_DIR):
-	@mkdir -p $(STAGING_DIR)/bin
-	@mkdir -p $(STAGING_DIR)/lib
-	@ln -snf lib $(STAGING_DIR)/$(LIB_SYMLINK)
-	@mkdir -p $(STAGING_DIR)/usr/lib
-	@ln -snf lib $(STAGING_DIR)/usr/$(LIB_SYMLINK)
-	@mkdir -p $(STAGING_DIR)/usr/include
-	@mkdir -p $(STAGING_DIR)/usr/bin
-	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
+$(STAGINGNOPKG_DIR):
+	@mkdir -p $@/bin $@/lib $@/usr/lib $@/usr/include $@/usr/bin
+	@ln -snf lib $@/$(LIB_SYMLINK)
+	@ln -snf lib $@/usr/$(LIB_SYMLINK)
+	@ln -snf $@ $(BASE_DIR)/staging
 
 ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 TARGET_SKELETON = $(BR2_ROOTFS_SKELETON_CUSTOM_PATH)

package/Makefile.in +1 -0

index 924c808..169c6b3 100644
@@ -109,6 +109,7 @@ endif
 
 STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
 STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
+STAGINGNOPKG_DIR := $(STAGING_DIR)
 
 TARGET_OPTIMIZATION := $(call qstrip,$(BR2_TARGET_OPTIMIZATION))

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #16 lore raw

RFC v4 15/16 pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages

In reply to Fabio Porcedda

This will simplify the commit that add the per-package staging directory.

Signed-off-by Fabio Porcedda <[email protected]>

---
 package/pkg-generic.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

package/pkg-generic.mk +6 -3

index 3da5a87..376b025 100644
@@ -394,17 +394,20 @@ $(2)_REDISTRIBUTE		?= YES
 # When a target package is a toolchain dependency set this variable to
 # 'NO' so the 'toolchain' dependency is not added to prevent a circular
 # dependency
-$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+ifeq ($(4),target)
+ $(2)_ADD_TOOLCHAIN_DEPENDENCY ?= YES
+else
+ $(2)_ADD_TOOLCHAIN_DEPENDENCY ?= NO
+endif
 
 ifeq ($(4),host)
 $(2)_DEPENDENCIES ?= $$(filter-out  host-toolchain $(1),\
 	$$(patsubst host-host-%,host-%,$$(addprefix host-,$$($(3)_DEPENDENCIES))))
 endif
-ifeq ($(4),target)
+
 ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
 $(2)_DEPENDENCIES += toolchain
 endif
-endif
 
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))

Signature · 1 line

2.4.3

Fabio Porcedda

author

2015-06-28 19:42 UTC #17 lore raw

RFC v4 16/16 infra: add per-package staging feature

In reply to Fabio Porcedda

To improve the build reproducibility, instead of sharing a common staging are, every package has a private staging directory where only the declared dependency are present.

Having a per-package staging directory ensure that every package use only the declared dependency, because even if a undeclared dependency is build before, it will be not included in the private package directory.

This feature is useful for both top-level non-parallel make and top-level parallel make.

This feature is useful for target dependencies but not for host dependencies, to solve that a similar feature for host packages must be developed.

Implementation details:
To minimize change the STAGING_DIR variable will be redefined for each package to point to the per-package staging directory.

At the beginning of each configuration stage the per-package staging directory will be created copying using hard links all the declared target dependencies.
Also each path of every copied configuration files will be changed to point to the per-package

Signed-off-by Fabio Porcedda <[email protected]>

---
 Makefile               |  2 +-
 package/Makefile.in    |  1 +
 package/pkg-cmake.mk   |  2 +-
 package/pkg-generic.mk | 34 +++++++++++++++++++++++++++++++++-
 4 files changed, 36 insertions(+), 3 deletions(-)

Makefile +1 -1

index b4e5a10..01a7198 100644
@@ -800,7 +800,7 @@ printvars:
 
 clean:
 	rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
-		$(BUILD_DIR) $(BASE_DIR)/staging \
+		$(BUILD_DIR) $(BASE_DIR)/staging $(STAGINGPKG_DIR) \
 		$(LEGAL_INFO_DIR) $(GRAPHS_DIR)
 
 distclean: clean

package/Makefile.in +1 -0

index 169c6b3..ea26772 100644
@@ -110,6 +110,7 @@ endif
 STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
 STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
 STAGINGNOPKG_DIR := $(STAGING_DIR)
+STAGINGPKG_DIR = $(BASE_DIR)/stagingpkg
 
 TARGET_OPTIMIZATION := $(call qstrip,$(BR2_TARGET_OPTIMIZATION))

package/pkg-cmake.mk +1 -1

index 8af1e48..eb80466 100644
@@ -85,7 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	GCC_SYSROOT=$$(STAGING_DIR) \
+	GCC_SYSROOT=$$($(2)_STAGING_DIR) \
 	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \

package/pkg-generic.mk +33 -1

index 376b025..edac936 100644
@@ -221,7 +221,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
-	rm -Rf $(@D)
+	rm -Rf $(@D) $(STAGING_DIR)
 
 ################################################################################
 # virt-provides-single -- check that provider-pkg is the declared provider for

@@ -414,6 +414,33 @@ $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
 $(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))
 
+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+  $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
+	$$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
+	$$($$(call UPPERCASE,$$(dep))_STAGING_DIR)/))
+
+  $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
+
+  define $(2)_PREPARE_STAGING_DIR
+	mkdir -p $$($(2)_STAGING_DIR)/usr/mkspecs/qws
+	cp -rl $(STAGING_DIR)/* $$($(2)_STAGING_DIR)
+	$$(if $$($(2)_STAGING_DIRS), $$(foreach dir,$$($(2)_STAGING_DIRS),\
+		rsync -au --link-dest=$$(dir) $$(dir) $$($(2)_STAGING_DIR); ))
+	find $$($(2)_STAGING_DIR)/usr/{bin,lib,mkspecs/qws} \
+		-ignore_readdir_race \
+		-name "*[-_]config" -or -name "*.la" -or -name "*.pc" -or \
+		-name "*.prl" -or -name "qmake.conf" | xargs -r sed -i -r \
+		-e "s|$$(STAGINGPKG_DIR)/[^/]*/+usr|$$($(2)_STAGING_DIR)/usr|g" \
+		-e "s|$$(BUILD_DIR)/[^ ]+/([^/ ]+).la|$$($(2)_STAGING_DIR)/usr/lib/\1.la|g" \
+		-e "s|/lib/libpulsecommon|/lib/pulseaudio/libpulsecommon|g"
+  endef
+
+  $(2)_PRE_CONFIGURE_HOOKS := \
+	$(2)_PREPARE_STAGING_DIR $$($(2)_PRE_CONFIGURE_HOOKS)
+else
+  $(2)_STAGING_DIR = $$(STAGING_DIR)
+endif
+
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
 $(2)_INSTALL_TARGET		?= YES

@@ -617,11 +644,15 @@ $(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
+$$($(2)_TARGET_INSTALL_TARGET):		STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
+$$($(2)_TARGET_INSTALL_STAGING):	STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
+$$($(2)_TARGET_BUILD):			STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
+$$($(2)_TARGET_CONFIGURE):		STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_RSYNC):                  SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
 $$($(2)_TARGET_RSYNC):                  PKG=$(2)
 $$($(2)_TARGET_PATCH):			PKG=$(2)

@@ -631,6 +662,7 @@ $$($(2)_TARGET_EXTRACT):		PKG=$(2)
 $$($(2)_TARGET_SOURCE):			PKG=$(2)
 $$($(2)_TARGET_SOURCE):			PKGDIR=$(pkgdir)
 $$($(2)_TARGET_DIRCLEAN):		PKG=$(2)
+$$($(2)_TARGET_DIRCLEAN):		STAGING_DIR:=$$($(2)_STAGING_DIR)
 
 # Compute the name of the Kconfig option that correspond to the
 # package being enabled. We handle three cases: the special Linux

Signature · 1 line

2.4.3

Romain Naour

2015-07-11 22:56 UTC #18 lore raw

RFC v4 16/16 infra: add per-package staging feature

Hi Fabio,

Le 28/06/2015 21:42, Fabio Porcedda a ?crit :

Quoted 33 lines

To improve the build reproducibility, instead of sharing a common staging are, every package has a private staging directory where only the declared dependency are present.
Having a per-package staging directory ensure that every package use only the declared dependency, because even if a undeclared dependency is build before, it will be not included in the private package directory.
This feature is useful for both top-level non-parallel make and top-level parallel make.
This feature is useful for target dependencies but not for host dependencies, to solve that a similar feature for host packages must be developed.
Implementation details:
To minimize change the STAGING_DIR variable will be redefined for each package to point to the per-package staging directory.
At the beginning of each configuration stage the per-package staging directory will be created copying using hard links all the declared target dependencies.
Also each path of every copied configuration files will be changed to point to the per-package
Signed-off-by Fabio Porcedda <[email protected]>
---
 Makefile               |  2 +-
 package/Makefile.in    |  1 +
 package/pkg-cmake.mk   |  2 +-
 package/pkg-generic.mk | 34 +++++++++++++++++++++++++++++++++-
 4 files changed, 36 insertions(+), 3 deletions(-)

[snip]

Quoted 17 lines

package/pkg-generic.mk +1 -1

index 376b025..edac936 100644
@@ -221,7 +221,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
-	rm -Rf $(@D)
+	rm -Rf $(@D) $(STAGING_DIR)
 
 ################################################################################
 # virt-provides-single -- check that provider-pkg is the declared provider for

@@ -414,6 +414,33 @@ $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
 $(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))

package/pkg-generic.mk $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES)) · line 426

Quoted 10 lines

package/pkg-generic.mk +10 -0

+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+  $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
+	$$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
+	$$($$(call UPPERCASE,$$(dep))_STAGING_DIR)/))
+
+  $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
+
+  define $(2)_PREPARE_STAGING_DIR
+	mkdir -p $$($(2)_STAGING_DIR)/usr/mkspecs/qws
+	cp -rl $(STAGING_DIR)/* $$($(2)_STAGING_DIR)

I tried your series with codesourcery ARM 2014.05 with thumb2 "ARM instruction set" enabled:

BR2_arm=y
BR2_cortex_a9=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM201405
BR2_ARM_INSTRUCTIONS_THUMB2

And the build fail immediately when copying the first stagingpkg directory due to a symlink in staging.

Quoted 1 line

Quoted 1 line
Quoted 1 line
dash 0.5.8 Updating config.sub and config.guess

for file in config.guess config.sub; do for i in $(find /home/naourr/git/buildroot/output/build/dash-0.5.8 -name $file); do cp support/gnuconfig/$file $i; done; done

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
dash 0.5.8 Patching libtool
dash 0.5.8 Configuring

mkdir -p /home/naourr/git/buildroot/output/stagingpkg/dash/usr/mkspecs/qws
cp -rl /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/* /home/naourr/git/buildroot/output/stagingpkg/dash
cp: cannot copy cyclic symbolic link ? /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/thumb2/thumb2 ? package/pkg-generic.mk:146: recipe for target '/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured' failed
make: *** [/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured] Error 1

Here is the content of the staging directory:
ls -l output/staging/
total 20
drwxr-xr-x 2 naourr naourr 4096 12 juil. 00:35 bin
drwxr-xr-x 2 naourr naourr 4096 30 mai 2014 etc
drwxr-xr-x 2 naourr naourr 4096 30 mai 2014 lib
lrwxrwxrwx 1 naourr naourr 3 12 juil. 00:35 lib32 -> lib
drwxr-xr-x 2 naourr naourr 4096 30 mai 2014 sbin
lrwxrwxrwx 1 naourr naourr 2 12 juil. 00:35 thumb2 -> ./
drwxr-xr-x 8 naourr naourr 4096 30 mai 2014 usr

If you use "cp -drl" or better yet "cp -al" (Thanks Yann) in order to not follow symbolic links and preserve the link, the issue is fixed :)

Thoughts ?

Best regards,
Romain Naour

Quoted 11 lines

package/pkg-generic.mk +11 -0

+	$$(if $$($(2)_STAGING_DIRS), $$(foreach dir,$$($(2)_STAGING_DIRS),\
+		rsync -au --link-dest=$$(dir) $$(dir) $$($(2)_STAGING_DIR); ))
+	find $$($(2)_STAGING_DIR)/usr/{bin,lib,mkspecs/qws} \
+		-ignore_readdir_race \
+		-name "*[-_]config" -or -name "*.la" -or -name "*.pc" -or \
+		-name "*.prl" -or -name "qmake.conf" | xargs -r sed -i -r \
+		-e "s|$$(STAGINGPKG_DIR)/[^/]*/+usr|$$($(2)_STAGING_DIR)/usr|g" \
+		-e "s|$$(BUILD_DIR)/[^ ]+/([^/ ]+).la|$$($(2)_STAGING_DIR)/usr/lib/\1.la|g" \
+		-e "s|/lib/libpulsecommon|/lib/pulseaudio/libpulsecommon|g"
+  endef
+

[snip]

Fabio Porcedda

author

2015-07-13 09:42 UTC #19 lore raw

RFC v4 16/16 infra: add per-package staging feature

Quoted 100 lines · Romain Naour

Hi Fabio,
Le 28/06/2015 21:42, Fabio Porcedda a ?crit :
Quoted 33 lines
To improve the build reproducibility, instead of sharing a common staging are, every package has a private staging directory where only the declared dependency are present.
Having a per-package staging directory ensure that every package use only the declared dependency, because even if a undeclared dependency is build before, it will be not included in the private package directory.
This feature is useful for both top-level non-parallel make and top-level parallel make.
This feature is useful for target dependencies but not for host dependencies, to solve that a similar feature for host packages must be developed.
Implementation details:
To minimize change the STAGING_DIR variable will be redefined for each package to point to the per-package staging directory.
At the beginning of each configuration stage the per-package staging directory will be created copying using hard links all the declared target dependencies.
Also each path of every copied configuration files will be changed to point to the per-package
Signed-off-by Fabio Porcedda <[email protected]>
---
 Makefile               |  2 +-
 package/Makefile.in    |  1 +
 package/pkg-cmake.mk   |  2 +-
 package/pkg-generic.mk | 34 +++++++++++++++++++++++++++++++++-
 4 files changed, 36 insertions(+), 3 deletions(-)
[snip]
Quoted 17 lines
package/pkg-generic.mk +1 -1
index 376b025..edac936 100644
@@ -221,7 +221,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
-     rm -Rf $(@D)
+     rm -Rf $(@D) $(STAGING_DIR)
 
 ################################################################################
 # virt-provides-single -- check that provider-pkg is the declared provider for
@@ -414,6 +414,33 @@ $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
 $(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))
 
package/pkg-generic.mk $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES)) · line 426
Quoted 10 lines
package/pkg-generic.mk +10 -0
+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+  $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
+     $$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
+     $$($$(call UPPERCASE,$$(dep))_STAGING_DIR)/))
+
+  $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
+
+  define $(2)_PREPARE_STAGING_DIR
+     mkdir -p $$($(2)_STAGING_DIR)/usr/mkspecs/qws
+     cp -rl $(STAGING_DIR)/* $$($(2)_STAGING_DIR)
I tried your series with codesourcery ARM 2014.05 with thumb2 "ARM instruction set" enabled:
BR2_arm=y
BR2_cortex_a9=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM201405
BR2_ARM_INSTRUCTIONS_THUMB2
And the build fail immediately when copying the first stagingpkg directory due to a symlink in staging.
Quoted 1 line
Quoted 1 line
Quoted 1 line
dash 0.5.8 Updating config.sub and config.guess
for file in config.guess config.sub; do for i in $(find /home/naourr/git/buildroot/output/build/dash-0.5.8 -name $file); do cp support/gnuconfig/$file $i; done; done
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
dash 0.5.8 Patching libtool
dash 0.5.8 Configuring
mkdir -p /home/naourr/git/buildroot/output/stagingpkg/dash/usr/mkspecs/qws
cp -rl /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/* /home/naourr/git/buildroot/output/stagingpkg/dash
cp: cannot copy cyclic symbolic link ? /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/thumb2/thumb2 ? package/pkg-generic.mk:146: recipe for target '/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured' failed
make: *** [/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured] Error 1
Here is the content of the staging directory:
ls -l output/staging/
total 20
drwxr-xr-x 2 naourr naourr 4096 12 juil. 00:35 bin
drwxr-xr-x 2 naourr naourr 4096 30 mai 2014 etc
drwxr-xr-x 2 naourr naourr 4096 30 mai 2014 lib
lrwxrwxrwx 1 naourr naourr 3 12 juil. 00:35 lib32 -> lib
drwxr-xr-x 2 naourr naourr 4096 30 mai 2014 sbin
lrwxrwxrwx 1 naourr naourr 2 12 juil. 00:35 thumb2 -> ./
drwxr-xr-x 8 naourr naourr 4096 30 mai 2014 usr

Quoted 5 lines

If you use "cp -drl" or better yet "cp -al" (Thanks Yann) in order to not follow symbolic links and preserve the link, the issue is fixed :)
Thoughts ?

Hi Romain,
thanks for the report and for the solution :), I will use "cp -al" in the updated version of this patch set.

[snip]

Best regards

Signature · 1 line

Fabio Porcedda

Fabio Porcedda

author

2015-06-28 19:53 UTC #20 lore raw

v4 cover Add per-package staging feature

In reply to Fabio Porcedda

Quoted 83 lines · Fabio Porcedda

Hi all,
this patch set aims to improve build reproducibility by using a per-package staging directory.
Also this feature aims to enable the top-level parallel make.
This patch set is available at:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4
Every package instead of installing the files in the shared staging directory (STAGING_DIR) install all files in a subdirectory of the STAGINGPKG_DIR.
At the beginning of the configure stage, all the dependencies staging directory are copied in the package staging directory, e.g.:
zlib-install-staging: install files in /output/stagingpkg/zlib/
cramsfs-configure: copy all files from /output/stagingpkg/zlib/* to
/output/stagingpkg/cramfs/
The STAGING_DIR variable is redefined for every package in order to point to the per-package directory, so a package cannot access automatically to staging files of packages that aren't in the (PKG)_DENDENDENCIES variable.
To improve performance and reduce space utilization hard links are used.
To evaluate performance and space utilization I've done some tests.
HW-MED:
CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
RAM: 16GB
SSD: 512GB
HW-HIGH:
CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
RAM: 32GB
SATA: RAID0 2x2TB 3.5" 7200rpm
defconfig-small, long chain dependency, unfavorable scenario for parallelization:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_LIBGTK3=y
defconfig-full:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
make allyespackageconfig
Also i removed few packages that do not built on my system. make show-targets | wc -w
1366
Test about output using hard links or note using the "defconfig-full" on HW-MED:
|  35GB |  99m | master branch
|  37GB | 100m | patch set
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all
Test about performances of this patch set vs master branch:
| 199m | HW-MED | defconfig-full | master branch | no top-level make |
|  99m | HW-MED | defconfig-full | master branch | top-level make    |
| 100m | HW-MED | defconfig-full | patch set     | top-level make    |
| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
|  73m | HW-HIGH | defconfig-full | master branch | top-level make    |
|  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |
| 10m | HW-MED | defconfig-small | master branch | no top-level make |
|  5m | HW-MED | defconfig-small | master branch | top-level make    |
|  5m | HW-MED | defconfig-samll | patch set     | top-level make    |
| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
|  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    |
|  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

Quoted 4 lines

TODO:
- use the toolchaing wrapper even for the internal toolchain
- rebase over master (this patch set is based on 2015.05-161-gca5f11e).
- more testing (tested only for arm target).

v4:
- Rewritten the path set to use the gcc --sysroot option trough the
toolchain wrapper.

Quoted 101 lines

Best regads

 Makefile                                           | 16 ++--
 package/Makefile.in                                |  5 +-
 package/aiccu/aiccu.mk                             |  2 +-
 package/blktrace/blktrace.mk                       |  2 +-
 package/boost/boost.mk                             |  2 +-
 package/botan/botan.mk                             |  2 +-
 package/bsdiff/bsdiff.mk                           |  4 +-
 package/dbus-python/dbus-python.mk                 |  8 +-
 package/dhcpdump/dhcpdump.mk                       |  3 +-
 package/dtc/dtc.mk                                 |  3 +-
 package/exim/exim.mk                               |  5 +-
 package/fbv/fbv.mk                                 |  2 +-
 package/feh/feh.mk                                 |  4 +-
 package/gadgetfs-test/gadgetfs-test.mk             |  2 +-
 package/gdk-pixbuf/gdk-pixbuf.mk                   |  2 +-
 package/iodine/iodine.mk                           |  2 +-
 package/iw/iw.mk                                   |  2 +-
 package/jack2/jack2.mk                             |  3 +-
 package/libmicrohttpd/libmicrohttpd.mk             |  2 +-
 package/libnss/libnss.mk                           |  4 +-
 package/libupnpp/libupnpp.mk                       |  2 +-
 package/lmbench/lmbench.mk                         |  8 +-
 package/logrotate/logrotate.mk                     |  2 +-
 package/lua-msgpack-native/lua-msgpack-native.mk   |  3 +-
 ...ctions-to-the-global-environment-of-the-c.patch | 54 ++++++++++++
 package/luarocks/0002-fixed-typo.patch             | 27 ++++++
 ...e-environment-table-for-the-system-config.patch | 76 +++++++++++++++++
 ...is-not-being-integrated-bug-introduced-in.patch | 62 ++++++++++++++
 package/luarocks/luarocks.mk                       | 11 ++-
 package/madplay/madplay.mk                         |  2 +-
 package/moarvm/moarvm.mk                           |  2 +-
 package/mongoose/mongoose.mk                       |  8 +-
 package/mpg123/mpg123.mk                           |  2 +-
 package/mplayer/mplayer.mk                         |  2 +-
 package/ncurses/ncurses.mk                         |  4 +-
 package/neardal/neardal.mk                         |  2 +-
 package/nginx/nginx.mk                             |  1 +
 package/nmap/nmap.mk                               |  2 +-
 package/open2300/open2300.mk                       |  2 +-
 package/openldap/openldap.mk                       |  2 +-
 package/openssl/openssl.mk                         |  2 +-
 package/opentyrian/opentyrian.mk                   |  2 +-
 package/pax-utils/pax-utils.mk                     |  3 +-
 package/php/php.mk                                 |  4 +-
 package/pifmrds/pifmrds.mk                         |  3 +-
 package/pkg-cmake.mk                               |  1 +
 package/pkg-generic.mk                             | 43 +++++++++-
 package/pkg-perl.mk                                |  2 +-
 package/pkg-python.mk                              |  2 +
 package/pkgconf/pkg-config.in                      |  4 +-
 package/pkgconf/pkgconf.mk                         |  5 +-
 package/poco/poco.mk                               |  3 +-
 package/poppler/poppler.mk                         |  2 +-
 package/pppd/pppd.mk                               |  2 +-
 package/python-gobject/python-gobject.mk           |  4 +-
 package/python-pygame/python-pygame.mk             |  6 +-
 package/python-pyqt/python-pyqt.mk                 |  1 +
 package/python-sip/python-sip.mk                   |  2 +-
 package/qt/qt.mk                                   | 18 ++--
 package/qtuio/qtuio.mk                             |  3 +-
 package/rtmpdump/rtmpdump.mk                       |  2 +-
 package/scons/scons.mk                             |  3 +-
 package/scons/site_init.py                         | 19 +++++
 package/speex/speex.mk                             |  3 +-
 package/statserial/statserial.mk                   |  3 +-
 package/tcpreplay/tcpreplay.mk                     |  6 +-
 package/ti-utils/ti-utils.mk                       |  6 +-
 package/tvheadend/tvheadend.mk                     |  4 +-
 package/upmpdcli/upmpdcli.mk                       |  2 +-
 package/ushare/ushare.mk                           |  3 +-
 package/vsftpd/vsftpd.mk                           |  4 +-
 support/misc/toolchainfile.cmake.in                |  9 +-
 .../toolchain-external/ext-toolchain-wrapper.c     | 96 ++++++++++++++--------
 toolchain/toolchain-external/toolchain-external.mk |  2 +-
 74 files changed, 476 insertions(+), 147 deletions(-)
 create mode 100644 package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
 create mode 100644 package/luarocks/0002-fixed-typo.patch
 create mode 100644 package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
 create mode 100644 package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch
 create mode 100644 package/scons/site_init.py

Quoted 3 lines

--
2.4.3

Best regards

Signature · 1 line

Fabio Porcedda

Thomas Petazzoni

2015-06-28 20:34 UTC #21 lore raw

v4 cover Add per-package staging feature

In reply to Fabio Porcedda

Dear Fabio Porcedda,

I haven't looked at the implementation yet, so I'll just comment on the results.

Quoted 31 lines · Fabio Porcedda

To improve performance and reduce space utilization hard links are used.
To evaluate performance and space utilization I've done some tests.
HW-MED:
CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
RAM: 16GB
SSD: 512GB
HW-HIGH:
CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
RAM: 32GB
SATA: RAID0 2x2TB 3.5" 7200rpm
defconfig-small, long chain dependency, unfavorable scenario for parallelization:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_LIBGTK3=y
defconfig-full:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
make allyespackageconfig
Also i removed few packages that do not built on my system. make show-targets | wc -w
1366

Quoted 7 lines

Test about output using hard links or note using the "defconfig-full" on HW-MED:

|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all

So clearly hardlinks are great, as they limit a lot the increase of disk usage. Going from 35 GB to 37 GB for a full defconfig is quite reasonable IMO, especially considering the additional safety wrt optional dependencies that per-package staging gives (even without considering the top-level parallel build feature).

Also, I guess you are keeping all the temporary per-package staging directories, right? We could probably save even more disk-space by getting rid of the per-package staging areas after each package is built (but that would be this per-package staging area would have to be re-created every time you do "make <pkg>-rebuild", which isn't nice).

Quoted 8 lines

Test about performances of this patch set vs master branch:

| 199m | HW-MED | defconfig-full | master branch | no top-level make |
|  99m | HW-MED | defconfig-full | master branch | top-level make    | 
| 100m | HW-MED | defconfig-full | patch set     | top-level make    |

| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
|  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
|  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |

So your high-end hardware in fact needs more time than the medium-end hardware to build with top-level parallel, probably because the I/Os come from HDDs instead of SSD. However, with top-level parallel build, you high-end hardware performs better.

Quoted 7 lines

| 10m | HW-MED | defconfig-small | master branch | no top-level make |
|  5m | HW-MED | defconfig-small | master branch | top-level make    | 
|  5m | HW-MED | defconfig-samll | patch set     | top-level make    |

| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
|  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
|  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

So all in all, top-level parallel build makes the build from twice faster to three times faster (or even more for defconfig-full on HW-HIGH). And the time impact of the per-package staging feature is minimal.

So to me, this is very encouraging, and clearly shows that it is worth diving into your implementation and see how we can progressively merge that.

Can I ask a last test to be done: what happens if you apply your patch set, but do not use top-level parallel make ? Just to see if there would be a significant impact to people not enabling top-level parallel make.

Thanks a lot!

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Yann E. MORIN

2015-06-28 20:46 UTC #22 lore raw

v4 cover Add per-package staging feature

In reply to Fabio Porcedda

Fabio, All,

On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:

Quoted 3 lines

this patch set aims to improve build reproducibility by using a per-package staging directory.
Also this feature aims to enable the top-level parallel make.

I haven't yet had time to look in the different patches, but I have a preliminary comment about your benchmarks:

[--SNIP--]

Quoted 28 lines

HW-MED:
CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
RAM: 16GB
SSD: 512GB

HW-HIGH:

CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads

RAM: 32GB
SATA: RAID0 2x2TB 3.5" 7200rpm
1366

Test about output using hard links or note using the "defconfig-full" on HW-MED:

|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all

Test about performances of this patch set vs master branch:

| 199m | HW-MED | defconfig-full | master branch | no top-level make |
|  99m | HW-MED | defconfig-full | master branch | top-level make    | 
| 100m | HW-MED | defconfig-full | patch set     | top-level make    |

| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
|  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
|  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |

Quoted 7 lines

| 10m | HW-MED | defconfig-small | master branch | no top-level make |
|  5m | HW-MED | defconfig-small | master branch | top-level make    | 
|  5m | HW-MED | defconfig-samll | patch set     | top-level make    |

| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
|  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
|  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

OK, so those benchmarks show that:

- HDD are terribly slow when compared to SSDs

- the build-time sppedup of parallel builds is huge, up to 4.5 on HW-HIGH, but mostly around 2 when HDDs do not play in the equation (i.e. on SSDs, or with all cached in RAM (small config)).

- the build-time overhead is low, 5% in the worst case (73min -> 77min)

- the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without hardlinks

- we're missing the benchmarks for this patchset without top-level parallel make (especially for the size overhead). Unless it no longer makes sense?

So, I'm really a bit skeptical. About five time the size for only about twice the speedup, is it worth it? Sure some people will easily favour speed over anything else, still the size overhead is really huge.

But I still think the idea is worth investigating.

Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which might be better as a HW_HIGH system. I can spin a test-build with this patchset on this machine to see what we get...

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Yann E. MORIN

2015-06-28 20:49 UTC #23 lore raw

v4 cover Add per-package staging feature

Fabio, All,

On 2015-06-28 22:46 +0200, Yann E. MORIN spake thusly:

Quoted 1 line

On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:

[--SNIP--]

Quoted 4 lines

|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all

[--SNIP--]

Quoted 2 lines

- the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without hardlinks

Oh wait, I read the wrong line...

OK, disregard my comments about the size overhead. It's only 2GiB, i.e. roughly 5%.

Sorry for the confusion...

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Thomas Petazzoni

2015-06-28 20:57 UTC #24 lore raw

v4 cover Add per-package staging feature

Hello,

Quoted 7 lines · Yann E. MORIN

On 2015-06-28 22:46 +0200, Yann E. MORIN spake thusly:
Quoted 1 line
On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:
[--SNIP--]
Quoted 4 lines
Quoted 4 lines
|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all

Quoted 8 lines

[--SNIP--]
Quoted 2 lines
- the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without hardlinks
Oh wait, I read the wrong line...
OK, disregard my comments about the size overhead. It's only 2GiB, i.e. roughly 5%.

Yes, the size overhead is IMO very reasonable. So do you reconsider the general opinion you gave in your previous e-mail?

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Yann E. MORIN

2015-06-28 21:04 UTC #25 lore raw

v4 cover Add per-package staging feature

Thomas, Fabio, All,

On 2015-06-28 22:57 +0200, Thomas Petazzoni spake thusly:

Quoted 13 lines · Yann E. MORIN

Quoted 12 lines
On 2015-06-28 22:46 +0200, Yann E. MORIN spake thusly:
Quoted 1 line
On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:
[--SNIP--]
Quoted 4 lines
Quoted 4 lines
|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all
[--SNIP--]
Quoted 2 lines
- the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without hardlinks

Quoted 7 lines

Quoted 4 lines
Oh wait, I read the wrong line...
OK, disregard my comments about the size overhead. It's only 2GiB, i.e. roughly 5%.
Yes, the size overhead is IMO very reasonable. So do you reconsider the general opinion you gave in your previous e-mail?

Yes, all I said about the size overhead, and thus my opinion, is void.

As a conclusion: yes, this is *very* good, and I think we must get that in the tree. :-)

Sorry for the confusion... :-/

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Thomas Petazzoni

2015-06-28 21:00 UTC #26 lore raw

v4 cover Add per-package staging feature

In reply to Yann E. MORIN

Dear Yann E. MORIN,

Quoted 18 lines · Yann E. MORIN

Quoted 18 lines

|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all

Test about performances of this patch set vs master branch:

| 199m | HW-MED | defconfig-full | master branch | no top-level make |
|  99m | HW-MED | defconfig-full | master branch | top-level make    | 
| 100m | HW-MED | defconfig-full | patch set     | top-level make    |

| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
|  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
|  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |

| 10m | HW-MED | defconfig-small | master branch | no top-level make |
|  5m | HW-MED | defconfig-small | master branch | top-level make    | 
|  5m | HW-MED | defconfig-samll | patch set     | top-level make    |

Quoted 7 lines

Quoted 3 lines

| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
|  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
|  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

OK, so those benchmarks show that:

- HDD are terribly slow when compared to SSDs

Yes, and the top-level parallel make gives the highest benefit on HDDs:
from 350m to 77m and from 21m to 8m on the HW-HIGH configuration.

Quoted 1 line

- the build-time overhead is low, 5% in the worst case (73min -> 77min)

Yes.

Quoted 2 lines

- the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without hardlinks

No, as you replied, the size overhead is very reasonable. You misread Fabio's table here.

Quoted 3 lines

- we're missing the benchmarks for this patchset without top-level parallel make (especially for the size overhead). Unless it no longer makes sense?

Correct, I asked the same thing.

Quoted 3 lines

So, I'm really a bit skeptical. About five time the size for only about twice the speedup, is it worth it? Sure some people will easily favour speed over anything else, still the size overhead is really huge.

Do you reconsider this comment now that you know the reality about the size impact ? :-)

Quoted 3 lines

Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which might be better as a HW_HIGH system. I can spin a test-build with this patchset on this machine to see what we get...

Would definitely be useful. Maybe Fabio can provide you his defconfig-full test case (he gave the defconfig-small case).

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Yann E. MORIN

2015-06-28 21:25 UTC #27 lore raw

v4 cover Add per-package staging feature

Thomas, Fabio, All,

On 2015-06-28 23:00 +0200, Thomas Petazzoni spake thusly:

Quoted 7 lines · Yann E. MORIN

Quoted 3 lines
So, I'm really a bit skeptical. About five time the size for only about twice the speedup, is it worth it? Sure some people will easily favour speed over anything else, still the size overhead is really huge.
Do you reconsider this comment now that you know the reality about the size impact ? :-)

Hehe! ;-)

Of course, ~5% size overhead for 2+ times the speedup is totally acceptable.

Quoted 6 lines

Quoted 3 lines
Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which might be better as a HW_HIGH system. I can spin a test-build with this patchset on this machine to see what we get...
Would definitely be useful. Maybe Fabio can provide you his defconfig-full test case (he gave the defconfig-small case).

Fabio, if you could share your two defconfig (or .config) files, that'd be great!

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Fabio Porcedda

author

2015-06-29 08:35 UTC #28 lore raw

v4 cover Add per-package staging feature

Quoted 8 lines · Yann E. MORIN

Thomas, Fabio, All,
On 2015-06-28 23:00 +0200, Thomas Petazzoni spake thusly:
Quoted 5 lines · Yann E. MORIN
Quoted 3 lines
So, I'm really a bit skeptical. About five time the size for only about twice the speedup, is it worth it? Sure some people will easily favour speed over anything else, still the size overhead is really huge.

Quoted 11 lines

Quoted 2 lines
Do you reconsider this comment now that you know the reality about the size impact ? :-)
Hehe! ;-)
Of course, ~5% size overhead for 2+ times the speedup is totally acceptable.
Quoted 3 lines
Quoted 3 lines
Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which might be better as a HW_HIGH system. I can spin a test-build with this patchset on this machine to see what we get...

Nice machine :)
It have less cores but at higher frequency and it have three SSD in RAID0! I'm curious to know how much faster is it.

I think the bottleneck on my HW-HIGH is the hard disk.

Quoted 5 lines

Quoted 2 lines
Would definitely be useful. Maybe Fabio can provide you his defconfig-full test case (he gave the defconfig-small case).
Fabio, if you could share your two defconfig (or .config) files, that'd be great!

Configs:
https://www.dropbox.com/s/tmdzhluxv69fdib/arm_external_longchaindep_defconfig?dl=0
https://www.dropbox.com/s/9ntz73wo5ua80k9/arm_external_allyespackageconfig_defconfig?dl=0

Branch with this patch set:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4

Branch without this patch set, on which this patch set was applied, it has already all the patches for using top-level-parallel make:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4-base

Before testing the pps-v4 branch or enabling the top-level make i suggest to test the arm_external_allyespackageconfig_defconfig to find if all packages builds on your machine, this defconfig builds fine on a Fedora 22 and Ubuntu 14.04 (using gnu make 4.1 built from source).

Command line for testing "arm_external_allyespackageconfig_defconfig" enabling top-level parallel make:

{ time { b="make -j$((nproc+1)) -Orecurse -s" && $b libevas gst1-plugins-base gst1-plugins-ugly && $b && echo Build OK; }; } 2>&1 | tee log

Pay attention to the fact that the packages "libevas" "gst1-plugins-base" "gst1-plugins-ugly" must be built for first otherwise they fail to build (I did not have enough time to analyze the problem or check if using pps help on this matter).

Thanks

Signature · 1 line

Fabio Porcedda

Yann E. MORIN

2015-06-29 17:27 UTC #29 lore raw

v4 cover Add per-package staging feature

Fabio, All,

On 2015-06-29 10:35 +0200, Fabio Porcedda spake thusly:

Quoted 8 lines · Yann E. MORIN

Quoted 2 lines
Fabio, if you could share your two defconfig (or .config) files, that'd be great!
Configs:
https://www.dropbox.com/s/tmdzhluxv69fdib/arm_external_longchaindep_defconfig?dl=0
https://www.dropbox.com/s/9ntz73wo5ua80k9/arm_external_allyespackageconfig_defconfig?dl=0

Quoted 6 lines

Branch with this patch set:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4
Branch without this patch set, on which this patch set was applied, it has already all the patches for using top-level-parallel make:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4-base

OK, thanks! :-) I'll spawn a test build later tonight (UTC+2) so it crunches during the night.

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Fabio Porcedda

author

2015-06-29 08:38 UTC #30 lore raw

v4 cover Add per-package staging feature

In reply to Thomas Petazzoni

Quoted 4 lines · Thomas Petazzoni

Dear Yann E. MORIN,
On Sun, 28 Jun 2015 22:46:22 +0200, Yann E. MORIN wrote:

[snip]

Quoted 5 lines

Quoted 3 lines
- we're missing the benchmarks for this patchset without top-level parallel make (especially for the size overhead). Unless it no longer makes sense?
Correct, I asked the same thing.

I will test it, it just takes time.

[snip]

Signature · 1 line

Fabio Porcedda

Fabio Porcedda

author

2015-07-05 08:51 UTC #31 lore raw

v4 cover Add per-package staging feature

In reply to Fabio Porcedda

On Sun, Jun 28, 2015 at 9:42 PM, Fabio Porcedda
<[email protected]> wrote:

[snip]

Quoted 48 lines

To improve performance and reduce space utilization hard links are used.
To evaluate performance and space utilization I've done some tests.
HW-MED:
CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
RAM: 16GB
SSD: 512GB
HW-HIGH:
CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
RAM: 32GB
SATA: RAID0 2x2TB 3.5" 7200rpm
defconfig-small, long chain dependency, unfavorable scenario for parallelization:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_LIBGTK3=y
defconfig-full:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
make allyespackageconfig
Also i removed few packages that do not built on my system. make show-targets | wc -w
1366
Test about output using hard links or note using the "defconfig-full" on HW-MED:
|  35GB |  99m | master branch
|  37GB | 100m | patch set
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all
Test about performances of this patch set vs master branch:
| 199m | HW-MED | defconfig-full | master branch | no top-level make |
|  99m | HW-MED | defconfig-full | master branch | top-level make    |
| 100m | HW-MED | defconfig-full | patch set     | top-level make    |
| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
|  73m | HW-HIGH | defconfig-full | master branch | top-level make    |
|  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |

Quoted 7 lines

| 10m | HW-MED | defconfig-small | master branch | no top-level make |
|  5m | HW-MED | defconfig-small | master branch | top-level make    |
|  5m | HW-MED | defconfig-samll | patch set     | top-level make    |

| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
|  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    |
|  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

[snip]

I've redone all the tests including the case of the patch set without using top-level make.

I've repeated the the test at least two times and taken the minimum value.

defconfig-small:

| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 21m17s    | 10m13s   | no                  | no           |
| 21m24s    | 10m17s   | no                  | yes         |
|   7m57s    |  5m25s   | yes                 | no           |
|   7m48s    |  5m22s   | yes                 | yes         |

defconfig-full:

| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 350m       | 199m       | no                  | no           |
| 355m       | 200m       | no                  | yes         |
|   74m       |  99m        | yes                 | no           |
|   76m       |  99m        | yes                 | yes         |

In these tests the penalty of using pps is 0%-3%.
Also the reduction of build time using top-level make is 50%-78%

I think they are nice numbers :)

Best regards

Signature · 1 line

Fabio Porcedda

Thomas Petazzoni

2015-07-05 09:01 UTC #32 lore raw

v4 cover Add per-package staging feature

Hello,

Quoted 4 lines · Fabio Porcedda

I've redone all the tests including the case of the patch set without using top-level make.
I've repeated the the test at least two times and taken the minimum value.

How much variation did you had between the test builds for a given scenario? Taking the minimal value is a somewhat strange choice :)

Quoted 16 lines

defconfig-small:

| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 21m17s    | 10m13s   | no                  | no           |
| 21m24s    | 10m17s   | no                  | yes         |
|   7m57s    |  5m25s   | yes                 | no           |
|   7m48s    |  5m22s   | yes                 | yes         |

defconfig-full:

| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 350m       | 199m       | no                  | no           |
| 355m       | 200m       | no                  | yes         |
|   74m       |  99m        | yes                 | no           |
|   76m       |  99m        | yes                 | yes         |

Quoted 4 lines

In these tests the penalty of using pps is 0%-3%.
Also the reduction of build time using top-level make is 50%-78%
I think they are nice numbers :)

Yes, they are very good. I guess that if we implement per-package host dir there will be a small additional impact, but clearly the benefits of top-level parallel largely outweighs the impact of per-package staging/host directories.

Thanks a lot for having done those additional tests, it clearly means we should progressively merge your work on this topic!

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Fabio Porcedda

author

2015-07-05 09:26 UTC #33 lore raw

v4 cover Add per-package staging feature

Quoted 11 lines · Thomas Petazzoni

Hello,
Quoted 4 lines · Fabio Porcedda
I've redone all the tests including the case of the patch set without using top-level make.
I've repeated the the test at least two times and taken the minimum value.
How much variation did you had between the test builds for a given scenario? Taking the minimal value is a somewhat strange choice :)

Not very much, it was done because in some cases every time a repeat the build the time was reduced, i will show you the full data, build times are sorted by their value, usually the lowest value is the last value taken:

defconfig-small, HW-MED:

| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            | 10m13s 10m16s 10m16s 10m17s
| no                  | yes          |  10m17s 10m18s 10m20s 11m23s
| yes                 | no           |   5m25s 5m27s 5m28s 5m31s
| yes                 | yes         |  5m22s 5m23s 5m24s 5m26s

defconfig-small, HW-HIGH:

| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            | 21m17s 21m20s 21m42s
| no                  | yes          |  21m24s 21m25s 21m28s
| yes                 | no           |  7m57s 7m57s 7m58s 7m58s
| yes                 | yes         |   7m48s 7m50s 7m50s 7m55s 7m57s 7m58s

defconfig-full, HW-MED:

| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            |  199m 199m
| no                  | yes          |   200m 200m
| yes                 | no           |  99m 101m
| yes                 | yes         |  99m 100m

defconfig-full, HW-HIGH:

| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            | 350m 351m
| no                  | yes          | 355m 358m
| yes                 | no           | 74m 75m
| yes                 | yes         | 76m 76m 76m

Quoted 21 lines

Quoted 16 lines

defconfig-small:

| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 21m17s    | 10m13s   | no                  | no           |
| 21m24s    | 10m17s   | no                  | yes         |
|   7m57s    |  5m25s   | yes                 | no           |
|   7m48s    |  5m22s   | yes                 | yes         |

defconfig-full:

| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 350m       | 199m       | no                  | no           |
| 355m       | 200m       | no                  | yes         |
|   74m       |  99m        | yes                 | no           |
|   76m       |  99m        | yes                 | yes         |

Quoted 4 lines

In these tests the penalty of using pps is 0%-3%.
Also the reduction of build time using top-level make is 50%-78%
I think they are nice numbers :)

Quoted 7 lines

Yes, they are very good. I guess that if we implement per-package host dir there will be a small additional impact, but clearly the benefits of top-level parallel largely outweighs the impact of per-package staging/host directories.
Thanks a lot for having done those additional tests, it clearly means we should progressively merge your work on this topic!

Great!

I work now to use the toolchain wrapper even for the internal toolchain.

Best regards

Signature · 1 line

Fabio Porcedda

Arnout Vandecappelle

2015-10-04 17:21 UTC #34 lore raw

v4 cover Add per-package staging feature

In reply to Fabio Porcedda

Quoted 12 lines · Fabio Porcedda

Hi all,
this patch set aims to improve build reproducibility by using a per-package staging directory.
Also this feature aims to enable the top-level parallel make.
This patch set is available at:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4
Every package instead of installing the files in the shared staging directory (STAGING_DIR) install all files in a subdirectory of the STAGINGPKG_DIR.

[snip]

Quoted 4 lines

TODO:
- use the toolchaing wrapper even for the internal toolchain
- rebase over master (this patch set is based on 2015.05-161-gca5f11e).
- more testing (tested only for arm target).

[snip]

Hi Fabio,

Now that we have a toolchain wrapper for the internal toolchain, would you care
to rebase this series and use the toolchain wrapper for pps support?

Thomas commented that there should also be a per-package host directory, but
after some discussion here at the BR developer meeting we decided that we could start with just a per-package staging and see how far we get. We anyway don't have complete reproducibility at the moment, since the build of host packages depends on what is installed on the system.

Note that we are really excited about the possibilities of per-package staging
and top-level parallel build, so we hope you'll continue working on it.

I've marked this series as Changes Requested in patchwork.

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Bjørn Forsman

2015-10-04 17:51 UTC #35 lore raw

v4 cover Add per-package staging feature

Quoted 3 lines · Arnout Vandecappelle

[...] We anyway don't
have complete reproducibility at the moment, since the build of host packages depends on what is installed on the system.

To get a known set of host packages, Buildroot could run in an environment setup by Nix[1].

With some extra tricks, you can get _very_ reproducible environments:

https://garbas.si/2015/reproducible-development-environments.html

I think that's close to as reproducible as it gets, if not emulating CPU/RAM/filesystem the build runs in :-)

[1] http://nixos.org/nix/

Disclaimer: I'm a Nix(OS) user/contributor.

- Bj?rn

Fabio Porcedda

author

2015-10-05 21:13 UTC #36 lore raw

v4 cover Add per-package staging feature

In reply to Arnout Vandecappelle

Quoted 21 lines · Arnout Vandecappelle

Quoted 12 lines · Fabio Porcedda
Hi all,
this patch set aims to improve build reproducibility by using a per-package staging directory.
Also this feature aims to enable the top-level parallel make.
This patch set is available at:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4
Every package instead of installing the files in the shared staging directory (STAGING_DIR) install all files in a subdirectory of the STAGINGPKG_DIR.
[snip]
Quoted 4 lines
TODO:
- use the toolchaing wrapper even for the internal toolchain
- rebase over master (this patch set is based on 2015.05-161-gca5f11e).
- more testing (tested only for arm target).

Quoted 9 lines

[snip]
Hi Fabio,
Now that we have a toolchain wrapper for the internal toolchain, would
you care
to rebase this series and use the toolchain wrapper for pps support?

Thanks for the toolchain wrapper for the internal toolchain.

I will rebase this patch for sure and send an updated version.

Quoted 10 lines

Thomas commented that there should also be a per-package host directory,
but
after some discussion here at the BR developer meeting we decided that we could
start with just a per-package staging and see how far we get. We anyway don't
have complete reproducibility at the moment, since the build of host packages
depends on what is installed on the system.

Quoted 4 lines

Note that we are really excited about the possibilities of per-package
staging
and top-level parallel build, so we hope you'll continue working on it.

Great, it's nice to hear it!

Quoted 2 lines

I've marked this series as Changes Requested in patchwork.

Signature · 4 lines

Fabio Porcedda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151005/83693684/attachment-0001.html>