Skip to contentSkip to conversation

rsync: use system zlib and popt, also do not lose debug options

PATCH Superseded Denis Mingulov 3 participants 8 messages Oct 16, 2013 – Nov 1, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rsync: use system zlib and popt, also do not lose debug options

rsync by default compiles statically its own zlib and popt libraries, it is better to use system wide libraries instead - both for disk and memory consumption. Change rsync package accordingly.

Also previously a debug configure option has been lost, as overwritten by '--with-included-popt' option - fixed.

For example, on ARMv5 rsync binary size is reduced by about 25%.

---
 package/rsync/Config.in | 2 ++
 package/rsync/rsync.mk  | 4 +++-
 2 files changed, 5 insertions(+), 1 deletion(-)
package/rsync/Config.in +2 -0
index be95a7c..b952974 100644
@@ -2,6 +2,8 @@ config BR2_PACKAGE_RSYNC
 	bool "rsync"
 	# fork()
 	depends on BR2_USE_MMU
+	select BR2_PACKAGE_ZLIB
+	select BR2_PACKAGE_POPT
 	help
 	  File transfer program to keep remote files in sync.
 
package/rsync/rsync.mk +3 -1
index d8e5f3f..4f336bc 100644
@@ -8,7 +8,9 @@ RSYNC_VERSION = 3.1.0
 RSYNC_SITE = http://rsync.samba.org/ftp/rsync/src
 RSYNC_LICENSE = GPLv3+
 RSYNC_LICENSE_FILES = COPYING
+RSYNC_DEPENDENCIES = zlib popt
 RSYNC_CONF_OPT = $(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug)
-RSYNC_CONF_OPT = --with-included-popt
+RSYNC_CONF_OPT += --with-included-zlib=no
+RSYNC_CONF_OPT += --with-included-popt=no
 
 $(eval $(autotools-package))
Signature · 1 line
1.8.4
#1 lore raw

PATCH 1/1 rsync: use system zlib and popt, also do not lose debug options

Dear Denis Mingulov,

Quoted 11 lines · Denis Mingulov

rsync by default compiles statically its own zlib and popt libraries, it is better to use system wide libraries instead - both for disk and memory consumption. Change rsync package accordingly.

Also previously a debug configure option has been lost, as overwritten by '--with-included-popt' option - fixed.

For example, on ARMv5 rsync binary size is reduced by about 25%.

Thanks, looks good, but I have one suggestion.

package/rsync/rsync.mk RSYNC_VERSION = 3.1.0 · line 14

Quoted 4 lines

RSYNC_CONF_OPT = $(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug)

package/rsync/rsync.mk +2 -1
-RSYNC_CONF_OPT = --with-included-popt
+RSYNC_CONF_OPT += --with-included-zlib=no
+RSYNC_CONF_OPT += --with-included-popt=no

Make this:

RSYNC_CONF_OPT = \
	$(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug) \
	--with-included-zlib=no \
	--with-included-popt=no

while it makes sense to append to <pkg>_CONF_OPT using += when such appends are done under conditions, for the general configuration options, just one assignment is clearer and that's what we use more or less everywhere in Buildroot.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#2 lore raw

PATCH v2 1/1 rsync: use system zlib and popt, also do not lose debug options

In reply to Denis Mingulov

rsync by default compiles statically its own zlib and popt libraries, it is better to use system wide libraries instead - both for disk and memory consumption. Change rsync package accordingly.

Also previously a debug configure option has been lost, as overwritten by '--with-included-popt' option - fixed.

For example, on ARMv5 rsync binary size is reduced by about 25%.

---
 package/rsync/Config.in | 2 ++
 package/rsync/rsync.mk  | 7 +++++--
 2 files changed, 7 insertions(+), 2 deletions(-)
package/rsync/Config.in +2 -0
index be95a7c..b952974 100644
@@ -2,6 +2,8 @@ config BR2_PACKAGE_RSYNC
 	bool "rsync"
 	# fork()
 	depends on BR2_USE_MMU
+	select BR2_PACKAGE_ZLIB
+	select BR2_PACKAGE_POPT
 	help
 	  File transfer program to keep remote files in sync.
 
package/rsync/rsync.mk +5 -2
index d8e5f3f..35f128c 100644
@@ -8,7 +8,10 @@ RSYNC_VERSION = 3.1.0
 RSYNC_SITE = http://rsync.samba.org/ftp/rsync/src
 RSYNC_LICENSE = GPLv3+
 RSYNC_LICENSE_FILES = COPYING
-RSYNC_CONF_OPT = $(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug)
-RSYNC_CONF_OPT = --with-included-popt
+RSYNC_DEPENDENCIES = zlib popt
+RSYNC_CONF_OPT = \
+	$(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug) \
+	--with-included-zlib=no \
+	--with-included-popt=no
 
 $(eval $(autotools-package))
Signature · 1 line
1.8.4
#3 lore raw

PATCH v2 1/1 rsync: use system zlib and popt, also do not lose debug options

Quoted 11 lines · Denis Mingulov

rsync by default compiles statically its own zlib and popt libraries, it is better to use system wide libraries instead - both for disk and memory consumption. Change rsync package accordingly.

Also previously a debug configure option has been lost, as overwritten by '--with-included-popt' option - fixed.

For example, on ARMv5 rsync binary size is reduced by about 25%.

(tested with Sourcery ARM toolchain)

Quoted 36 lines
---
  package/rsync/Config.in | 2 ++
  package/rsync/rsync.mk  | 7 +++++--
  2 files changed, 7 insertions(+), 2 deletions(-)
package/rsync/Config.in +2 -0
index be95a7c..b952974 100644
@@ -2,6 +2,8 @@ config BR2_PACKAGE_RSYNC
  	bool "rsync"
  	# fork()
  	depends on BR2_USE_MMU
+	select BR2_PACKAGE_ZLIB
+	select BR2_PACKAGE_POPT
  	help
  	  File transfer program to keep remote files in sync.
 
package/rsync/rsync.mk +5 -2
index d8e5f3f..35f128c 100644
@@ -8,7 +8,10 @@ RSYNC_VERSION = 3.1.0
  RSYNC_SITE = http://rsync.samba.org/ftp/rsync/src
  RSYNC_LICENSE = GPLv3+
  RSYNC_LICENSE_FILES = COPYING
-RSYNC_CONF_OPT = $(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug)
-RSYNC_CONF_OPT = --with-included-popt
+RSYNC_DEPENDENCIES = zlib popt
+RSYNC_CONF_OPT = \
+	$(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug) \
+	--with-included-zlib=no \
+	--with-included-popt=no
 
  $(eval $(autotools-package))
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#4 lore raw

PATCH v2 1/1 rsync: use system zlib and popt, also do not lose debug options

In reply to Denis Mingulov

Dear Denis Mingulov,

Quoted 8 lines · Denis Mingulov

rsync by default compiles statically its own zlib and popt libraries, it is better to use system wide libraries instead - both for disk and memory consumption. Change rsync package accordingly.

Also previously a debug configure option has been lost, as overwritten by '--with-included-popt' option - fixed.

Quoted 7 lines

For example, on ARMv5 rsync binary size is reduced by about 25%.

---
 package/rsync/Config.in | 2 ++
 package/rsync/rsync.mk  | 7 +++++--
 2 files changed, 7 insertions(+), 2 deletions(-)

Applied to for-peter-2013.11, thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#5 lore raw

PATCH 1/1 rsync: use system zlib and popt, also do not lose debug options

In reply to Denis Mingulov
Quoted 9 lines · Denis Mingulov

rsync by default compiles statically its own zlib and popt libraries, it is better to use system wide libraries instead - both for disk and memory consumption. Change rsync package accordingly.

Also previously a debug configure option has been lost, as overwritten by '--with-included-popt' option - fixed.

For example, on ARMv5 rsync binary size is reduced by about 25%.

I just did a check by building only rsync before and after your patch,
and although the binary becomes smaller with your patch, the overall filesystem becomes larger. I'd therefore propose to only use the system library if it is anyway selected. So:

Quoted 12 lines
---
  package/rsync/Config.in | 2 ++
  package/rsync/rsync.mk  | 4 +++-
  2 files changed, 5 insertions(+), 1 deletion(-)
package/rsync/Config.in +0 -0
index be95a7c..b952974 100644

package/rsync/Config.in config BR2_PACKAGE_RSYNC · line 6

Quoted 5 lines
package/rsync/Config.in +2 -0
@@ -2,6 +2,8 @@ config BR2_PACKAGE_RSYNC
  	bool "rsync"
  	# fork()
  	depends on BR2_USE_MMU
+	select BR2_PACKAGE_ZLIB
+	select BR2_PACKAGE_POPT

Remove this.

package/rsync/rsync.mk RSYNC_VERSION = 3.1.0 · line 11

Quoted 12 lines
package/rsync/Config.in +0 -0
  	help
  	  File transfer program to keep remote files in sync.
 
package/rsync/rsync.mk +1 -0
index d8e5f3f..4f336bc 100644
@@ -8,7 +8,9 @@ RSYNC_VERSION = 3.1.0
  RSYNC_SITE = http://rsync.samba.org/ftp/rsync/src
  RSYNC_LICENSE = GPLv3+
  RSYNC_LICENSE_FILES = COPYING
+RSYNC_DEPENDENCIES = zlib popt

Remove this.

package/rsync/rsync.mk RSYNC_VERSION = 3.1.0 · line 14

Quoted 4 lines
package/rsync/rsync.mk +2 -1
  RSYNC_CONF_OPT = $(if $(BR2_ENABLE_DEBUG),--enable-debug,--disable-debug)
-RSYNC_CONF_OPT = --with-included-popt
+RSYNC_CONF_OPT += --with-included-zlib=no
+RSYNC_CONF_OPT += --with-included-popt=no

Replace this with something like:

ifeq($(BR2_PACKAGE_ZLIB),y)
RSYNC_CONF_OPT += --with-included-zlib=no
RSYNC_DEPENDENCIES += zlib
else
RSYNC_CONF_OPT += --with-included-zlib=yes
endif

And similar for popt.

Of course, this is untested and YMMV.

Regards,
Arnout

Quoted 3 lines
package/rsync/rsync.mk +0 -0
  $(eval $(autotools-package))
 
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#6 lore raw

PATCH 1/1 rsync: use system zlib and popt, also do not lose debug options

Hi,

Quoted 1 line

building only rsync before and after your patch, and although the binary

becomes smaller with your patch, the overall filesystem becomes larger.

Yes, in case of the only selected package it will be so. Due to overhead of shared library / not removed unused components.
But for this particular case - you most probably have openssh (depends to zlib) or dropbear (depends to zlib if not 'small' version is selected separately) also already.

Quoted 1 line

I'd therefore propose to only use the system library if it is anyway

selected.

Sorry I am disagree, even without considering the security issue, just because zlib/popt for rsync are not optional libraries, they are mandatory, and will be compiled in any case.
In the current buildroot .mk files such 'if' use case might be useful for optional components only, something like 'nice to have, will be supported if already present'. Otherwise there might be tens of components compiling its own 'zlib' version (tens different versions) - and the Buildroot user must somehow found FAQ or understand himself, that he must 'include zlib to the package manually' (zlib or tens of other libraries).

Also it will be opposite to the current Buildroot configuration. In the latest master there are 9 packages with 'select BR2_PACKAGE_POPT' (cryptsetup, gptfdisk, libiscsi, librsync, logrotate, lttng-tools, oprofile, rpm, samba) and 50 packages with the same for zlib (of course not every one has its own zlib/popt).

Might be, later it would be nice to have something like 'select_if_multiple' in Config files, to select other package if more than 1 component is selecting it, but it will do Configs more complex, and the use case is quite rare.

Quoted 95 lines · Arnout Vandecappelle
Quoted 10 lines · Denis Mingulov

rsync by default compiles statically its own zlib and popt libraries, it is better to use system wide libraries instead - both for disk and memory consumption. Change rsync package accordingly.

Also previously a debug configure option has been lost, as overwritten by '--with-included-popt' option - fixed.

For example, on ARMv5 rsync binary size is reduced by about 25%.

I just did a check by building only rsync before and after your patch,
and although the binary becomes smaller with your patch, the overall filesystem becomes larger. I'd therefore propose to only use the system library if it is anyway selected. So:

Quoted 11 lines
---
  package/rsync/Config.in | 2 ++
  package/rsync/rsync.mk  | 4 +++-
  2 files changed, 5 insertions(+), 1 deletion(-)
package/rsync/Config.in +0 -0
index be95a7c..b952974 100644

package/rsync/Config.in config BR2_PACKAGE_RSYNC · line 7

Quoted 6 lines
package/rsync/Config.in +2 -0
@@ -2,6 +2,8 @@ config BR2_PACKAGE_RSYNC
        bool "rsync"
        # fork()
        depends on BR2_USE_MMU
+       select BR2_PACKAGE_ZLIB
+       select BR2_PACKAGE_POPT
 

Remove this.

help

package/rsync/rsync.mk RSYNC_VERSION = 3.1.0 · line 12

Quoted 12 lines
package/rsync/Config.in +0 -0
          File transfer program to keep remote files in sync.
 
package/rsync/rsync.mk +1 -0
index d8e5f3f..4f336bc 100644
@@ -8,7 +8,9 @@ RSYNC_VERSION = 3.1.0
  RSYNC_SITE = http://rsync.samba.org/ftp/**rsync/src<http://rsync.samba.org/ftp/rsync/src>
  RSYNC_LICENSE = GPLv3+
  RSYNC_LICENSE_FILES = COPYING
+RSYNC_DEPENDENCIES = zlib popt
 

Remove this.

   RSYNC_CONF_OPT = $(if $(BR2_ENABLE_DEBUG),--enable-**

package/rsync/rsync.mk RSYNC_VERSION = 3.1.0 · line 15

Quoted 5 lines
package/rsync/rsync.mk +2 -1
 debug,--disable-debug)
-RSYNC_CONF_OPT = --with-included-popt
+RSYNC_CONF_OPT += --with-included-zlib=no
+RSYNC_CONF_OPT += --with-included-popt=no
 

Replace this with something like:

ifeq($(BR2_PACKAGE_ZLIB),y)
RSYNC_CONF_OPT += --with-included-zlib=no
RSYNC_DEPENDENCIES += zlib
else
RSYNC_CONF_OPT += --with-included-zlib=yes
endif

And similar for popt.

Of course, this is untested and YMMV.

Regards,
Arnout

Quoted 3 lines
  $(eval $(autotools-package))

--

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/**arnoutvandecappelle<http://www.linkedin.com/in/arnoutvandecappelle>
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
Signature · 5 lines
Best Regards,
Denis Mingulov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131017/ba4a2d01/attachment-0001.html>
#7 lore raw

PATCH 1/1 rsync: use system zlib and popt, also do not lose debug options

Quoted 31 lines · Denis Mingulov

Hi,

Quoted 1 line

building only rsync before and after your patch, and although the

binary becomes smaller with your patch, the overall filesystem becomes larger.

Yes, in case of the only selected package it will be so. Due to overhead of shared library / not removed unused components.
But for this particular case - you most probably have openssh (depends to zlib) or dropbear (depends to zlib if not 'small' version is selected separately) also already.

Quoted 1 line

I'd therefore propose to only use the system library if it is anyway

selected.

Sorry I am disagree, even without considering the security issue, just because zlib/popt for rsync are not optional libraries, they are mandatory, and will be compiled in any case.
In the current buildroot .mk files such 'if' use case might be useful for optional components only, something like 'nice to have, will be supported if already present'. Otherwise there might be tens of components compiling its own 'zlib' version (tens different versions) - and the Buildroot user must somehow found FAQ or understand himself, that he must 'include zlib to the package manually' (zlib or tens of other libraries).

Also it will be opposite to the current Buildroot configuration. In the latest master there are 9 packages with 'select BR2_PACKAGE_POPT' (cryptsetup, gptfdisk, libiscsi, librsync, logrotate, lttng-tools, oprofile, rpm, samba) and 50 packages with the same for zlib (of course not every one has its own zlib/popt).

Quoted 4 lines

Might be, later it would be nice to have something like 'select_if_multiple' in Config files, to select other package if more than 1 component is selecting it, but it will do Configs more complex, and the use case is quite rare.

Yeah, you're right.

It was the "binary size is reduced by 25%" thing in your commit message
that triggered me. That difference is probably irrelevant, because it hardly affects the overall filesystem size (rsync binary is relatively small).

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F