Skip to contentSkip to conversation

mklibs library optimization for all arches

PATCH In review Nitin A Kamble 2 participants 7 messages Feb 1, 2011 – Feb 2, 2011

lore
Share & tools
lore

Nitin A Kamble

author
#0 lore raw

cover mklibs library optimization for all arches

From: Nitin A Kamble <[email protected]>

I tested this with poky-image-minimal and I see some disk space savings by
using the mklibs library optimizations. (76MB rootfs became 75MB)

The minimal poky image had 106 dynamically linked executables. And the
footprint savings will go up as number of dynamically linked executables in an image decrease.

I did not see any improvement on mips and ppc, but could see the footprint
improvement on x86, arm & x86-64.

Also checked that the executables were not broken after the mklibs
optimization.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: nitin/mklibs
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=nitin/mklibs

Thanks,
Nitin A Kamble <[email protected]>
---

Nitin A Kamble (4):
distro_tracking: update latest upstream versions
elfutils: add a missing depedancy
mklibs-native: new recipe for optimizing size of library files image-mklibs.bbclass: add the library optimization functionality

 meta/classes/image-mklibs.bbclass                  |   57 ++++++++++++++++++++
 .../conf/distro/include/distro_tracking_fields.inc |   43 ++++++++-------
 meta/conf/local.conf.sample                        |    2 +
 meta/recipes-devtools/elfutils/elfutils_0.148.bb   |    3 +-
 .../mklibs/files/ac_init_fix.patch                 |   17 ++++++
 .../mklibs/mklibs-native_0.1.31.bb                 |   23 ++++++++
 6 files changed, 124 insertions(+), 21 deletions(-)
 create mode 100644 meta/classes/image-mklibs.bbclass
 create mode 100644 meta/recipes-devtools/mklibs/files/ac_init_fix.patch
 create mode 100644 meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb
Signature · 1 line
1.7.2.2
author
#5 lore raw

PATCH 3/4 mklibs-native: new recipe for optimizing size of library files

From: Nitin A Kamble <[email protected]>

Signed-off-by Nitin A Kamble <[email protected]>
---
 .../mklibs/files/ac_init_fix.patch                 |   17 ++++++++++++++
 .../mklibs/mklibs-native_0.1.31.bb                 |   23 ++++++++++++++++++++
 2 files changed, 40 insertions(+), 0 deletions(-)
 create mode 100644 meta/recipes-devtools/mklibs/files/ac_init_fix.patch
 create mode 100644 meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb
meta/recipes-devtools/mklibs/files/ac_init_fix.patch new file +17 -0
new file mode 100644
index 0000000..7325931
@@ -0,0 +1,17 @@
+Get the version of mklibs by simpler means. The MKLIBS_VERSION string in the 
+configure.ac file is replaced with real version string by the 
+do_configure_prepend() function from the recipe .bb file.
+
+Nitin A Kamble <[email protected]>
+Date: 2011/01/24
+
+Index: mklibs/configure.ac
+===================================================================
+--- mklibs.orig/configure.ac	2010-02-21 17:34:56.000000000 -0800
++++ mklibs/configure.ac	2011-01-24 18:52:19.943242079 -0800
+@@ -1,4 +1,4 @@
+-AC_INIT([mklibs],m4_esyscmd(dpkg-parsechangelog | perl -ne 'print $1 if m/^Version: (.*)$/;'))
++AC_INIT([mklibs], MKLIBS_VERSION)
+ AM_INIT_AUTOMAKE([foreign no-define])
+ AC_CONFIG_HEADERS([config.h])
+ AM_MAINTAINER_MODE
meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb new file +23 -0
new file mode 100644
index 0000000..2044d12
@@ -0,0 +1,23 @@
+DESCRIPTION = "mklibs produces cut-down shared libraries that contain only the routines required by a particular set of executables."
+HOMEPAGE = "https://code.launchpad.net/mklibs"
+SECTION = "devel"
+LICENSE = "GPLv2+"
+LIC_FILES_CHKSUM = "file://debian/copyright;md5=98d31037b13d896e33890738ef01af64"
+DEPENDS = "python-native"
+
+PR = "r0"
+
+SRC_URI = "http://ftp.de.debian.org/debian/pool/main/m/mklibs/${BPN}_${PV}.tar.gz \
+	file://ac_init_fix.patch\
+"
+
+SRC_URI[md5sum] = "f4df0307ccbdf60070e42277513f27ed"
+SRC_URI[sha256sum] = "8f5595621eb09d52871c771861e81b032d10c31d15e5dd61fa7f5a9e5b7de405"
+
+S = "${WORKDIR}/${BPN}"
+
+inherit autotools gettext native
+
+do_configure_prepend() {
+	sed "s+MKLIBS_VERSION+${PV}+" ${S}/configure.ac
+}
Signature · 1 line
1.7.2.2
author
#6 lore raw

PATCH 4/4 image-mklibs.bbclass: add the library optimization functionality

In reply to Nitin A Kamble

From: Nitin A Kamble <[email protected]>

Thanks to mark hatle for his help in implementation of this.

Signed-off-by Nitin A Kamble <[email protected]>
---
 meta/classes/image-mklibs.bbclass |   57 +++++++++++++++++++++++++++++++++++++
 meta/conf/local.conf.sample       |    2 +
 2 files changed, 59 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/image-mklibs.bbclass
meta/classes/image-mklibs.bbclass new file +57 -0
new file mode 100644
index 0000000..9fd2bc0
@@ -0,0 +1,57 @@
+do_rootfs[depends] += "mklibs-native:do_populate_sysroot"
+
+IMAGE_PREPROCESS_COMMAND += "mklibs_optimize_image; "
+
+mklibs_optimize_image () {
+	rm -rf ${WORKDIR}/mklibs
+	mkdir -p ${WORKDIR}/mklibs/dest
+	cd ${IMAGE_ROOTFS}
+	du -bs > ${WORKDIR}/mklibs/du.before.mklibs.txt
+	for i in `find .`; do file $i; done \
+		| grep ELF \
+		| grep "LSB executable" \
+		| grep "dynamically linked" \
+		| sed "s/:.*//" \
+		| sed "s+^\./++" \
+		> ${WORKDIR}/mklibs/executables.list
+
+	case ${TARGET_ARCH} in
+		powerpc | mips )
+			dynamic_loader="/lib/ld.so.1"
+			;;
+		x86_64)
+			dynamic_loader="/lib/ld-linux-x86-64.so.2"
+			;;
+		i586 )
+			dynamic_loader="/lib/ld-linux.so.2"
+			;;
+		arm )
+			dynamic_loader="/lib/ld-linux.so.3"
+			;;
+		* )
+			dynamic_loader="/unknown_dynamic_linker"
+			;;
+	esac
+
+	mklibs -v \
+		--ldlib ${dynamic_loader} \
+		--sysroot ${PKG_CONFIG_SYSROOT_DIR} \
+		--root ${IMAGE_ROOTFS} \
+		--target `echo ${TARGET_PREFIX} | sed 's/-$//' ` \
+		-d ${WORKDIR}/mklibs/dest \
+		`cat ${WORKDIR}/mklibs/executables.list`
+
+	cd ${WORKDIR}/mklibs/dest
+	for i in *
+	do
+		cp $i `find ${IMAGE_ROOTFS} -name $i`
+	done
+
+	cd ${IMAGE_ROOTFS}
+	du -bs > ${WORKDIR}/mklibs/du.after.mklibs.txt
+
+	echo rootfs size before mklibs optimization: `cat ${WORKDIR}/mklibs/du.before.mklibs.txt`
+	echo rootfs size after mklibs optimization: `cat ${WORKDIR}/mklibs/du.after.mklibs.txt`
+}
+
+EXPORT_FUNCTIONS mklibs_optimize_image
meta/conf/local.conf.sample +2 -0
index f9953d1..a3667e9 100644
@@ -74,8 +74,10 @@ EXTRA_IMAGE_FEATURES_mx31ads = "tools-testapps debug-tweaks"
 PACKAGE_CLASSES ?= "package_rpm package_ipk"
 
 # A list of additional classes to use when building the system
+# include 'image-mklibs' to reduce shared library files size for an image
 # include 'image-prelink' in order to prelink the filesystem image
 # include 'image-swab' to perform host system intrusion detection
+# NOTE: if listing mklibs & prelink both, then make sure mklibs is before prelink
 USER_CLASSES ?= "image-prelink"
 
 # POKYMODE controls the characteristics of the generated packages/images by
Signature · 1 line
1.7.2.2
author
#1 lore raw

PATCH 1/4 distro_tracking: update latest upstream versions

In reply to Nitin A Kamble

From: Nitin A Kamble <[email protected]>

Signed-off-by Nitin A Kamble <[email protected]>
---
 .../conf/distro/include/distro_tracking_fields.inc |   43 ++++++++++---------
 1 files changed, 23 insertions(+), 20 deletions(-)
meta/conf/distro/include/distro_tracking_fields.inc +23 -20
index 9f99e50..1bdf521 100644
@@ -102,9 +102,14 @@ RECIPE_LATEST_VERSION_pn-run-postinsts = "check"
 RECIPE_LAST_UPDATE_pn-run-postinsts = "Nov 22, 2010"
 RECIPE_MAINTAINER_pn-run-postinsts = "Dongxiao Xu <[email protected]"
 
-RECIPE_STATUS_pn-subversion = "green"
-RECIPE_LATEST_VERSION_pn-subversion = "1.6.13"
-RECIPE_LAST_UPDATE_pn-subversion = "Nov 18, 2010"
+RECIPE_STATUS_pn-screen = "red"
+RECIPE_LATEST_VERSION_pn-screen = "check"
+RECIPE_LAST_UPDATE_pn-screen = "Aug 31, 2010"
+RECIPE_MAINTAINER_pn-screen = "Qing He <[email protected]>"
+
+RECIPE_STATUS_pn-subversion = "red"
+RECIPE_LATEST_VERSION_pn-subversion = "1.6.15"
+RECIPE_MANUAL_CHECK_DATE_pn-subversion = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-subversion = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-trace-cmd = "red"

@@ -2807,17 +2812,16 @@ RECIPE_STATUS_pn-ldconfig-native = "red"
 RECIPE_LAST_UPDATE_pn-ldconfig-native = "Nov 23, 2007"
 DISTRO_PN_ALIAS_pn-ldconfig-native = "Ubuntu=libc-bin Fedora=glibc"
 
-RECIPE_STATUS_pn-gcc="green" 
-RECIPE_LATEST_VERSION_pn-gcc="4.5.1"
-RECIPE_NO_OF_PATCHES_pn-gcc="8"
-RECIPE_TIME_BETWEEN_LAST_TWO_RELEASES_pn-gcc="5 months"
+RECIPE_STATUS_pn-gcc="red" 
+RECIPE_LATEST_VERSION_pn-gcc="4.5.2"
 RECIPE_LATEST_RELEASE_DATE_pn-gcc="2010/03/03"
 RECIPE_LAST_UPDATE_pn-gcc = "Nov 8, 2010"
 RECIPE_MAINTAINER_pn-gcc = "Nitin A Kamble <[email protected]>"
 
-RECIPE_STATUS_pn-autoconf="green" 
+RECIPE_STATUS_pn-autoconf="red" 
 RECIPE_LATEST_VERSION_pn-autoconf="2.68"
 RECIPE_LAST_UPDATE_pn-autoconf = "Jan 29, 2010"
+RECIPE_NO_UPDATE_REASON_autoconf="seeing rpm issue with the 2.68 version, a bug is open"
 RECIPE_MAINTAINER_pn-autoconf = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-automake="green" 

@@ -2887,7 +2891,7 @@ RECIPE_MAINTAINER_pn-postinsts = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-nasm="green" 
 RECIPE_LATEST_VERSION_pn-nasm="2.07"
-RECIPE_LAST_UPDATE_pn-nasm = "Jun 23, 2010"
+RECIPE_MANUAL_CHECK_DATE_pn-nasm = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-nasm = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-perl="red" # upgrade needed

@@ -2898,7 +2902,7 @@ RECIPE_MAINTAINER_pn-perl = "Nitin A Kamble <[email protected]>"
 RECIPE_STATUS_pn-prelink="red" 
 RECIPE_LATEST_VERSION_pn-prelink="1.0+git0+0x909470ee441237563d6236c505cb2d02ddc48704"
 RECIPE_LAST_UPDATE_pn-perl = "Jul 23, 2010"
-RECIPE_MAINTAINER_pn-prelink = "Nitin A Kamble <[email protected]>"
+RECIPE_MAINTAINER_pn-prelink = "Mark Hatle <[email protected]>"
 
 RECIPE_STATUS_pn-python-dbus="red" 
 RECIPE_LATEST_VERSION_pn-python-dbus="0.83.1"

@@ -2906,8 +2910,8 @@ RECIPE_LAST_UPDATE_pn-python-dbus = "Jul 7, 2010"
 RECIPE_MAINTAINER_pn-python-dbus = "Nitin A Kamble <[email protected]>"
 DISTRO_PN_ALIAS_pn-python-dbus = "Ubuntu=python-dbus Debian=python-dbus Mandriva=python-dbus"
 
-RECIPE_STATUS_pn-python-gst="green" 
-RECIPE_LATEST_VERSION_pn-python-gst="0.10.19"
+RECIPE_STATUS_pn-python-gst="red" 
+RECIPE_LATEST_VERSION_pn-python-gst="0.10.21"
 DISTRO_PN_ALIAS_pn-python-gst = "OpenSuSE=python-gstreamer Ubuntu=gst0.10-python Debian=gst0.10-python"
 RECIPE_LAST_UPDATE_pn-python-gst = "Nov 8, 2010"
 RECIPE_MAINTAINER_pn-python-gst = "Nitin A Kamble <[email protected]>"

@@ -2948,9 +2952,6 @@ RECIPE_MAINTAINER_pn-python-scons = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-python="red"
 RECIPE_LATEST_VERSION_pn-python="2.7.1"
-RECIPE_NO_OF_PATCHES_pn-python="8"
-RECIPE_LATEST_RELEASE_DATE_pn-python="2010/03/18"
-RECIPE_TIME_BETWEEN_LAST_TWO_RELEASES_pn-python="5 months"
 RECIPE_LAST_UPDATE_pn-python = "Nov 8, 2010"
 RECIPE_MAINTAINER_pn-python = "Nitin A Kamble <[email protected]>"
 

@@ -2976,7 +2977,7 @@ RECIPE_MAINTAINER_pn-gnu-config = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-mpfr="green"
 RECIPE_LATEST_VERSION_pn-mpfr="3.0.0"
-RECIPE_LAST_UPDATE_pn-mpfr = "Aug 27, 2010"
+RECIPE_MANUAL_CHECK_DATE_pn-mpfr = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-mpfr = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-gmp="green"

@@ -2986,16 +2987,18 @@ RECIPE_MAINTAINER_pn-gmp = "Nitin A Kamble <[email protected]>"
 
 RECIPE_STATUS_pn-libmpc="green"
 RECIPE_LATEST_VERSION_pn-libmpc="0.8.2"
-RECIPE_LAST_UPDATE_pn-libmpc = "Jul 19, 2010"
+RECIPE_MANUAL_CHECK_DATE_pn-libmpc = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-libmpc = "Nitin A Kamble <[email protected]>"
 DISTRO_PN_ALIAS_pn-libmpc = "Fedora=libmpc OpenSuse=libmpc2"
 
-RECIPE_STATUS_pn-byacc="green" 
-RECIPE_LATEST_VERSION_pn-byacc="20101127"
+RECIPE_STATUS_pn-byacc="red" 
+RECIPE_LATEST_VERSION_pn-byacc="20101229"
 RECIPE_MAINTAINER_pn-byacc = "Nitin A Kamble <[email protected]>"
 
-RECIPE_STATUS_pn-libconvert-asn1-perl="green" # 0.22 version does not build, so created a recipe for version 0.21
+RECIPE_STATUS_pn-libconvert-asn1-perl="green" 
+# 0.22 version does not build, so created a recipe for version 0.21
 RECIPE_LATEST_VERSION_pn-libconvert-asn1-perl="0.22"
+RECIPE_NO_UPDATE_REASON_pn-libconvert-asn1-perl = "0.22 version does not build, so created a recipe for version 0.21"
 RECIPE_LAST_UPDATE_pn-libconvert-asn1-perl = "Aug 13, 2010"
 RECIPE_MAINTAINER_pn-libconvert-asn1-perl = "Nitin A Kamble <[email protected]>"
Signature · 1 line
1.7.2.2
author
#2 lore raw

PATCH 2/4 elfutils: add a missing depedancy

In reply to Nitin A Kamble

From: Nitin A Kamble <[email protected]>

while implementing the mklibs optimization, found out that an executable from elfutils package, viz eu-findtextrel is dynamically linked to libbz2.so library. But the poky-image-minimal did not have that library.
adding the depedency to the bzip2 to get that library for eu-fundtextrel
executable.

Signed-off-by Nitin A Kamble <[email protected]>
---
 meta/recipes-devtools/elfutils/elfutils_0.148.bb |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
meta/recipes-devtools/elfutils/elfutils_0.148.bb +2 -1
index 44b3c52..14060e6 100644
@@ -5,8 +5,9 @@ LICENSE = "GPLv2 with exceptions"
 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3\
                     file://EXCEPTION;md5=570adcb0c1218ab57f2249c67d0ce417"
 DEPENDS = "libtool zlib"
+RDEPENDS = "bzip2"
 
-PR = "r1"
+PR = "r2"
 
 SRC_URI = "https://fedorahosted.org/releases/e/l/elfutils/elfutils-${PV}.tar.bz2"
Signature · 1 line
1.7.2.2
#3 lore raw

Re: [PATCH 2/4] elfutils: add a missing depedancy

Quoted 18 lines · Nitin A Kamble

From: Nitin A Kamble <[email protected]>

while implementing the mklibs optimization, found out that an executable from elfutils package, viz eu-findtextrel is dynamically linked to libbz2.so library. But the poky-image-minimal did not have that library.
adding the depedency to the bzip2 to get that library for eu-fundtextrel
executable.

Signed-off-by Nitin A Kamble <[email protected]>
---
 meta/recipes-devtools/elfutils/elfutils_0.148.bb |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
meta/recipes-devtools/elfutils/elfutils_0.148.bb +0 -0
index 44b3c52..14060e6 100644

meta/recipes-devtools/elfutils/elfutils_0.148.bb LICENSE = "GPLv2 with exceptions" · line 8

Quoted 4 lines
meta/recipes-devtools/elfutils/elfutils_0.148.bb +1 -0
@@ -5,8 +5,9 @@ LICENSE = "GPLv2 with exceptions"
 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3\
                     file://EXCEPTION;md5=570adcb0c1218ab57f2249c67d0ce417"
 DEPENDS = "libtool zlib"
+RDEPENDS = "bzip2"

There should be a package name here, e.g. RDEPENDS_${PN}.

Also, why is the shlibs code not picking up the dependency automatically? Is it dlopen'd?

Cheers,

Richard

author
#4 lore raw

Re: [PATCH 2/4] elfutils: add a missing depedancy

Quoted 28 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Tuesday, February 01, 2011 2:00 PM
To: Kamble, Nitin A
Cc: [email protected]
Subject: Re: [poky] [PATCH 2/4] elfutils: add a missing depedancy

Quoted 3 lines · Nitin A Kamble

From: Nitin A Kamble <[email protected]>

while implementing the mklibs optimization, found out that an

executable from

Quoted 1 line

elfutils package, viz eu-findtextrel is dynamically linked to

libbz2.so

Quoted 2 lines

library. But the poky-image-minimal did not have that library.
adding the depedency to the bzip2 to get that library for eu-

fundtextrel

Review

Quoted 8 lines

executable.

Signed-off-by Nitin A Kamble <[email protected]>
---
 meta/recipes-devtools/elfutils/elfutils_0.148.bb |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
(quoted diff) +0 -0

b/meta/recipes-devtools/elfutils/elfutils_0.148.bb

Quoted 11 lines

meta/recipes-devtools/elfutils/elfutils_0.148.bb LICENSE = "GPLv2 with exceptions" · line 5

Quoted 5 lines

index 44b3c52..14060e6 100644

meta/recipes-devtools/elfutils/elfutils_0.148.bb +0 -0
@@ -5,8 +5,9 @@ LICENSE = "GPLv2 with exceptions"
 LIC_FILES_CHKSUM =

"file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3\

Quoted 1 line

file://EXCEPTION;md5=570adcb0c1218ab57f2249c67d0ce417"

Quoted 2 lines

DEPENDS = "libtool zlib"
+RDEPENDS = "bzip2"

Thanks Richard for the review,

Quoted 1 line

There should be a package name here, e.g. RDEPENDS_${PN}.

I will fix the RDEPENDS_${PN}, and also fix the gplv3 recipe

Quoted 3 lines

Also, why is the shlibs code not picking up the dependency automatically? Is it dlopen'd?

Don't know. Not sure what you are asking.

Thanks,
Nitin

Quoted 4 lines

Cheers,

Richard