Skip to contentSkip to conversation

Postinstall improvements

PATCH In review Laurentiu Palcu 4 participants 12 messages Dec 4, 2012 – Dec 14, 2012

lore
Share & tools
lore
#0 lore raw

cover Postinstall improvements

Hi,

This is the former RFC that was on the mailing list a couple of months ago. Hopefully, this time the patches will make it in since last time the release was to close to have these changes in.

So, the main improvement this patchset brings is running the icon cache generation on host, at rootfs time, after all packages have been installed. This will cut off a lot from the target's first boot time. Also, postinstalls for gconf and gdk-pixbuf were also changed to run on host at rootfs time.

With these changes, just a handful of packages will need to run the postinstalls on target (did the tests with core-image-sato). I prefer to approach the remaining packages one by one.

My preliminary tests show no issues but, since there are lots of setups out there, testing is always welcomed. Andreas, I know you have such a setup, could you take these patches for a spin if you find some time?

Thanks,
Laurentiu

The following changes since commit da6c266b1184eed8b1bc851f24a9ee17ab56d562:

xserver-xorg: restore packaging for the DRI/DRI2/DBE extensions (2012-12-03 16:38:35 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib lpalcu/postinst
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/postinst

Laurentiu Palcu (5):
image.bbclass: run postinst scripts once, at rootfs time rootfs generation: export two new variables to postinst scriptlets
gtk-icon-cache: run the icon generation at rootfs time
gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time gconf.bbclass: run postinstalls at rootfs time

 meta/classes/gconf.bbclass                         |   11 +++++++----
 meta/classes/gtk-icon-cache.bbclass                |   18 ++++++++++++++----
 meta/classes/image.bbclass                         |   20 +++++++++++++++++---
 meta/classes/package_rpm.bbclass                   |    2 ++
 meta/classes/rootfs_deb.bbclass                    |    2 ++
 meta/classes/rootfs_ipk.bbclass                    |    2 ++
 meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb |   17 +++++++++++++++--
 7 files changed, 59 insertions(+), 13 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/5 image.bbclass: run postinst scripts once, at rootfs time

This patch will allow the repeating postinst scriptlets to be run only once, on host, at do_rootfs time. This will lower the time for rootfs generation and, also, instead of running some time consuming scriptlets at target's first boot, we will do on the host.

---
 meta/classes/image.bbclass |   20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
meta/classes/image.bbclass +17 -3
index 7b24e4e..719d871 100644
@@ -156,7 +156,7 @@ inherit ${IMAGE_CLASSES}
 
 IMAGE_POSTPROCESS_COMMAND ?= ""
 MACHINE_POSTPROCESS_COMMAND ?= ""
-ROOTFS_POSTPROCESS_COMMAND ?= ""
+ROOTFS_POSTPROCESS_COMMAND_prepend = "run_intercept_scriptlets; "
 
 # some default locales
 IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
@@ -166,15 +166,29 @@ LINGUAS_INSTALL ?= "${@" ".join(map(lambda s: "locale-base-%s" % s, d.getVar('IM
 PSEUDO_PASSWD = "${IMAGE_ROOTFS}"
 
 do_rootfs[nostamp] = "1"
-do_rootfs[dirs] = "${TOPDIR}"
+do_rootfs[dirs] = "${TOPDIR} ${WORKDIR}/intercept_scripts"
 do_rootfs[lockfiles] += "${IMAGE_ROOTFS}.lock"
-do_rootfs[cleandirs] += "${S}"
+do_rootfs[cleandirs] += "${S} ${WORKDIR}/intercept_scripts"
 do_build[nostamp] = "1"
 
 # Must call real_do_rootfs() from inside here, rather than as a separate
 # task, so that we have a single fakeroot context for the whole process.
 do_rootfs[umask] = "022"
 
+
+run_intercept_scriptlets () {
+	if [ -d ${WORKDIR}/intercept_scripts ]; then
+		cd ${WORKDIR}/intercept_scripts
+		echo "Running intercept scripts:"
+		for script in *; do
+			if [ "$script" = "*" ]; then break; fi
+			echo "> Executing $script"
+			chmod +x $script
+			./$script
+		done
+	fi
+}
+
 fakeroot do_rootfs () {
 	#set -x
 	# When use the rpm incremental image generation, don't remove the rootfs
Signature · 1 line
1.7.9.5
#2 lore raw

Re: [PATCH 1/5] image.bbclass: run postinst scripts once, at rootfs time

Quoted 37 lines · Laurentiu Palcu

This patch will allow the repeating postinst scriptlets to be run only once, on host, at do_rootfs time. This will lower the time for rootfs generation and, also, instead of running some time consuming scriptlets at target's first boot, we will do on the host.

---
 meta/classes/image.bbclass |   20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
meta/classes/image.bbclass +1 -1
index 7b24e4e..719d871 100644
@@ -156,7 +156,7 @@ inherit ${IMAGE_CLASSES}
 
 IMAGE_POSTPROCESS_COMMAND ?= ""
 MACHINE_POSTPROCESS_COMMAND ?= ""
-ROOTFS_POSTPROCESS_COMMAND ?= ""
+ROOTFS_POSTPROCESS_COMMAND_prepend = "run_intercept_scriptlets; "
 
 # some default locales
 IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
@@ -166,15 +166,29 @@ LINGUAS_INSTALL ?= "${@" ".join(map(lambda s:

"locale-base-%s" % s, d.getVar('IM
PSEUDO_PASSWD = "${IMAGE_ROOTFS}"

do_rootfs[nostamp] = "1"

meta/classes/image.bbclass +2 -2
-do_rootfs[dirs] = "${TOPDIR}"
+do_rootfs[dirs] = "${TOPDIR} ${WORKDIR}/intercept_scripts"
 do_rootfs[lockfiles] += "${IMAGE_ROOTFS}.lock"
-do_rootfs[cleandirs] += "${S}"
+do_rootfs[cleandirs] += "${S} ${WORKDIR}/intercept_scripts"
 do_build[nostamp] = "1"
 
 # Must call real_do_rootfs() from inside here, rather than as a separate

meta/classes/image.bbclass LINGUAS_INSTALL ?= "${@" ".join(map(lambda s: "locale-base-%s" % s, d.getVar('IM · line 184

Quoted 10 lines
meta/classes/image.bbclass +7 -0
 # task, so that we have a single fakeroot context for the whole process.
 do_rootfs[umask] = "022"
 
+
+run_intercept_scriptlets () {
+       if [ -d ${WORKDIR}/intercept_scripts ]; then
+               cd ${WORKDIR}/intercept_scripts
+               echo "Running intercept scripts:"
+               for script in *; do
+                       if [ "$script" = "*" ]; then break; fi

This is personal choice but I'd use:

if [ ... ]; then
break
fi

or

[ ... ] && break

The if in a single line seems confusing to read for me.

Quoted 20 lines
meta/classes/image.bbclass +7 -0
+                       echo "> Executing $script"
+                       chmod +x $script
+                       ./$script
+               done
+       fi
+}
+
 fakeroot do_rootfs () {
        #set -x
        # When use the rpm incremental image generation, don't remove the

rootfs
--
1.7.9.5

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Signature · 3 lines
Otavio Salvador                             O.S. Systems
E-mail: [email protected]  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br
#3 lore raw

PATCH 2/5 rootfs generation: export two new variables to postinst scriptlets

In reply to Laurentiu Palcu

In order for the postinst scriptlets to be able to run once we need to export the location of the intercept scripts and also the location of native sysrootfs. The gdk-pixbuf binaries will need the latter because in order to generate the loaders.cache it will need to scan some shared libraries that must be native. Even though the output is a text file.

---
 meta/classes/package_rpm.bbclass |    2 ++
 meta/classes/rootfs_deb.bbclass  |    2 ++
 meta/classes/rootfs_ipk.bbclass  |    2 ++
 3 files changed, 6 insertions(+)
meta/classes/package_rpm.bbclass +2 -0
index 1ff92ce..c7ac07a 100644
@@ -475,6 +475,8 @@ export D="${target_rootfs}"
 export OFFLINE_ROOT="\$D"
 export IPKG_OFFLINE_ROOT="\$D"
 export OPKG_OFFLINE_ROOT="\$D"
+export INTERCEPT_DIR="${WORKDIR}/intercept_scripts"
+export NATIVE_ROOT=${STAGING_DIR_NATIVE}
 
 \$2 \$1/\$3 \$4
 if [ \$? -ne 0 ]; then
meta/classes/rootfs_deb.bbclass +2 -0
index 881fdbd..955382f 100644
@@ -48,6 +48,8 @@ fakeroot rootfs_deb_do_rootfs () {
 	export OFFLINE_ROOT=${IMAGE_ROOTFS}
 	export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS}
 	export OPKG_OFFLINE_ROOT=${IMAGE_ROOTFS}
+	export INTERCEPT_DIR=${WORKDIR}/intercept_scripts
+	export NATIVE_ROOT=${STAGING_DIR_NATIVE}
 
 	# Attempt to run preinsts
 	# Mark packages with preinst failures as unpacked
meta/classes/rootfs_ipk.bbclass +2 -0
index fc69b7e..8766d24 100644
@@ -69,6 +69,8 @@ fakeroot rootfs_ipk_do_rootfs () {
 	export OFFLINE_ROOT=${IMAGE_ROOTFS}
 	export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS}
 	export OPKG_OFFLINE_ROOT=${IPKG_OFFLINE_ROOT}
+	export INTERCEPT_DIR=${WORKDIR}/intercept_scripts
+	export NATIVE_ROOT=${STAGING_DIR_NATIVE}
 
 	package_install_internal_ipk
 
Signature · 1 line
1.7.9.5
#4 lore raw

Re: [PATCH 2/5] rootfs generation: export two new variables to postinst scriptlets

Quoted 19 lines · Laurentiu Palcu

In order for the postinst scriptlets to be able to run once we need to export the location of the intercept scripts and also the location of native sysrootfs. The gdk-pixbuf binaries will need the latter because in order to generate the loaders.cache it will need to scan some shared libraries that must be native. Even though the output is a text file.

---
 meta/classes/package_rpm.bbclass |    2 ++
 meta/classes/rootfs_deb.bbclass  |    2 ++
 meta/classes/rootfs_ipk.bbclass  |    2 ++
 3 files changed, 6 insertions(+)
meta/classes/package_rpm.bbclass +0 -0
index 1ff92ce..c7ac07a 100644

meta/classes/package_rpm.bbclass export D="${target_rootfs}" · line 480

Quoted 6 lines
meta/classes/package_rpm.bbclass +2 -0
@@ -475,6 +475,8 @@ export D="${target_rootfs}"
 export OFFLINE_ROOT="\$D"
 export IPKG_OFFLINE_ROOT="\$D"
 export OPKG_OFFLINE_ROOT="\$D"
+export INTERCEPT_DIR="${WORKDIR}/intercept_scripts"
+export NATIVE_ROOT=${STAGING_DIR_NATIVE}
 

The NATIVE_ROOT seems a duplication, I'd prefer you to use STAGING_DIR_NATIVE in code as this is a known variable name and makes it easy to understand.

Quoted 39 lines
meta/classes/package_rpm.bbclass +0 -0
 \$2 \$1/\$3 \$4
 if [ \$? -ne 0 ]; then
meta/classes/rootfs_deb.bbclass +2 -0
index 881fdbd..955382f 100644
@@ -48,6 +48,8 @@ fakeroot rootfs_deb_do_rootfs () {
        export OFFLINE_ROOT=${IMAGE_ROOTFS}
        export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS}
        export OPKG_OFFLINE_ROOT=${IMAGE_ROOTFS}
+       export INTERCEPT_DIR=${WORKDIR}/intercept_scripts
+       export NATIVE_ROOT=${STAGING_DIR_NATIVE}
 
        # Attempt to run preinsts
        # Mark packages with preinst failures as unpacked
meta/classes/rootfs_ipk.bbclass +2 -1
index fc69b7e..8766d24 100644
@@ -69,6 +69,8 @@ fakeroot rootfs_ipk_do_rootfs () {
        export OFFLINE_ROOT=${IMAGE_ROOTFS}
        export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS}
        export OPKG_OFFLINE_ROOT=${IPKG_OFFLINE_ROOT}
+       export INTERCEPT_DIR=${WORKDIR}/intercept_scripts
+       export NATIVE_ROOT=${STAGING_DIR_NATIVE}
 
        package_install_internal_ipk
 
--

1.7.9.5

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Signature · 3 lines
Otavio Salvador                             O.S. Systems
E-mail: [email protected]  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br
#5 lore raw

Re: [PATCH 2/5] rootfs generation: export two new variables to postinst scriptlets

Quoted 36 lines · Otavio Salvador

On Tue, Dec 4, 2012 at 11:59 AM, Laurentiu Palcu <[email protected]> wrote:
In order for the postinst scriptlets to be able to run once we need to
export the location of the intercept scripts and also the location of
native sysrootfs. The gdk-pixbuf binaries will need the latter because
in order to generate the loaders.cache it will need to scan some shared
libraries that must be native. Even though the output is a text file.

---
meta/classes/package_rpm.bbclass | 2 ++
meta/classes/rootfs_deb.bbclass | 2 ++
meta/classes/rootfs_ipk.bbclass | 2 ++
3 files changed, 6 insertions(+)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 1ff92ce..c7ac07a 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -475,6 +475,8 @@ export D="${target_rootfs}"
export OFFLINE_ROOT="\$D"
export IPKG_OFFLINE_ROOT="\$D"
export OPKG_OFFLINE_ROOT="\$D"
+export INTERCEPT_DIR="${WORKDIR}/intercept_scripts" +export NATIVE_ROOT=${STAGING_DIR_NATIVE}

Quoted 3 lines

The NATIVE_ROOT seems a duplication, I'd prefer you to use STAGING_DIR_NATIVE in code as this is a known variable name and makes it easy to understand.

I'm going to disagree on that, I think this does make sense in the context of the functions we're changing here.

On the other hand I'd love to simplify the whole *OFFLINE_ROOT mess but that is for a different patch...

Cheers,

Richard

#6 lore raw

PATCH 3/5 gtk-icon-cache: run the icon generation at rootfs time

In reply to Laurentiu Palcu

This change will allow for the icon cache generation at rootfs time and only once, at the end. So, even though there will be many packages depending on gtk+, the icon cache generation will be done once. Hopefully, this will lower the target's first boot time significantly by using the power of the host machine to generate the cache.

---
 meta/classes/gtk-icon-cache.bbclass |   18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
meta/classes/gtk-icon-cache.bbclass +14 -4
index 01fb2f3..f87a30f 100644
@@ -1,12 +1,22 @@
 FILES_${PN} += "${datadir}/icons/hicolor"
 
-DEPENDS += "${@['hicolor-icon-theme', '']['${BPN}' == 'hicolor-icon-theme']}"
+DEPENDS += "${@['hicolor-icon-theme', '']['${BPN}' == 'hicolor-icon-theme']} gtk+-native"
 
-# This could run on the host as icon cache files are architecture independent,
-# but there is no gtk-update-icon-cache built natively.
 gtk_icon_cache_postinst() {
 if [ "x$D" != "x" ]; then
-        exit 1
+    if [ ! -f $INTERCEPT_DIR/update_icon_cache ]; then
+        cat << "EOF" > $INTERCEPT_DIR/update_icon_cache
+#!/bin/sh
+
+# update native pixbuf loaders
+gdk-pixbuf-query-loaders --update-cache
+
+for icondir in $D/usr/share/icons/*/ ; do
+	gtk-update-icon-cache -fqt  $icondir
+done
+EOF
+    fi
+    exit 0
 fi
 
 # Update the pixbuf loaders in case they haven't been registered yet
Signature · 1 line
1.7.9.5
#7 lore raw

PATCH 4/5 gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time

In reply to Laurentiu Palcu

This will generate the loaders.cache file for pixbuf, at rootfs time.

---
 meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb |   17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb +15 -2
index 65a2d68..2377d0e 100644
@@ -58,9 +58,21 @@ FILES_${PN}-dbg += " \
 
 postinst_pixbufloader () {
 if [ "x$D" != "x" ]; then
-    exit 1
+# Update the target's pixbuf loader's cache. Since the native binary will
+# throw an error if the shared objects do not belong to the same ELF class,
+# we trick the gdk-pixbuf-query-loaders into scanning the native shared
+# objects and then we remove the NATIVE_ROOT prefix from the paths in
+# loaders.cache.
+gdk-pixbuf-query-loaders $(find $D/${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders \
+        -name *.so | sed -e "s:$D:$NATIVE_ROOT:g") > \
+        $D/${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders.cache || exit 1
+
+sed -i -e "s:$NATIVE_ROOT:/:g" $D/${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders.cache
+
+exit 0
 fi
 
+# Update the pixbuf loaders in case they haven't been registered yet
 GDK_PIXBUF_MODULEDIR=${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders gdk-pixbuf-query-loaders --update-cache
 
 if [ -x ${bindir}/gtk-update-icon-cache ] && [ -d ${datadir}/icons ]; then
@@ -93,6 +105,7 @@ do_install_append_class-native() {
 		GDK_PIXBUF_MODULE_FILE=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders.cache
 
 	create_wrapper ${D}/${bindir}/gdk-pixbuf-query-loaders \
-		GDK_PIXBUF_MODULE_FILE=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders.cache
+		GDK_PIXBUF_MODULE_FILE=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders.cache \
+		GDK_PIXBUF_MODULEDIR=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders
 }
 BBCLASSEXTEND = "native"
Signature · 1 line
1.7.9.5
#8 lore raw

Re: [PATCH 4/5] gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time

Hi,

meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb FILES_${PN}-dbg += " \ · line 65

Quoted 5 lines · Laurentiu Palcu
meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb +5 -0
+# Update the target's pixbuf loader's cache. Since the native binary will
+# throw an error if the shared objects do not belong to the same ELF class,
+# we trick the gdk-pixbuf-query-loaders into scanning the native shared
+# objects and then we remove the NATIVE_ROOT prefix from the paths in
+# loaders.cache.

It's probably worth explaining why it's fine that only the loaders from gdk-pixbuf-native are used and for example librsvg-native doesn't need to be built.

Ross

#9 lore raw

Re: [PATCH 4/5] gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time

Quoted 3 lines · Ross

It's probably worth explaining why it's fine that only the loaders from gdk-pixbuf-native are used and for example librsvg-native doesn't need to be built.

First off, the librsvg postinstall scriptlet is still postponed to run on target, at first boot. I didn't touch that one, yet. We will deal with this one (and others) in subsequent patches. Secondly, the gdk-pixbuf postinstall scriptlet will exit 1 if the command fails. So, if one has librsvg built and the native one is not present, the gdk-pixbuf scriptlet will fail on host because it will not find the native SVG pixbuf loader and it will be postponed to run on target at first boot. No harm done, this is the current behavior anyway.

However, if we want to be able to run all postinstall scriptlets on host in order to support read-only rootfs, we might need to have librsvg-native built too so that we generate the pixbuf loader cache on host at rootfs install time.

Let's take it step-by-step though and see how the current changes behave. What do you think?

Thanks,
Laurentiu

#10 lore raw

Re: [PATCH 4/5] gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time

Quoted 4 lines · Laurentiu Palcu

However, if we want to be able to run all postinstall scriptlets on host in order to support read-only rootfs, we might need to have librsvg-native built too so that we generate the pixbuf loader cache on host at rootfs install time.

Not true - the icon cache explicitly only loads .png and .xpm, so my point was that it's entirely safe to run gtk-update-icon-cache at rootfs time as those loaders are part of gdk-pixbuf, which must be available natively if we're running gtk-update-icon-cache.

Ross

#11 lore raw

PATCH 5/5 gconf.bbclass: run postinstalls at rootfs time

In reply to Laurentiu Palcu
---
 meta/classes/gconf.bbclass |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/gconf.bbclass +7 -4
index 7a3ee3c..38097bc 100644
@@ -15,13 +15,16 @@ export GCONF_DISABLE_MAKEFILE_SCHEMA_INSTALL = "1"
 
 gconf_postinst() {
 if [ "x$D" != "x" ]; then
-	exit 1
+	export GCONF_BACKEND_DIR=${STAGING_LIBDIR_NATIVE}/GConf/2
+	export GCONF_CONFIG_SOURCE="xml::$D${sysconfdir}/gconf/gconf.xml.defaults"
+else
+	export GCONF_CONFIG_SOURCE=`gconftool-2 --get-default-source`
 fi
-SCHEMA_LOCATION=/etc/gconf/schemas
+
+SCHEMA_LOCATION=$D/etc/gconf/schemas
 for SCHEMA in ${SCHEMA_FILES}; do
 	if [ -e $SCHEMA_LOCATION/$SCHEMA ]; then
-		HOME=/root GCONF_CONFIG_SOURCE=`gconftool-2 --get-default-source` \
-			gconftool-2 \
+		HOME=$D/root gconftool-2 \
 			--makefile-install-rule $SCHEMA_LOCATION/$SCHEMA > /dev/null
 	fi
 done
Signature · 1 line
1.7.9.5