Postinstall improvements
cover Postinstall improvements
Hi,
This is the former RFC that was on the mailing list a couple of months ago. Hopefully, this time the patches will make it in since last time the release was to close to have these changes in.
So, the main improvement this patchset brings is running the icon cache generation on host, at rootfs time, after all packages have been installed. This will cut off a lot from the target's first boot time. Also, postinstalls for gconf and gdk-pixbuf were also changed to run on host at rootfs time.
With these changes, just a handful of packages will need to run the postinstalls on target (did the tests with core-image-sato). I prefer to approach the remaining packages one by one.
My preliminary tests show no issues but, since there are lots of setups out there, testing is always welcomed. Andreas, I know you have such a setup, could you take these patches for a spin if you find some time?
Thanks,
Laurentiu
The following changes since commit da6c266b1184eed8b1bc851f24a9ee17ab56d562:
xserver-xorg: restore packaging for the DRI/DRI2/DBE extensions (2012-12-03 16:38:35 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib lpalcu/postinst
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/postinst
Laurentiu Palcu (5):
image.bbclass: run postinst scripts once, at rootfs time rootfs generation: export two new variables to postinst scriptlets
gtk-icon-cache: run the icon generation at rootfs time
gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time gconf.bbclass: run postinstalls at rootfs time
meta/classes/gconf.bbclass | 11 +++++++---- meta/classes/gtk-icon-cache.bbclass | 18 ++++++++++++++---- meta/classes/image.bbclass | 20 +++++++++++++++++--- meta/classes/package_rpm.bbclass | 2 ++ meta/classes/rootfs_deb.bbclass | 2 ++ meta/classes/rootfs_ipk.bbclass | 2 ++ meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb | 17 +++++++++++++++-- 7 files changed, 59 insertions(+), 13 deletions(-)
Signature · 1 line
1.7.9.5
PATCH 1/5 image.bbclass: run postinst scripts once, at rootfs time
This patch will allow the repeating postinst scriptlets to be run only once, on host, at do_rootfs time. This will lower the time for rootfs generation and, also, instead of running some time consuming scriptlets at target's first boot, we will do on the host.
--- meta/classes/image.bbclass | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
meta/classes/image.bbclass +17 -3
@@ -156,7 +156,7 @@ inherit ${IMAGE_CLASSES} IMAGE_POSTPROCESS_COMMAND ?= "" MACHINE_POSTPROCESS_COMMAND ?= "" -ROOTFS_POSTPROCESS_COMMAND ?= "" +ROOTFS_POSTPROCESS_COMMAND_prepend = "run_intercept_scriptlets; " # some default locales IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
@@ -166,15 +166,29 @@ LINGUAS_INSTALL ?= "${@" ".join(map(lambda s: "locale-base-%s" % s, d.getVar('IM PSEUDO_PASSWD = "${IMAGE_ROOTFS}" do_rootfs[nostamp] = "1" -do_rootfs[dirs] = "${TOPDIR}" +do_rootfs[dirs] = "${TOPDIR} ${WORKDIR}/intercept_scripts" do_rootfs[lockfiles] += "${IMAGE_ROOTFS}.lock" -do_rootfs[cleandirs] += "${S}" +do_rootfs[cleandirs] += "${S} ${WORKDIR}/intercept_scripts" do_build[nostamp] = "1" # Must call real_do_rootfs() from inside here, rather than as a separate # task, so that we have a single fakeroot context for the whole process. do_rootfs[umask] = "022" + +run_intercept_scriptlets () { + if [ -d ${WORKDIR}/intercept_scripts ]; then + cd ${WORKDIR}/intercept_scripts + echo "Running intercept scripts:" + for script in *; do + if [ "$script" = "*" ]; then break; fi + echo "> Executing $script" + chmod +x $script + ./$script + done + fi +} + fakeroot do_rootfs () { #set -x # When use the rpm incremental image generation, don't remove the rootfs
Signature · 1 line
1.7.9.5
Re: [PATCH 1/5] image.bbclass: run postinst scripts once, at rootfs time
Quoted 37 lines · Laurentiu Palcu
This patch will allow the repeating postinst scriptlets to be run only once, on host, at
do_rootfstime. This will lower the time for rootfs generation and, also, instead of running some time consuming scriptlets at target's first boot, we will do on the host.Signed-off-by Laurentiu Palcu <[email protected]>--- meta/classes/image.bbclass | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)meta/classes/image.bbclass +1
-1@@ -156,7 +156,7 @@ inherit ${IMAGE_CLASSES} IMAGE_POSTPROCESS_COMMAND ?= "" MACHINE_POSTPROCESS_COMMAND ?= "" -ROOTFS_POSTPROCESS_COMMAND ?= "" +ROOTFS_POSTPROCESS_COMMAND_prepend = "run_intercept_scriptlets; " # some default locales IMAGE_LINGUAS ?= "de-de fr-fr en-gb"@@ -166,15 +166,29 @@ LINGUAS_INSTALL ?= "${@" ".join(map(lambda s:"locale-base-%s" % s, d.getVar('IM
PSEUDO_PASSWD= "${IMAGE_ROOTFS}"do_rootfs[nostamp] = "1"
meta/classes/image.bbclass +2
-2-do_rootfs[dirs] = "${TOPDIR}" +do_rootfs[dirs] = "${TOPDIR} ${WORKDIR}/intercept_scripts" do_rootfs[lockfiles] += "${IMAGE_ROOTFS}.lock" -do_rootfs[cleandirs] += "${S}" +do_rootfs[cleandirs] += "${S} ${WORKDIR}/intercept_scripts" do_build[nostamp] = "1" # Must call real_do_rootfs() from inside here, rather than as a separate
meta/classes/image.bbclass LINGUAS_INSTALL ?= "${@" ".join(map(lambda s: "locale-base-%s" % s, d.getVar('IM · line 184
Quoted 10 lines
meta/classes/image.bbclass +7
-0# task, so that we have a single fakeroot context for the whole process. do_rootfs[umask] = "022" + +run_intercept_scriptlets () { + if [ -d ${WORKDIR}/intercept_scripts ]; then + cd ${WORKDIR}/intercept_scripts + echo "Running intercept scripts:" + for script in *; do + if [ "$script" = "*" ]; then break; fi
This is personal choice but I'd use:
if [ ... ]; then
break
fi
or
[ ... ] && break
The if in a single line seems confusing to read for me.
Quoted 20 lines
meta/classes/image.bbclass +7
-0+ echo "> Executing $script" + chmod +x $script + ./$script + done + fi +} + fakeroot do_rootfs () { #set -x # When use the rpm incremental image generation, don't remove therootfs
--
1.7.9.5_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 3 lines
Otavio Salvador O.S. Systems E-mail: [email protected] http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
PATCH 2/5 rootfs generation: export two new variables to postinst scriptlets
In reply to Laurentiu PalcuIn order for the postinst scriptlets to be able to run once we need to export the location of the intercept scripts and also the location of native sysrootfs. The gdk-pixbuf binaries will need the latter because in order to generate the loaders.cache it will need to scan some shared libraries that must be native. Even though the output is a text file.
--- meta/classes/package_rpm.bbclass | 2 ++ meta/classes/rootfs_deb.bbclass | 2 ++ meta/classes/rootfs_ipk.bbclass | 2 ++ 3 files changed, 6 insertions(+)
meta/classes/package_rpm.bbclass +2 -0
@@ -475,6 +475,8 @@ export D="${target_rootfs}" export OFFLINE_ROOT="\$D" export IPKG_OFFLINE_ROOT="\$D" export OPKG_OFFLINE_ROOT="\$D" +export INTERCEPT_DIR="${WORKDIR}/intercept_scripts" +export NATIVE_ROOT=${STAGING_DIR_NATIVE} \$2 \$1/\$3 \$4 if [ \$? -ne 0 ]; then
meta/classes/rootfs_deb.bbclass +2 -0
@@ -48,6 +48,8 @@ fakeroot rootfs_deb_do_rootfs () { export OFFLINE_ROOT=${IMAGE_ROOTFS} export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS} export OPKG_OFFLINE_ROOT=${IMAGE_ROOTFS} + export INTERCEPT_DIR=${WORKDIR}/intercept_scripts + export NATIVE_ROOT=${STAGING_DIR_NATIVE} # Attempt to run preinsts # Mark packages with preinst failures as unpacked
meta/classes/rootfs_ipk.bbclass +2 -0
@@ -69,6 +69,8 @@ fakeroot rootfs_ipk_do_rootfs () { export OFFLINE_ROOT=${IMAGE_ROOTFS} export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS} export OPKG_OFFLINE_ROOT=${IPKG_OFFLINE_ROOT} + export INTERCEPT_DIR=${WORKDIR}/intercept_scripts + export NATIVE_ROOT=${STAGING_DIR_NATIVE} package_install_internal_ipk
Signature · 1 line
1.7.9.5
Re: [PATCH 2/5] rootfs generation: export two new variables to postinst scriptlets
Quoted 19 lines · Laurentiu Palcu
In order for the postinst scriptlets to be able to run once we need to export the location of the intercept scripts and also the location of native sysrootfs. The gdk-pixbuf binaries will need the latter because in order to generate the loaders.cache it will need to scan some shared libraries that must be native. Even though the output is a text file.
Signed-off-by Laurentiu Palcu <[email protected]>--- meta/classes/package_rpm.bbclass | 2 ++ meta/classes/rootfs_deb.bbclass | 2 ++ meta/classes/rootfs_ipk.bbclass | 2 ++ 3 files changed, 6 insertions(+)meta/classes/package_rpm.bbclass +0
-0
meta/classes/package_rpm.bbclass export D="${target_rootfs}" · line 480
Quoted 6 lines
meta/classes/package_rpm.bbclass +2
-0@@ -475,6 +475,8 @@ export D="${target_rootfs}" export OFFLINE_ROOT="\$D" export IPKG_OFFLINE_ROOT="\$D" export OPKG_OFFLINE_ROOT="\$D" +export INTERCEPT_DIR="${WORKDIR}/intercept_scripts" +export NATIVE_ROOT=${STAGING_DIR_NATIVE}
The NATIVE_ROOT seems a duplication, I'd prefer you to use STAGING_DIR_NATIVE in code as this is a known variable name and makes it easy to understand.
Quoted 39 lines
meta/classes/package_rpm.bbclass +0
-0\$2 \$1/\$3 \$4 if [ \$? -ne 0 ]; thenmeta/classes/rootfs_deb.bbclass +2
-0@@ -48,6 +48,8 @@ fakeroot rootfs_deb_do_rootfs () { export OFFLINE_ROOT=${IMAGE_ROOTFS} export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS} export OPKG_OFFLINE_ROOT=${IMAGE_ROOTFS} + export INTERCEPT_DIR=${WORKDIR}/intercept_scripts + export NATIVE_ROOT=${STAGING_DIR_NATIVE} # Attempt to run preinsts # Mark packages with preinst failures as unpackedmeta/classes/rootfs_ipk.bbclass +2
-1@@ -69,6 +69,8 @@ fakeroot rootfs_ipk_do_rootfs () { export OFFLINE_ROOT=${IMAGE_ROOTFS} export IPKG_OFFLINE_ROOT=${IMAGE_ROOTFS} export OPKG_OFFLINE_ROOT=${IPKG_OFFLINE_ROOT} + export INTERCEPT_DIR=${WORKDIR}/intercept_scripts + export NATIVE_ROOT=${STAGING_DIR_NATIVE} package_install_internal_ipk --1.7.9.5
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 3 lines
Otavio Salvador O.S. Systems E-mail: [email protected] http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
Re: [PATCH 2/5] rootfs generation: export two new variables to postinst scriptlets
Quoted 36 lines · Otavio Salvador
On Tue, Dec 4, 2012 at 11:59 AM, Laurentiu Palcu <[email protected]> wrote:
In order for the postinst scriptlets to be able to run once we need to
export the location of the intercept scripts and also the location of
native sysrootfs. The gdk-pixbuf binaries will need the latter because
in order to generate the loaders.cache it will need to scan some shared
libraries that must be native. Even though the output is a text file.Signed-off-by Laurentiu Palcu <[email protected]>---
meta/classes/package_rpm.bbclass | 2 ++
meta/classes/rootfs_deb.bbclass | 2 ++
meta/classes/rootfs_ipk.bbclass | 2 ++
3 files changed, 6 insertions(+)diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 1ff92ce..c7ac07a 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -475,6 +475,8 @@ export D="${target_rootfs}"
export OFFLINE_ROOT="\$D"
export IPKG_OFFLINE_ROOT="\$D"
export OPKG_OFFLINE_ROOT="\$D"
+export INTERCEPT_DIR="${WORKDIR}/intercept_scripts" +export NATIVE_ROOT=${STAGING_DIR_NATIVE}
Quoted 3 lines
The
NATIVE_ROOTseems a duplication, I'd prefer you to useSTAGING_DIR_NATIVEin code as this is a known variable name and makes it easy to understand.
I'm going to disagree on that, I think this does make sense in the context of the functions we're changing here.
On the other hand I'd love to simplify the whole *OFFLINE_ROOT mess but that is for a different patch...
Cheers,
Richard
PATCH 3/5 gtk-icon-cache: run the icon generation at rootfs time
In reply to Laurentiu PalcuThis change will allow for the icon cache generation at rootfs time and only once, at the end. So, even though there will be many packages depending on gtk+, the icon cache generation will be done once. Hopefully, this will lower the target's first boot time significantly by using the power of the host machine to generate the cache.
--- meta/classes/gtk-icon-cache.bbclass | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
meta/classes/gtk-icon-cache.bbclass +14 -4
@@ -1,12 +1,22 @@ FILES_${PN} += "${datadir}/icons/hicolor" -DEPENDS += "${@['hicolor-icon-theme', '']['${BPN}' == 'hicolor-icon-theme']}" +DEPENDS += "${@['hicolor-icon-theme', '']['${BPN}' == 'hicolor-icon-theme']} gtk+-native" -# This could run on the host as icon cache files are architecture independent, -# but there is no gtk-update-icon-cache built natively. gtk_icon_cache_postinst() { if [ "x$D" != "x" ]; then - exit 1 + if [ ! -f $INTERCEPT_DIR/update_icon_cache ]; then + cat << "EOF" > $INTERCEPT_DIR/update_icon_cache +#!/bin/sh + +# update native pixbuf loaders +gdk-pixbuf-query-loaders --update-cache + +for icondir in $D/usr/share/icons/*/ ; do + gtk-update-icon-cache -fqt $icondir +done +EOF + fi + exit 0 fi # Update the pixbuf loaders in case they haven't been registered yet
Signature · 1 line
1.7.9.5
PATCH 4/5 gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time
In reply to Laurentiu PalcuThis will generate the loaders.cache file for pixbuf, at rootfs time.
--- meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb +15 -2
@@ -58,9 +58,21 @@ FILES_${PN}-dbg += " \ postinst_pixbufloader () { if [ "x$D" != "x" ]; then - exit 1 +# Update the target's pixbuf loader's cache. Since the native binary will +# throw an error if the shared objects do not belong to the same ELF class, +# we trick the gdk-pixbuf-query-loaders into scanning the native shared +# objects and then we remove the NATIVE_ROOT prefix from the paths in +# loaders.cache. +gdk-pixbuf-query-loaders $(find $D/${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders \ + -name *.so | sed -e "s:$D:$NATIVE_ROOT:g") > \ + $D/${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders.cache || exit 1 + +sed -i -e "s:$NATIVE_ROOT:/:g" $D/${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders.cache + +exit 0 fi +# Update the pixbuf loaders in case they haven't been registered yet GDK_PIXBUF_MODULEDIR=${libdir}/gdk-pixbuf-2.0/${LIBV}/loaders gdk-pixbuf-query-loaders --update-cache if [ -x ${bindir}/gtk-update-icon-cache ] && [ -d ${datadir}/icons ]; then
@@ -93,6 +105,7 @@ do_install_append_class-native() { GDK_PIXBUF_MODULE_FILE=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders.cache create_wrapper ${D}/${bindir}/gdk-pixbuf-query-loaders \ - GDK_PIXBUF_MODULE_FILE=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders.cache + GDK_PIXBUF_MODULE_FILE=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders.cache \ + GDK_PIXBUF_MODULEDIR=${STAGING_LIBDIR_NATIVE}/gdk-pixbuf-2.0/${LIBV}/loaders } BBCLASSEXTEND = "native"
Signature · 1 line
1.7.9.5
Re: [PATCH 4/5] gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time
Hi,
meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb FILES_${PN}-dbg += " \ · line 65
Quoted 5 lines · Laurentiu Palcu
meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.24.1.bb +5
-0+# Update the target's pixbuf loader's cache. Since the native binary will +# throw an error if the shared objects do not belong to the same ELF class, +# we trick the gdk-pixbuf-query-loaders into scanning the native shared +# objects and then we remove the NATIVE_ROOT prefix from the paths in +# loaders.cache.
It's probably worth explaining why it's fine that only the loaders from gdk-pixbuf-native are used and for example librsvg-native doesn't need to be built.
Ross
Re: [PATCH 4/5] gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time
Quoted 3 lines · Ross
It's probably worth explaining why it's fine that only the loaders from gdk-pixbuf-native are used and for example librsvg-native doesn't need to be built.
First off, the librsvg postinstall scriptlet is still postponed to run on target, at first boot. I didn't touch that one, yet. We will deal with this one (and others) in subsequent patches. Secondly, the gdk-pixbuf postinstall scriptlet will exit 1 if the command fails. So, if one has librsvg built and the native one is not present, the gdk-pixbuf scriptlet will fail on host because it will not find the native SVG pixbuf loader and it will be postponed to run on target at first boot. No harm done, this is the current behavior anyway.
However, if we want to be able to run all postinstall scriptlets on host in order to support read-only rootfs, we might need to have librsvg-native built too so that we generate the pixbuf loader cache on host at rootfs install time.
Let's take it step-by-step though and see how the current changes behave. What do you think?
Thanks,
Laurentiu
Re: [PATCH 4/5] gdk-pixbuf: generate the pixbuf loader's cache ar rootfs time
Quoted 4 lines · Laurentiu Palcu
However, if we want to be able to run all postinstall scriptlets on host in order to support read-only rootfs, we might need to have librsvg-native built too so that we generate the pixbuf loader cache on host at rootfs install time.
Not true - the icon cache explicitly only loads .png and .xpm, so my point was that it's entirely safe to run gtk-update-icon-cache at rootfs time as those loaders are part of gdk-pixbuf, which must be available natively if we're running gtk-update-icon-cache.
Ross
PATCH 5/5 gconf.bbclass: run postinstalls at rootfs time
In reply to Laurentiu Palcu--- meta/classes/gconf.bbclass | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/gconf.bbclass +7 -4
@@ -15,13 +15,16 @@ export GCONF_DISABLE_MAKEFILE_SCHEMA_INSTALL = "1" gconf_postinst() { if [ "x$D" != "x" ]; then - exit 1 + export GCONF_BACKEND_DIR=${STAGING_LIBDIR_NATIVE}/GConf/2 + export GCONF_CONFIG_SOURCE="xml::$D${sysconfdir}/gconf/gconf.xml.defaults" +else + export GCONF_CONFIG_SOURCE=`gconftool-2 --get-default-source` fi -SCHEMA_LOCATION=/etc/gconf/schemas + +SCHEMA_LOCATION=$D/etc/gconf/schemas for SCHEMA in ${SCHEMA_FILES}; do if [ -e $SCHEMA_LOCATION/$SCHEMA ]; then - HOME=/root GCONF_CONFIG_SOURCE=`gconftool-2 --get-default-source` \ - gconftool-2 \ + HOME=$D/root gconftool-2 \ --makefile-install-rule $SCHEMA_LOCATION/$SCHEMA > /dev/null fi done
Signature · 1 line
1.7.9.5