Skip to contentSkip to conversation

rootfs.py: Allow to override postinst-intercepts location

RFC PATCH In review Martin Jansa 3 participants 10 messages Sep 4, 2015 – Oct 14, 2015

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/3 rootfs.py: Allow to override postinst-intercepts location

* useful when we need to overlay/extend intercept scripts from oe-core

---
 meta/lib/oe/rootfs.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
meta/lib/oe/rootfs.py +4 -2
index c29843b..76950ec 100644
@@ -164,6 +164,9 @@ class Rootfs(object):
         pre_process_cmds = self.d.getVar("ROOTFS_PREPROCESS_COMMAND", True)
         post_process_cmds = self.d.getVar("ROOTFS_POSTPROCESS_COMMAND", True)
 
+        postinst_intercepts_dir = self.d.getVar("POSTINST_INTERCEPTS_DIR", True)
+        if not postinst_intercepts_dir:
+            postinst_intercepts_dir = self.d.expand("${COREBASE}/scripts/postinst-intercepts")
         intercepts_dir = os.path.join(self.d.getVar('WORKDIR', True),
                                       "intercept_scripts")
 
@@ -173,8 +176,7 @@ class Rootfs(object):
 
         bb.utils.mkdirhier(self.deploy_dir_image)
 
-        shutil.copytree(self.d.expand("${COREBASE}/scripts/postinst-intercepts"),
-                        intercepts_dir)
+        shutil.copytree(postinst_intercepts_dir, intercepts_dir)
 
         shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.txt"),
                     self.deploy_dir_image +
Signature · 1 line
2.5.0
#1 lore raw

RFC PATCH 2/3 postinst_intercept: allow to pass variables with spaces

* trying to pass foo="a b" through postinst_intercept ends with the actual script header to containing:
b
foo=a
which fails because "b" command doesn't exist.

---
 scripts/postinst-intercepts/postinst_intercept | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/postinst-intercepts/postinst_intercept +1 -1
index a257198..b18e806 100755
@@ -48,7 +48,7 @@ if [ -n "$pkgs_line" ]; then
 		sed -i -e "s/##PKGS:.*/\0${package_name} /" $intercept_script
 	fi
 else
-	for var in $@; do
+	for var in "$@"; do
 		sed -i -e "\%^#\!/bin/.*sh%a $var" $intercept_script
 	done
 	echo "##PKGS: ${package_name} " >> $intercept_script
Signature · 1 line
2.5.0
#2 lore raw

RFC PATCH 3/3 fontcache: allow to pass extra parameters and environment to fc-cache

In reply to Martin Jansa

* this can be useful for passing extra parameters, pass
-v by default to see what's going on in do_rootfs
* we need to use this for extra parameter we implemented
in fontconfig:
--ignore-mtime always use cache file regardless of font directory mtime because the checksum of fontcache generated in do_rootfs doesn't match with /usr/share/fonts directory as seen on target device causing fontconfig to re-create the cache when fontconfig is used for first time or worse create
new cache in every user's home directory when /usr/ filesystem is read only and cache cannot be updated.

Running FC_DEBUG=16 fc-cache -v on such device shows:
FcCacheTimeValid dir "/usr/share/fonts" cache checksum 1441207803 dir checksum 1441206149
* my guess is that the checksum is different, because pseudo
(which is unloaded when running qemuwrapper) or because some
influence of running the rootfs under qemu.

---
 meta/classes/fontcache.bbclass                | 19 +++++++++++++++----
 scripts/postinst-intercepts/update_font_cache |  4 ++--
 2 files changed, 17 insertions(+), 6 deletions(-)
meta/classes/fontcache.bbclass +15 -4
index d122387..1dd1418 100644
@@ -9,12 +9,23 @@ inherit qemu
 FONT_PACKAGES ??= "${PN}"
 FONT_EXTRA_RDEPENDS ?= "fontconfig-utils"
 FONTCONFIG_CACHE_DIR ?= "${localstatedir}/cache/fontconfig"
+FONTCONFIG_CACHE_PARAMS ?= "-v"
+# You can change this to e.g. FC_DEBUG=16 to debug fc-cache issues,
+# something has to be set, because qemuwrapper is using this variable after -E
+# multiple variables aren't allowed because for qemu they are separated
+# by comma and in -n "$D" case they should be separated by space
+FONTCONFIG_CACHE_ENV ?= "FC_DEBUG=1"
 fontcache_common() {
-if [ "x$D" != "x" ] ; then
-	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} bindir=${bindir} \
-		libdir=${libdir} base_libdir=${base_libdir} fontconfigcachedir=${FONTCONFIG_CACHE_DIR}
+if [ -n "$D" ] ; then
+	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} \
+		'bindir="${bindir}'" \
+		'libdir="${libdir}'" \
+		'base_libdir="${base_libdir}'" \
+		'fontconfigcachedir="${FONTCONFIG_CACHE_DIR}'" \
+		'fontconfigcacheparams="${FONTCONFIG_CACHE_PARAMS}'" \
+		'fontconfigcacheenv="${FONTCONFIG_CACHE_ENV}'"
 else
-	fc-cache
+	${FONTCONFIG_CACHE_ENV} fc-cache ${FONTCONFIG_CACHE_PARAMS}
 fi
 }
 
scripts/postinst-intercepts/update_font_cache +2 -2
index c8c6018..0deab3c 100644
@@ -1,5 +1,5 @@
 #!/bin/sh
 
-PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir}\
-					$D${bindir}/fc-cache --sysroot=$D
+PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir} \
+					-E ${fontconfigcacheenv} $D${bindir}/fc-cache --sysroot=$D ${fontconfigcacheparams}
 chown -R root:root $D${fontconfigcachedir}
Signature · 1 line
2.5.0
#3 lore raw

PATCH v2 1/1 fontcache: allow to pass extra parameters and environment to fc-cache

* this can be useful for passing extra parameters, pass
-v by default to see what's going on in do_rootfs
* we need to use this for extra parameter we implemented
in fontconfig:
--ignore-mtime always use cache file regardless of font directory mtime because the checksum of fontcache generated in do_rootfs doesn't match with /usr/share/fonts directory as seen on target device causing fontconfig to re-create the cache when fontconfig is used for first time or worse create
new cache in every user's home directory when /usr/ filesystem is read only and cache cannot be updated.

Running FC_DEBUG=16 fc-cache -v on such device shows:
FcCacheTimeValid dir "/usr/share/fonts" cache checksum 1441207803 dir checksum 1441206149
* my guess is that the checksum is different, because pseudo
(which is unloaded when running qemuwrapper) or because some
influence of running the rootfs under qemu.

---
 meta/classes/fontcache.bbclass                | 19 +++++++++++++++----
 scripts/postinst-intercepts/update_font_cache |  4 ++--
 2 files changed, 17 insertions(+), 6 deletions(-)
meta/classes/fontcache.bbclass +15 -4
index d122387..8ebdfc4 100644
@@ -9,12 +9,23 @@ inherit qemu
 FONT_PACKAGES ??= "${PN}"
 FONT_EXTRA_RDEPENDS ?= "fontconfig-utils"
 FONTCONFIG_CACHE_DIR ?= "${localstatedir}/cache/fontconfig"
+FONTCONFIG_CACHE_PARAMS ?= "-v"
+# You can change this to e.g. FC_DEBUG=16 to debug fc-cache issues,
+# something has to be set, because qemuwrapper is using this variable after -E
+# multiple variables aren't allowed because for qemu they are separated
+# by comma and in -n "$D" case they should be separated by space
+FONTCONFIG_CACHE_ENV ?= "FC_DEBUG=1"
 fontcache_common() {
-if [ "x$D" != "x" ] ; then
-	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} bindir=${bindir} \
-		libdir=${libdir} base_libdir=${base_libdir} fontconfigcachedir=${FONTCONFIG_CACHE_DIR}
+if [ -n "$D" ] ; then
+	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} \
+		'bindir="${bindir}"' \
+		'libdir="${libdir}"' \
+		'base_libdir="${base_libdir}"' \
+		'fontconfigcachedir="${FONTCONFIG_CACHE_DIR}"' \
+		'fontconfigcacheparams="${FONTCONFIG_CACHE_PARAMS}"' \
+		'fontconfigcacheenv="${FONTCONFIG_CACHE_ENV}"'
 else
-	fc-cache
+	${FONTCONFIG_CACHE_ENV} fc-cache ${FONTCONFIG_CACHE_PARAMS}
 fi
 }
 
scripts/postinst-intercepts/update_font_cache +2 -2
index c8c6018..0deab3c 100644
@@ -1,5 +1,5 @@
 #!/bin/sh
 
-PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir}\
-					$D${bindir}/fc-cache --sysroot=$D
+PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir} \
+					-E ${fontconfigcacheenv} $D${bindir}/fc-cache --sysroot=$D ${fontconfigcacheparams}
 chown -R root:root $D${fontconfigcachedir}
Signature · 1 line
2.5.1
#4 lore raw

Re: [PATCHv2] fontcache: allow to pass extra parameters and environment to fc-cache

Quoted 12 lines · Martin Jansa

* this can be useful for passing extra parameters, pass
-v by default to see what's going on in do_rootfs
* we need to use this for extra parameter we implemented
in fontconfig:
--ignore-mtime always use cache file regardless of font directory mtime because the checksum of fontcache generated in do_rootfs doesn't match with /usr/share/fonts directory as seen on target device causing fontconfig to re-create the cache when fontconfig is used for first time or worse create
new cache in every user's home directory when /usr/ filesystem is read only and cache cannot be updated.

Quoted 5 lines

Running FC_DEBUG=16 fc-cache -v on such device shows:
FcCacheTimeValid dir "/usr/share/fonts" cache checksum 1441207803 dir checksum 1441206149
* my guess is that the checksum is different, because pseudo
(which is unloaded when running qemuwrapper) or because some
influence of running the rootfs under qemu.

ping

was this postponed to 2.0? Any feedback would be nice.

But thanks for applying the other 2, now we were able to implement this in our layer with overlayed fontcache.bbclass and intercepts scripts.

Quoted 53 lines
---
 meta/classes/fontcache.bbclass                | 19 +++++++++++++++----
 scripts/postinst-intercepts/update_font_cache |  4 ++--
 2 files changed, 17 insertions(+), 6 deletions(-)
meta/classes/fontcache.bbclass +15 -4
index d122387..8ebdfc4 100644
@@ -9,12 +9,23 @@ inherit qemu
 FONT_PACKAGES ??= "${PN}"
 FONT_EXTRA_RDEPENDS ?= "fontconfig-utils"
 FONTCONFIG_CACHE_DIR ?= "${localstatedir}/cache/fontconfig"
+FONTCONFIG_CACHE_PARAMS ?= "-v"
+# You can change this to e.g. FC_DEBUG=16 to debug fc-cache issues,
+# something has to be set, because qemuwrapper is using this variable after -E
+# multiple variables aren't allowed because for qemu they are separated
+# by comma and in -n "$D" case they should be separated by space
+FONTCONFIG_CACHE_ENV ?= "FC_DEBUG=1"
 fontcache_common() {
-if [ "x$D" != "x" ] ; then
-	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} bindir=${bindir} \
-		libdir=${libdir} base_libdir=${base_libdir} fontconfigcachedir=${FONTCONFIG_CACHE_DIR}
+if [ -n "$D" ] ; then
+	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} \
+		'bindir="${bindir}"' \
+		'libdir="${libdir}"' \
+		'base_libdir="${base_libdir}"' \
+		'fontconfigcachedir="${FONTCONFIG_CACHE_DIR}"' \
+		'fontconfigcacheparams="${FONTCONFIG_CACHE_PARAMS}"' \
+		'fontconfigcacheenv="${FONTCONFIG_CACHE_ENV}"'
 else
-	fc-cache
+	${FONTCONFIG_CACHE_ENV} fc-cache ${FONTCONFIG_CACHE_PARAMS}
 fi
 }
 
scripts/postinst-intercepts/update_font_cache +2 -2
index c8c6018..0deab3c 100644
@@ -1,5 +1,5 @@
 #!/bin/sh
 
-PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir}\
-					$D${bindir}/fc-cache --sysroot=$D
+PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir} \
+					-E ${fontconfigcacheenv} $D${bindir}/fc-cache --sysroot=$D ${fontconfigcacheparams}
 chown -R root:root $D${fontconfigcachedir}
Signature · 1 line
2.5.1
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#5 lore raw

Re: [PATCHv2] fontcache: allow to pass extra parameters and environment to fc-cache

Quoted 19 lines · Martin Jansa
Quoted 12 lines · Martin Jansa

* this can be useful for passing extra parameters, pass
-v by default to see what's going on in do_rootfs
* we need to use this for extra parameter we implemented
in fontconfig:
--ignore-mtime always use cache file regardless of font directory mtime because the checksum of fontcache generated in do_rootfs doesn't match with /usr/share/fonts directory as seen on target device causing fontconfig to re-create the cache when fontconfig is used for first time or worse create
new cache in every user's home directory when /usr/ filesystem is read only and cache cannot be updated.

Quoted 5 lines

Running FC_DEBUG=16 fc-cache -v on such device shows:
FcCacheTimeValid dir "/usr/share/fonts" cache checksum 1441207803 dir checksum 1441206149
* my guess is that the checksum is different, because pseudo
(which is unloaded when running qemuwrapper) or because some
influence of running the rootfs under qemu.

Quoted 6 lines

ping

was this postponed to 2.0? Any feedback would be nice.

But thanks for applying the other 2, now we were able to implement this in our layer with overlayed fontcache.bbclass and intercepts scripts.

ping2

Quoted 56 lines
Quoted 53 lines
---
 meta/classes/fontcache.bbclass                | 19 +++++++++++++++----
 scripts/postinst-intercepts/update_font_cache |  4 ++--
 2 files changed, 17 insertions(+), 6 deletions(-)
meta/classes/fontcache.bbclass +15 -4
index d122387..8ebdfc4 100644
@@ -9,12 +9,23 @@ inherit qemu
 FONT_PACKAGES ??= "${PN}"
 FONT_EXTRA_RDEPENDS ?= "fontconfig-utils"
 FONTCONFIG_CACHE_DIR ?= "${localstatedir}/cache/fontconfig"
+FONTCONFIG_CACHE_PARAMS ?= "-v"
+# You can change this to e.g. FC_DEBUG=16 to debug fc-cache issues,
+# something has to be set, because qemuwrapper is using this variable after -E
+# multiple variables aren't allowed because for qemu they are separated
+# by comma and in -n "$D" case they should be separated by space
+FONTCONFIG_CACHE_ENV ?= "FC_DEBUG=1"
 fontcache_common() {
-if [ "x$D" != "x" ] ; then
-	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} bindir=${bindir} \
-		libdir=${libdir} base_libdir=${base_libdir} fontconfigcachedir=${FONTCONFIG_CACHE_DIR}
+if [ -n "$D" ] ; then
+	$INTERCEPT_DIR/postinst_intercept update_font_cache ${PKG} mlprefix=${MLPREFIX} \
+		'bindir="${bindir}"' \
+		'libdir="${libdir}"' \
+		'base_libdir="${base_libdir}"' \
+		'fontconfigcachedir="${FONTCONFIG_CACHE_DIR}"' \
+		'fontconfigcacheparams="${FONTCONFIG_CACHE_PARAMS}"' \
+		'fontconfigcacheenv="${FONTCONFIG_CACHE_ENV}"'
 else
-	fc-cache
+	${FONTCONFIG_CACHE_ENV} fc-cache ${FONTCONFIG_CACHE_PARAMS}
 fi
 }
 
scripts/postinst-intercepts/update_font_cache +2 -2
index c8c6018..0deab3c 100644
@@ -1,5 +1,5 @@
 #!/bin/sh
 
-PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir}\
-					$D${bindir}/fc-cache --sysroot=$D
+PSEUDO_UNLOAD=1 qemuwrapper -L $D -E LD_LIBRARY_PATH=$D/${libdir}:$D/${base_libdir} \
+					-E ${fontconfigcacheenv} $D${bindir}/fc-cache --sysroot=$D ${fontconfigcacheparams}
 chown -R root:root $D${fontconfigcachedir}
Signature · 1 line
2.5.1
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#6 lore raw
In reply to Martin Jansa
Quoted 1 line · Martin Jansa

* useful when we need to overlay/extend intercept scripts from oe-core

2/3 and 3/3 could be a bit dangerous or controversial (main reason why this was sent as RFC), but this one is really useful for us to apply these fixes in advance (and without backporting them all to fido and dizzy) - so if you agree please merge 1/3 to master, fido and dizzy.

Quoted 32 lines
---
 meta/lib/oe/rootfs.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
meta/lib/oe/rootfs.py +4 -2
index c29843b..76950ec 100644
@@ -164,6 +164,9 @@ class Rootfs(object):
         pre_process_cmds = self.d.getVar("ROOTFS_PREPROCESS_COMMAND", True)
         post_process_cmds = self.d.getVar("ROOTFS_POSTPROCESS_COMMAND", True)
 
+        postinst_intercepts_dir = self.d.getVar("POSTINST_INTERCEPTS_DIR", True)
+        if not postinst_intercepts_dir:
+            postinst_intercepts_dir = self.d.expand("${COREBASE}/scripts/postinst-intercepts")
         intercepts_dir = os.path.join(self.d.getVar('WORKDIR', True),
                                       "intercept_scripts")
 
@@ -173,8 +176,7 @@ class Rootfs(object):
 
         bb.utils.mkdirhier(self.deploy_dir_image)
 
-        shutil.copytree(self.d.expand("${COREBASE}/scripts/postinst-intercepts"),
-                        intercepts_dir)
+        shutil.copytree(postinst_intercepts_dir, intercepts_dir)
 
         shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.txt"),
                     self.deploy_dir_image +
Signature · 1 line
2.5.0
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#7 lore raw

All 3 in master so backporting to Dizzy.

- armin

Quoted 43 lines · Martin Jansa
Quoted 1 line · Martin Jansa

* useful when we need to overlay/extend intercept scripts from oe-core

2/3 and 3/3 could be a bit dangerous or controversial (main reason why this was sent as RFC), but this one is really useful for us to apply these fixes in advance (and without backporting them all to fido and dizzy) - so if you agree please merge 1/3 to master, fido and dizzy.

Quoted 32 lines
---
 meta/lib/oe/rootfs.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
meta/lib/oe/rootfs.py +4 -2
index c29843b..76950ec 100644
@@ -164,6 +164,9 @@ class Rootfs(object):
         pre_process_cmds = self.d.getVar("ROOTFS_PREPROCESS_COMMAND", True)
         post_process_cmds = self.d.getVar("ROOTFS_POSTPROCESS_COMMAND", True)
 
+        postinst_intercepts_dir = self.d.getVar("POSTINST_INTERCEPTS_DIR", True)
+        if not postinst_intercepts_dir:
+            postinst_intercepts_dir = self.d.expand("${COREBASE}/scripts/postinst-intercepts")
         intercepts_dir = os.path.join(self.d.getVar('WORKDIR', True),
                                       "intercept_scripts")
 
@@ -173,8 +176,7 @@ class Rootfs(object):
 
         bb.utils.mkdirhier(self.deploy_dir_image)
 
-        shutil.copytree(self.d.expand("${COREBASE}/scripts/postinst-intercepts"),
-                        intercepts_dir)
+        shutil.copytree(postinst_intercepts_dir, intercepts_dir)
 
         shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.txt"),
                     self.deploy_dir_image +
Signature · 1 line
2.5.0
#8 lore raw

RFC PATCH 1/1 rootfs.py: show intercept script output in log.do_rootfs

In reply to Martin Jansa

* without this the output wasn't shown anywhere even when the bb.warn says:
"See log for details!"

---
 meta/lib/oe/rootfs.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/rootfs.py +1 -1
index 76950ec..2372226 100644
@@ -279,7 +279,7 @@ class Rootfs(object):
             bb.note("> Executing %s intercept ..." % script)
 
             try:
-                subprocess.check_output(script_full)
+                subprocess.check_call(script_full)
             except subprocess.CalledProcessError as e:
                 bb.warn("The postinstall intercept hook '%s' failed (exit code: %d)! See log for details!" %
                         (script, e.returncode))
Signature · 1 line
2.5.0
#9 lore raw
In reply to Martin Jansa
Quoted 4 lines · Martin Jansa

* useful when we need to overlay/extend intercept scripts from oe -core

Patch queued in my joshuagl/fido-next tree - thanks!

Joshua

http://cgit.openembedded.org/openembedded-core-contrib/log/?h=joshuagl/
fido-next

Quoted 39 lines
---
 meta/lib/oe/rootfs.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
meta/lib/oe/rootfs.py +3 -0
index c29843b..76950ec 100644
@@ -164,6 +164,9 @@ class Rootfs(object):
         pre_process_cmds =
 self.d.getVar("ROOTFS_PREPROCESS_COMMAND", True)
         post_process_cmds =
 self.d.getVar("ROOTFS_POSTPROCESS_COMMAND", True)
 
+        postinst_intercepts_dir =
 self.d.getVar("POSTINST_INTERCEPTS_DIR", True)
+        if not postinst_intercepts_dir:
+            postinst_intercepts_dir =

self.d.expand("${COREBASE}/scripts/postinst-intercepts")
intercepts_dir = os.path.join(self.d.getVar('WORKDIR',
True),
"intercept_scripts")

meta/lib/oe/rootfs.py +1 -3
@@ -173,8 +176,7 @@ class Rootfs(object):
 
         bb.utils.mkdirhier(self.deploy_dir_image)
 
-        shutil.copytree(self.d.expand("${COREBASE}/scripts/postinst
-intercepts"),
-                        intercepts_dir)
+        shutil.copytree(postinst_intercepts_dir, intercepts_dir)
 
 
 shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.t

xt"),
self.deploy_dir_image +

Signature · 1 line
2.5.0