Fixes about unsafe-references QA warnings
cover Fixes about unsafe-references QA warnings
From: Chen Qi <[email protected]>
This solution is based on the following two principles.
1. With /usr on a seperate partition, system should still boot without any error.
2. Without /usr, system should be able to boot into single user mode with error.
When a QA warning about unsafe references is encountered, the above two principles, together with FHS, are taken into consideration before making a decision.
For example, libkmod is moved to /lib because /lib/udevd requires it; libgudevd is moved to /usr/lib because it's a GObject wrapper for libudev and it's obviously not necessary for booting into single user mode.
This patchset also contains several fixes for the sysvinit system.
The overall effect of all these patches:
1. Performing a world build doesn't report any warning about unsafe references.
2. System can boot into runlevel 3 without any error if /usr is on another partition.
3. System can boot into single user mode without any error even if /usr is missing.
---------------------------------------------------------------------------
The following changes since commit 4fdc3d77d4a875b7236536bf78849a4d1f6a7449:
kbd: Fix stdarg related errors on uclibc (2013-11-08 17:31:36 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib ChenQi/unsafe-reference
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/unsafe-reference
Chen Qi (8):
udev: fix dependency and location of udevadm
initscripts: add setup-commands.sh
zlib: install into base_libdir
kmod: install libkmod into base_libdir
udev: fix unsafe reference by installing libgudev in libdir insane.bbclass: make the checking stricter for unsafe references in
scripts
iputils: fix program location and QA warning
busybox: install ping6 into bindir by default
meta/classes/insane.bbclass | 2 +- .../busybox-install-ping6-to-usr-bin.patch | 25 +++++++++++++++++ meta/recipes-core/busybox/busybox_1.21.1.bb | 1 + .../initscripts/initscripts-1.0/setup-commands.sh | 24 ++++++++++++++++ meta/recipes-core/initscripts/initscripts_1.0.bb | 3 ++ meta/recipes-core/udev/udev.inc | 16 ++++++++--- meta/recipes-core/zlib/zlib_1.2.8.bb | 2 +- meta/recipes-extended/iputils/iputils_s20121221.bb | 29 ++++++++++++-------- meta/recipes-kernel/kmod/kmod_git.bb | 7 +++-- 9 files changed, 89 insertions(+), 20 deletions(-) create mode 100644 meta/recipes-core/busybox/busybox-1.21.1/busybox-install-ping6-to-usr-bin.patch create mode 100644 meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh
Signature · 1 line
1.7.9.5
PATCH 1/8 udev: fix dependency and location of udevadm
From: Chen Qi <[email protected]>
As udev's init script needs udevadm, we should make udev RDEPEND on udev-utils instead of just recommends it.
Also, move udevadm from /usr/bin to /bin. The purpose is to make udev start correctly even if /usr is missing.
--- meta/recipes-core/udev/udev.inc | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
meta/recipes-core/udev/udev.inc +7 -3
@@ -55,7 +55,8 @@ INITSCRIPT_NAME_udev-cache = "udev-cache" INITSCRIPT_PARAMS_udev-cache = "start 36 S ." FILES_${PN} += "${libexecdir} ${libdir}/ConsoleKit ${nonarch_base_libdir}/udev" -RRECOMMENDS_${PN} += "udev-utils udev-cache" +RRECOMMENDS_${PN} += "udev-cache" +RDEPENDS_${PN} += "udev-utils" FILES_${PN}-dbg += "${libexecdir}/.debug" FILES_${PN}-dbg += "${base_libdir}/udev/.debug/"
@@ -72,7 +73,7 @@ FILES_libgudev-dev = "${includedir}/gudev* ${libdir}/libgudev*.so ${libdir}/libg ${libdir}/libgudev*.a ${libdir}/pkgconfig/gudev*.pc" FILES_udev-cache = "${sysconfdir}/init.d/udev-cache ${sysconfdir}/default/udev-cache" -FILES_udev-utils = "${bindir}/udevadm" +FILES_udev-utils = "${base_bindir}/udevadm" do_install_append () { install -d ${D}${sysconfdir}/init.d
@@ -87,9 +88,12 @@ do_install_append () { touch ${D}${sysconfdir}/udev/cache.data install -d ${D}${sysconfdir}/udev/rules.d/ - install -m 0644 ${WORKDIR}/local.rules ${D}${sysconfdir}/udev/rules.d/local.rules + install -d ${D}${base_bindir} + mv ${D}${bindir}/udevadm ${D}${base_bindir}/udevadm + rmdir ${D}${bindir} + # hid2hci has moved to bluez4. removed in udev as of version 169 rm -f ${D}${base_libdir}/udev/hid2hci
Signature · 1 line
1.7.9.5
PATCH 2/8 initscripts: add setup-commands.sh
In reply to [email protected]From: Chen Qi <[email protected]>
The purpose of this script is to set up commands that busybox provides if /usr is missing. In this way, the system can still boot into single user mode for recovery or repair if the /usr partition is broken.
For normal use case, that is, /usr is on the same partition as /, or /usr is correctly mounted at boot time, this script has no real effect.
--- .../initscripts/initscripts-1.0/setup-commands.sh | 24 ++++++++++++++++++++ meta/recipes-core/initscripts/initscripts_1.0.bb | 3 +++ 2 files changed, 27 insertions(+) create mode 100644 meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh
meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh new file +24 -0
@@ -0,0 +1,24 @@ +#!/bin/sh + +# set up necessary links to busybox if fundamental command from /usr +# are not available. + +[ -e /etc/default/rcS ] && . /etc/default/rcS + +for dir in /usr/bin /usr/sbin; do + if [ ! -e $dir ]; then + if [ "$VERBOSE" != "no" ]; then + echo "WARN: $dir missing, setting up links to busybox" + fi + mkdir -p $dir + for suffix in ".nosuid" ".suid" ""; do + if [ ! -e /etc/busybox.links${suffix} ]; then + continue + fi + usr_commands=`grep "$dir" /etc/busybox.links${suffix}` + for command in $usr_commands; do + ln -sf /bin/busybox${suffix} $command + done + done + fi +done
meta/recipes-core/initscripts/initscripts_1.0.bb +3 -0
@@ -33,6 +33,7 @@ SRC_URI = "file://functions \ file://GPLv2.patch \ file://dmesg.sh \ file://logrotate-dmesg.conf \ + file://setup-commands.sh \ " SRC_URI_append_arm = " file://alignment.sh"
@@ -93,6 +94,7 @@ do_install () { install -m 0644 ${WORKDIR}/volatiles ${D}${sysconfdir}/default/volatiles/00_core install -m 0755 ${WORKDIR}/dmesg.sh ${D}${sysconfdir}/init.d install -m 0644 ${WORKDIR}/logrotate-dmesg.conf ${D}${sysconfdir}/ + install -m 0755 ${WORKDIR}/setup-commands.sh ${D}${sysconfdir}/init.d if [ "${TARGET_ARCH}" = "arm" ]; then install -m 0755 ${WORKDIR}/alignment.sh ${D}${sysconfdir}/init.d
@@ -123,6 +125,7 @@ do_install () { update-rc.d -r ${D} populate-volatile.sh start 37 S . update-rc.d -r ${D} read-only-rootfs-hook.sh start 29 S . update-rc.d -r ${D} devpts.sh start 38 S . + update-rc.d -r ${D} setup-commands.sh start 03 S . if [ "${TARGET_ARCH}" = "arm" ]; then update-rc.d -r ${D} alignment.sh start 06 S . fi
Signature · 1 line
1.7.9.5
PATCH 3/8 zlib: install into base_libdir
In reply to [email protected]From: Chen Qi <[email protected]>
Install zlib into base_libdir as kmod needs it. The purpose is to make system boot into single user mode even if /usr is missing.
--- meta/recipes-core/zlib/zlib_1.2.8.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-core/zlib/zlib_1.2.8.bb +1 -1
@@ -15,7 +15,7 @@ SRC_URI[md5sum] = "28f1205d8dd2001f26fec1e8c2cebe37" SRC_URI[sha256sum] = "831df043236df8e9a7667b9e3bb37e1fcb1220a0f163b6de2626774b9590d057" do_configure (){ - ./configure --prefix=${prefix} --shared --libdir=${libdir} + ./configure --prefix=${base_prefix} --shared --libdir=${base_libdir} } do_compile (){
Signature · 1 line
1.7.9.5
PATCH 4/8 kmod: install libkmod into base_libdir
In reply to [email protected]From: Chen Qi <[email protected]>
Install libkmod into base_libdir because udevd requires it. The purpose is to make system boot into single user mode even if /usr is missing.
This patch fixes the following QA issue about unsafe references in binaries.
/lib/udevd requires libkmod.so.2 (=> /usr/lib/libkmod.so.2)
--- meta/recipes-kernel/kmod/kmod_git.bb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
meta/recipes-kernel/kmod/kmod_git.bb +4 -3
@@ -12,9 +12,7 @@ RREPLACES_${PN} += "module-init-tools-insmod-static module-init-tools-depmod mod RCONFLICTS_libkmod2 += "module-init-tools-insmod-static module-init-tools-depmod module-init-tools" # autotools set prefix to /usr, however we want them in /bin and /sbin -bindir = "${base_bindir}" -sbindir = "${base_sbindir}" -# libdir = "${base_libdir}" +EXTRA_OECONF += "--bindir=${base_bindir} --sbindir=${base_sbindir} --libdir=${base_libdir}" do_install_append () { install -dm755 ${D}${base_bindir}
@@ -33,6 +31,9 @@ do_install_append () { # install depmod.d file for search/ dir install -Dm644 "${WORKDIR}/depmod-search.conf" "${D}${base_libdir}/depmod.d/search.conf" + # mv the .pc file to ${libdir} + install -d ${D}${libdir} + mv ${D}${base_libdir}/pkgconfig ${D}${libdir} } do_compile_prepend() {
Signature · 1 line
1.7.9.5
PATCH 5/8 udev: fix unsafe reference by installing libgudev in libdir
In reply to [email protected]From: Chen Qi <[email protected]>
This patch fixes the following QA issue about unsafe reference in binaries.
/lib/libgudev-1.0.so.0.1.1 requires /usr/lib/libgobject-2.0.so.0.
As libgudev is a GObject wrapper for libudev, and the purpose of it is to make the development of GObject based applications easier, there's no reason why it should remain in /lib. So this patches moves the libgudev to /usr/lib.
This patch also removes a duplicate line in the EXTRA_OECONF.
--- meta/recipes-core/udev/udev.inc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/recipes-core/udev/udev.inc +5 -1
@@ -39,7 +39,6 @@ EXTRA_OECONF = "--disable-introspection \ ac_cv_file__usr_share_misc_pci_ids=yes \ --sbindir=${base_sbindir} \ --libexecdir=${nonarch_base_libdir} \ - --with-rootlibdir=${base_libdir} \ --with-rootprefix= \ --without-systemdsystemunitdir \ "
@@ -97,6 +96,11 @@ do_install_append () { # hid2hci has moved to bluez4. removed in udev as of version 169 rm -f ${D}${base_libdir}/udev/hid2hci + # install libgudev.so.* into libdir. + so_img_name=`basename $(readlink ${D}${libdir}/libgudev-1.0.so)` + mv ${D}${base_libdir}/libgudev-1.0.so.* ${D}${libdir} + ln -sf $so_img_name ${D}${libdir}/libgudev-1.0.so + echo 'udev_run="/var/run/udev"' >> ${D}${sysconfdir}/udev/udev.conf }
Signature · 1 line
1.7.9.5
PATCH 6/8 insane.bbclass: make the checking stricter for unsafe references in scripts
In reply to [email protected]From: Chen Qi <[email protected]>
Previously, the checking for unsafe references is not strict enough. It only checks whether '/usr/' is in the script. As a result, any script containing statements like below will match this check.
PATH="/bin:/sbin:/usr/bin:/usr/sbin"
However, as we can see, this is actually not an unsafe reference. What we really want to check is something like '/usr/bin/tail', so we should make the checking stricter.
This patch solves the QA warning in gzip and nfs-utils.
--- meta/classes/insane.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/insane.bbclass +1 -1
@@ -367,7 +367,7 @@ def package_qa_check_unsafe_references_in_scripts(path, name, d, elf, messages): if bool(statinfo.st_mode & stat.S_IXUSR): # grep shell scripts for possible references to /exec_prefix/ exec_prefix = d.getVar('exec_prefix', True) - statement = "grep -e '%s/' %s > /dev/null" % (exec_prefix, path) + statement = "grep -e '%s/[^ :]\{1,\}/[^ :]\{1,\}' %s > /dev/null" % (exec_prefix, path) if subprocess.call(statement, shell=True) == 0: error_msg = pn + ": Found a reference to %s/ in %s" % (exec_prefix, path) package_qa_handle_error("unsafe-references-in-scripts", error_msg, d)
Signature · 1 line
1.7.9.5
PATCH 7/8 iputils: fix program location and QA warning
In reply to [email protected]From: Chen Qi <[email protected]>
According to FHS, only ping' is required to be under /bin. So this patch move other commands like ping6' and 'traceroute6' to /usr/bin.
Doing so will fix QA warnings of unsafe reference of binaries in ping6.
/bin/ping6.iputils requires /usr/lib/libgnutls.so.26 /bin/ping6.iputils requires /usr/lib/libgcrypt.so.11 /bin/ping6.iputils requires /usr/lib/libgpg-error.so.0 /bin/ping6.iputils requires /usr/lib/libtasn1.so.6
--- meta/recipes-extended/iputils/iputils_s20121221.bb | 29 ++++++++++++-------- 1 file changed, 18 insertions(+), 11 deletions(-)
meta/recipes-extended/iputils/iputils_s20121221.bb +18 -11
@@ -29,15 +29,16 @@ do_compile () { } do_install () { - install -m 0755 -d ${D}${base_bindir} ${D}${mandir}/man8 + install -m 0755 -d ${D}${base_bindir} ${D}${mandir}/man8 ${D}${bindir} ${D}${sbindir} # SUID root programs install -m 4555 ping ${D}${base_bindir}/ping - install -m 4555 ping6 ${D}${base_bindir}/ping6 - install -m 4555 traceroute6 ${D}${base_bindir}/ + install -m 4555 ping6 ${D}${bindir}/ping6 + install -m 4555 traceroute6 ${D}${bindir}/ # Other programgs - for i in arping tracepath tracepath6; do - install -m 0755 $i ${D}${base_bindir}/ + for i in tracepath tracepath6; do + install -m 0755 $i ${D}${bindir}/ done + install -m 0755 arping ${D}${sbindir}/ # Manual pages for things we build packages for for i in tracepath.8 traceroute6.8 ping.8 arping.8; do install -m 0644 doc/$i ${D}${mandir}/man8/ || true
@@ -52,7 +53,13 @@ ALTERNATIVE_${PN}-ping = "ping" ALTERNATIVE_LINK_NAME[ping] = "${base_bindir}/ping" ALTERNATIVE_${PN}-ping6 = "ping6" -ALTERNATIVE_LINK_NAME[ping6] = "${base_bindir}/ping6" +ALTERNATIVE_LINK_NAME[ping6] = "${bindir}/ping6" + +ALTERNATIVE_${PN}-traceroute6 = "traceroute6" +ALTERNATIVE_LINK_NAME[traceroute6] = "${bindir}/traceroute6" + +ALTERNATIVE_${PN}-arping = "arping" +ALTERNATIVE_LINK_NAME[arping] = "${sbindir}/arping" PACKAGES += "${PN}-ping ${PN}-ping6 ${PN}-arping ${PN}-tracepath ${PN}-tracepath6 ${PN}-traceroute6"
@@ -61,9 +68,9 @@ RDEPENDS_${PN} += "${PN}-ping ${PN}-ping6 ${PN}-arping ${PN}-tracepath ${PN}-tra FILES_${PN} = "" FILES_${PN}-ping = "${base_bindir}/ping.${BPN}" -FILES_${PN}-ping6 = "${base_bindir}/ping6.${BPN}" -FILES_${PN}-arping = "${base_bindir}/arping" -FILES_${PN}-tracepath = "${base_bindir}/tracepath" -FILES_${PN}-tracepath6 = "${base_bindir}/tracepath6" -FILES_${PN}-traceroute6 = "${base_bindir}/traceroute6" +FILES_${PN}-ping6 = "${bindir}/ping6.${BPN}" +FILES_${PN}-arping = "${sbindir}/arping.${BPN}" +FILES_${PN}-tracepath = "${bindir}/tracepath" +FILES_${PN}-tracepath6 = "${bindir}/tracepath6" +FILES_${PN}-traceroute6 = "${bindir}/traceroute6.${BPN}" FILES_${PN}-doc = "${mandir}/man8"
Signature · 1 line
1.7.9.5
PATCH 8/8 busybox: install ping6 into bindir by default
In reply to [email protected]From: Chen Qi <[email protected]>
Install `ping6' provided by busybox into /usr/bin to sync with the iputils package.
--- .../busybox-install-ping6-to-usr-bin.patch | 25 ++++++++++++++++++++ meta/recipes-core/busybox/busybox_1.21.1.bb | 1 + 2 files changed, 26 insertions(+) create mode 100644 meta/recipes-core/busybox/busybox-1.21.1/busybox-install-ping6-to-usr-bin.patch
meta/recipes-core/busybox/busybox-1.21.1/busybox-install-ping6-to-usr-bin.patch new file +25 -0
@@ -0,0 +1,25 @@ +Subject: [PATCH] busybox: install ping6 to /usr/bin by default + +Upstream-Status: Pending + +Signed-off-by: Chen Qi <[email protected]> +--- + networking/ping.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/networking/ping.c b/networking/ping.c +index 5e4771f..471c347 100644 +--- a/networking/ping.c ++++ b/networking/ping.c +@@ -70,7 +70,7 @@ + + /* Needs socket(AF_INET, SOCK_RAW, IPPROTO_ICMP), therefore BB_SUID_MAYBE: */ + //applet:IF_PING(APPLET(ping, BB_DIR_BIN, BB_SUID_MAYBE)) +-//applet:IF_PING6(APPLET(ping6, BB_DIR_BIN, BB_SUID_MAYBE)) ++//applet:IF_PING6(APPLET(ping6, BB_DIR_USR_BIN, BB_SUID_MAYBE)) + + //kbuild:lib-$(CONFIG_PING) += ping.o + //kbuild:lib-$(CONFIG_PING6) += ping.o +-- +1.7.9.5 +
meta/recipes-core/busybox/busybox_1.21.1.bb +1 -0
@@ -34,6 +34,7 @@ SRC_URI = "http://www.busybox.net/downloads/busybox-${PV}.tar.bz2;name=tarball \ file://login-utilities.cfg \ file://busybox-list-suid-and-non-suid-app-configs.patch \ file://busybox-sed-fix-sed-clusternewline-testcase.patch \ + file://busybox-install-ping6-to-usr-bin.patch \ " SRC_URI[tarball.md5sum] = "795394f83903b5eec6567d51eebb417e"
Signature · 1 line
1.7.9.5
Re: [PATCH 1/8] udev: fix dependency and location of udevadm
In reply to [email protected]meta/recipes-core/udev/udev.inc do_install_append () { · line 95
Quoted 3 lines · On Sat, 2013-11-09 at 13:28 +0800, [email protected] wrote:
meta/recipes-core/udev/udev.inc +3
-0+ install -d ${D}${base_bindir} + mv ${D}${bindir}/udevadm ${D}${base_bindir}/udevadm + rmdir ${D}${bindir}
This will fail if ${bindir} and ${base_bindir} are the same.
p.
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to [email protected]Quoted 4 lines · On Sat, 2013-11-09 at 13:28 +0800, [email protected] wrote:
meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh new file +4
-0+for dir in /usr/bin /usr/sbin; do + if [ ! -e $dir ]; then + if [ "$VERBOSE" != "no" ]; then + echo "WARN: $dir missing, setting up links to busybox"
meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh line 21
Quoted 10 lines
meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh new file +10
-0+ fi + mkdir -p $dir + for suffix in ".nosuid" ".suid" ""; do + if [ ! -e /etc/busybox.links${suffix} ]; then + continue + fi + usr_commands=`grep "$dir" /etc/busybox.links${suffix}` + for command in $usr_commands; do + ln -sf /bin/busybox${suffix} $command + done
This seems slightly bogus for a number of reasons:
1. initscript doesn't obviously rdepend on busybox so it's not obvious that the latter will always be available;
2. it should probably be using ${base_bindir} and ${bindir} rather than hardcoding absolute paths.
3. the whole idea of creating a shadow "/usr/bin" underneath what's meant to be a mountpoint seems rather dubious to me.
4. this seems like distro policy and not something that really belongs in oe-core at all. For systems where ${bindir} and ${base_bindir} are on the same filesystem (or even are the same directory) this script will just make bootup slower without achieving anything useful.
p.
Re: [PATCH 1/8] udev: fix dependency and location of udevadm
In reply to Phil BlundellQuoted 11 lines · Phil Blundell
meta/recipes-core/udev/udev.inc do_install_append () { · line 95
Quoted 3 lines · On Sat, 2013-11-09 at 13:28 +0800, [email protected] wrote:
meta/recipes-core/udev/udev.inc +3
-0+ install -d ${D}${base_bindir} + mv ${D}${bindir}/udevadm ${D}${base_bindir}/udevadm + rmdir ${D}${bindir}This will fail if ${bindir} and ${base_bindir} are the same.
p.
In udev recipe, they are not defined as the same one.
And moving something from bindir to base_bindir doesn't seem uncommon in OE, you can grep the project using the following command.
grep -Ri 'mv.*bindir.*base_bindir' meta/*
Best Regards,
Chen Qi
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to Phil BlundellQuoted 15 lines · Phil Blundell
Quoted 4 lines · On Sat, 2013-11-09 at 13:28 +0800, [email protected] wrote:
meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh new file +4
-0+for dir in /usr/bin /usr/sbin; do + if [ ! -e $dir ]; then + if [ "$VERBOSE" != "no" ]; then + echo "WARN: $dir missing, setting up links to busybox"meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh line 21
Quoted 10 lines
meta/recipes-core/initscripts/initscripts-1.0/setup-commands.sh new file +10
-0+ fi + mkdir -p $dir + for suffix in ".nosuid" ".suid" ""; do + if [ ! -e /etc/busybox.links${suffix} ]; then + continue + fi + usr_commands=`grep "$dir" /etc/busybox.links${suffix}` + for command in $usr_commands; do + ln -sf /bin/busybox${suffix} $command + done
Hi Phil,
This script is an effort to try to make system boot up even if /usr is not there. It relies on busybox because in most cases, busybox is there in an OE based system. And please see comments below.
Quoted 4 lines
This seems slightly bogus for a number of reasons:
1. initscript doesn't obviously rdepend on busybox so it's not obvious that the latter will always be available;
Yes. Initscript doesn't rdepend on busybox. But note it also doesn't rdepend on sed or awk or grep. So I think it's reasonable to assume the presence of busybox.
Quoted 2 lines
2. it should probably be using ${base_bindir} and ${bindir} rather than hardcoding absolute paths.
In init scripts, we usually "hardcode" things, because these scripts are destined to run on target. In recipes we try not to hardcode things because the recipe may need to extend to "native" or "nativesdk".
Quoted 2 lines
3. the whole idea of creating a shadow "
/usr/bin" underneath what's meant to be a mountpoint seems rather dubious to me.
Agree.
The problem here is that the init scripts under /etc/rcS.d/ need to execute commands like awk, dirname, and readlink which are from /usr.
As it's not appropriate to move these commands into /bin, basically there are only two options I can see here. One is to modify these scripts to use only commands from /bin or /sbin; the other is to make use of busybox, as busybox is located under /bin as it provides these commands. I chose to the latter one because I thought that solution would have the less impact.
What do you think? Do we need to modify the init scripts? Or any other solution?
Quoted 7 lines
4. this seems like distro policy and not something that really belongs in oe-core at all. For systems where ${bindir} and ${base_bindir} are on the same filesystem (or even are the same directory) this script will just make bootup slower without achieving anything useful.
p.
If /usr and / are on the same file system, this script has no real effect because /usr will always be there. So I think it will not take much time at boot.
If ${base_bindir} and ${bindir} are the same, that means that there's no /usr. In this case, there should no /usr/xxx entries in /etc/busybox.links, so this script should also function correctly and it will not take much time at boot.
(I just configured ${bindir} and ${base_bindir} to be the same and performed a build, it failed. I'm not sure whether it's valid to make such configurations in OE.)
Best Regards,
Chen Qi
Quoted 2 lines
Re: [PATCH 1/8] udev: fix dependency and location of udevadm
In reply to ChenQiQuoted 7 lines · ChenQi
Quoted 6 lines · Phil Blundell
meta/recipes-core/udev/udev.inc do_install_append () { · line 95
Quoted 3 lines · On Sat, 2013-11-09 at 13:28 +0800, [email protected] wrote:
meta/recipes-core/udev/udev.inc +3
-0+ install -d ${D}${base_bindir} + mv ${D}${bindir}/udevadm ${D}${base_bindir}/udevadm + rmdir ${D}${bindir}This will fail if ${bindir} and ${base_bindir} are the same.
Quoted 7 lines
Quoted 5 lines
p.
In udev recipe, they are not defined as the same one.
Those variables are part of the distro configuration. Individual recipes don't, in general, set them.
Quoted 4 lines
And moving something from bindir to
base_bindirdoesn't seem uncommon in OE, you can grep the project using the following command.grep -Ri 'mv.*bindir.*base_bindir' meta/*
A better command to use would be:
grep -C 4 -Ri 'mv.*bindir.*base_bindir' meta/*
which reveals that most of these "mv" commands are enclosed in a conditional that checks whether the two directories are indeed different before trying to move them.
It's true that a few of the things in recipes-extended do appear to be broken. cpio, for example, was broken by 6dee3050a4a0c4f3cc9fec23a0bc02155d680863; gzip was broken by e0626a0270fb0f4ff128e761c13d44162723434c; mktemp was broken by 4807d938023ce06f2924c8a0503c32d083be23b5. All of these three patches seem to be well-intentioned attempts to improve the handling of update-alternatives and I guess those recipes are obscure enough that nobody has noticed before now that there is anything wrong with them.
p.
Quoted 3 lines · On 9 November 2013 05:28, <[email protected]> wrote:
This solution is based on the following two principles.
1. With /usr on a seperate partition, system should still boot without any error.
2. Without /usr, system should be able to boot into single user mode with error.
Presumably this is with sysvinit. Have you tested systemd with these principles?
Ross
Quoted 10 lines · Ross
Quoted 3 lines · On 9 November 2013 05:28, <[email protected]> wrote:
This solution is based on the following two principles.
1. With /usr on a seperate partition, system should still boot without any error.
2. Without /usr, system should be able to boot into single user mode with error.Presumably this is with sysvinit. Have you tested systemd with these principles?
Ross
Not yet ...
I can do such testing in a day or two.
I'll then file a bug and make patches if needed.
Best Regards,
Chen Qi
Re: [PATCH 1/8] udev: fix dependency and location of udevadm
In reply to Phil BlundellQuoted 17 lines · Phil Blundell
Quoted 7 lines · ChenQi
Quoted 6 lines · Phil Blundell
meta/recipes-core/udev/udev.inc do_install_append () { · line 95
Quoted 3 lines · On Sat, 2013-11-09 at 13:28 +0800, [email protected] wrote:
meta/recipes-core/udev/udev.inc +3
-0+ install -d ${D}${base_bindir} + mv ${D}${bindir}/udevadm ${D}${base_bindir}/udevadm + rmdir ${D}${bindir}This will fail if ${bindir} and ${base_bindir} are the same.
Quoted 6 lines
Quoted 5 lines
p.
In udev recipe, they are not defined as the same one.
Those variables are part of the distro configuration. Individual recipes don't, in general, set them.
Quoted 7 lines
Quoted 4 lines
And moving something from bindir to
base_bindirdoesn't seem uncommon in OE, you can grep the project using the following command.grep -Ri 'mv.*bindir.*base_bindir' meta/*
A better command to use would be:
grep -C 4 -Ri 'mv.*bindir.*base_bindir' meta/*
Thanks for pointing it out :)
Currently I'm not sure whether we support configuring ${bindir} to equal to ${base_bindir}, but maybe we will support this such configuration in the future. So I'll send out V2 of this patch.
Thanks,
Chen Qi
Quoted 18 lines
which reveals that most of these "mv" commands are enclosed in a conditional that checks whether the two directories are indeed different before trying to move them.
It's true that a few of the things in recipes-extended do appear to be broken. cpio, for example, was broken by
6dee3050a4a0c4f3cc9fec23a0bc02155d680863; gzip was broken bye0626a0270fb0f4ff128e761c13d44162723434c; mktemp was broken by4807d938023ce06f2924c8a0503c32d083be23b5. All of these three patches seem to be well-intentioned attempts to improve the handling of update-alternatives and I guess those recipes are obscure enough that nobody has noticed before now that there is anything wrong with them.p.
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to ChenQiQuoted 7 lines · ChenQi
Quoted 2 lines · Phil Blundell
1. initscript doesn't obviously rdepend on busybox so it's not obvious that the latter will always be available;
Yes. Initscript doesn't rdepend on busybox. But note it also doesn't rdepend on sed or awk or grep.
So I think it's reasonable to assume the presence of busybox.
I think one could argue that it's also a bug that it doesn't rdepend on the three things you mentioned (and indeed on /bin/sh itself, for non-rpm systems). However, the usage of grep and sed in particular, and perhaps to a slightly lesser extent awk, is so deeply ingrained into so much software that I think it's probably fair to say those utilities will almost always be present.
By contrast, it is perfectly feasible to build a system which doesn't use busybox (by using the full GNU implementations of everything that busybox provides) and indeed I think there might even be a task package in oe-core which does exactly that. So it seems entirely possible that /bin/busybox might not be installed unless you have a dependency on it.
Quoted 7 lines
Quoted 2 lines
2. it should probably be using ${base_bindir} and ${bindir} rather than hardcoding absolute paths.
In init scripts, we usually "hardcode" things, because these scripts are destined to run on target.
In recipes we try not to hardcode things because the recipe may need to extend to "native" or "nativesdk".
I'm not quite sure I follow the logic here. Even if the script is intended to run on the target it still ought to be respecting the values of ${bindir}, ${sysconfdir} and such like.
Quoted 16 lines
Quoted 2 lines
3. the whole idea of creating a shadow "
/usr/bin" underneath what's meant to be a mountpoint seems rather dubious to me.Agree.
The problem here is that the init scripts under
/etc/rcS.d/need to execute commands like awk, dirname, and readlink which are from /usr.As it's not appropriate to move these commands into /bin, basically there are only two options I can see here. One is to modify these scripts to use only commands from /bin or /sbin; the other is to make use of busybox, as busybox is located under /bin as it provides these commands.
I chose to the latter one because I thought that solution would have the less impact.
Quoted 2 lines
What do you think? Do we need to modify the init scripts? Or any other solution?
I thought that last time this topic came up on the mailing list, the eventual conclusion was that Wind River (being more-or-less the only people who seemed to feel strongly that supporting / and /usr on different partitions was important) were going to come up with some sort of overall strategy for solving the whole problem rather than just fixing up individual bits in a piecemeal fashion. I think that strategy would need to include a policy for which utilities initscripts can legitimately expect to be available before /usr is mounted, and if this set is different from what we have today then it would need to include a plan for sorting that out.
To answer the particular question at hand it seems that someone would need to do some analysis of things like:
- how many scripts actually need those commands
- how hard would it be to change them to not need them
- what would be the impact of moving them into ${base_bindir} (for example, would this cause a whole load of libraries to get dragged into ${base_libdir} as well?)
Offhand I don't know the answer to any of those things.
Quoted 8 lines
Quoted 4 lines
4. this seems like distro policy and not something that really belongs in oe-core at all. For systems where ${bindir} and ${base_bindir} are on the same filesystem (or even are the same directory) this script will just make bootup slower without achieving anything useful.
If /usr and / are on the same file system, this script has no real effect because /usr will always be there. So I think it will not take much time at boot.
Just spawning a new copy of the shell and reading the script does take a finite (and measureable) time. If you can determine statically at build time that the script isn't going to do anything useful then I don't think it's appropriate to install it.
Quoted 8 lines
If ${base_bindir} and ${bindir} are the same, that means that there's no /usr. In this case, there should no
/usr/xxxentries
in/etc/busybox.links, so this script should also function correctly and it will not take much time at boot.(I just configured ${bindir} and ${base_bindir} to be the same and performed a build, it failed. I'm not sure whether it's valid to make such configurations in OE.)
It is a valid configuration, and this is the way that most of the images I build are configured. It probably is true that there are still some number of recipes in oe-core that don't support this properly, but I don't want to see that situation get any worse.
p.
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to ChenQiQuoted 2 lines · ChenQi
The problem here is that the init scripts under
/etc/rcS.d/need to execute commands like awk, dirname, and readlink which are from /usr.
Yes. So why do you really need to support split /usr, and why isn't an initramfs a suitable alternative? People seem to insist that split /usr is trivial but these patches are clearly showing it's not.
Ross
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to Phil BlundellHi Phil,
First of all, thank you for your careful review and explanation.
To conclude, you suggest modifying the init scripts and moving commands around if needed, right?
And please see comments inline.
Quoted 14 lines · Phil Blundell
Quoted 6 lines · ChenQi
Quoted 2 lines · Phil Blundell
1. initscript doesn't obviously rdepend on busybox so it's not obvious that the latter will always be available;
Yes. Initscript doesn't rdepend on busybox. But note it also doesn't rdepend on sed or awk or grep.
So I think it's reasonable to assume the presence of busybox.I think one could argue that it's also a bug that it doesn't rdepend on the three things you mentioned (and indeed on
/bin/shitself, for non-rpm systems). However, the usage of grep and sed in particular, and perhaps to a slightly lesser extent awk, is so deeply ingrained into so much software that I think it's probably fair to say those utilities will almost always be present.
Quoted 7 lines
By contrast, it is perfectly feasible to build a system which doesn't use busybox (by using the full GNU implementations of everything that busybox provides) and indeed I think there might even be a task package in oe-core which does exactly that. So it seems entirely possible that
/bin/busyboxmight not be installed unless you have a dependency on it.
Agree. I'm gonna drop this patch and try to come up with another solution :)
Quoted 27 lines
Quoted 6 lines
Quoted 2 lines
2. it should probably be using ${base_bindir} and ${bindir} rather than hardcoding absolute paths.
In init scripts, we usually "hardcode" things, because these scripts are destined to run on target.
In recipes we try not to hardcode things because the recipe may need to extend to "native" or "nativesdk".I'm not quite sure I follow the logic here. Even if the script is intended to run on the target it still ought to be respecting the values of ${bindir}, ${sysconfdir} and such like.
Quoted 15 lines
Quoted 2 lines
3. the whole idea of creating a shadow "
/usr/bin" underneath what's meant to be a mountpoint seems rather dubious to me.Agree.
The problem here is that the init scripts under
/etc/rcS.d/need to execute commands like awk, dirname, and readlink which are from /usr.As it's not appropriate to move these commands into /bin, basically there are only two options I can see here. One is to modify these scripts to use only commands from /bin or /sbin; the other is to make use of busybox, as busybox is located under /bin as it provides these commands.
I chose to the latter one because I thought that solution would have the less impact.Quoted 2 lines
What do you think? Do we need to modify the init scripts? Or any other solution?
Quoted 10 lines
I thought that last time this topic came up on the mailing list, the eventual conclusion was that Wind River (being more-or-less the only people who seemed to feel strongly that supporting / and /usr on different partitions was important) were going to come up with some sort of overall strategy for solving the whole problem rather than just fixing up individual bits in a piecemeal fashion. I think that strategy would need to include a policy for which utilities initscripts can legitimately expect to be available before /usr is mounted, and if this set is different from what we have today then it would need to include a plan for sorting that out.
Basically we don't have any problem if /usr is going to be mounted, because the mountall.sh takes place really early.
What I want to achieve here is that we can boot into single user mode for recovery or repair if /usr is not available (maybe just because the drive's broken).
Quoted 6 lines
To answer the particular question at hand it seems that someone would need to do some analysis of things like:
- how many scripts actually need those commands
- how hard would it be to change them to not need them
Only init scripts that have links under /etc/rcS.d/ need to be examined, so that wouldn't be a lot of work.
I'm just not sure whether we should make such a restriction of which commands could be used and which could not. After all, commands like readlink', dirname' and `awk' are also very commonly used commands.
Quoted 3 lines
- what would be the impact of moving them into ${base_bindir} (for example, would this cause a whole load of libraries to get dragged into ${base_libdir} as well?)
If I can easily modify the init scripts to make things work, I don't want to move things around. But I guess statements using awk are hard to be replaced unless I use a block of codes ....
Quoted 22 lines
Offhand I don't know the answer to any of those things.
Quoted 7 lines
Quoted 4 lines
4. this seems like distro policy and not something that really belongs in oe-core at all. For systems where ${bindir} and ${base_bindir} are on the same filesystem (or even are the same directory) this script will just make bootup slower without achieving anything useful.
If /usr and / are on the same file system, this script has no real effect because /usr will always be there. So I think it will not take much time at boot.
Just spawning a new copy of the shell and reading the script does take a finite (and measureable) time. If you can determine statically at build time that the script isn't going to do anything useful then I don't think it's appropriate to install it.
Quoted 8 lines
If ${base_bindir} and ${bindir} are the same, that means that there's no /usr. In this case, there should no
/usr/xxxentries
in/etc/busybox.links, so this script should also function correctly and it will not take much time at boot.(I just configured ${bindir} and ${base_bindir} to be the same and performed a build, it failed. I'm not sure whether it's valid to make such configurations in OE.)
Quoted 10 lines
It is a valid configuration, and this is the way that most of the images I build are configured. It probably is true that there are still some number of recipes in oe-core that don't support this properly, but I don't want to see that situation get any worse.
p.
Could you please share your configuration so that when I make V2 I can test the case of ${bindir} and ${base_bindir} being the same. (Currently I really can't build out an image with such configuration ... )
Best Regards,
Chen Qi
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to Burton, RossQuoted 10 lines · Ross
Quoted 2 lines · ChenQi
The problem here is that the init scripts under
/etc/rcS.d/need to execute commands like awk, dirname, and readlink which are from /usr.Yes. So why do you really need to support split /usr, and why isn't an initramfs a suitable alternative? People seem to insist that split /usr is trivial but these patches are clearly showing it's not.
Ross
If /usr is going to be mounted, we don't have any problem here because mountall.sh takes place really early. In this situation, splitting /usr is trivial.
What I want to do here is to make system boot up even if /usr is totally missing. Do you think it's suitable for us to support such use case?
Best Regards,
Chen Qi
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to ChenQiQuoted 5 lines · ChenQi
Hi Phil,
First of all, thank you for your careful review and explanation. To conclude, you suggest modifying the init scripts and moving commands around if needed, right?
Well, I'm not sure that I'm necessarily suggesting any particular change. What I'm saying is that if we're going to try to solve the split-/usr problem in oe-core (as opposed to Wind River just doing it in their internal distro layers) then:
1. It needs to be done in a way that doesn't have any negative impact for people who don't have a split /usr (or indeed who don't have /usr at all)
2. The changes need to be presented as a coherent package with the reasoning clearly explained.
3. We need to be clear that it's solving the "whole problem" (whatever that might be) rather than just some part of it.
Quoted 3 lines
Could you please share your configuration so that when I make V2 I can test the case of ${bindir} and ${base_bindir} being the same. (Currently I really can't build out an image with such configuration ... )
I think it should basically just be a case of setting:
prefix = ""exec_prefix = ""prefix_native = ""
in your distro configuration.
p.
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to Phil BlundellQuoted 2 lines · Phil Blundell
Quoted 1 line · ChenQi
On 11/10/2013 07:00 AM, Phil Blundell wrote:
...
Quoted 10 lines
I thought that last time this topic came up on the mailing list, the eventual conclusion was that Wind River (being more-or-less the only people who seemed to feel strongly that supporting / and /usr on different partitions was important) were going to come up with some sort of overall strategy for solving the whole problem rather than just fixing up individual bits in a piecemeal fashion. I think that strategy would need to include a policy for which utilities initscripts can legitimately expect to be available before /usr is mounted, and if this set is different from what we have today then it would need to include a plan for sorting that out.
There is a bug open for the Yocto Project 1.6 to explicitly do this. Come up with an overall strategy for these kinds of things, as well as a test plan to verify that whatever we end up doing works properly long term.
As you said, busybox is not required on a system -- just something that has a reasonable set of software packages. Also a lot of this stuff is simply limited to 'early boot'. At some point we do need to define 'early boot'. (Generally I define it as until /usr would normally be mounted.)
And unfortunately you will see patches in piecemeal at this point. Until such a strategy is generated, everything is being done reactively. We see a QA error/warning and someone is assigned to solve it. Yocto Project Compliance (and our own internal) guidelines then indicate since we've patched something it has to go out to the community.
Quoted 8 lines
To answer the particular question at hand it seems that someone would need to do some analysis of things like:
- how many scripts actually need those commands
- how hard would it be to change them to not need them
- what would be the impact of moving them into ${base_bindir} (for example, would this cause a whole load of libraries to get dragged into ${base_libdir} as well?)
That was my thought exactly. Lets look at moving things -or- stop using them. Whichever is more reasonable.
Quoted 21 lines
Offhand I don't know the answer to any of those things.
Quoted 8 lines
Quoted 4 lines
4. this seems like distro policy and not something that really belongs in oe-core at all. For systems where ${bindir} and ${base_bindir} are on the same filesystem (or even are the same directory) this script will just make bootup slower without achieving anything useful.
If /usr and / are on the same file system, this script has no real effect because /usr will always be there. So I think it will not take much time at boot.
Just spawning a new copy of the shell and reading the script does take a finite (and measureable) time. If you can determine statically at build time that the script isn't going to do anything useful then I don't think it's appropriate to install it.
Quoted 5 lines
If ${base_bindir} and ${bindir} are the same, that means that there's no /usr. In this case, there should no
/usr/xxxentries
in/etc/busybox.links, so this script should also function correctly and it will not take much time at boot.
Quoted 8 lines
Quoted 3 lines
(I just configured ${bindir} and ${base_bindir} to be the same and performed a build, it failed. I'm not sure whether it's valid to make such configurations in OE.)
It is a valid configuration, and this is the way that most of the images I build are configured. It probably is true that there are still some number of recipes in oe-core that don't support this properly, but I don't want to see that situation get any worse.
And yes, all scripts need to (at build time) bind to the bindir and base_bindir of that distribution. Hardcoding those values into the scripts before build time is wrong.
--Mark
Quoted 8 lines
p.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [PATCH 2/8] initscripts: add setup-commands.sh
In reply to Burton, RossQuoted 7 lines · Ross
Quoted 2 lines · ChenQi
The problem here is that the init scripts under
/etc/rcS.d/need to execute commands like awk, dirname, and readlink which are from /usr.Yes. So why do you really need to support split /usr, and why isn't an initramfs a suitable alternative? People seem to insist that split /usr is trivial but these patches are clearly showing it's not.
Just to be clear, I've never said it was trivial -- but what I have said is that it -should- be trivial if people pay attention to this stuff early.
The only thing that has to be done for the split is 'early boot'. (See my email response to Phil..)
As for why, we still have people developing systems with small per-device memory that need to do the early boot from a small system, then mount the /usr partition from a shared location. (/usr is often times still mounted RO.. and is shared between multiple CPUs within a single device or chasis.)
The usage of this is getting less and less, but we do still get issues from customers when things don't work right, so we know people are still doing it.
--Mark
Quoted 6 lines
Ross
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core