Implement merged /usr
cover Implement merged /usr
Hi,
this patchset implements Fedora-style merged /usr with symlinks, and fixes the kernel and busybox recipes to work with this. There are probably more packages which need similar fixes.
The creation of the symlinks is controlled by the merged-usr DISTRO_FEATURE as suggested in [YOCTO #7040]. The variables base_bindir, base_sbindir, sbindir, base_libdir and nonarch_base_libdir need to be set explicitly in the distro config in addition to the DISTRO_FEATURE to make this feature as flexible as possible (see the commit message of PATCH 4/4).
Matthias Schiffer (4):
busybox: generalize recipe to work with arbitrary install directories
kernel: allow kernel module and firmware installation with
${base_libdir} != "/lib" base-files: create ${base_bindir} etc. instead of /bin, /sbin and /lib
base-files: create typical merged /usr symlinks if the "merged-usr"
distro feature is set
meta/classes/kernel-module-split.bbclass | 4 +-- meta/classes/kernel.bbclass | 25 +++++++++++----- meta/recipes-core/base-files/base-files_3.0.14.bb | 26 +++++++++++++++-- meta/recipes-core/busybox/busybox.inc | 35 ++++++++--------------- 4 files changed, 56 insertions(+), 34 deletions(-)
Signature · 1 line
2.6.4
PATCH 1/4 busybox: generalize recipe to work with arbitrary install directories
Allow configuring base_bindir, base_sbindir, bindir and sbindir arbitrarily.
Also change the temporary symlinks created in the postinst script from relative to absolute to make the code simpler and more flexible. There was no reason to use relative links anyways, as directly afterwards update-alternatives would replace them by absolute links anyways.
--- meta/recipes-core/busybox/busybox.inc | 35 ++++++++++++----------------------- 1 file changed, 12 insertions(+), 23 deletions(-)
meta/recipes-core/busybox/busybox.inc +12 -23
@@ -171,20 +171,20 @@ do_compile() { } do_install () { - if [ "${prefix}" != "/usr" ]; then - sed -i "s:^/usr/:${prefix}/:" busybox.links* - fi - if [ "${base_sbindir}" != "/sbin" ]; then - sed -i "s:^/sbin/:${base_sbindir}/:" busybox.links* - fi - if [ "${base_bindir}" != "/bin" ]; then - sed -i "s:^/bin/:${base_bindir}/:" busybox.links* - fi + sed -i "s:^/bin/:BASE_BINDIR/:" busybox.links* + sed -i "s:^/sbin/:BASE_SBINDIR/:" busybox.links* + sed -i "s:^/usr/bin/:BINDIR/:" busybox.links* + sed -i "s:^/usr/sbin/:SBINDIR/:" busybox.links* + + sed -i "s:^BASE_BINDIR/:${base_bindir}/:" busybox.links* + sed -i "s:^BASE_SBINDIR/:${base_sbindir}/:" busybox.links* + sed -i "s:^BINDIR/:${bindir}/:" busybox.links* + sed -i "s:^SBINDIR/:${sbindir}/:" busybox.links* install -d ${D}${sysconfdir}/init.d if ! grep -q "CONFIG_FEATURE_INDIVIDUAL=y" ${B}/.config; then - # Install /bin/busybox, and the /bin/sh link so the postinst script + # Install ${base_bindir}/busybox, and the ${base_bindir}/sh link so the postinst script # can run. Let update-alternatives handle the rest. install -d ${D}${base_bindir} if [ "${BUSYBOX_SPLIT_SUID}" = "1" ]; then
@@ -212,7 +212,7 @@ do_install () { # We make this symlink here to eliminate the error when upgrading together # with busybox-syslog. Without this symlink, the opkg may think of the # busybox.nosuid as obsolete and remove it, resulting in dead links like - # /bin/sed -> /bin/busybox.nosuid. This will make upgrading busybox-syslog fail. + # ${base_bindir}/sed -> ${base_bindir}/busybox.nosuid. This will make upgrading busybox-syslog fail. # This symlink will be safely deleted in postinst, thus no negative effect. ln -sf busybox ${D}${base_bindir}/busybox.nosuid fi
@@ -368,21 +368,10 @@ pkg_postinst_${PN} () { if test -e ${sysconfdir}/busybox.links$suffix; then while read link; do if test ! -e "$link"; then - case "$link" in - /*/*/*) - to="../..${base_bindir}/busybox$suffix" - ;; - /bin/*) - to="busybox$suffix" - ;; - /*/*) - to="..${base_bindir}/busybox$suffix" - ;; - esac # we can use busybox here because even if we are using splitted busybox # we've made a symlink from /bin/busybox to /bin/busybox.nosuid. busybox rm -f $link - busybox ln -s $to $link + busybox ln -s "${base_bindir}/busybox$suffix" $link fi done < ${sysconfdir}/busybox.links$suffix fi
Signature · 1 line
2.6.4
PATCH 2/4 kernel: allow kernel module and firmware installation with ${base_libdir} != "/lib"
In reply to Matthias Schiffer--- meta/classes/kernel-module-split.bbclass | 4 ++-- meta/classes/kernel.bbclass | 25 ++++++++++++++++++------- 2 files changed, 20 insertions(+), 9 deletions(-)
meta/classes/kernel-module-split.bbclass +2 -2
@@ -109,7 +109,7 @@ python split_kernel_module_packages () { # file no longer includes PKGD file = file.replace(d.getVar('PKGD', True) or '', '', 1) # instead is prefixed with /lib/modules/${KERNEL_VERSION} - file = file.replace("/lib/modules/%s/" % d.getVar('KERNEL_VERSION', True) or '', '', 1) + file = file.replace("%s/modules/%s/" % (d.getVar('base_libdir', True), d.getVar('KERNEL_VERSION', True)), '', 1) if file in module_deps: dependencies = []
@@ -186,7 +186,7 @@ python split_kernel_module_packages () { postinst = d.getVar('pkg_postinst_modules', True) postrm = d.getVar('pkg_postrm_modules', True) - modules = do_split_packages(d, root='/lib/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='kernel-%s' % (d.getVar("KERNEL_VERSION", True))) + modules = do_split_packages(d, root='%s/modules' % d.getVar('base_libdir', True), file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='kernel-%s' % (d.getVar("KERNEL_VERSION", True))) if modules: metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE', True) d.appendVar('RDEPENDS_' + metapkg, ' '+' '.join(modules))
meta/classes/kernel.bbclass +18 -7
@@ -242,6 +242,11 @@ kernel_do_install() { rm "${D}/lib/modules/${KERNEL_VERSION}/source" # If the kernel/ directory is empty remove it to prevent QA issues rmdir --ignore-fail-on-non-empty "${D}/lib/modules/${KERNEL_VERSION}/kernel" + + if [ "${base_libdir}" != "/lib" ]; then + install -d "${D}${base_libdir}" + mv "${D}/lib/modules" "${D}${base_libdir}" + fi else bbnote "no modules to install" fi
@@ -258,6 +263,12 @@ kernel_do_install() { [ -e Module.symvers ] && install -m 0644 Module.symvers ${D}/boot/Module.symvers-${KERNEL_VERSION} install -d ${D}${sysconfdir}/modules-load.d install -d ${D}${sysconfdir}/modprobe.d + + if [ "${base_libdir}" != "/lib" ]; then + install -d "${D}${base_libdir}" + mv "${D}/lib/firmware" "${D}${base_libdir}" + rmdir --ignore-fail-on-non-empty "${D}/lib" + fi } do_install[prefuncs] += "package_get_auto_pr"
@@ -364,9 +375,9 @@ EXPORT_FUNCTIONS do_compile do_install do_configure # kernel-image becomes kernel-image-${KERNEL_VERISON} PACKAGES = "kernel kernel-base kernel-vmlinux kernel-image kernel-dev kernel-modules" FILES_${PN} = "" -FILES_kernel-base = "/lib/modules/${KERNEL_VERSION}/modules.order /lib/modules/${KERNEL_VERSION}/modules.builtin" +FILES_kernel-base = "${base_libdir}/modules/${KERNEL_VERSION}/modules.order ${base_libdir}/modules/${KERNEL_VERSION}/modules.builtin" FILES_kernel-image = "/boot/${KERNEL_IMAGETYPE}*" -FILES_kernel-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} /lib/modules/${KERNEL_VERSION}/build" +FILES_kernel-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${base_libdir}/modules/${KERNEL_VERSION}/build" FILES_kernel-vmlinux = "/boot/vmlinux*" FILES_kernel-modules = "" RDEPENDS_kernel = "kernel-base"
@@ -384,8 +395,8 @@ ALLOW_EMPTY_kernel-modules = "1" DESCRIPTION_kernel-modules = "Kernel modules meta package" pkg_postinst_kernel-base () { - if [ ! -e "$D/lib/modules/${KERNEL_VERSION}" ]; then - mkdir -p $D/lib/modules/${KERNEL_VERSION} + if [ ! -e "$D${base_libdir}/modules/${KERNEL_VERSION}" ]; then + mkdir -p $D${base_libdir}/modules/${KERNEL_VERSION} fi if [ -n "$D" ]; then depmodwrapper -a -b $D ${KERNEL_VERSION}
@@ -405,7 +416,7 @@ pkg_postrm_kernel-image () { PACKAGESPLITFUNCS_prepend = "split_kernel_packages " python split_kernel_packages () { - do_split_packages(d, root='/lib/firmware', file_regex='^(.*)\.(bin|fw|cis|dsp)$', output_pattern='kernel-firmware-%s', description='Firmware for %s', recursive=True, extra_depends='') + do_split_packages(d, root='%s/firmware' % d.getVar('base_libdir', True), file_regex='^(.*)\.(bin|fw|cis|dsp)$', output_pattern='kernel-firmware-%s', description='Firmware for %s', recursive=True, extra_depends='') } do_strip() {
@@ -469,8 +480,8 @@ MODULE_TARBALL_DEPLOY ?= "1" kernel_do_deploy() { install -m 0644 ${KERNEL_OUTPUT} ${DEPLOYDIR}/${KERNEL_IMAGE_BASE_NAME}.bin if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then - mkdir -p ${D}/lib - tar -cvzf ${DEPLOYDIR}/${MODULE_TARBALL_BASE_NAME} -C ${D} lib + mkdir -p ${D}${base_libdir} + tar -cvzf ${DEPLOYDIR}/${MODULE_TARBALL_BASE_NAME} -C ${D} ${@d.getVar('base_libdir', True)[1:]} ln -sf ${MODULE_TARBALL_BASE_NAME} ${DEPLOYDIR}/${MODULE_TARBALL_SYMLINK_NAME} fi
Signature · 1 line
2.6.4
Re: [PATCH 2/4] kernel: allow kernel module and firmware installation with ${base_libdir} != "/lib"
В письме от 16 декабря 2015 19:11:58 пользователь Matthias Schiffer написал:
Quoted 20 lines
meta/classes/kernel.bbclass +0
-0@@ -242,6 +242,11 @@ kernel_do_install() { rm "${D}/lib/modules/${KERNEL_VERSION}/source" # If the kernel/ directory is empty remove it to prevent QA issues rmdir --ignore-fail-on-non-empty"${D}/lib/modules/${KERNEL_VERSION}/kernel" +
meta/classes/kernel.bbclass +4
-0+ if [ "${base_libdir}" != "/lib" ]; then + install -d "${D}${base_libdir}" + mv "${D}/lib/modules" "${D}${base_libdir}" + fi else bbnote "no modules to install" fi@@ -258,6 +263,12 @@ kernel_do_install() { [ -e Module.symvers ] && install -m 0644 Module.symvers${D}/boot/Module.symvers-${KERNEL_VERSION} install -d ${D}${sysconfdir}/modules-load.d
meta/classes/kernel.bbclass kernel_do_install() { · line 273
Quoted 9 lines
meta/classes/kernel.bbclass +6
-0install -d ${D}${sysconfdir}/modprobe.d + + if [ "${base_libdir}" != "/lib" ]; then + install -d "${D}${base_libdir}" + mv "${D}/lib/firmware" "${D}${base_libdir}" + rmdir --ignore-fail-on-non-empty "${D}/lib" + fi } do_install[prefuncs] += "package_get_auto_pr"
I think this hunks are not needed if you're to fix fs-perms.txt.
Signature · 3 lines
http://roman.khimov.ru mailto: [email protected] gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3
Re: [PATCH 2/4] kernel: allow kernel module and firmware installation with ${base_libdir} != "/lib"
In reply to Matthias SchifferВ письме от 16 декабря 2015 19:11:58 пользователь Matthias Schiffer написал:
Quoted 4 lines
# instead is prefixed with /lib/modules/${KERNEL_VERSION}
- file = file.replace("/lib/modules/%s/" % d.getVar('KERNEL_VERSION',
True) or '', '', 1) + file = file.replace("%s/modules/%s/" %
(d.getVar('base_libdir', True), d.getVar('KERNEL_VERSION', True)), '', 1)And, BTW, the comment still references /lib.
Signature · 3 lines
http://roman.khimov.ru mailto: [email protected] gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3
PATCH 3/4 base-files: create ${base_bindir} etc. instead of /bin, /sbin and /lib
In reply to Matthias SchifferThese directories conflict with the symlinks created for merged-usr setups.
--- meta/recipes-core/base-files/base-files_3.0.14.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/recipes-core/base-files/base-files_3.0.14.bb +2 -2
@@ -33,8 +33,8 @@ INHIBIT_DEFAULT_DEPS = "1" docdir_append = "/${P}" dirs1777 = "/tmp ${localstatedir}/volatile/tmp" dirs2775 = "" -dirs755 = "/bin /boot /dev ${sysconfdir} ${sysconfdir}/default \ - ${sysconfdir}/skel /lib /mnt /proc ${ROOT_HOME} /run /sbin \ +dirs755 = "${base_bindir} /boot /dev ${sysconfdir} ${sysconfdir}/default \ + ${sysconfdir}/skel ${nonarch_base_libdir} /mnt /proc ${ROOT_HOME} /run ${base_sbindir} \ ${prefix} ${bindir} ${docdir} /usr/games ${includedir} \ ${libdir} ${sbindir} ${datadir} \ ${datadir}/common-licenses ${datadir}/dict ${infodir} \
Signature · 1 line
2.6.4
Re: [PATCH 3/4] base-files: create ${base_bindir} etc. instead of /bin, /sbin and /lib
Quoted 6 lines · Matthias Schiffer
meta/recipes-core/base-files/base-files_3.0.14.bb +2
-2-dirs755 = "/bin /boot /dev ${sysconfdir} ${sysconfdir}/default \ - ${sysconfdir}/skel /lib /mnt /proc ${ROOT_HOME} /run /sbin \ +dirs755 = "${base_bindir} /boot /dev ${sysconfdir} ${sysconfdir}/default \ + ${sysconfdir}/skel ${nonarch_base_libdir} /mnt /proc${ROOT_HOME} /run ${base_sbindir} \
I think the *intention* of /lib was to be $(base_libdir).
Ross
Re: [PATCH 3/4] base-files: create ${base_bindir} etc. instead of /bin, /sbin and /lib
Quoted 12 lines · Ross
Quoted 6 lines · Matthias Schiffer
meta/recipes-core/base-files/base-files_3.0.14.bb +2
-2-dirs755 = "/bin /boot /dev ${sysconfdir} ${sysconfdir}/default \ - ${sysconfdir}/skel /lib /mnt /proc ${ROOT_HOME} /run /sbin \ +dirs755 = "${base_bindir} /boot /dev ${sysconfdir} ${sysconfdir}/default \ + ${sysconfdir}/skel ${nonarch_base_libdir} /mnt /proc${ROOT_HOME} /run ${base_sbindir} \
I think the *intention* of /lib was to be $(base_libdir).
Quoted 2 lines
Ross
Maybe adding both ${base_libdir} and ${nonarch_base_libdir} would make sense?
Re: [PATCH 3/4] base-files: create ${base_bindir} etc. instead of /bin, /sbin and /lib
Quoted 3 lines · Matthias Schiffer
Maybe adding both ${base_libdir} and ${nonarch_base_libdir} would make sense?
Sure.
Ross
PATCH 4/4 base-files: create typical merged /usr symlinks if the "merged-usr" distro feature is set
In reply to Matthias SchifferThe typical symlinks are:
* /bin -> usr/bin
* /sbin -> usr/sbin
* /lib -> usr/lib
* /usr/sbin -> bin
It is checked that only sane symlinks are created, so a half-merged /usr (e.g, merging bin and sbin, but not lib) is feasible as well.
The following options in a distro configuration create a full Fedora-style merged /usr:
DISTRO_FEATURES_DEFAULT += "merged-usr"
base_bindir = "${exec_prefix}/bin"
base_sbindir = "${exec_prefix}/bin"
sbindir = "${exec_prefix}/bin"
base_libdir = "${exec_prefix}/${baselib}"
nonarch_base_libdir = "${exec_prefix}/lib"[YOCTO #7040]
--- meta/recipes-core/base-files/base-files_3.0.14.bb | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
meta/recipes-core/base-files/base-files_3.0.14.bb +22 -0
@@ -134,6 +134,28 @@ do_install () { fi ln -sf /proc/mounts ${D}${sysconfdir}/mtab + + if ${@bb.utils.contains('DISTRO_FEATURES','merged-usr','true','false',d)}; then + if [ "${base_bindir}" != "/bin" ]; then + ln -snf "${@d.getVar('base_bindir', True)[1:]}" ${D}/bin + fi + + if [ "${base_sbindir}" != "/sbin" ]; then + ln -snf "${@d.getVar('base_sbindir', True)[1:]}" ${D}/sbin + fi + + if [ "${nonarch_base_libdir}" != "/lib" ]; then + ln -snf "${@d.getVar('nonarch_base_libdir', True)[1:]}" ${D}/lib + fi + + if [ "${base_libdir}" != "/${baselib}" ]; then + ln -snf "${@d.getVar('base_libdir', True)[1:]}" ${D}/${baselib} + fi + + if [ "${bindir}" = "${exec_prefix}/bin" -a "${sbindir}" = "${exec_prefix}/bin" ]; then + ln -snf bin ${D}${exec_prefix}/sbin + fi + fi } DISTRO_VERSION[vardepsexclude] += "DATE"
Signature · 1 line
2.6.4
Re: [PATCH 4/4] base-files: create typical merged /usr symlinks if the "merged-usr" distro feature is set
В письме от 16 декабря 2015 19:12:00 пользователь Matthias Schiffer написал:
Quoted 9 lines
The following options in a distro configuration create a full Fedora-style merged /usr:
DISTRO_FEATURES_DEFAULT += "merged-usr" base_bindir = "${exec_prefix}/bin" base_sbindir = "${exec_prefix}/bin" sbindir = "${exec_prefix}/bin" base_libdir = "${exec_prefix}/${baselib}" nonarch_base_libdir = "${exec_prefix}/lib"
Hmm. Why not just change the base_prefix? We've done it this way (not merging bin with sbin, but that's a separate thing):
+base_prefix = "/usr"
+sysconfdir = "/etc"
+localstatedir = "/var"
Also, you probably want to add this symlinks to fs-perms.txt. I don't quite remember which packages failed for me without it, but there were some.
Signature · 3 lines
http://roman.khimov.ru mailto: [email protected] gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3
Re: [PATCH 4/4] base-files: create typical merged /usr symlinks if the "merged-usr" distro feature is set
Quoted 14 lines · Roman Khimov
В письме от 16 декабря 2015 19:12:00 пользователь Matthias Schiffer написал:
Quoted 9 lines
The following options in a distro configuration create a full Fedora-style merged /usr:
DISTRO_FEATURES_DEFAULT += "merged-usr" base_bindir = "${exec_prefix}/bin" base_sbindir = "${exec_prefix}/bin" sbindir = "${exec_prefix}/bin" base_libdir = "${exec_prefix}/${baselib}" nonarch_base_libdir = "${exec_prefix}/lib"Hmm. Why not just change the
base_prefix? We've done it this way (not merging bin with sbin, but that's a separate thing):
Review
Quoted 3 lines
(quoted diff) +3
-0+base_prefix = "/usr" +sysconfdir = "/etc" +localstatedir = "/var"
My patches don't care how base_bindir etc. are set, your way would work with it just fine. My recommendation is to set the dirs explicitly instead of changing the base_prefix, as other things might depend on the base_prefix, and symlinks are created for /bin, /sbin, /lib and /${baselib} only.
Quoted 4 lines
Also, you probably want to add this symlinks to fs-perms.txt. I don't quite remember which packages failed for me without it, but there were some.
Is it even possible to have fs-perms.txt entries depending on a DISTRO_FEATURE? And can fs-perms.txt move files from one directory to another (as you suggested in your reply for PATCH 2)?
AFAICT, fs-perms.txt is only there to fix some common issues with directory permissions. Most packages already respect base_libdir etc. just fine, so in my opinion it would be preferable to just fix the packages which don't if adding such things to fs-perms.txt is non-trivial.
Re: [PATCH 4/4] base-files: create typical merged /usr symlinks if the "merged-usr" distro feature is set
В письме от 17 декабря 2015 10:47:28 пользователь Matthias Schiffer написал:
Quoted 9 lines · Roman Khimov
Quoted 5 lines
В письме от 16 декабря 2015 19:12:00 пользователь Matthias Schiffer написал:
Also, you probably want to add this symlinks to fs-perms.txt. I don't quite
remember which packages failed for me without it, but there were some.Is it even possible to have fs-perms.txt entries depending on a
DISTRO_FEATURE?
Probably not.
Quoted 2 lines
And can fs-perms.txt move files from one directory to another (as you suggested in your reply for PATCH 2)?
IIRC, yes.
Quoted 4 lines
AFAICT, fs-perms.txt is only there to fix some common issues with directory permissions. Most packages already respect
base_libdiretc. just fine, so in my opinion it would be preferable to just fix the packages which don't if adding such things to fs-perms.txt is non-trivial.
fs-perms can fix things automatically, with no additional code in recipes (and that's a nice thing IMO), but, yes, at the moment it probably can't do or don't do things based on DISTRO_FEATURE settings. Maybe using FILESYSTEM_PERMS_TABLES can help or even generating fs-perms on the fly taking into account real ${somedir} values.