Skip to contentSkip to conversation

glibc: make nscd optional

PATCH In review Ross Burton 3 participants 10 messages Sep 28, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/7 glibc: make nscd optional

NSCD itself is already packaged into its own package, but if it is never going to be needed then it can be disabled and it won't be called from glibc at all. Add a PACKAGECONFIG for NSCD that is enabled by default.

As a side effect if NSCD is disable, glibc with and without systemd is binary identical, which helps sstate reuse.

---
 meta/recipes-core/glibc/glibc_2.32.bb | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/recipes-core/glibc/glibc_2.32.bb +3 -1
index 7049e61625..2a0e464385 100644
@@ -69,13 +69,15 @@ EXTRA_OECONF = "--enable-kernel=${OLDEST_KERNEL} \
                 --enable-stackguard-randomization \
                 --disable-crypt \
                 --with-default-link \
-                --enable-nscd \
                 ${@bb.utils.contains_any('SELECTED_OPTIMIZATION', '-O0 -Og', '--disable-werror', '', d)} \
                 ${GLIBCPIE} \
                 ${GLIBC_EXTRA_OECONF}"
 
 EXTRA_OECONF += "${@get_libc_fpu_setting(bb, d)}"
 
+PACKAGECONFIG ??= "nscd"
+PACKAGECONFIG[nscd] = "--enable-nscd,--disable-nscd"
+
 do_patch_append() {
     bb.build.exec_func('do_fix_readlib_c', d)
 }
Signature · 1 line
2.25.1
#1 lore raw

PATCH 2/7 utils: respect scheduler affinity in cpu_count()

cpu_count() returns multiprocessing.cpu_count() but that is simply returns os.cpu_count() so we could use that directly.

However this returns the number of CPUs on the host, not the number of usable CPUs on the host. If the user is using scheduler affinity then the number of usable CPUs may be less, so when determining how many cores we can use check the affinity instead.

---
 meta/lib/oe/utils.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta/lib/oe/utils.py +1 -2
index 13f4271da0..468c76f30f 100644
@@ -249,8 +249,7 @@ def trim_version(version, num_parts=2):
     return trimmed
 
 def cpu_count(at_least=1):
-    import multiprocessing
-    cpus = multiprocessing.cpu_count()
+    cpus = len(os.sched_getaffinity(0))
     return max(cpus, at_least)
 
 def execute_pre_post_process(d, cmds):
Signature · 1 line
2.25.1
#2 lore raw

PATCH 3/7 rpm: disable libarchive use

In reply to Ross Burton

libarchive is only needed for the rpm2archive tool, as this is of limited use and libarchive is a non-trivial build dependency make this optional and disabled by default.

---
 meta/recipes-devtools/rpm/rpm_4.15.1.bb | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
meta/recipes-devtools/rpm/rpm_4.15.1.bb +4 -3
index c9258632d2..71e2c033fd 100644
@@ -49,7 +49,7 @@ SRCREV = "ab2179452c5be276a6b96c591afded485c7e58c3"
 
 S = "${WORKDIR}/git"
 
-DEPENDS = "openssl libarchive db file popt xz bzip2 dbus elfutils python3"
+DEPENDS = "openssl db file popt xz bzip2 dbus elfutils python3"
 DEPENDS_append_class-native = " file-replacement-native bzip2-replacement-native"
 
 inherit autotools gettext pkgconfig python3native
@@ -74,6 +74,7 @@ BBCLASSEXTEND = "native nativesdk"
 
 PACKAGECONFIG ??= ""
 PACKAGECONFIG[imaevm] = "--with-imaevm,,ima-evm-utils"
+PACKAGECONFIG[rpm2archive] = "--with-archive,--without-archive,libarchive"
 
 ASNEEDED = ""
 
@@ -95,7 +96,7 @@ WRAPPER_TOOLS = " \
 
 do_install_append_class-native() {
         for tool in ${WRAPPER_TOOLS}; do
-                create_wrapper ${D}$tool \
+                test -x ${D}$tool && create_wrapper ${D}$tool \
                         RPM_CONFIGDIR=${STAGING_LIBDIR_NATIVE}/rpm \
                         RPM_ETCCONFIGDIR=${STAGING_DIR_NATIVE} \
                         MAGIC=${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc \
@@ -105,7 +106,7 @@ do_install_append_class-native() {
 
 do_install_append_class-nativesdk() {
         for tool in ${WRAPPER_TOOLS}; do
-                create_wrapper ${D}$tool \
+                test -x ${D}$tool && create_wrapper ${D}$tool \
                         RPM_CONFIGDIR='`dirname $''realpath`'/${@os.path.relpath(d.getVar('libdir'), d.getVar('bindir'))}/rpm \
                         RPM_ETCCONFIGDIR='$'{RPM_ETCCONFIGDIR-'`dirname $''realpath`'/${@os.path.relpath(d.getVar('sysconfdir'), d.getVar('bindir'))}/..} \
                         MAGIC='`dirname $''realpath`'/${@os.path.relpath(d.getVar('datadir'), d.getVar('bindir'))}/misc/magic.mgc \
Signature · 1 line
2.25.1
#3 lore raw

PATCH 4/7 rpm: add PACKAGECONFIG for the systemd inhibit plugin

In reply to Ross Burton

RPM ships a systemd inhibit plugin, that will tell systemd to inhibit a reboot or sleep during a package upgrades.

For native RPM this is entirely useless, and for target it's only useful if you're using systemd+logind+rpm.

This plugin uses DBus which means it depends on expat -> cmake-native -> libarchive-native curl-native, which is quite a dependency tree to need in early build (required to build packages via rpm-native).

It was previously forcibly disabled for native packages but the build dependency on DBus remained. Remove that logic and add a PACKAGECONFIG for the plugin that is only enabled for target builds with systemd and explicitly disabled for native/nativesdk builds.

---
 meta/recipes-devtools/rpm/rpm_4.15.1.bb | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
meta/recipes-devtools/rpm/rpm_4.15.1.bb +9 -8
index 71e2c033fd..a5b811087b 100644
@@ -49,7 +49,7 @@ SRCREV = "ab2179452c5be276a6b96c591afded485c7e58c3"
 
 S = "${WORKDIR}/git"
 
-DEPENDS = "openssl db file popt xz bzip2 dbus elfutils python3"
+DEPENDS = "openssl db file popt xz bzip2 elfutils python3"
 DEPENDS_append_class-native = " file-replacement-native bzip2-replacement-native"
 
 inherit autotools gettext pkgconfig python3native
@@ -62,18 +62,19 @@ EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl"
 EXTRA_OECONF_append_libc-musl = " --disable-nls --disable-openmp"
 
 # --sysconfdir prevents rpm from attempting to access machine-specific configuration in sysroot/etc; we need to have it in rootfs
-#
 # --localstatedir prevents rpm from writing its database to native sysroot when building images
-#
-# Disable dbus for native, so that rpm doesn't attempt to inhibit shutdown via session dbus even when plugins support is enabled.
-# Also disable plugins by default for native.
-EXTRA_OECONF_append_class-native = " --sysconfdir=/etc --localstatedir=/var --disable-plugins"
-EXTRA_OECONF_append_class-nativesdk = " --sysconfdir=/etc --disable-plugins"
+EXTRA_OECONF_append_class-native = " --sysconfdir=/etc --localstatedir=/var"
+EXTRA_OECONF_append_class-nativesdk = " --sysconfdir=/etc"
 
 BBCLASSEXTEND = "native nativesdk"
 
-PACKAGECONFIG ??= ""
+PACKAGECONFIG ??= "${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'inhibit', '', d)}"
+# The inhibit plugin serves no purpose outside of the target
+PACKAGECONFIG_remove_class-native = "inhibit"
+PACKAGECONFIG_remove_class-nativesdk = "inhibit"
+
 PACKAGECONFIG[imaevm] = "--with-imaevm,,ima-evm-utils"
+PACKAGECONFIG[inhibit] = "--enable-inhibit-plugin,--disable-inhibit-plugin,dbus"
 PACKAGECONFIG[rpm2archive] = "--with-archive,--without-archive,libarchive"
 
 ASNEEDED = ""
Signature · 1 line
2.25.1
#4 lore raw

PATCH 5/7 utils: add umask changing context manager

In reply to Ross Burton

Add a umask context manager which can be used to temporarily change the umask in a 'with' block.

---
 bitbake/lib/bb/utils.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
bitbake/lib/bb/utils.py +11 -0
index 0b79f92e25..f73d31fb73 100644
@@ -944,6 +944,17 @@ def which(path, item, direction = 0, history = False, executable=False):
         return "", hist
     return ""
 
+@contextmanager
+def umask(new_mask):
+    """
+    Context manager to set the umask to a specific mask, and restore it afterwards.
+    """
+    current_mask = os.umask(new_mask)
+    try:
+        yield
+    finally:
+        os.umask(current_mask)
+
 def to_boolean(string, default=None):
     if not string:
         return default
Signature · 1 line
2.25.1
#5 lore raw

PATCH 6/7 siggen: use correct umask when writing siginfo

In reply to Ross Burton

We try to write sstate with group-write permissions so that sstate-cache can be shared between multiple users. However the siginfo files are created in various tasks which may set their own umask (such as do_populate_sysroot, 0022). This results in no group write permission on the intermediate directories, which is fatal when sharing a cache.

Fix this by wrapping the siginfo mkdir in a umask change to 0002.

---
 bitbake/lib/bb/siggen.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
bitbake/lib/bb/siggen.py +2 -1
index 1456324a70..86e0e16f39 100644
@@ -358,7 +358,8 @@ class SignatureGeneratorBasic(SignatureGenerator):
         else:
             sigfile = stampbase + "." + task + ".sigbasedata" + "." + self.basehash[tid]
 
-        bb.utils.mkdirhier(os.path.dirname(sigfile))
+        with bb.utils.umask(0o002):
+            bb.utils.mkdirhier(os.path.dirname(sigfile))
 
         data = {}
         data['task'] = task
Signature · 1 line
2.25.1
#6 lore raw

PATCH 7/7 sstate: set mode explicitly when creating directories in sstate-cache

In reply to Ross Burton

When creating directories in the sstate-cache, explicitly set the mode passed to mkdir to 0775 so that the directories are group writable, as otherwise they cannot be shared with other users.

---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/sstate.bbclass +1 -1
index 66a96a7603..a8ae75101d 100644
@@ -787,7 +787,7 @@ sstate_create_package () {
 		return
 	fi
 
-	mkdir -p `dirname ${SSTATE_PKG}`
+	mkdir --mode=0775 -p `dirname ${SSTATE_PKG}`
 	TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
 
 	# Use pigz if available
Signature · 1 line
2.25.1
#7 lore raw

Re: [OE-core] [PATCH 7/7] sstate: set mode explicitly when creating directories in sstate-cache

I worry that this could be problematic for other ways. Security requirements for an org or even users who want to share the sstate case 0777 etc.

Wouldn't it be better to warn the user that their umask won't allow them to share this with others, and give them instructions and opening the umask?

--Mark

Quoted 27 lines · Ross Burton

When creating directories in the sstate-cache, explicitly set the mode passed to mkdir to 0775 so that the directories are group writable, as otherwise they cannot be shared with other users.

---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/sstate.bbclass +1 -1
index 66a96a7603..a8ae75101d 100644
@@ -787,7 +787,7 @@ sstate_create_package () {
 		return
 	fi
 
-	mkdir -p `dirname ${SSTATE_PKG}`
+	mkdir --mode=0775 -p `dirname ${SSTATE_PKG}`
 	TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
 
 	# Use pigz if available
#8 lore raw

Re: [OE-core] [PATCH 7/7] sstate: set mode explicitly when creating directories in sstate-cache

Quoted 5 lines · Mark Hatle

I worry that this could be problematic for other ways. Security requirements for an org or even users who want to share the sstate case 0777 etc.

Wouldn't it be better to warn the user that their umask won't allow them to share this with others, and give them instructions and opening the umask?

FWIW, ten lines lower we chmod 664 the actual data, so this is just in keeping with existing behaviour.

Ross

#9 lore raw
In reply to Ross Burton
Quoted 24 lines · Ross Burton

NSCD itself is already packaged into its own package, but if it is never going to be needed then it can be disabled and it won't be called from glibc at all. Add a PACKAGECONFIG for NSCD that is enabled by default.

As a side effect if NSCD is disable, glibc with and without systemd is binary identical, which helps sstate reuse.

---
 meta/recipes-core/glibc/glibc_2.32.bb | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/recipes-core/glibc/glibc_2.32.bb +0 -1
index 7049e61625..2a0e464385 100644
@@ -69,13 +69,15 @@ EXTRA_OECONF = "--enable-kernel=${OLDEST_KERNEL} \
                 --enable-stackguard-randomization \
                 --disable-crypt \
                 --with-default-link \
-                --enable-nscd \
                 ${@bb.utils.contains_any('SELECTED_OPTIMIZATION', '-O0 -Og', '--disable-werror', '', d)} \
                 ${GLIBCPIE} \

meta/recipes-core/glibc/glibc_2.32.bb EXTRA_OECONF = "--enable-kernel=${OLDEST_KERNEL} \ · line 83

Quoted 10 lines
meta/recipes-core/glibc/glibc_2.32.bb +3 -0
                 ${GLIBC_EXTRA_OECONF}"
 
 EXTRA_OECONF += "${@get_libc_fpu_setting(bb, d)}"
 
+PACKAGECONFIG ??= "nscd"
+PACKAGECONFIG[nscd] = "--enable-nscd,--disable-nscd"
+
 do_patch_append() {
     bb.build.exec_func('do_fix_readlib_c', d)
 }

The patch looks good to me but
I wonder if this goes beyond glibc and more like a distro features, where you can also ignore nsswitch.conf and
packages being aware of it.

Quoted 6 lines

--
2.25.1