Skip to contentSkip to conversation

useradd fixes, tests and documentation

PATCH In review Eilís 'pidge' Ní Fhlannagáin 4 participants 7 messages Dec 7, 2023 – Dec 8, 2023

lore
Share & tools
lore
#0 lore raw

cover useradd fixes, tests and documentation

This patch series fixes and/or tests for some issues in useradd dependencies. It adds a new variable "USERADD_DEPENDS" that is required now for recipes that have dependencies on other recipes useradd.

Fixes/Tests:
Bug 13419 - recipes that add users to groups cannot rely on other recipes creating those groups (when population from sstate happens) Bug 13904 - do_prepare_recipe_sysroot: postinst-useradd-* does not run in order of dependency and sometimes fails Bug 13279 - Make sure users/groups exist for package_write_* tasks
Bug 15084 - For some reason using of same user in two recipes does not work properly

Tests:
Bug 14961 - addtask between do_populate_sysroot and do_package breaks useradd class

Documentation for useradd is also included.

Eilís 'pidge' Ní Fhlannagáin (3):
useradd: Fix issues with useradd dependencies usergrouptests.py: Add test for switching between static-ids variables.rst: Add USERADD_DEPENDS

 documentation/ref-manual/variables.rst        | 15 +++++
 .../selftest-users/creategroup1.bb            | 32 +++++++++++
 .../selftest-users/creategroup2.bb            | 33 +++++++++++
 .../selftest-users/useraddbadtask.bb          | 20 +++++++
 meta/classes/useradd.bbclass                  |  4 +-
 .../lib/oeqa/selftest/cases/usergrouptests.py | 55 +++++++++++++++++++
 6 files changed, 158 insertions(+), 1 deletion(-)
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup1.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup2.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/useraddbadtask.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/usergrouptests.py
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 useradd: Fix issues with useradd dependencies

If recipe A requires the useradd actions of recipe B we need to ensure that recipe B is part of the recipe A dependancy chain. In order to do that, we introduce USERADD_DEPENDS. This makes sure that the do_populate_sysroot_setscene of recipe B exists for recipe A in case of a missing TMPDIR. This requires changes made in runqueue.py by RP.

This commit along with the runqueue fixes effects:
Bug 13419 - recipes that add users to groups cannot rely on other recipes creating those groups (when population from sstate happens) Bug 13904 - do_prepare_recipe_sysroot: postinst-useradd-* does not run in order of dependency and sometimes fails Bug 13279 - Make sure users/groups exist for package_write_* tasks
Bug 15084 - For some reason using of same user in two recipes does not work properly

I've included the start of self-testing for useradd by adding tests for 13419 (which ends up testing 13904, 13279, 15084 by virtue of them all
having the same root cause)

I've also added (and disabled), a test for 14961 - addtask between do_populate_sysroot and do_package breaks useradd class. A fix is still needed for this, but that fix is a better error message that stops people from doing this.

---
 .../selftest-users/creategroup1.bb            | 32 ++++++++++++++++++
 .../selftest-users/creategroup2.bb            | 33 +++++++++++++++++++
 .../selftest-users/useraddbadtask.bb          | 20 +++++++++++
 meta/classes/useradd.bbclass                  |  4 ++-
 .../lib/oeqa/selftest/cases/usergrouptests.py | 26 +++++++++++++++
 5 files changed, 114 insertions(+), 1 deletion(-)
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup1.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup2.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/useraddbadtask.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/usergrouptests.py
meta-selftest/recipes-test/selftest-users/creategroup1.bb new file +32 -0
new file mode 100644
index 00000000000..ebbbfaa83d3
@@ -0,0 +1,32 @@
+SUMMARY = "creategroup pt 1"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+LICENSE = "MIT"
+
+DEPENDS = "coreutils-native"
+
+S = "${WORKDIR}"
+
+inherit useradd allarch
+
+USERADD_PACKAGES = "${PN}"
+USERADD_PARAM:${PN} = "-u 5555 --gid grouptest gt1"
+GROUPADD_PARAM:${PN} = "-r grouptest"
+
+TESTDIR = "${D}${sysconfdir}/creategroup"
+
+do_install() {
+	install -d   ${TESTDIR}
+	install -d   ${TESTDIR}/dir
+	touch        ${TESTDIR}/file
+	ln -s ./file ${TESTDIR}/symlink
+	install -d   ${TESTDIR}/fifotest
+	mkfifo       ${TESTDIR}/fifotest/fifo
+
+	chown    gt1:grouptest ${TESTDIR}/file
+	chown -R gt1:grouptest ${TESTDIR}/dir
+	chown -h gt1:grouptest ${TESTDIR}/symlink
+	chown -R gt1:grouptest ${TESTDIR}/fifotest
+}
+
+FILES:${PN} = "${sysconfdir}/creategroup/*"
meta-selftest/recipes-test/selftest-users/creategroup2.bb new file +33 -0
new file mode 100644
index 00000000000..ef697f09b4d
@@ -0,0 +1,33 @@
+SUMMARY = "creategroup pt 2"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+LICENSE = "MIT"
+
+DEPENDS = "coreutils-native"
+USERADD_DEPENDS = "creategroup1"
+
+S = "${WORKDIR}"
+
+inherit useradd allarch
+
+USERADD_PACKAGES = "${PN}"
+USERADD_PARAM:${PN} = "-u 5556 --gid grouptest gt2"
+
+TESTDIR = "${D}${sysconfdir}/creategroup"
+
+do_install() {
+	install -d   ${TESTDIR}
+	install -d   ${TESTDIR}/dir
+	touch        ${TESTDIR}/file
+	ln -s ./file ${TESTDIR}/symlink
+	install -d   ${TESTDIR}/fifotest
+	mkfifo       ${TESTDIR}/fifotest/fifo
+
+	chown    gt2:grouptest ${TESTDIR}/file
+	chown -R gt2:grouptest ${TESTDIR}/dir
+	chown -h gt2:grouptest ${TESTDIR}/symlink
+	chown -R gt2:grouptest ${TESTDIR}/fifotest
+}
+
+FILES:${PN} = "${sysconfdir}/creategroup/*"
+
meta-selftest/recipes-test/selftest-users/useraddbadtask.bb new file +20 -0
new file mode 100644
index 00000000000..99e04a80b34
@@ -0,0 +1,20 @@
+SUMMARY = "UserAddBadTask"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+LICENSE = "MIT"
+
+DEPENDS:append = "coreutils-native"
+
+S = "${WORKDIR}"
+
+inherit useradd allarch
+
+USERADD_PACKAGES = "${PN}"
+USERADD_PARAM:${PN} = "-u 5555 --gid groupaddtask useraddtask"
+GROUPADD_PARAM:${PN} = "-r groupaddtask"
+
+do_badthingshappen() {
+ echo "foo"
+}
+
+addtask badthingshappen after do_populate_sysroot before do_package
meta/classes/useradd.bbclass +3 -1
index 4d3bd9a5f56..a4b8a2d6d60 100644
@@ -177,9 +177,11 @@ SYSROOT_PREPROCESS_FUNCS += "${SYSROOTFUNC}"
 
 SSTATEPREINSTFUNCS:append:class-target = " useradd_sysroot_sstate"
 
+USERADD_DEPENDS ??= ""
+DEPENDS += "${USERADD_DEPENDS}"
 do_package_setscene[depends] += "${USERADDSETSCENEDEPS}"
 do_populate_sysroot_setscene[depends] += "${USERADDSETSCENEDEPS}"
-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
+USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"
 USERADDSETSCENEDEPS = ""
 
 # Recipe parse-time sanity checks
meta/lib/oeqa/selftest/cases/usergrouptests.py new file +26 -0
new file mode 100644
index 00000000000..a63fce54dc9
@@ -0,0 +1,26 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var
+import bb.utils
+import os
+
+class UserGroupTests(OESelftestTestCase):
+    def test_group_from_dep_package(self):
+        self.logger.info("Building creategroup2")
+        bitbake(' creategroup2 creategroup1')
+        bitbake(' creategroup2 creategroup1 -c clean')
+        self.logger.info("Packaging creategroup2")
+        self.assertTrue(bitbake(' creategroup2 -c package'))
+
+    def _test_add_task_between_p_sysroot_and_package(self):
+        self.logger.info("Cleaning sstate for useraddbadtask")
+        #bitbake(' useraddbadtask -f -c cleansstate')
+        self.logger.info("Building useraddbadtask")
+        # This is expected to fail due to bug #14961
+        self.assertTrue(bitbake(' useraddbadtask -C fetch'))
+    
Signature · 1 line
2.34.1
#5 lore raw

PATCH 2/3 usergrouptests.py: Add test for switching between static-ids

In reply to Eilís 'pidge' Ní Fhlannagáin

This test is related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=12107

At the moment it doesn't seem to be able to actually replicate this issue in the bug, which tells me it's likely fixed.

---
 .../lib/oeqa/selftest/cases/usergrouptests.py | 45 +++++++++++++++----
 1 file changed, 37 insertions(+), 8 deletions(-)
meta/lib/oeqa/selftest/cases/usergrouptests.py +37 -8
index a63fce54dc9..c74b5678d00 100644
@@ -5,11 +5,15 @@
 #
 
 from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var
-import bb.utils
+from oeqa.utils.commands import bitbake, get_bb_var, get_test_layer
 import os
 
 class UserGroupTests(OESelftestTestCase):
+    def _test_add_task_between_p_sysroot_and_package(self):
+        self.logger.info("Building useraddbadtask")
+        # This is expected to fail due to bug #14961
+        self.assertTrue(bitbake(' useraddbadtask -C fetch'))
+
     def test_group_from_dep_package(self):
         self.logger.info("Building creategroup2")
         bitbake(' creategroup2 creategroup1')
@@ -17,10 +21,35 @@ class UserGroupTests(OESelftestTestCase):
         self.logger.info("Packaging creategroup2")
         self.assertTrue(bitbake(' creategroup2 -c package'))
 
-    def _test_add_task_between_p_sysroot_and_package(self):
-        self.logger.info("Cleaning sstate for useraddbadtask")
-        #bitbake(' useraddbadtask -f -c cleansstate')
-        self.logger.info("Building useraddbadtask")
-        # This is expected to fail due to bug #14961
-        self.assertTrue(bitbake(' useraddbadtask -C fetch'))
+    def test_static_useradd_from_dynamic(self):
+        metaselftestpath = get_test_layer()
+        self.logger.info("Building core-image-minimal to generate passwd/group file")
+        bitbake(' core-image-minimal')
+        self.logger.info("Setting up useradd-staticids")
+        repropassdir = os.path.join(metaselftestpath, "conf/include")
+        os.makedirs(repropassdir)
+        etcdir=os.path.join(os.path.join(os.path.join(get_bb_var("TMPDIR"), "work"), \
+                            os.path.join(get_bb_var("MACHINE").replace("-","_")+"-poky-linux", "core-image-minimal/1.0/rootfs/etc")))
+        import shutil
+        shutil.copy(os.path.join(etcdir, "passwd"), os.path.join(repropassdir, "reproducable-passwd"))
+        shutil.copy(os.path.join(etcdir, "group"), os.path.join(repropassdir, "reproducable-group"))
+        # Copy the original local.conf
+        shutil.copyfile(os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf'), os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf.orig'))
+
+        self.write_config("USERADDEXTENSION = \"useradd-staticids\"")
+        self.write_config("USERADD_ERROR_DYNAMIC ??= \"error\"")
+        self.write_config("USERADD_UID_TABLES += \"conf/include/reproducible-passwd\"")
+        self.write_config("USERADD_GID_TABLES += \"conf/include/reproducible-group\"")
+        self.logger.info("Rebuild with staticids")
+        bitbake(' core-image-minimal')
+        shutil.copyfile(os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf.orig'), os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf'))
+        self.logger.info("Rebuild without staticids")
+        bitbake(' core-image-minimal')
+        self.write_config("USERADDEXTENSION = \"useradd-staticids\"")
+        self.write_config("USERADD_ERROR_DYNAMIC ??= \"error\"")
+        self.write_config("USERADD_UID_TABLES += \"files/static-passwd\"")
+        self.write_config("USERADD_GID_TABLES += \"files/static-group\"")
+        self.logger.info("Rebuild with other staticids")
+        self.assertTrue(bitbake(' core-image-minimal'))
+    
     
Signature · 1 line
2.34.1
#6 lore raw

PATCH 3/3 variables.rst: Add USERADD_DEPENDS

In reply to Eilís 'pidge' Ní Fhlannagáin

Adding USERADD_DEPENDS to the variables document

---
 documentation/ref-manual/variables.rst | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
documentation/ref-manual/variables.rst +15 -0
index b394d310990..9ec0ce4b4ca 100644
@@ -9736,6 +9736,21 @@ system and gives an overview of their function and contents.
       ``meta-poky/conf/templates/default/local.conf.sample`` in the
       :term:`Source Directory`.
 
+   :term:`USERADD_DEPENDS`
+      Lists a recipe's USERADD dependencies. These are dependencies on
+      other recipes whose create users and groups via useradd.bbclass and are
+      needed by the recipe at build time. This variable is now required for
+      recipes that rely on other recipes for part of their useradd.
+      
+      As an example, consider a recipe ``foo``. Recipe ``bar`` creates a group
+      that foo requires for it's useradd tasks. In order to ensure that the
+      useradd tasks exist for foo, we add the the following assignment::
+
+          $USERADD_DEPENDS = "bar"
+
+      The practical effect of this assignment is that bar's sysroot_setscene is
+      added to :term:`USERADDSETSCENEDEPS` variable.
+
    :term:`USERADD_ERROR_DYNAMIC`
       If set to ``error``, forces the OpenEmbedded build system to produce
       an error if the user identification (``uid``) and group
Signature · 1 line
2.34.1
#2 lore raw

Re: [OE-core] [PATCH 1/3] useradd: Fix issues with useradd dependencies

In reply to Eilís 'pidge' Ní Fhlannagáin

meta/classes/useradd.bbclass SYSROOT_PREPROCESS_FUNCS += "${SYSROOTFUNC}" · line 184

Quoted 2 lines · Eilís 'pidge' Ní Fhlannagáin
meta/classes/useradd.bbclass +1 -1
-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
+USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"

I'd like to clarify: do all these recent fixes mean that we can also add dynamic library dependencies of shadow-native into this list (it becomes an issue with latest versions of shadow currently not in oe-core)?

Specifically, this commit

https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/package-version-updates&id=9fdbe321fa53785cc35bd1bb4054366383ef20e5

doing

-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene" +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene \
+ pseudo-native:do_populate_sysroot_setscene \
+ shadow-native:do_populate_sysroot_setscene \
+ attr-native:do_populate_sysroot_setscene \
+ libbsd-native:do_populate_sysroot_setscene \
+ libmd-native:do_populate_sysroot_setscene \
+ ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"

has been rejected by RP with the suggestion that shadow should be linked statically, but now I'm not sure if the argument still holds.

Alex

#3 lore raw

Re: [OE-core] [PATCH 1/3] useradd: Fix issues with useradd dependencies

Quoted 30 lines · Alexander Kanavin

meta/classes/useradd.bbclass SYSROOT_PREPROCESS_FUNCS += "${SYSROOTFUNC}" · line 184

Quoted 2 lines · Eilís 'pidge' Ní Fhlannagáin
meta/classes/useradd.bbclass +1 -1
-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
+USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"

I'd like to clarify: do all these recent fixes mean that we can also add dynamic library dependencies of shadow-native into this list (it becomes an issue with latest versions of shadow currently not in oe-core)?

Specifically, this commit

https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/package-version-updates&id=9fdbe321fa53785cc35bd1bb4054366383ef20e5

doing

-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene" +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene \
+ pseudo-native:do_populate_sysroot_setscene \
+ shadow-native:do_populate_sysroot_setscene \
+ attr-native:do_populate_sysroot_setscene \
+ libbsd-native:do_populate_sysroot_setscene \
+ libmd-native:do_populate_sysroot_setscene \
+ ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"

Quoted 2 lines

has been rejected by RP with the suggestion that shadow should be linked statically, but now I'm not sure if the argument still holds.

After my recent change, we could in theory do this as bitbake might be able to make it work. It does run the risk of circular dependencies and these setscene inter-task dependencies are *horrible* in general from an sstate perspective though as my other comments about these dependencies leading to poor behaviour still stand.

To be clear about what you're saying above, you're saying that if any of those native dependencies are missing, we cannot use/install the sstate object. We have to defer installing that sstate until all the others are downloaded and available too.

My preference is still to statically link shadow to avoid this pretty horrible side effect.

Cheers,

Richard

#4 lore raw

Re: [PATCH 1/3] useradd: Fix issues with useradd dependencies

In reply to Eilís 'pidge' Ní Fhlannagáin
Quoted 6 lines

If recipe A requires the useradd actions of recipe B we need to ensure that recipe B is part of the recipe A dependancy chain. In order to do that, we introduce USERADD_DEPENDS. This makes sure that the do_populate_sysroot_setscene of recipe B exists for recipe A in case of a missing TMPDIR. This requires changes made in runqueue.py by RP.

I was wondering if all required changes including these would be considered suitable to backport to e.g. kirkstone? Or am I going to have to look forward to scarthgap for these :)

Sincerely,

Jermain