useradd fixes, tests and documentation
cover useradd fixes, tests and documentation
This patch series fixes and/or tests for some issues in useradd dependencies. It adds a new variable "USERADD_DEPENDS" that is required now for recipes that have dependencies on other recipes useradd.
Fixes/Tests:
Bug 13419 - recipes that add users to groups cannot rely on other recipes creating those groups (when population from sstate happens) Bug 13904 - do_prepare_recipe_sysroot: postinst-useradd-* does not run in order of dependency and sometimes fails Bug 13279 - Make sure users/groups exist for package_write_* tasks
Bug 15084 - For some reason using of same user in two recipes does not work properly
Tests:
Bug 14961 - addtask between do_populate_sysroot and do_package breaks useradd class
Documentation for useradd is also included.
Eilís 'pidge' Ní Fhlannagáin (3):
useradd: Fix issues with useradd dependencies usergrouptests.py: Add test for switching between static-ids variables.rst: Add USERADD_DEPENDS
documentation/ref-manual/variables.rst | 15 +++++ .../selftest-users/creategroup1.bb | 32 +++++++++++ .../selftest-users/creategroup2.bb | 33 +++++++++++ .../selftest-users/useraddbadtask.bb | 20 +++++++ meta/classes/useradd.bbclass | 4 +- .../lib/oeqa/selftest/cases/usergrouptests.py | 55 +++++++++++++++++++ 6 files changed, 158 insertions(+), 1 deletion(-) create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup1.bb create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup2.bb create mode 100644 meta-selftest/recipes-test/selftest-users/useraddbadtask.bb create mode 100644 meta/lib/oeqa/selftest/cases/usergrouptests.py
Signature · 1 line
2.34.1
PATCH 1/3 useradd: Fix issues with useradd dependencies
If recipe A requires the useradd actions of recipe B we need to ensure that recipe B is part of the recipe A dependancy chain. In order to do that, we introduce USERADD_DEPENDS. This makes sure that the do_populate_sysroot_setscene of recipe B exists for recipe A in case of a missing TMPDIR. This requires changes made in runqueue.py by RP.
This commit along with the runqueue fixes effects:
Bug 13419 - recipes that add users to groups cannot rely on other recipes creating those groups (when population from sstate happens) Bug 13904 - do_prepare_recipe_sysroot: postinst-useradd-* does not run in order of dependency and sometimes fails Bug 13279 - Make sure users/groups exist for package_write_* tasks
Bug 15084 - For some reason using of same user in two recipes does not work properly
I've included the start of self-testing for useradd by adding tests for 13419 (which ends up testing 13904, 13279, 15084 by virtue of them all
having the same root cause)
I've also added (and disabled), a test for 14961 - addtask between do_populate_sysroot and do_package breaks useradd class. A fix is still needed for this, but that fix is a better error message that stops people from doing this.
--- .../selftest-users/creategroup1.bb | 32 ++++++++++++++++++ .../selftest-users/creategroup2.bb | 33 +++++++++++++++++++ .../selftest-users/useraddbadtask.bb | 20 +++++++++++ meta/classes/useradd.bbclass | 4 ++- .../lib/oeqa/selftest/cases/usergrouptests.py | 26 +++++++++++++++ 5 files changed, 114 insertions(+), 1 deletion(-) create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup1.bb create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup2.bb create mode 100644 meta-selftest/recipes-test/selftest-users/useraddbadtask.bb create mode 100644 meta/lib/oeqa/selftest/cases/usergrouptests.py
meta-selftest/recipes-test/selftest-users/creategroup1.bb new file +32 -0
@@ -0,0 +1,32 @@ +SUMMARY = "creategroup pt 1" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +LICENSE = "MIT" + +DEPENDS = "coreutils-native" + +S = "${WORKDIR}" + +inherit useradd allarch + +USERADD_PACKAGES = "${PN}" +USERADD_PARAM:${PN} = "-u 5555 --gid grouptest gt1" +GROUPADD_PARAM:${PN} = "-r grouptest" + +TESTDIR = "${D}${sysconfdir}/creategroup" + +do_install() { + install -d ${TESTDIR} + install -d ${TESTDIR}/dir + touch ${TESTDIR}/file + ln -s ./file ${TESTDIR}/symlink + install -d ${TESTDIR}/fifotest + mkfifo ${TESTDIR}/fifotest/fifo + + chown gt1:grouptest ${TESTDIR}/file + chown -R gt1:grouptest ${TESTDIR}/dir + chown -h gt1:grouptest ${TESTDIR}/symlink + chown -R gt1:grouptest ${TESTDIR}/fifotest +} + +FILES:${PN} = "${sysconfdir}/creategroup/*"
meta-selftest/recipes-test/selftest-users/creategroup2.bb new file +33 -0
@@ -0,0 +1,33 @@ +SUMMARY = "creategroup pt 2" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +LICENSE = "MIT" + +DEPENDS = "coreutils-native" +USERADD_DEPENDS = "creategroup1" + +S = "${WORKDIR}" + +inherit useradd allarch + +USERADD_PACKAGES = "${PN}" +USERADD_PARAM:${PN} = "-u 5556 --gid grouptest gt2" + +TESTDIR = "${D}${sysconfdir}/creategroup" + +do_install() { + install -d ${TESTDIR} + install -d ${TESTDIR}/dir + touch ${TESTDIR}/file + ln -s ./file ${TESTDIR}/symlink + install -d ${TESTDIR}/fifotest + mkfifo ${TESTDIR}/fifotest/fifo + + chown gt2:grouptest ${TESTDIR}/file + chown -R gt2:grouptest ${TESTDIR}/dir + chown -h gt2:grouptest ${TESTDIR}/symlink + chown -R gt2:grouptest ${TESTDIR}/fifotest +} + +FILES:${PN} = "${sysconfdir}/creategroup/*" +
meta-selftest/recipes-test/selftest-users/useraddbadtask.bb new file +20 -0
@@ -0,0 +1,20 @@ +SUMMARY = "UserAddBadTask" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +LICENSE = "MIT" + +DEPENDS:append = "coreutils-native" + +S = "${WORKDIR}" + +inherit useradd allarch + +USERADD_PACKAGES = "${PN}" +USERADD_PARAM:${PN} = "-u 5555 --gid groupaddtask useraddtask" +GROUPADD_PARAM:${PN} = "-r groupaddtask" + +do_badthingshappen() { + echo "foo" +} + +addtask badthingshappen after do_populate_sysroot before do_package
meta/classes/useradd.bbclass +3 -1
@@ -177,9 +177,11 @@ SYSROOT_PREPROCESS_FUNCS += "${SYSROOTFUNC}" SSTATEPREINSTFUNCS:append:class-target = " useradd_sysroot_sstate" +USERADD_DEPENDS ??= "" +DEPENDS += "${USERADD_DEPENDS}" do_package_setscene[depends] += "${USERADDSETSCENEDEPS}" do_populate_sysroot_setscene[depends] += "${USERADDSETSCENEDEPS}" -USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene" +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}" USERADDSETSCENEDEPS = "" # Recipe parse-time sanity checks
meta/lib/oeqa/selftest/cases/usergrouptests.py new file +26 -0
@@ -0,0 +1,26 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import bitbake, get_bb_var +import bb.utils +import os + +class UserGroupTests(OESelftestTestCase): + def test_group_from_dep_package(self): + self.logger.info("Building creategroup2") + bitbake(' creategroup2 creategroup1') + bitbake(' creategroup2 creategroup1 -c clean') + self.logger.info("Packaging creategroup2") + self.assertTrue(bitbake(' creategroup2 -c package')) + + def _test_add_task_between_p_sysroot_and_package(self): + self.logger.info("Cleaning sstate for useraddbadtask") + #bitbake(' useraddbadtask -f -c cleansstate') + self.logger.info("Building useraddbadtask") + # This is expected to fail due to bug #14961 + self.assertTrue(bitbake(' useraddbadtask -C fetch')) +
Signature · 1 line
2.34.1
PATCH 2/3 usergrouptests.py: Add test for switching between static-ids
In reply to Eilís 'pidge' Ní FhlannagáinThis test is related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=12107
At the moment it doesn't seem to be able to actually replicate this issue in the bug, which tells me it's likely fixed.
--- .../lib/oeqa/selftest/cases/usergrouptests.py | 45 +++++++++++++++---- 1 file changed, 37 insertions(+), 8 deletions(-)
meta/lib/oeqa/selftest/cases/usergrouptests.py +37 -8
@@ -5,11 +5,15 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, get_bb_var -import bb.utils +from oeqa.utils.commands import bitbake, get_bb_var, get_test_layer import os class UserGroupTests(OESelftestTestCase): + def _test_add_task_between_p_sysroot_and_package(self): + self.logger.info("Building useraddbadtask") + # This is expected to fail due to bug #14961 + self.assertTrue(bitbake(' useraddbadtask -C fetch')) + def test_group_from_dep_package(self): self.logger.info("Building creategroup2") bitbake(' creategroup2 creategroup1')
@@ -17,10 +21,35 @@ class UserGroupTests(OESelftestTestCase): self.logger.info("Packaging creategroup2") self.assertTrue(bitbake(' creategroup2 -c package')) - def _test_add_task_between_p_sysroot_and_package(self): - self.logger.info("Cleaning sstate for useraddbadtask") - #bitbake(' useraddbadtask -f -c cleansstate') - self.logger.info("Building useraddbadtask") - # This is expected to fail due to bug #14961 - self.assertTrue(bitbake(' useraddbadtask -C fetch')) + def test_static_useradd_from_dynamic(self): + metaselftestpath = get_test_layer() + self.logger.info("Building core-image-minimal to generate passwd/group file") + bitbake(' core-image-minimal') + self.logger.info("Setting up useradd-staticids") + repropassdir = os.path.join(metaselftestpath, "conf/include") + os.makedirs(repropassdir) + etcdir=os.path.join(os.path.join(os.path.join(get_bb_var("TMPDIR"), "work"), \ + os.path.join(get_bb_var("MACHINE").replace("-","_")+"-poky-linux", "core-image-minimal/1.0/rootfs/etc"))) + import shutil + shutil.copy(os.path.join(etcdir, "passwd"), os.path.join(repropassdir, "reproducable-passwd")) + shutil.copy(os.path.join(etcdir, "group"), os.path.join(repropassdir, "reproducable-group")) + # Copy the original local.conf + shutil.copyfile(os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf'), os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf.orig')) + + self.write_config("USERADDEXTENSION = \"useradd-staticids\"") + self.write_config("USERADD_ERROR_DYNAMIC ??= \"error\"") + self.write_config("USERADD_UID_TABLES += \"conf/include/reproducible-passwd\"") + self.write_config("USERADD_GID_TABLES += \"conf/include/reproducible-group\"") + self.logger.info("Rebuild with staticids") + bitbake(' core-image-minimal') + shutil.copyfile(os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf.orig'), os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf')) + self.logger.info("Rebuild without staticids") + bitbake(' core-image-minimal') + self.write_config("USERADDEXTENSION = \"useradd-staticids\"") + self.write_config("USERADD_ERROR_DYNAMIC ??= \"error\"") + self.write_config("USERADD_UID_TABLES += \"files/static-passwd\"") + self.write_config("USERADD_GID_TABLES += \"files/static-group\"") + self.logger.info("Rebuild with other staticids") + self.assertTrue(bitbake(' core-image-minimal')) +
Signature · 1 line
2.34.1
PATCH 3/3 variables.rst: Add USERADD_DEPENDS
In reply to Eilís 'pidge' Ní FhlannagáinAdding USERADD_DEPENDS to the variables document
--- documentation/ref-manual/variables.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
documentation/ref-manual/variables.rst +15 -0
@@ -9736,6 +9736,21 @@ system and gives an overview of their function and contents. ``meta-poky/conf/templates/default/local.conf.sample`` in the :term:`Source Directory`. + :term:`USERADD_DEPENDS` + Lists a recipe's USERADD dependencies. These are dependencies on + other recipes whose create users and groups via useradd.bbclass and are + needed by the recipe at build time. This variable is now required for + recipes that rely on other recipes for part of their useradd. + + As an example, consider a recipe ``foo``. Recipe ``bar`` creates a group + that foo requires for it's useradd tasks. In order to ensure that the + useradd tasks exist for foo, we add the the following assignment:: + + $USERADD_DEPENDS = "bar" + + The practical effect of this assignment is that bar's sysroot_setscene is + added to :term:`USERADDSETSCENEDEPS` variable. + :term:`USERADD_ERROR_DYNAMIC` If set to ``error``, forces the OpenEmbedded build system to produce an error if the user identification (``uid``) and group
Signature · 1 line
2.34.1
Re: [OE-core] [PATCH 1/3] useradd: Fix issues with useradd dependencies
In reply to Eilís 'pidge' Ní Fhlannagáinmeta/classes/useradd.bbclass SYSROOT_PREPROCESS_FUNCS += "${SYSROOTFUNC}" · line 184
Quoted 2 lines · Eilís 'pidge' Ní Fhlannagáin
meta/classes/useradd.bbclass +1
-1-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene" +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"
I'd like to clarify: do all these recent fixes mean that we can also add dynamic library dependencies of shadow-native into this list (it becomes an issue with latest versions of shadow currently not in oe-core)?
Specifically, this commit
doing
-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene" +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene \
+ pseudo-native:do_populate_sysroot_setscene \
+ shadow-native:do_populate_sysroot_setscene \
+ attr-native:do_populate_sysroot_setscene \
+ libbsd-native:do_populate_sysroot_setscene \
+ libmd-native:do_populate_sysroot_setscene \
+ ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
has been rejected by RP with the suggestion that shadow should be linked statically, but now I'm not sure if the argument still holds.
Alex
Re: [OE-core] [PATCH 1/3] useradd: Fix issues with useradd dependencies
Quoted 30 lines · Alexander Kanavin
meta/classes/useradd.bbclass SYSROOT_PREPROCESS_FUNCS += "${SYSROOTFUNC}" · line 184
Quoted 2 lines · Eilís 'pidge' Ní Fhlannagáin
meta/classes/useradd.bbclass +1
-1-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene" +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"I'd like to clarify: do all these recent fixes mean that we can also add dynamic library dependencies of shadow-native into this list (it becomes an issue with latest versions of shadow currently not in oe-core)?
Specifically, this commit
doing
-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene" +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene \
+ pseudo-native:do_populate_sysroot_setscene \
+ shadow-native:do_populate_sysroot_setscene \
+ attr-native:do_populate_sysroot_setscene \
+ libbsd-native:do_populate_sysroot_setscene \
+ libmd-native:do_populate_sysroot_setscene \
+ ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
Quoted 2 lines
has been rejected by RP with the suggestion that shadow should be linked statically, but now I'm not sure if the argument still holds.
After my recent change, we could in theory do this as bitbake might be able to make it work. It does run the risk of circular dependencies and these setscene inter-task dependencies are *horrible* in general from an sstate perspective though as my other comments about these dependencies leading to poor behaviour still stand.
To be clear about what you're saying above, you're saying that if any of those native dependencies are missing, we cannot use/install the sstate object. We have to defer installing that sstate until all the others are downloaded and available too.
My preference is still to statically link shadow to avoid this pretty horrible side effect.
Cheers,
Richard
Re: [PATCH 1/3] useradd: Fix issues with useradd dependencies
In reply to Eilís 'pidge' Ní FhlannagáinQuoted 6 lines
If recipe A requires the useradd actions of recipe B we need to ensure that recipe B is part of the recipe A dependancy chain. In order to do that, we introduce
USERADD_DEPENDS. This makes sure that thedo_populate_sysroot_setsceneof recipe B exists for recipe A in case of a missing TMPDIR. This requires changes made inrunqueue.pyby RP.
I was wondering if all required changes including these would be considered suitable to backport to e.g. kirkstone? Or am I going to have to look forward to scarthgap for these :)
Sincerely,
Jermain