Revert cleaning of users/groups
v2 cover Revert cleaning of users/groups
About a week ago the "solution" to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=9262 was accepted into OE-Core. This is supposed to solve the problem that users/groups are not removed from the sysroot when the recipe is cleansstated. The solution, however, totally fails to handle the case where multiple recipes have installed the same user/group causing the build to fail all over the place.
We have a lot of recipes that installs users and groups, many of which are created by multiple recipes. An example of how this change broke for us:
Recipe libwsd creates the wsd group. Recipe wsd creates the wsd group and the wsd user which has the wsd group as it primary group. When the libwsd recipe needs to be rebuilt the following error occurs:
groupdel: cannot remove the primary group of user 'wsd'
and the build aborts.
If a solution like this is supposed to be implemented it needs to keep track of all recipes that install a particular user/group and only when all of them have been removed can the user/group be removed. Until that has been solved, the current solution needs to be reverted...
PATCHv2: Correct a typo in the commit message.
//Peter
The following changes since commit 5602f64ab8c3654737273c0e7ac1c966fa8afb6a:
archiver: Fix ASSUME_PROVIDED issues (2016-04-11 09:03:58 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib pkj/clean_users
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/clean_users
Peter Kjellerstedt (1):
Revert "useradd.bbclass: remove user/group created by the package in
clean* task"
meta/classes/sstate.bbclass | 5 ----- meta/classes/useradd.bbclass | 29 ----------------------------- 2 files changed, 34 deletions(-)
Signature · 1 line
2.1.0
PATCH v2 1/1 Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Removal of users/group when cleansstating a recipe as implemented here totally breaks when multiple recipes install the same user/groups.
This reverts commit b5304ce438666a7418746f4ddd32703ae3188089.
--- meta/classes/sstate.bbclass | 5 ----- meta/classes/useradd.bbclass | 29 ----------------------------- 2 files changed, 34 deletions(-)
meta/classes/sstate.bbclass +0 -5
@@ -51,7 +51,6 @@ SSTATEPREINSTFUNCS = "" SSTATEPOSTUNPACKFUNCS = "sstate_hardcode_path_unpack" SSTATEPOSTINSTFUNCS = "" EXTRA_STAGING_FIXMES ?= "" -SSTATECLEANFUNCS = "" # Check whether sstate exists for tasks that support sstate and are in the # locked signatures file.
@@ -451,10 +450,6 @@ def sstate_clean(ss, d): stfile.endswith(rm_nohash): oe.path.remove(stfile) - # Removes the users/groups created by the package - for cleanfunc in (d.getVar('SSTATECLEANFUNCS', True) or '').split(): - bb.build.exec_func(cleanfunc, d) - sstate_clean[vardepsexclude] = "SSTATE_MANFILEPREFIX" CLEANFUNCS += "sstate_cleanall"
meta/classes/useradd.bbclass +0 -29
@@ -127,35 +127,6 @@ useradd_sysroot_sstate () { fi } -userdel_sysroot_sstate () { -if test "x${STAGING_DIR_TARGET}" != "x"; then - if [ "${BB_CURRENTTASK}" = "configure" -o "${BB_CURRENTTASK}" = "clean" ]; then - export PSEUDO="${FAKEROOTENV} PSEUDO_LOCALSTATEDIR=${STAGING_DIR_TARGET}${localstatedir}/pseudo ${STAGING_DIR_NATIVE}${bindir}/pseudo" - OPT="--root ${STAGING_DIR_TARGET}" - - # Remove groups and users defined for package - GROUPADD_PARAM="${@get_all_cmd_params(d, 'groupadd')}" - USERADD_PARAM="${@get_all_cmd_params(d, 'useradd')}" - - if test "x`echo $USERADD_PARAM | tr -d '[:space:]'`" != "x"; then - user=`echo "$USERADD_PARAM" | cut -d ';' -f 1 | awk '{ print $NF }'` - perform_userdel "${STAGING_DIR_TARGET}" "$OPT $user" - fi - - if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then - group=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1 | awk '{ print $NF }'` - perform_groupdel "${STAGING_DIR_TARGET}" "$OPT $group" - fi - - fi -fi -} - -SSTATECLEANFUNCS = "userdel_sysroot_sstate" -SSTATECLEANFUNCS_class-cross = "" -SSTATECLEANFUNCS_class-native = "" -SSTATECLEANFUNCS_class-nativesdk = "" - do_install[prefuncs] += "${SYSROOTFUNC}" SYSROOTFUNC = "useradd_sysroot" SYSROOTFUNC_class-cross = ""
Signature · 1 line
2.1.0
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 6 lines · Peter Kjellerstedt
Removal of users/group when cleansstating a recipe as implemented here totally breaks when multiple recipes install the same user/groups.
This reverts commit
b5304ce438666a7418746f4ddd32703ae3188089.Signed-off-by Peter Kjellerstedt <[email protected]>
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
In reply to Peter KjellerstedtQuoted 11 lines · Peter Kjellerstedt
Removal of users/group when cleansstating a recipe as implemented here
totally breaks when multiple recipes install the same user/groups.This reverts commit
b5304ce438666a7418746f4ddd32703ae3188089.Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/sstate.bbclass | 5 ----- meta/classes/useradd.bbclass | 29 ----------------------------- 2 files changed, 34 deletions(-)
This is one of these situations where we can't win.
Without this code, there is no way to clean up these users out the sysroot. Having to tell people to "wipe TMPDIR" when the parameters used to create a user change for example is rather bad. It also has bad implications for build determinism and the expectation that a "clean" undoes the changes a recipe makes.
Have we ever documented we support the same user being created by multiple recipes?
I'm open to proposals about how to fix this but an outright revert doesn't seem like the right thing to do. Making it optional so you can disable it for your multiple recipes case would perhaps be a better option for example.
Cheers,
Richard
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 21 lines · Richard Purdie
Quoted 11 lines · Peter Kjellerstedt
Removal of users/group when cleansstating a recipe as implemented here
totally breaks when multiple recipes install the same user/groups.This reverts commit
b5304ce438666a7418746f4ddd32703ae3188089.Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/sstate.bbclass | 5 ----- meta/classes/useradd.bbclass | 29 ----------------------------- 2 files changed, 34 deletions(-)This is one of these situations where we can't win.
Without this code, there is no way to clean up these users out the sysroot. Having to tell people to "wipe TMPDIR" when the parameters used to create a user change for example is rather bad. It also has bad implications for build determinism and the expectation that a "clean" undoes the changes a recipe makes.
Quoted 7 lines
Have we ever documented we support the same user being created by multiple recipes?
I'm open to proposals about how to fix this but an outright revert doesn't seem like the right thing to do. Making it optional so you can disable it for your multiple recipes case would perhaps be a better option for example.
I am not against the change by the timing; it should be merged in master as soon it is open for 2.2 work.
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 8 lines
-----Original Message-----
From: Otavio Salvador [mailto:[email protected]]
Sent: den 12 april 2016 18:36
To: Richard Purdie
Cc: Peter Kjellerstedt; Patches and discussions about the oe-core layer
Subject: Re: [OE-core] [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 21 lines
Quoted 19 lines · Richard Purdie
Quoted 10 lines · Peter Kjellerstedt
Removal of users/group when cleansstating a recipe as implemented here totally breaks when multiple recipes install the same user/groups.
This reverts commit
b5304ce438666a7418746f4ddd32703ae3188089.Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/sstate.bbclass | 5 ----- meta/classes/useradd.bbclass | 29 ----------------------------- 2 files changed, 34 deletions(-)This is one of these situations where we can't win.
Without this code, there is no way to clean up these users out the sysroot. Having to tell people to "wipe TMPDIR" when the parameters used to create a user change for example is rather bad. It also has bad implications for build determinism and the expectation that a "clean" undoes the changes a recipe makes.
Well, build determinism is also broken when it comes to creating users and groups if two recipes tries to create the same user (or group) with different definitions, since the first recipe that happens to be installed wins...
In our case we have solved this by using the useradd-staticids.bbclass, which allows us to have common definitions for the users, in addition to only use GROUPMEMS_PARAM to add users to groups. But there should really be some QA warning/error to catch this from happening...
Quoted 2 lines
Quoted 2 lines
Have we ever documented we support the same user being created by multiple recipes?
I am pretty sure you have not documented that it is NOT supported... Because if it was not supported, then I would have expected errors if two recipes tried to install the same user/group. And I cannot see the logic behind not allowing it (except because it makes it hard to uninstall users and groups), because if a package needs a user/group, then of course it should be allowed to make sure the user/group exists...
Quoted 4 lines
Quoted 4 lines
I'm open to proposals about how to fix this but an outright revert doesn't seem like the right thing to do. Making it optional so you can disable it for your multiple recipes case would perhaps be a better option for example.
Making it a configuration option does not sound like a good idea because having multiple recipes install the same user/group can lead to build errors and that is not something you should need to configure around.
One way to solve the problem of both creating users/groups and being able to remove them in deterministic ways would be to only allow them to be created from special user/group recipes. Then the normal dependency handling would make sure they are installed and removed as needed. The drawback of course would be all the extra packages needed to create users and groups (which in our case would be a considerable number of packages).
An alternative solution would be to create package specific files somewhere in the sysroot which keep track of what users and groups a particular package has created, i.e., something like "users/systemd-timesync/systemd" which would be created by the systemd package when it creates the systemd-timesync user. That way it would be easy to only remove the systemd-timesync user when "users/systemd-timesync" is empty. A benefit of this solution is that it would only require changes to useradd.bbclass so it can be implemented without having to change any recipes. A minor drawback is that it would not solve the problem of different recipes creating the same user differently.
Quoted 2 lines
I am not against the change by the timing; it should be merged in master as soon it is open for 2.2 work.
I have to agree with this. Above are two solutions on how to make this work, but doing it at this stage of the release schedule is probably asking a bit much (especially the first solution)...
//Peter
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
I am pretty frustrated with this thread. The reasons are perhaps not immediately obvious though.
The issue is that there are only a limited number of people who actually dive in and try and fix some of the underlying "core architecture" bugs. There is what I believe to be a pretty good patch here which does fix real world issues which have been reported into the bugzilla (its related to at least two bug reports). As such it has been seen as a bugfix. Its now clear it does have some side effects which weren't envisaged, some causing issues for a small number of meta-oe recipes, the others breaking a companies internal code.
Otavio wants it deferred to 2.2, Peter wants it abandoned entirely.
If I revert this, Peter is then happy and has zero incentive to do anything further. The pressure is still on the reopened bugs to try and fix this somehow and falls back to the usual suspects. There is a real world usability problem there.
In a single isolated case, fine, we'd figure a way through this. I think I'm so frustrated as we see this all the time. Making a change to the core architecture is hard and gets ever harder, then we wonder why we don't have contributors. Part of this is having so many different workflows and corner cases.
I have pushed very hard to have more test cases, then its easier to determine if a patch causes regressions. Again though, few people are contributing to them outside the usual suspects.
I'm therefore starting to think the correct answer to this thread is simply this:
The patch doesn't break any of the current regression tests. If you have use cases like this you care about, you really should make sure we have test coverage for them, else you run the risk of exactly the problem we have here.
I haven't honestly decided what to do but this latter conclusion is very tempting from where I'm sitting...
Cheers,
Richard
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 20 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 13 april 2016 13:05
To: Peter Kjellerstedt; Otavio Salvador
Cc: OE Core ([email protected])
Subject: Re: [OE-core] [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"I am pretty frustrated with this thread. The reasons are perhaps not immediately obvious though.
The issue is that there are only a limited number of people who actually dive in and try and fix some of the underlying "core architecture" bugs. There is what I believe to be a pretty good patch here which does fix real world issues which have been reported into the bugzilla (its related to at least two bug reports). As such it has been seen as a bugfix. Its now clear it does have some side effects which weren't envisaged, some causing issues for a small number of meta-oe recipes, the others breaking a companies internal code.
Quoted 6 lines
Otavio wants it deferred to 2.2, Peter wants it abandoned entirely.
If I revert this, Peter is then happy and has zero incentive to do anything further. The pressure is still on the reopened bugs to try and fix this somehow and falls back to the usual suspects. There is a real world usability problem there.
Hold your horses. I definitely see the problem the change tried to address as one that needs to be fixed, and I am already looking at how to solve this properly (currently based on my second suggested solution). However, I do not know if I can fix it in time for Krogoth. Which is why I agree with Otavio that the change was introduced too late in the process, especially as it causes breakage for existing users.
Quoted 9 lines
In a single isolated case, fine, we'd figure a way through this. I think I'm so frustrated as we see this all the time. Making a change to the core architecture is hard and gets ever harder, then we wonder why we don't have contributors. Part of this is having so many different workflows and corner cases.
I have pushed very hard to have more test cases, then its easier to determine if a patch causes regressions. Again though, few people are contributing to them outside the usual suspects.
Here I must show my lack of knowledge. How and where should I go about adding a regression test that verifies the support for that multiple recipes can add the same user/group? Since this does not test a specific recipe, but rather a part of the build framework, I do not know if, e.g., ptest is applicable (of which I have no experience either).
Quoted 8 lines
I'm therefore starting to think the correct answer to this thread is simply this:
The patch doesn't break any of the current regression tests. If you have use cases like this you care about, you really should make sure we have test coverage for them, else you run the risk of exactly the problem we have here.
Quoted 6 lines
I haven't honestly decided what to do but this latter conclusion is very tempting from where I'm sitting...
Cheers,
Richard
//Peter
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Hi Peter,
Quoted 27 lines · Peter Kjellerstedt
Quoted 20 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 13 april 2016 13:05
To: Peter Kjellerstedt; Otavio Salvador
Cc: OE Core ([email protected])
Subject: Re: [OE-core] [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"I am pretty frustrated with this thread. The reasons are perhaps not immediately obvious though.
The issue is that there are only a limited number of people who actually dive in and try and fix some of the underlying "core architecture" bugs. There is what I believe to be a pretty good patch here which does fix real world issues which have been reported into the bugzilla (its related to at least two bug reports). As such it has been seen as a bugfix. Its now clear it does have some side effects which weren't envisaged, some causing issues for a small number of meta-oe recipes, the others breaking a companies internal code.
Quoted 6 lines
Otavio wants it deferred to 2.2, Peter wants it abandoned entirely.
If I revert this, Peter is then happy and has zero incentive to do anything further. The pressure is still on the reopened bugs to try and fix this somehow and falls back to the usual suspects. There is a real world usability problem there.
Quoted 7 lines
Hold your horses. I definitely see the problem the change tried to address as one that needs to be fixed, and I am already looking at how to solve this properly (currently based on my second suggested solution). However, I do not know if I can fix it in time for Krogoth. Which is why I agree with Otavio that the change was introduced too late in the process, especially as it causes breakage for existing users.
As the author of that patch, I am responsible for that regression and would like to assist in fixing that particular problem. As you can see, I had no option to test your particular scenario as that recipe was not part of my oe-core "world" build test. RP has already sent a patch to fix the "cyrus-sasl" recipe which had some troubles with this change.
Quoted 10 lines
Quoted 9 lines
In a single isolated case, fine, we'd figure a way through this. I think I'm so frustrated as we see this all the time. Making a change to the core architecture is hard and gets ever harder, then we wonder why we don't have contributors. Part of this is having so many different workflows and corner cases.
I have pushed very hard to have more test cases, then its easier to determine if a patch causes regressions. Again though, few people are contributing to them outside the usual suspects.
Quoted 6 lines
Here I must show my lack of knowledge. How and where should I go about adding a regression test that verifies the support for that multiple recipes can add the same user/group? Since this does not test a specific recipe, but rather a part of the build framework, I do not know if, e.g., ptest is applicable (of which I have no experience either).
Just want to let you know that I am interested in working together to fix this one. As per my understanding, you have a library and an application, which depends on that library. Both these packages creates the same user.
At this point, a package manager (eg: smart) doesn't remove the users when we remove the package. However, I am not sure about the situation where a package manager removes the users added by a package when we remove it from the target device. In that case,wouldn't the removal of application, remove the user which the library depends on ?
Best Regards,
Maxin
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
In reply to Peter KjellerstedtQuoted 36 lines · Peter Kjellerstedt
Quoted 27 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 13 april 2016 13:05
To: Peter Kjellerstedt; Otavio Salvador
Cc: OE Core ([email protected])
Subject: Re: [OE-core] [PATCHv2 1/1] Revert "useradd.bbclass:
remove
user/group created by the package in clean* task"I am pretty frustrated with this thread. The reasons are perhaps not
immediately obvious though.The issue is that there are only a limited number of people who actually dive in and try and fix some of the underlying "core architecture" bugs. There is what I believe to be a pretty good patch
here which does fix real world issues which have been reported into the
bugzilla (its related to at least two bug reports). As such it has been
seen as a bugfix. Its now clear it does have some side effects which
weren't envisaged, some causing issues for a small number of meta -oe
recipes, the others breaking a companies internal code.Quoted 8 lines
Otavio wants it deferred to 2.2, Peter wants it abandoned entirely.
If I revert this, Peter is then happy and has zero incentive to do anything further. The pressure is still on the reopened bugs to try and
fix this somehow and falls back to the usual suspects. There is a real
world usability problem there.
Quoted 8 lines
Hold your horses. I definitely see the problem the change tried to address as one that needs to be fixed, and I am already looking at how to solve this properly (currently based on my second suggested solution). However, I do not know if I can fix it in time for Krogoth.
Which is why I agree with Otavio that the change was introduced too late in the process, especially as it causes breakage for existing users.
There is a comparatively neat way we could use pkgdata to track the provider of a given user, specifically:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip
&id=5cd646ea185eaafaa341f26310f2eddc75766175
The above is a quick patch I've just put together which illustrates what could be done so that the user gets better warnings about conflicting users. It needs cleaning up but thought it worth sharing now as if might give some ideas.
This is in keeping with how bitbake detects multiple providers of the same thing.
Quoted 7 lines
Here I must show my lack of knowledge. How and where should I go about
adding a regression test that verifies the support for that multiple recipes can add the same user/group? Since this does not test a specific recipe, but rather a part of the build framework, I do not know if, e.g., ptest is applicable (of which I have no experience either).
oe-selftest would be the place to add something like this, see the
/meta/lib/oeqa/selftest directory. We could add some test useradd
recipes to meta-selftest.
Cheers,
Richard
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 7 lines · Richard Purdie
There is a comparatively neat way we could use pkgdata to track the provider of a given user, specifically:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/w
ip
&id=5cd646ea185eaafaa341f26310f2eddc75766175
Quoted 7 lines
The above is a quick patch I've just put together which illustrates what could be done so that the user gets better warnings about conflicting users. It needs cleaning up but thought it worth sharing now as if might give some ideas.
This is in keeping with how bitbake detects multiple providers of the same thing.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip
&id=a362e6da0eec1f7a7d4db19d47f16b1c729562ba
is an improved version.
Quoted 9 lines
Quoted 8 lines
Here I must show my lack of knowledge. How and where should I go about
adding a regression test that verifies the support for that multiple
recipes can add the same user/group? Since this does not test a specific recipe, but rather a part of the build framework, I do not know if, e.g., ptest is applicable (of which I have no experience either).
Quoted 3 lines
oe-selftest would be the place to add something like this, see the
/meta/lib/oeqa/selftest directory. We could add some test useradd
recipes to meta-selftest.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wip
&id=23134a7d837bc7985ba8e7069d405913c2055a04
is a really simplistic new selftest for useradd. It does however test the bug the original patch fixes. There are clearly a much wider range of things it could test, I just wanted to show this isn't hard.
Cheers,
Richard
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 41 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 14 april 2016 12:41
To: Peter Kjellerstedt; Otavio Salvador
Cc: OE Core ([email protected])
Subject: Re: [OE-core] [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"Quoted 5 lines · Richard Purdie
There is a comparatively neat way we could use pkgdata to track the provider of a given user, specifically:
Quoted 7 lines
The above is a quick patch I've just put together which illustrates what could be done so that the user gets better warnings about conflicting users. It needs cleaning up but thought it worth sharing now as if might give some ideas.
This is in keeping with how bitbake detects multiple providers of the same thing.
is an improved version.
Quoted 9 lines
Quoted 8 lines
Here I must show my lack of knowledge. How and where should I go about
adding a regression test that verifies the support for that multiple
recipes can add the same user/group? Since this does not test a specific recipe, but rather a part of the build framework, I do not know if, e.g., ptest is applicable (of which I have no experience either).Quoted 3 lines
oe-selftest would be the place to add something like this, see the
/meta/lib/oeqa/selftest directory. We could add some test useradd
recipes to meta-selftest.
Quoted 7 lines
is a really simplistic new selftest for useradd. It does however test the bug the original patch fixes. There are clearly a much wider range of things it could test, I just wanted to show this isn't hard.
Cheers,
Richard
Ok, this looks like a good start for me to continue working on.
What is the easiest way to execute the useradd selftest?
//Peter
Re: [PATCHv2 1/1] Revert "useradd.bbclass: remove user/group created by the package in clean* task"
Quoted 3 lines · Peter Kjellerstedt
Ok, this looks like a good start for me to continue working on.
What is the easiest way to execute the useradd selftest?
oe-selftest --run-tests useradd.Useradd.test_useradd
Cheers,
Richard