Add checks for "host user contamination"
cover Add checks for "host user contamination"
From: Christopher Larson <[email protected]>
This adds a QA test, as well as a rootfs_ function for use in ROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by the same user/group as the user running bitbake. This is useful to catch stuff which is written outside of pseudo's control, for example.
The following changes since commit c38acd720b3f6ffbeb544063692eb471dada8593:
binconfig-disabled: write an message to stderr to help confused developers (2015-08-19 17:57:58 +0100)
are available in the git repository at:
git://github.com/kergoth/openembedded-core host-user-contaminated
https://github.com/kergoth/openembedded-core/tree/host-user-contaminated
Christopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: add rootfs_check_host_user_contaminated
meta/classes/image.bbclass | 14 ++++++++++++++ meta/classes/insane.bbclass | 38 +++++++++++++++++++++++++++++++++++++- 2 files changed, 51 insertions(+), 1 deletion(-)
Signature · 1 line
2.2.1
PATCH 1/3 insane.bbclass: handle tests which need fakeroot
From: Christopher Larson <[email protected]>
If any tests listed in FAKEROOT_QA are enabled (listed in ALL_QA), then run do_package_qa under fakeroot.
--- meta/classes/insane.bbclass | 8 ++++++++ 1 file changed, 8 insertions(+)
meta/classes/insane.bbclass +8 -0
@@ -38,6 +38,9 @@ ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \ split-strip packages-list pkgv-undefined var-undefined \ version-going-backwards expanded-d \ " +FAKEROOT_QA = "" +FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \ +enabled tests are listed here, the do_package_qa task will run under fakeroot." ALL_QA = "${WARN_QA} ${ERROR_QA}"
@@ -1210,6 +1213,11 @@ python () { for var in 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RCONFLICTS', 'RPROVIDES', 'RREPLACES', 'FILES', 'pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm', 'ALLOW_EMPTY': if d.getVar(var, False): issues.append(var) + + fakeroot_tests = d.getVar('FAKEROOT_QA', True).split() + if set(tests) & set(fakeroot_tests): + d.setVarFlag('do_package_qa', 'fakeroot', '1') + d.appendVarFlag('do_package_qa', 'depends', ' virtual/fakeroot-native:do_populate_sysroot') else: d.setVarFlag('do_package_qa', 'rdeptask', '') for i in issues:
Signature · 1 line
2.2.1
PATCH 2/3 insane.bbclass: add host-user-contaminated test
From: Christopher Larson <[email protected]>
- Add a test which checks for any paths outside of /home which are owned by the user running bitbake.
- Add the test to WARN_QA by default.
This test has been in meta-mentor for some time, and in our ERROR_QA for our builds, and has caught a number of issues for us.
--- meta/classes/insane.bbclass | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
meta/classes/insane.bbclass +30 -2
@@ -31,14 +31,14 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \ installed-vs-shipped compile-host-path install-host-path \ pn-overrides infodir build-deps file-rdeps \ unknown-configure-option symlink-to-sysroot multilib \ - invalid-pkgconfig \ + invalid-pkgconfig host-user-contaminated \ " ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \ perms dep-cmp pkgvarcheck perm-config perm-line perm-link \ split-strip packages-list pkgv-undefined var-undefined \ version-going-backwards expanded-d \ " -FAKEROOT_QA = "" +FAKEROOT_QA = "host-user-contaminated" FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \ enabled tests are listed here, the do_package_qa task will run under fakeroot."
@@ -950,6 +950,34 @@ def package_qa_check_expanded_d(path,name,d,elf,messages): sane = False return sane +HOST_USER_UID := "${@os.getuid()}" +HOST_USER_GID := "${@os.getgid()}" + +QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user" +def package_qa_check_host_user(path, name, d, elf, messages): + """Check for paths outside of /home which are owned by the user running bitbake.""" + + if not os.path.lexists(path): + return + + check_uid = int(d.getVar('HOST_USER_UID', True)) + check_gid = int(d.getVar('HOST_USER_GID', True)) + + dest = d.getVar('PKGDEST', True) + home = os.path.join(dest, 'home') + if path == home or path.startswith(home + os.sep): + return + + stat = os.lstat(path) + if stat.st_uid == check_uid: + messages["host-user-contaminated"] = "%s is owned by uid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_uid) + return False + + if stat.st_gid == check_gid: + messages["host-user-contaminated"] = "%s is owned by gid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_gid) + return False + return True + # The PACKAGE FUNC to scan each package python do_package_qa () { import subprocess
Signature · 1 line
2.2.1
PATCH 3/3 image.bbclass: add rootfs_check_host_user_contaminated
In reply to Christopher LarsonFrom: Christopher Larson <[email protected]>
This function is intended to be used in ROOTFS_POSTPROCESS_COMMAND, and checks for any paths outside of /home which are owned by the user running bitbake.
--- meta/classes/image.bbclass | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
meta/classes/image.bbclass +14 -0
@@ -458,6 +458,20 @@ rootfs_trim_schemas () { done } +rootfs_check_host_user_contaminated () { + contaminated="${WORKDIR}/host-user-contaminated.txt" + HOST_USER_UID="$(PSEUDO_UNLOAD=1 id -u)" + HOST_USER_GID="$(PSEUDO_UNLOAD=1 id -g)" + + find "${IMAGE_ROOTFS}" -wholename "${IMAGE_ROOTFS}/home" -prune \ + -user "$HOST_USER_UID" -o -group "$HOST_USER_GID" >"$contaminated" + + if [ -s "$contaminated" ]; then + echo "WARNING: Paths in the rootfs are owned by the same user or group as the user running bitbake. See the logfile for the specific paths." + cat "$contaminated" | sed "s,^, ," + fi +} + # Make any absolute links in a sysroot relative rootfs_sysroot_relativelinks () { sysroot-relativelinks.py ${SDK_OUTPUT}/${SDKTARGETSYSROOT}
Signature · 1 line
2.2.1
Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test
In reply to Christopher LarsonQuoted 5 lines · Christopher Larson
From: Christopher Larson <[email protected]>
- Add a test which checks for any paths outside of /home which are owned by the user running bitbake.
- Add the test toWARN_QAby default.
I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?
It would be nice to be able to bless my build directory and still benefit from this check.
Regards,
Joshua
Quoted 65 lines
This test has been in meta-mentor for some time, and in our
ERROR_QAfor our builds, and has caught a number of issues for us.Signed-off-by Christopher Larson <[email protected]>--- meta/classes/insane.bbclass | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)meta/classes/insane.bbclass +30
-2@@ -31,14 +31,14 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \ installed-vs-shipped compile-host-path install-host-path \ pn-overrides infodir build-deps file-rdeps \ unknown-configure-option symlink-to-sysroot multilib \ - invalid-pkgconfig \ + invalid-pkgconfig host-user-contaminated \ " ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \ perms dep-cmp pkgvarcheck perm-config perm-line perm-link \ split-strip packages-list pkgv-undefined var-undefined \ version-going-backwards expanded-d \ " -FAKEROOT_QA = "" +FAKEROOT_QA = "host-user-contaminated" FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \ enabled tests are listed here, the do_package_qa task will run under fakeroot."@@ -950,6 +950,34 @@ def package_qa_check_expanded_d(path,name,d,elf,messages): sane = False return sane +HOST_USER_UID := "${@os.getuid()}" +HOST_USER_GID := "${@os.getgid()}" + +QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user" +def package_qa_check_host_user(path, name, d, elf, messages): + """Check for paths outside of /home which are owned by the user running bitbake.""" + + if not os.path.lexists(path): + return + + check_uid = int(d.getVar('HOST_USER_UID', True)) + check_gid = int(d.getVar('HOST_USER_GID', True)) + + dest = d.getVar('PKGDEST', True) + home = os.path.join(dest, 'home') + if path == home or path.startswith(home + os.sep): + return + + stat = os.lstat(path) + if stat.st_uid == check_uid: + messages["host-user-contaminated"] = "%s is owned by uid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_uid) + return False + + if stat.st_gid == check_gid: + messages["host-user-contaminated"] = "%s is owned by gid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_gid) + return False + return True + # The PACKAGE FUNC to scan each package python do_package_qa () { import subprocess
Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test
Quoted 4 lines · Joshua Lock
I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?
The paths are prefixed with ${D} so pretend the commit log says "in packages".
Ross
Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test
Quoted 10 lines · Ross
Quoted 4 lines · Joshua Lock
I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?
The paths are prefixed with ${D} so pretend the commit log says "in packages".
Heh, indeed, it's a package QA test. do_install runs under pseudo, so any newly created files there, or files chown'd to root, will be fine. If, however, a recipe does a cp -a or so to install without doing a chown, you'll end up with files in your rootfs owned by the user that did the build -- not good.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test
Quoted 15 lines · Christopher Larson
Quoted 10 lines · Ross
Quoted 4 lines · Joshua Lock
I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?
The paths are prefixed with ${D} so pretend the commit log says "in packages".
Quoted 5 lines
Heh, indeed, it's a package QA test.
do_installruns under pseudo, so any newly created files there, or files chown'd to root, will be fine. If, however, a recipe does a cp -a or so to install without doing a chown, you'll end up with files in your rootfs owned by the user that did the build -- not good.
I can re-submit with that commit message clarification, if needed? I rather thought the fact that it was in insane.bbclass, not sanity.bbclass, carried the necessary implication.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test
Quoted 21 lines · Christopher Larson
On Wed, Aug 26, 2015 at 7:19 AM, Christopher Larson <[email protected] <mailto:[email protected]>> wrote:
On Wed, Aug 26, 2015 at 4:01 AM, Burton, Ross <[email protected] <mailto:[email protected]>> wrote:
On 26 August 2015 at 09:44, Joshua Lock <[email protected] <mailto:[email protected]>> wrote:
I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?
The paths are prefixed with ${D} so pretend the commit log says "in packages".
Quoted 5 lines
Heh, indeed, it's a package QA test.
do_installruns under pseudo, so any newly created files there, or files chown'd to root, will be fine. If, however, a recipe does a cp -a or so to install without doing a chown, you'll end up with files in your rootfs owned by the user that did the build -- not good.
Indeed. Thanks for taking the time to clarify.
Quoted 4 lines
I can re-submit with that commit message clarification, if needed? I rather thought the fact that it was in insane.bbclass, not sanity.bbclass, carried the necessary implication.
Personally I don't feel that's necessary - I should review with more care (and coffee).
Thanks for the offer though.
Regards,
Joshua
Quoted 13 lines · Christopher Larson
From: Christopher Larson <[email protected]>
This adds a QA test, as well as a
rootfs_function for use inROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by the same user/group as the user running bitbake. This is useful to catch stuff which is written outside of pseudo's control, for example.The following changes since commit
c38acd720b3f6ffbeb544063692eb471dada8593:binconfig-disabled: write an message to stderr to help confused developers (2015-08-19 17:57:58 +0100)
are available in the git repository at:
Quoted 7 lines
git://github.com/kergoth/openembedded-corehost-user-contaminated
https://github.com/kergoth/openembedded-core/tree/host-user-contaminatedChristopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: addrootfs_check_host_user_contaminated
On the autobuilder we found:
http://errors.yoctoproject.org/Errors/Details/16318/
which looks like a race against the packaging code and the creation of package manager files. Not sure why we don't see this in other cases...
Cheers,
Richard
Quoted 22 lines · Richard Purdie
Quoted 13 lines · Christopher Larson
From: Christopher Larson <[email protected]>
This adds a QA test, as well as a
rootfs_function for use inROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by the same user/group as the user running bitbake. This is useful to catch stuff which is written outside of pseudo's control, for example.The following changes since commit
c38acd720b3f6ffbeb544063692eb471dada8593:binconfig-disabled: write an message to stderr to help confused developers (2015-08-19 17:57:58 +0100)
are available in the git repository at:
Quoted 7 lines
git://github.com/kergoth/openembedded-corehost-user-contaminated
https://github.com/kergoth/openembedded-core/tree/host-user-contaminatedChristopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: addrootfs_check_host_user_contaminated
Quoted 6 lines
On the autobuilder we found:
http://errors.yoctoproject.org/Errors/Details/16318/
which looks like a race against the packaging code and the creation of package manager files. Not sure why we don't see this in other cases...
Also http://errors.yoctoproject.org/Errors/Details/16310/ which looks like the same issue (racing dpkg rather than opkg).
Cheers,
Richard
Quoted 30 lines · Richard Purdie
Quoted 5 lines · Richard Purdie
Quoted 4 lines · Christopher Larson
From: Christopher Larson <[email protected]>
This adds a QA test, as well as a
rootfs_function for use inROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by thesame
Quoted 1 line
Quoted 1 line
user/group as the user running bitbake. This is useful to catch stuff
which is
Quoted 3 lines
Quoted 3 lines
written outside of pseudo's control, for example.
The following changes since commit
c38acd720b3f6ffbeb544063692eb471dada8593:Quoted 2 lines
Quoted 2 lines
binconfig-disabled: write an message to stderr to help confused
developers (2015-08-19 17:57:58 +0100)
Quoted 5 lines
Quoted 5 lines
are available in the git repository at:
git://github.com/kergoth/openembedded-corehost-user-contaminatedhttps://github.com/kergoth/openembedded-core/tree/host-user-contaminated
Quoted 8 lines
Quoted 5 lines
Christopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: addrootfs_check_host_user_contaminatedOn the autobuilder we found:
Quoted 7 lines
Quoted 4 lines
http://errors.yoctoproject.org/Errors/Details/16318/
which looks like a race against the packaging code and the creation of package manager files. Not sure why we don't see this in other cases...
Also http://errors.yoctoproject.org/Errors/Details/16310/ which looks like the same issue (racing dpkg rather than opkg).
Ah, indeed, sorry about that, I must have gotten lucky with our builds thus far. I'll re-submit, ignoring ENOENT.
Signature · 5 lines
Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics