Skip to contentSkip to conversation

Add checks for "host user contamination"

PATCH Superseded Christopher Larson 5 participants 12 messages Aug 24, 2015 – Sep 1, 2015

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Add checks for "host user contamination"

From: Christopher Larson <[email protected]>

This adds a QA test, as well as a rootfs_ function for use in ROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by the same user/group as the user running bitbake. This is useful to catch stuff which is written outside of pseudo's control, for example.

The following changes since commit c38acd720b3f6ffbeb544063692eb471dada8593:

binconfig-disabled: write an message to stderr to help confused developers (2015-08-19 17:57:58 +0100)

are available in the git repository at:

git://github.com/kergoth/openembedded-core host-user-contaminated
https://github.com/kergoth/openembedded-core/tree/host-user-contaminated

Christopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: add rootfs_check_host_user_contaminated

 meta/classes/image.bbclass  | 14 ++++++++++++++
 meta/classes/insane.bbclass | 38 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 51 insertions(+), 1 deletion(-)
Signature · 1 line
2.2.1
#1 lore raw

PATCH 1/3 insane.bbclass: handle tests which need fakeroot

From: Christopher Larson <[email protected]>

If any tests listed in FAKEROOT_QA are enabled (listed in ALL_QA), then run do_package_qa under fakeroot.

---
 meta/classes/insane.bbclass | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/classes/insane.bbclass +8 -0
index 9c05c86..cd773b7 100644
@@ -38,6 +38,9 @@ ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             split-strip packages-list pkgv-undefined var-undefined \
             version-going-backwards expanded-d \
             "
+FAKEROOT_QA = ""
+FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \
+enabled tests are listed here, the do_package_qa task will run under fakeroot."
 
 ALL_QA = "${WARN_QA} ${ERROR_QA}"
 
@@ -1210,6 +1213,11 @@ python () {
         for var in 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RCONFLICTS', 'RPROVIDES', 'RREPLACES', 'FILES', 'pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm', 'ALLOW_EMPTY':
             if d.getVar(var, False):
                 issues.append(var)
+
+        fakeroot_tests = d.getVar('FAKEROOT_QA', True).split()
+        if set(tests) & set(fakeroot_tests):
+            d.setVarFlag('do_package_qa', 'fakeroot', '1')
+            d.appendVarFlag('do_package_qa', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
     else:
         d.setVarFlag('do_package_qa', 'rdeptask', '')
     for i in issues:
Signature · 1 line
2.2.1
#2 lore raw

PATCH 2/3 insane.bbclass: add host-user-contaminated test

From: Christopher Larson <[email protected]>

- Add a test which checks for any paths outside of /home which are owned by the user running bitbake.
- Add the test to WARN_QA by default.

This test has been in meta-mentor for some time, and in our ERROR_QA for our builds, and has caught a number of issues for us.

---
 meta/classes/insane.bbclass | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
meta/classes/insane.bbclass +30 -2
index cd773b7..aec9800 100644
@@ -31,14 +31,14 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
             installed-vs-shipped compile-host-path install-host-path \
             pn-overrides infodir build-deps file-rdeps \
             unknown-configure-option symlink-to-sysroot multilib \
-            invalid-pkgconfig \
+            invalid-pkgconfig host-user-contaminated \
             "
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
             split-strip packages-list pkgv-undefined var-undefined \
             version-going-backwards expanded-d \
             "
-FAKEROOT_QA = ""
+FAKEROOT_QA = "host-user-contaminated"
 FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \
 enabled tests are listed here, the do_package_qa task will run under fakeroot."
 
@@ -950,6 +950,34 @@ def package_qa_check_expanded_d(path,name,d,elf,messages):
                         sane = False
     return sane
 
+HOST_USER_UID := "${@os.getuid()}"
+HOST_USER_GID := "${@os.getgid()}"
+
+QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user"
+def package_qa_check_host_user(path, name, d, elf, messages):
+    """Check for paths outside of /home which are owned by the user running bitbake."""
+
+    if not os.path.lexists(path):
+        return
+
+    check_uid = int(d.getVar('HOST_USER_UID', True))
+    check_gid = int(d.getVar('HOST_USER_GID', True))
+
+    dest = d.getVar('PKGDEST', True)
+    home = os.path.join(dest, 'home')
+    if path == home or path.startswith(home + os.sep):
+        return
+
+    stat = os.lstat(path)
+    if stat.st_uid == check_uid:
+        messages["host-user-contaminated"] = "%s is owned by uid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_uid)
+        return False
+
+    if stat.st_gid == check_gid:
+        messages["host-user-contaminated"] = "%s is owned by gid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_gid)
+        return False
+    return True
+
 # The PACKAGE FUNC to scan each package
 python do_package_qa () {
     import subprocess
Signature · 1 line
2.2.1
#8 lore raw

PATCH 3/3 image.bbclass: add rootfs_check_host_user_contaminated

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

This function is intended to be used in ROOTFS_POSTPROCESS_COMMAND, and checks for any paths outside of /home which are owned by the user running bitbake.

---
 meta/classes/image.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
meta/classes/image.bbclass +14 -0
index ff2ed0d..d1cd9fd 100644
@@ -458,6 +458,20 @@ rootfs_trim_schemas () {
 	done
 }
 
+rootfs_check_host_user_contaminated () {
+	contaminated="${WORKDIR}/host-user-contaminated.txt"
+	HOST_USER_UID="$(PSEUDO_UNLOAD=1 id -u)"
+	HOST_USER_GID="$(PSEUDO_UNLOAD=1 id -g)"
+
+	find "${IMAGE_ROOTFS}" -wholename "${IMAGE_ROOTFS}/home" -prune \
+	    -user "$HOST_USER_UID" -o -group "$HOST_USER_GID" >"$contaminated"
+
+	if [ -s "$contaminated" ]; then
+		echo "WARNING: Paths in the rootfs are owned by the same user or group as the user running bitbake. See the logfile for the specific paths."
+		cat "$contaminated" | sed "s,^,  ,"
+	fi
+}
+
 # Make any absolute links in a sysroot relative
 rootfs_sysroot_relativelinks () {
 	sysroot-relativelinks.py ${SDK_OUTPUT}/${SDKTARGETSYSROOT}
Signature · 1 line
2.2.1
#3 lore raw

Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test

In reply to Christopher Larson
Quoted 5 lines · Christopher Larson

From: Christopher Larson <[email protected]>

- Add a test which checks for any paths outside of /home which are owned by the user running bitbake.
- Add the test to WARN_QA by default.

I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?

It would be nice to be able to bless my build directory and still benefit from this check.

Regards,

Joshua

Quoted 65 lines

This test has been in meta-mentor for some time, and in our ERROR_QA for our builds, and has caught a number of issues for us.

---
  meta/classes/insane.bbclass | 32 ++++++++++++++++++++++++++++++--
  1 file changed, 30 insertions(+), 2 deletions(-)
meta/classes/insane.bbclass +30 -2
index cd773b7..aec9800 100644
@@ -31,14 +31,14 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
              installed-vs-shipped compile-host-path install-host-path \
              pn-overrides infodir build-deps file-rdeps \
              unknown-configure-option symlink-to-sysroot multilib \
-            invalid-pkgconfig \
+            invalid-pkgconfig host-user-contaminated \
              "
  ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
              perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
              split-strip packages-list pkgv-undefined var-undefined \
              version-going-backwards expanded-d \
              "
-FAKEROOT_QA = ""
+FAKEROOT_QA = "host-user-contaminated"
  FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \
  enabled tests are listed here, the do_package_qa task will run under fakeroot."
 
@@ -950,6 +950,34 @@ def package_qa_check_expanded_d(path,name,d,elf,messages):
                          sane = False
      return sane
 
+HOST_USER_UID := "${@os.getuid()}"
+HOST_USER_GID := "${@os.getgid()}"
+
+QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user"
+def package_qa_check_host_user(path, name, d, elf, messages):
+    """Check for paths outside of /home which are owned by the user running bitbake."""
+
+    if not os.path.lexists(path):
+        return
+
+    check_uid = int(d.getVar('HOST_USER_UID', True))
+    check_gid = int(d.getVar('HOST_USER_GID', True))
+
+    dest = d.getVar('PKGDEST', True)
+    home = os.path.join(dest, 'home')
+    if path == home or path.startswith(home + os.sep):
+        return
+
+    stat = os.lstat(path)
+    if stat.st_uid == check_uid:
+        messages["host-user-contaminated"] = "%s is owned by uid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_uid)
+        return False
+
+    if stat.st_gid == check_gid:
+        messages["host-user-contaminated"] = "%s is owned by gid %d, which is the same as the user running bitbake. This may be due to host contamination" % (path, check_gid)
+        return False
+    return True
+
  # The PACKAGE FUNC to scan each package
  python do_package_qa () {
      import subprocess
#4 lore raw

Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test

Quoted 4 lines · Joshua Lock

I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?

The paths are prefixed with ${D} so pretend the commit log says "in packages".

Ross

#5 lore raw

Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test

Quoted 10 lines · Ross
Quoted 4 lines · Joshua Lock

I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?

The paths are prefixed with ${D} so pretend the commit log says "in packages".

Heh, indeed, it's a package QA test. do_install runs under pseudo, so any newly created files there, or files chown'd to root, will be fine. If, however, a recipe does a cp -a or so to install without doing a chown, you'll end up with files in your rootfs owned by the user that did the build -- not good.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#6 lore raw

Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test

Quoted 15 lines · Christopher Larson
Quoted 10 lines · Ross
Quoted 4 lines · Joshua Lock

I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?

The paths are prefixed with ${D} so pretend the commit log says "in packages".

Quoted 5 lines

Heh, indeed, it's a package QA test. do_install runs under pseudo, so any newly created files there, or files chown'd to root, will be fine. If, however, a recipe does a cp -a or so to install without doing a chown, you'll end up with files in your rootfs owned by the user that did the build -- not good.

I can re-submit with that commit message clarification, if needed? I rather thought the fact that it was in insane.bbclass, not sanity.bbclass, carried the necessary implication.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#7 lore raw

Re: [PATCH 2/3] insane.bbclass: add host-user-contaminated test

Quoted 21 lines · Christopher Larson

On Wed, Aug 26, 2015 at 7:19 AM, Christopher Larson <[email protected] <mailto:[email protected]>> wrote:

On Wed, Aug 26, 2015 at 4:01 AM, Burton, Ross <[email protected] <mailto:[email protected]>> wrote:

On 26 August 2015 at 09:44, Joshua Lock <[email protected] <mailto:[email protected]>> wrote:

I do all of my builds on a separate partition in a directory hierarchy which is owned by my user - if I'm understanding this correctly I'll get QA WARNINGS for all of my builds with this change?

The paths are prefixed with ${D} so pretend the commit log says "in packages".

Quoted 5 lines

Heh, indeed, it's a package QA test. do_install runs under pseudo, so any newly created files there, or files chown'd to root, will be fine. If, however, a recipe does a cp -a or so to install without doing a chown, you'll end up with files in your rootfs owned by the user that did the build -- not good.

Indeed. Thanks for taking the time to clarify.

Quoted 4 lines

I can re-submit with that commit message clarification, if needed? I rather thought the fact that it was in insane.bbclass, not sanity.bbclass, carried the necessary implication.

Personally I don't feel that's necessary - I should review with more care (and coffee).

Thanks for the offer though.

Regards,

Joshua

#9 lore raw
In reply to Christopher Larson
Quoted 13 lines · Christopher Larson

From: Christopher Larson <[email protected]>

This adds a QA test, as well as a rootfs_ function for use in ROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by the same user/group as the user running bitbake. This is useful to catch stuff which is written outside of pseudo's control, for example.

The following changes since commit c38acd720b3f6ffbeb544063692eb471dada8593:

binconfig-disabled: write an message to stderr to help confused developers (2015-08-19 17:57:58 +0100)

are available in the git repository at:

Quoted 7 lines

git://github.com/kergoth/openembedded-core host-user-contaminated
https://github.com/kergoth/openembedded-core/tree/host-user-contaminated

Christopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: add rootfs_check_host_user_contaminated

On the autobuilder we found:

http://errors.yoctoproject.org/Errors/Details/16318/

which looks like a race against the packaging code and the creation of package manager files. Not sure why we don't see this in other cases...

Cheers,

Richard

#10 lore raw
Quoted 22 lines · Richard Purdie
Quoted 13 lines · Christopher Larson

From: Christopher Larson <[email protected]>

This adds a QA test, as well as a rootfs_ function for use in ROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by the same user/group as the user running bitbake. This is useful to catch stuff which is written outside of pseudo's control, for example.

The following changes since commit c38acd720b3f6ffbeb544063692eb471dada8593:

binconfig-disabled: write an message to stderr to help confused developers (2015-08-19 17:57:58 +0100)

are available in the git repository at:

Quoted 7 lines

git://github.com/kergoth/openembedded-core host-user-contaminated
https://github.com/kergoth/openembedded-core/tree/host-user-contaminated

Christopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: add rootfs_check_host_user_contaminated

Quoted 6 lines

On the autobuilder we found:

http://errors.yoctoproject.org/Errors/Details/16318/

which looks like a race against the packaging code and the creation of package manager files. Not sure why we don't see this in other cases...

Also http://errors.yoctoproject.org/Errors/Details/16310/ which looks like the same issue (racing dpkg rather than opkg).

Cheers,

Richard

#11 lore raw
Quoted 30 lines · Richard Purdie
Quoted 5 lines · Richard Purdie
Quoted 4 lines · Christopher Larson

From: Christopher Larson <[email protected]>

This adds a QA test, as well as a rootfs_ function for use in ROOTFS_POSTPROCESS_COMMAND, to warn/error when paths are owned by the

same

Quoted 1 line
Quoted 1 line

user/group as the user running bitbake. This is useful to catch stuff

which is

Quoted 3 lines
Quoted 3 lines

written outside of pseudo's control, for example.

The following changes since commit

c38acd720b3f6ffbeb544063692eb471dada8593:

Quoted 2 lines
Quoted 2 lines

binconfig-disabled: write an message to stderr to help confused

developers (2015-08-19 17:57:58 +0100)

Quoted 5 lines
Quoted 5 lines

are available in the git repository at:

git://github.com/kergoth/openembedded-core host-user-contaminated

https://github.com/kergoth/openembedded-core/tree/host-user-contaminated

Quoted 8 lines
Quoted 5 lines

Christopher Larson (3):
insane.bbclass: handle tests which need fakeroot insane.bbclass: add host-user-contaminated test image.bbclass: add rootfs_check_host_user_contaminated

On the autobuilder we found:

Quoted 7 lines
Quoted 4 lines

http://errors.yoctoproject.org/Errors/Details/16318/

which looks like a race against the packaging code and the creation of package manager files. Not sure why we don't see this in other cases...

Also http://errors.yoctoproject.org/Errors/Details/16310/ which looks like the same issue (racing dpkg rather than opkg).

Ah, indeed, sorry about that, I must have gotten lucky with our builds thus far. I'll re-submit, ignoring ENOENT.

Signature · 5 lines
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics