Patch review - pseudo file mode corruption fix V2
cover Patch review - pseudo file mode corruption fix V2
Issues with undetected file mode corruption in pseudo have been identified.
Fixes have been merged into master and gatesgarth over the past several months and things seem to have stabilized enough that we can consider backporting these fixes to dunfell.
This is a somewhat more invasive change than normal. Specifically the user will be required to clean TMPDIR after this merge. It may also expose issues with other layers, though master and gategarth have probably helped to pave the way for this change.
Personally I think that this fixes a serious enough issue that we should consider this series for dunfell.
This second version includes the wic patches which have been the gating item for merging this series.
Note also that there is an associated bitbake patch for stable/1.46:
lib/bb/utils.py: Do not preserve TERM in the environment
This is due to taking:
bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
which was needed to do clean cherry-picks of the pseudo changes.
Let me know if you think this is an issue, or if you are aware of any other pseudo issues which would necessitate delaying the merge of this series.
Please have comments back by Wednesday evening.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1779
The following changes since commit 897822233faef0f8f35dc1d8a39e1c4bc0550f1e:
ppp: Whitelist CVE-2020-15704 (2021-01-19 04:22:10 -1000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut
Jack Mitchell (1):
distutils3: allow setup.py to be run from a different directory to
${S}Jacob Kroon (1):
bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
Joe Slater (1):
pseudo: fix renaming to self
Martin Jansa (2):
base.bbclass: use os.path.normpath instead of just comparing WORKDIR
and S as strings
license.bbclass: Add COMMON_LICENSE_DIR and LICENSE_PATH dirs to
PSEUDO_IGNORE_PATHS
Mingli Yu (2):
tcl: adapt to potential pseudo changes bitbake.conf: Exclude ${CCACHE_DIR} from pseudo database
Paul Barker (8):
bitbake.conf: Prevent pyc file generation in pseudo context
wic: Add workdir argument
wic: Allow exec_native_cmd to run HOSTTOOLS
wic: Ensure internal workdir is not reused
image_types_wic: Move wic working directory
wic: Update pseudo db when excluding content from rootfs
wic: Copy rootfs dir if fstab needs updating
wic: Optimise fstab modification for ext2/3/4 and msdos partitions
Peter Kjellerstedt (4):
pseudo: Simplify pseudo_client_ignore_path_chroot() lib/oe/path: Add canonicalize()
bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
Ricardo Ribalda Delgado (4):
wic: Fix permissions when using exclude or include path
wic: Fix multi images .wks with bitbake
wic: Avoid creating invalid pseudo directory
wic: Add --change-directory argument
Richard Purdie (23):
pseudo: Switch to oe-core branch in git repo
pseudo: merge in fixes for setfacl issue
pseudo: Update to add OFC fcntl lock updates
pseudo: Ignore mismatched inodes from the db
pseudo: Add support for ignoring paths from the pseudo DB
pseudo: Abort on mismatch patch
psuedo: Add tracking of linked files for fds
pseudo: Fix xattr segfault
pseudo: Add may unlink patch
pseudo: Add pathfix patch
base/bitbake.conf: Enable pseudo path filtering
wic: Handle new PSEUDO_IGNORE_PATHS variable
pseudo: Fix statx function usage
bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta abi_version,sanity: Tell users TMPDIR must be clean after pseudo
changes
pseudo: Update to account for patches merged on branch
pseudo: Upgrade to include mkostemp64 wrapper oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database
uninative: Don't use single sstate for pseudo-native
pseudo: Drop patches merged into upstream branch bitbake.conf: Add /run/ to PSEUDO_IGNORE_PATHS
pseudo: Add lchmod wrapper
pseudo: Update for arm host and memleak fixes/cleanup
Tomasz Dziendzielski (1):
pseudo: Update to print PSEUDO_LOGFILE in abort message on path
mismatches
meta-selftest/lib/pseudo_pyc_test1.py | 1 + meta-selftest/lib/pseudo_pyc_test2.py | 1 + .../pseudo-pyc-test/pseudo-pyc-test.bb | 15 ++ meta/classes/archiver.bbclass | 2 +- meta/classes/base.bbclass | 6 + meta/classes/distutils3.bbclass | 10 +- meta/classes/image_types_wic.bbclass | 20 ++- meta/classes/license.bbclass | 1 + meta/classes/populate_sdk_base.bbclass | 2 + meta/classes/sanity.bbclass | 3 + meta/classes/sstate.bbclass | 4 + meta/conf/abi_version.conf | 2 +- meta/conf/bitbake.conf | 13 +- meta/lib/oe/path.py | 21 +++ meta/lib/oe/sstatesig.py | 4 +- meta/lib/oeqa/selftest/cases/pseudo.py | 27 ++++ meta/lib/oeqa/selftest/cases/runtime_test.py | 1 + .../pseudo/files/0001-Add-statx.patch | 106 -------------- ...001-maketables-wrappers-use-Python-3.patch | 34 ----- ...ixup-remove-files-that-do-not-exist-.patch | 49 ------- .../0001-pseudo_ipc.h-Fix-enum-typedef.patch | 31 ---- ...1-realpath.c-Remove-trailing-slashes.patch | 57 -------- ...xattr-adjust-for-attr-2.4.48-release.patch | 48 ------ .../pseudo/files/moreretries.patch | 19 --- .../pseudo/files/seccomp.patch | 137 ------------------ .../pseudo/files/toomanyfiles.patch | 71 --------- .../pseudo/files/xattr_version.patch | 54 ------- meta/recipes-devtools/pseudo/pseudo_git.bb | 14 +- meta/recipes-devtools/tcltk/tcl_8.6.10.bb | 1 + scripts/lib/wic/help.py | 6 + scripts/lib/wic/ksparser.py | 1 + scripts/lib/wic/misc.py | 7 +- scripts/lib/wic/partition.py | 66 ++++++--- scripts/lib/wic/plugins/imager/direct.py | 48 +++--- scripts/lib/wic/plugins/source/rootfs.py | 85 +++++++++-- scripts/postinst-intercepts/update_font_cache | 2 + scripts/wic | 2 + 37 files changed, 281 insertions(+), 690 deletions(-) create mode 100644 meta-selftest/lib/pseudo_pyc_test1.py create mode 100644 meta-selftest/lib/pseudo_pyc_test2.py create mode 100644 meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb create mode 100644 meta/lib/oeqa/selftest/cases/pseudo.py delete mode 100644 meta/recipes-devtools/pseudo/files/0001-Add-statx.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch delete mode 100644 meta/recipes-devtools/pseudo/files/moreretries.patch delete mode 100644 meta/recipes-devtools/pseudo/files/seccomp.patch delete mode 100644 meta/recipes-devtools/pseudo/files/toomanyfiles.patch delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_version.patch
Signature · 1 line
2.25.1
PATCH 1/47 pseudo: Switch to oe-core branch in git repo
From: Richard Purdie <[email protected]>
We have a significant number of outstanding patches to pseudo. Rather than queue these up as patches, create a branch in the upstream repo and use that until such times as we have someone with the time/skills to properly review these for master in the pseudo repo.
--- .../pseudo/files/0001-Add-statx.patch | 106 -------------- ...001-maketables-wrappers-use-Python-3.patch | 34 ----- ...ixup-remove-files-that-do-not-exist-.patch | 49 ------- .../0001-pseudo_ipc.h-Fix-enum-typedef.patch | 31 ---- ...1-realpath.c-Remove-trailing-slashes.patch | 57 -------- ...xattr-adjust-for-attr-2.4.48-release.patch | 48 ------ .../pseudo/files/moreretries.patch | 19 --- .../pseudo/files/seccomp.patch | 137 ------------------ .../pseudo/files/toomanyfiles.patch | 71 --------- .../pseudo/files/xattr_version.patch | 54 ------- meta/recipes-devtools/pseudo/pseudo_git.bb | 14 +- 11 files changed, 2 insertions(+), 618 deletions(-) delete mode 100644 meta/recipes-devtools/pseudo/files/0001-Add-statx.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch delete mode 100644 meta/recipes-devtools/pseudo/files/moreretries.patch delete mode 100644 meta/recipes-devtools/pseudo/files/seccomp.patch delete mode 100644 meta/recipes-devtools/pseudo/files/toomanyfiles.patch delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_version.patch
meta/recipes-devtools/pseudo/files/0001-Add-statx.patch deleted +0 -106
@@ -1,106 +0,0 @@ -From 4e41a05de1f34ba00a68ca4f20fb49c4d1cbd2d0 Mon Sep 17 00:00:00 2001 -From: Richard Purdie <[email protected]> -Date: Wed, 6 Nov 2019 12:17:46 +0000 -Subject: [PATCH] Add statx glibc/syscall support - -Modern distros (e.g. fedora30) are starting to use the new statx() syscall through -the newly exposed glibc wrapper function in software like coreutils (e.g. the ls -command). Add support to intercept this to pseudo. - -Signed-off-by: Richard Purdie <[email protected]> -Upstream-Status: Submitted [Emailed to seebs] ---- - ports/linux/guts/statx.c | 48 ++++++++++++++++++++++++++++++++++++++++ - ports/linux/portdefs.h | 1 + - ports/linux/wrapfuncs.in | 1 + - 3 files changed, 50 insertions(+) - create mode 100644 ports/linux/guts/statx.c - -diff --git a/ports/linux/statx/guts/statx.c b/ports/linux/statx/guts/statx.c -new file mode 100644 -index 0000000..a3259c4 ---- /dev/null -+++ b/ports/linux/statx/guts/statx.c -@@ -0,0 +1,42 @@ -+/* -+ * Copyright (c) 2019 Linux Foundation -+ * Author: Richard Purdie -+ * -+ * SPDX-License-Identifier: LGPL-2.1-only -+ * -+ * int -+ * statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf) { -+ * int rc = -1; -+ */ -+ pseudo_msg_t *msg; -+ PSEUDO_STATBUF buf; -+ int save_errno; -+ -+ rc = real_statx(dirfd, pathname, flags, mask, statxbuf); -+ save_errno = errno; -+ if (rc == -1) { -+ return rc; -+ } -+ -+ buf.st_uid = statxbuf->stx_uid; -+ buf.st_gid = statxbuf->stx_gid; -+ buf.st_dev = makedev(statxbuf->stx_dev_major, statxbuf->stx_dev_minor); -+ buf.st_ino = statxbuf->stx_ino; -+ buf.st_mode = statxbuf->stx_mode; -+ buf.st_rdev = makedev(statxbuf->stx_rdev_major, statxbuf->stx_rdev_minor); -+ buf.st_nlink = statxbuf->stx_nlink; -+ msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, pathname, &buf); -+ if (msg && msg->result == RESULT_SUCCEED) { -+ pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid); -+ statxbuf->stx_uid = msg->uid; -+ statxbuf->stx_gid = msg->gid; -+ statxbuf->stx_mode = msg->mode; -+ statxbuf->stx_rdev_major = major(msg->rdev); -+ statxbuf->stx_rdev_minor = minor(msg->rdev); -+ } else { -+ pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid); -+ } -+ errno = save_errno; -+/* return rc; -+ * } -+ */ -diff --git a/ports/linux/statx/portdefs.h b/ports/linux/statx/portdefs.h -new file mode 100644 -index 0000000..bf934dc ---- /dev/null -+++ b/ports/linux/statx/portdefs.h -@@ -0,0 +1,6 @@ -+/* -+ * SPDX-License-Identifier: LGPL-2.1-only -+ * -+ */ -+#include <sys/stat.h> -+#include <sys/sysmacros.h> -diff --git a/ports/linux/statx/wrapfuncs.in b/ports/linux/statx/wrapfuncs.in -new file mode 100644 -index 0000000..c9cd4c3 ---- /dev/null -+++ b/ports/linux/statx/wrapfuncs.in -@@ -0,0 +1 @@ -+int statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf); -diff --git a/ports/linux/subports b/ports/linux/subports -index a29044a..49081bf 100755 ---- a/ports/linux/subports -+++ b/ports/linux/subports -@@ -54,3 +54,13 @@ else - fi - rm -f dummy.c dummy.o - -+cat > dummy.c <<EOF -+#define _GNU_SOURCE -+#include <sys/stat.h> -+struct statx x; -+EOF -+if ${CC} -c -o dummy.o dummy.c >/dev/null 2>&1; then -+ echo "linux/statx" -+fi -+rm -f dummy.c dummy.o -+ --- -2.17.1 -
meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch deleted +0 -34
@@ -1,34 +0,0 @@ -From dbd34b1b2af8fbf44a0d5c37abe3448405819823 Mon Sep 17 00:00:00 2001 -From: Alexander Kanavin <[email protected]> -Date: Wed, 28 Aug 2019 19:20:29 +0200 -Subject: [PATCH] maketables/wrappers: use Python 3 - -Changelog indicates they should be compatible. - -Upstream-Status: Pending -Signed-off-by: Alexander Kanavin <[email protected]> ---- - maketables | 2 +- - makewrappers | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -diff --git a/maketables b/maketables -index a211772..52285e2 100755 ---- a/maketables -+++ b/maketables -@@ -1,4 +1,4 @@ --#!/usr/bin/env python -+#!/usr/bin/env python3 - # - # Copyright (c) 2008-2010, 2013 Wind River Systems, Inc. - # -diff --git a/makewrappers b/makewrappers -index e84607d..b34f7eb 100755 ---- a/makewrappers -+++ b/makewrappers -@@ -1,4 +1,4 @@ --#!/usr/bin/env python -+#!/usr/bin/env python3 - # - # Copyright (c) 2008-2011,2013 Wind River Systems, Inc. - #
meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch deleted +0 -49
49 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch deleted +0 -31
@@ -1,31 +0,0 @@ -From a491aececfedf7313d29b80d626e0964fb533548 Mon Sep 17 00:00:00 2001 -From: Jacob Kroon <[email protected]> -Date: Sun, 3 May 2020 06:24:03 +0200 -Subject: [PATCH] pseudo_ipc.h: Fix enum typedef - -'pseudo_access_t' is a type, so use typedef. - -Fixes building pseudo with gcc 10 where -fno-common is the default. - -Signed-off-by: Jacob Kroon <[email protected]> -Upstream-Status: Submitted [https://lists.openembedded.org/g/openembedded-core/message/137758] ---- - pseudo_ipc.h | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -diff --git a/pseudo_ipc.h b/pseudo_ipc.h -index caeae5c..d945257 100644 ---- a/pseudo_ipc.h -+++ b/pseudo_ipc.h -@@ -29,7 +29,7 @@ typedef struct { - char path[]; - } pseudo_msg_t; - --enum { -+typedef enum { - PSA_EXEC = 1, - PSA_WRITE = (PSA_EXEC << 1), - PSA_READ = (PSA_WRITE << 1), --- -2.26.2 -
meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch deleted +0 -57
57 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch deleted +0 -48
48 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/moreretries.patch deleted +0 -19
@@ -1,19 +0,0 @@ -Increase the number of retries in pseudo due to occasional slow -server shutdowns. - -Upstream-Status: Pending -RP 2016/2/28 - -Index: git/pseudo_client.c -=================================================================== ---- git.orig/pseudo_client.c -+++ git/pseudo_client.c -@@ -1282,7 +1282,7 @@ pseudo_client_setup(void) { - } - } - --#define PSEUDO_RETRIES 20 -+#define PSEUDO_RETRIES 250 - static pseudo_msg_t * - pseudo_client_request(pseudo_msg_t *msg, size_t len, const char *path) { - pseudo_msg_t *response = 0;
meta/recipes-devtools/pseudo/files/seccomp.patch deleted +0 -137
137 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/toomanyfiles.patch deleted +0 -71
71 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/xattr_version.patch deleted +0 -54
54 lines not on this page · raw message
meta/recipes-devtools/pseudo/pseudo_git.bb +2 -12
@@ -1,22 +1,12 @@ require pseudo.inc -SRC_URI = "git://git.yoctoproject.org/pseudo \ +SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0001-configure-Prune-PIE-flags.patch \ file://fallback-passwd \ file://fallback-group \ - file://moreretries.patch \ - file://toomanyfiles.patch \ - file://0001-maketables-wrappers-use-Python-3.patch \ - file://0001-Add-statx.patch \ - file://0001-realpath.c-Remove-trailing-slashes.patch \ - file://0006-xattr-adjust-for-attr-2.4.48-release.patch \ - file://seccomp.patch \ - file://0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch \ - file://0001-pseudo_ipc.h-Fix-enum-typedef.patch \ - file://xattr_version.patch \ " -SRCREV = "060058bb29f70b244e685b3c704eb0641b736f73" +SRCREV = "b94fa2fc81cde25865ee223ca437d07377229a53" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 2/47 pseudo: merge in fixes for setfacl issue
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
[YOCTO #13959]
--- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://fallback-group \ " -SRCREV = "b94fa2fc81cde25865ee223ca437d07377229a53" +SRCREV = "067950bd9415fafcafb2bf32c0e10df4f2ac936c" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 3/47 pseudo: Update to add OFC fcntl lock updates
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
This adds in support for the OFC fcntl ioctl lock flags, removing warnings on some more recent linux kernels.
--- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://fallback-group \ " -SRCREV = "067950bd9415fafcafb2bf32c0e10df4f2ac936c" +SRCREV = "8efb082863ff0ceec7b7e46f9a44750e12f48039" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 4/47 pseudo: fix renaming to self
In reply to Steve SakomanFrom: Joe Slater <[email protected]>
Pseudo tests for an item being renamed to itself only after information about it has been deleted. Move the test to before we change the database.
Note that pseudo does not support renameat2(), but neither does glibc.
--- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://fallback-group \ " -SRCREV = "8efb082863ff0ceec7b7e46f9a44750e12f48039" +SRCREV = "d6b1b13c268d7246f0288d32d6b5eccc658cff4e" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 5/47 pseudo: Ignore mismatched inodes from the db
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Currently, where pseudo finds a database entry for an inode but the path doesn't match, it reuses that database entry metadata. This is causing real world "corruption" of file attributes.
See [YOCTO #14057] for an example of this.
This can happen when files are deleted outside of pseudo context and the inode is reused by a new file which pseduo then "sees".
Its possible the opposite could happen, it needs to reuse attributes but this change would prevent it. As far as I can tell, we don't want pseuo to reuse these attributes though so this code should be safer and avoid bugs like the above.
--- .../pseudo/files/delete_mismatches.patch | 51 +++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 52 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/delete_mismatches.patch
meta/recipes-devtools/pseudo/files/delete_mismatches.patch new file +51 -0
@@ -0,0 +1,51 @@ +When we see cases where the inode no longer matches the file path, pseudo +notices but currently reuses the database entry. This can happen where for +example, a file is deleted and a new file created outside of pseudo where +the inode number is reused. + +Change this to ignore the likely stale database entry instead. We're +seeing bugs where inode reuse for deleted files causes permission corruption. +(See bug #14057 for example). We don't want to delete the database entry +as the permissions may need to be applied to that file (and testing shows +we do need the path matching code which handles that). + +I appreciate this should never happen under the original design of pseudo +where all file accesses are monitored by pseudo. The reality is to do that, +we'd have to run pseudo: + +a) for all tasks +b) as one pseudo database for all of TMPDIR + +Neither of these is realistically possible for performance reasons. + +I believe pseudo to be much better at catching all accesses than it +might once have been. As such, these "fixups" are in the cases I've +seen in the logs, always incorrect. + +It therefore makes more sense to ignore the database data rather than +corrupt the file permissions or worse. Looking at the pseudo logs +in my heavily reused build directories, the number of these +errors is staggering. This issue would explain many weird bugs we've +seen over the years. + +There is a risk that we could not map permissions in some case where +we currently would. I have not seen evidence of this in any logs I've +read though. This change, whilst going against the original design, +is in my view the safer option for the project at this point given we +don't use pseudo as originally designed and never will be able to. + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/pseudo.c +=================================================================== +--- git.orig/pseudo.c ++++ git/pseudo.c +@@ -699,6 +701,7 @@ pseudo_op(pseudo_msg_t *msg, const char + (unsigned long long) msg_header.ino, + path_by_ino ? path_by_ino : "no path", + msg->path); ++ found_ino = 0; + } + } + } else {
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -2,6 +2,7 @@ require pseudo.inc SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0001-configure-Prune-PIE-flags.patch \ + file://delete_mismatches.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 6/47 pseudo: Add support for ignoring paths from the pseudo DB
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Add PSEUDO_IGNORE_PATHS, a comma separated list of path prefixes, where any files underneath are not handled by pseudo. This allows files to be left out of the pseudo datanase where we know we don't need the fake root emulation. This is particularly useful if we know these files can be deleted outside of pseudo context.
--- .../pseudo/files/add_ignore_paths.patch | 298 ++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 299 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/add_ignore_paths.patch
meta/recipes-devtools/pseudo/files/add_ignore_paths.patch new file +298 -0
@@ -0,0 +1,298 @@ +Currently, pseudo considers any path accessed whist its running to be +a valid entry to track in its database. The way OpenEmbedded uses pseudo, +there are paths we care about accesses to from a pseudo perspective and paths +which we simply don't care about. + +This patch adds a PSEUDO_IGNORE_PATHS environment variable which is a comma +separated list of path prefixes to ignore accesses to. + +To do this, we add some functions which can check a path argument or a file +descriptor argument and use these in the pseudo wrappers where path or fd +arguments are present. Where paths are being ignored, we skip straight to +the underlying real function. + +Psuedo needs to keep track of the open fd mappings to files so we still need +to allow those cases into the pseudo_op function. Specficially this means +OP_CLOSE, OP_OPEN, OP_DUP and OP_CHDIR. + +Apart from OP_OPEN which could call the server, the other operations are client +side only so passed through. We 'tag' the functions using these operations so +that the path ignore code isn't triggered. For OP_OPEN we exit early and skip +the server op. We also have a catch all in client_op to ensure any operatings +we didn't manage to skip early still get skipped correctly. + +OP_CHROOT is a special case. Where ignored path prefixes are used as a chroot, +for the lifetime of the chroot, the path is effectively dropped from the +PSEUDO_IGNORE_PATHS list. Whilst slightly counter intuaitive, this turned out +to be the most effective way to do things due to commands like useradd and +their use of chroots. + +For sqlite3 and appropriate path filtering in OE, this took the database from +45,000 entries to about 180. For dbus this was 88,000 down to 760. Given the +number of client to server trips these numbers of paths involves, the win +is seemingly worthwhile. + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/pseudo_client.c +=================================================================== +--- git.orig/pseudo_client.c ++++ git/pseudo_client.c +@@ -1482,6 +1482,43 @@ base_path(int dirfd, const char *path, i + return newpath; + } + ++int pseudo_client_ignore_fd(int fd) { ++ if (fd >= 0 && fd <= nfds) ++ return pseudo_client_ignore_path(fd_path(fd)); ++ return 0; ++} ++ ++int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) { ++ char *env; ++ if (path) { ++ if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0) ++ return 0; ++ env = pseudo_get_value("PSEUDO_IGNORE_PATHS"); ++ if (env) { ++ char *p = env; ++ while (*p) { ++ char *next = strchr(p, ','); ++ if (!next) ++ next = strchr(p, '\0'); ++ if ((next - p) && !strncmp(path, p, next - p)) { ++ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path); ++ return 1; ++ } ++ if (next && *next != '\0') ++ p = next+1; ++ else ++ break; ++ } ++ free(env); ++ } ++ } ++ return 0; ++} ++ ++int pseudo_client_ignore_path(const char *path) { ++ return pseudo_client_ignore_path_chroot(path, 1); ++} ++ + pseudo_msg_t * + pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...) { + pseudo_msg_t *result = 0; +@@ -1522,6 +1559,16 @@ pseudo_client_op(pseudo_op_t op, int acc + } + } + ++ if (op != OP_CHROOT && op != OP_CHDIR && op != OP_CLOSE && op != OP_DUP ++ && pseudo_client_ignore_path_chroot(path, 0)) { ++ if (op == OP_OPEN) { ++ pseudo_client_path(fd, path); ++ } ++ /* reenable wrappers */ ++ pseudo_magic(); ++ return result; ++ } ++ + #ifdef PSEUDO_XATTRDB + if (buf) { + struct stat64 bufcopy = *buf; +Index: git/pseudo_util.c +=================================================================== +--- git.orig/pseudo_util.c ++++ git/pseudo_util.c +@@ -43,6 +43,7 @@ static struct pseudo_variables pseudo_en + { "PSEUDO_BINDIR", 13, NULL }, + { "PSEUDO_LIBDIR", 13, NULL }, + { "PSEUDO_LOCALSTATEDIR", 20, NULL }, ++ { "PSEUDO_IGNORE_PATHS", 19, NULL }, + { "PSEUDO_PASSWD", 13, NULL }, + { "PSEUDO_CHROOT", 13, NULL }, + { "PSEUDO_UIDS", 11, NULL }, +Index: git/pseudo_client.h +=================================================================== +--- git.orig/pseudo_client.h ++++ git/pseudo_client.h +@@ -7,6 +7,8 @@ + * + */ + extern pseudo_msg_t *pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...); ++extern int pseudo_client_ignore_path(const char *path); ++extern int pseudo_client_ignore_fd(int fd); + #if PSEUDO_STATBUF_64 + #define base_lstat real_lstat64 + #define base_fstat real_fstat64 +Index: git/ports/linux/wrapfuncs.in +=================================================================== +--- git.orig/ports/linux/wrapfuncs.in ++++ git/ports/linux/wrapfuncs.in +@@ -1,23 +1,23 @@ +-int open(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */ ++int open(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */ + char *get_current_dir_name(void); + int __xstat(int ver, const char *path, struct stat *buf); + int __lxstat(int ver, const char *path, struct stat *buf); /* flags=AT_SYMLINK_NOFOLLOW */ + int __fxstat(int ver, int fd, struct stat *buf); + int lchown(const char *path, uid_t owner, gid_t group); /* flags=AT_SYMLINK_NOFOLLOW */ + int __fxstatat(int ver, int dirfd, const char *path, struct stat *buf, int flags); +-int openat(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */ +-int __openat_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW */ ++int openat(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */ ++int __openat_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW, noignore_path=1 */ + int mknod(const char *path, mode_t mode, dev_t dev); /* real_func=pseudo_mknod */ + int mknodat(int dirfd, const char *path, mode_t mode, dev_t dev); /* real_func=pseudo_mknodat */ + int __xmknod(int ver, const char *path, mode_t mode, dev_t *dev); /* flags=AT_SYMLINK_NOFOLLOW */ + int __xmknodat(int ver, int dirfd, const char *path, mode_t mode, dev_t *dev); /* flags=AT_SYMLINK_NOFOLLOW */ +-int fcntl(int fd, int cmd, ...{struct flock *lock}); ++int fcntl(int fd, int cmd, ...{struct flock *lock}); /* noignore_path=1 */ + # just so we know the inums of symlinks + char *canonicalize_file_name(const char *filename); + int eaccess(const char *path, int mode); +-int open64(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */ +-int openat64(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */ +-int __openat64_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW */ ++int open64(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */ ++int openat64(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */ ++int __openat64_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW, noignore_path=1 */ + int creat64(const char *path, mode_t mode); + int stat(const char *path, struct stat *buf); /* real_func=pseudo_stat */ + int lstat(const char *path, struct stat *buf); /* real_func=pseudo_lstat, flags=AT_SYMLINK_NOFOLLOW */ +@@ -29,9 +29,9 @@ int __xstat64(int ver, const char *path, + int __lxstat64(int ver, const char *path, struct stat64 *buf); /* flags=AT_SYMLINK_NOFOLLOW */ + int __fxstat64(int ver, int fd, struct stat64 *buf); + int __fxstatat64(int ver, int dirfd, const char *path, struct stat64 *buf, int flags); +-FILE *fopen64(const char *path, const char *mode); +-int nftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int, struct FTW *), int nopenfd, int flag); +-FILE *freopen64(const char *path, const char *mode, FILE *stream); ++FILE *fopen64(const char *path, const char *mode); /* noignore_path=1 */ ++int nftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int, struct FTW *), int nopenfd, int flag); /* noignore_path=1 */ ++FILE *freopen64(const char *path, const char *mode, FILE *stream); /* noignore_path=1 */ + int ftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int), int nopenfd); + int glob64(const char *pattern, int flags, int (*errfunc)(const char *, int), glob64_t *pglob); + int scandir64(const char *path, struct dirent64 ***namelist, int (*filter)(const struct dirent64 *), int (*compar)()); +Index: git/templates/wrapfuncs.c +=================================================================== +--- git.orig/templates/wrapfuncs.c ++++ git/templates/wrapfuncs.c +@@ -60,9 +60,15 @@ ${maybe_async_skip} + ${rc_assign} (*real_${name})(${call_args}); + } else { + ${fix_paths} +- /* exec*() use this to restore the sig mask */ +- pseudo_saved_sigmask = saved; +- ${rc_assign} wrap_$name(${call_args}); ++ if (${ignore_paths}) { ++ /* call the real syscall */ ++ pseudo_debug(PDBGF_SYSCALL, "${name} ignored path, calling real syscall.\n"); ++ ${rc_assign} (*real_${name})(${call_args}); ++ } else { ++ /* exec*() use this to restore the sig mask */ ++ pseudo_saved_sigmask = saved; ++ ${rc_assign} wrap_$name(${call_args}); ++ } + } + ${variadic_end} + save_errno = errno; +Index: git/ports/unix/wrapfuncs.in +=================================================================== +--- git.orig/ports/unix/wrapfuncs.in ++++ git/ports/unix/wrapfuncs.in +@@ -1,14 +1,14 @@ + int creat(const char *path, mode_t mode); + char *getcwd(char *buf, size_t size); + char *getwd(char *buf); +-int close(int fd); ++int close(int fd); /* noignore_path=1 */ + int fchmod(int fd, mode_t mode); + int fchown(int fd, uid_t owner, gid_t group); + int lchown(const char *path, uid_t owner, gid_t group); /* flags=AT_SYMLINK_NOFOLLOW */ +-int dup2(int oldfd, int newfd); +-int dup(int fd); +-int chdir(const char *path); +-int fchdir(int dirfd); ++int dup2(int oldfd, int newfd); /* noignore_path=1 */ ++int dup(int fd); /* noignore_path=1 */ ++int chdir(const char *path); /* noignore_path=1 */ ++int fchdir(int dirfd); /* noignore_path=1 */ + int access(const char *path, int mode); + FTS *fts_open(char * const *path_argv, int options, int (*compar)(const FTSENT **, const FTSENT **)); /* inode64=1 */ + int ftw(const char *path, int (*fn)(const char *, const struct stat *, int), int nopenfd); +@@ -20,18 +20,18 @@ char *mktemp(char *template); + long pathconf(const char *path, int name); + char *realpath(const char *name, char *resolved_name); /* version="GLIBC_2.3" */ + int remove(const char *path); /* flags=AT_SYMLINK_NOFOLLOW */ +-DIR *opendir(const char *path); +-int closedir(DIR *dirp); ++DIR *opendir(const char *path); /* noignore_path=1 */ ++int closedir(DIR *dirp); /* noignore_path=1 */ + char *tempnam(const char *template, const char *pfx); + char *tmpnam(char *s); + int truncate(const char *path, off_t length); + int utime(const char *path, const struct utimbuf *buf); + int utimes(const char *path, const struct timeval *times); + # needed because libc stdio does horrible things with inline asm syscalls +-FILE *fopen(const char *path, const char *mode); +-int fclose(FILE *fp); +-FILE *freopen(const char *path, const char *mode, FILE *stream); +-int chroot(const char *path); ++FILE *fopen(const char *path, const char *mode); /* noignore_path=1 */ ++int fclose(FILE *fp); /* noignore_path=1 */ ++FILE *freopen(const char *path, const char *mode, FILE *stream); /* noignore_path=1 */ ++int chroot(const char *path); /* noignore_path=1 */ + int acct(const char *path); + int chmod(const char *path, mode_t mode); + int chown(const char *path, uid_t owner, gid_t group); +Index: git/makewrappers +=================================================================== +--- git.orig/makewrappers ++++ git/makewrappers +@@ -228,6 +228,8 @@ class Function: + self.real_func = None + self.paths_to_munge = [] + self.specific_dirfds = {} ++ self.fd_arg = False ++ self.noignore_path = False + self.hand_wrapped = None + self.async_skip = None + # used for the copyright date when creating stub functions +@@ -267,6 +269,11 @@ class Function: + self.flags = '(flags & AT_SYMLINK_NOFOLLOW)' + elif arg.name.endswith('path'): + self.paths_to_munge.append(arg.name) ++ elif arg.name == 'fd': ++ self.fd_arg = "fd" ++ elif arg.name == 'filedes': ++ self.fd_arg = "filedes" ++ + + # pick default values + if self.type == 'void': +@@ -361,6 +368,25 @@ class Function: + (path, prefix, self.dirfd, path, self.flags)) + return "\n\t\t".join(fix_paths) + ++ def ignore_paths(self): ++ if self.noignore_path: ++ return "0" ++ ++ mainpath = None ++ if "oldpath" in self.paths_to_munge: ++ mainpath = "oldpath" ++ elif "newpath" in self.paths_to_munge: ++ mainpath = "newpath" ++ elif "path" in self.paths_to_munge: ++ mainpath = "path" ++ ++ if mainpath: ++ return "pseudo_client_ignore_path(%s)" % mainpath ++ if self.fd_arg: ++ return "pseudo_client_ignore_fd(%s)" % self.fd_arg ++ ++ return "0" ++ + def real_predecl(self): + if self.real_func: + return self.decl().replace(self.name, self.real_func, 1) + ";"
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -3,6 +3,7 @@ require pseudo.inc SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0001-configure-Prune-PIE-flags.patch \ file://delete_mismatches.patch \ + file://add_ignore_paths.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 7/47 pseudo: Abort on mismatch patch
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Rather than doing what turns out to be a rather dangerous "fixup" if we see a file with a different path but the same inode as another file we've previously seen, throw and abort. Direct the user to a wiki page where we can maintain information about what this error means.
--- .../pseudo/files/abort_on_mismatch.patch | 64 +++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 65 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch
meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch new file +64 -0
@@ -0,0 +1,64 @@ +Rather than mapping mismatched inode entries to paths, thrown an abort() +instead. Add a new result type to allow the server to pass back +this instruction to the client. + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/pseudo.c +=================================================================== +--- git.orig/pseudo.c ++++ git/pseudo.c +@@ -695,17 +695,15 @@ pseudo_op(pseudo_msg_t *msg, const char + msg->path); + pdb_did_unlink_file(path_by_ino, &by_ino, by_ino.deleting); + } else { +- int flags = 0; +- if (msg->nlink > 1) { +- flags = PDBGF_FILE | PDBGF_VERBOSE; +- } +- pseudo_debug(flags, "path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n", ++ pseudo_diag("path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n", + msg->nlink, + msg->nlink == 1 ? "" : "s", + (unsigned long long) msg_header.ino, + path_by_ino ? path_by_ino : "no path", + msg->path); + found_ino = 0; ++ msg->result = RESULT_ABORT; ++ goto op_exit; + } + } + } else { +@@ -1025,6 +1023,7 @@ pseudo_op(pseudo_msg_t *msg, const char + break; + } + ++op_exit: + /* in the case of an exact match, we just used the pointer + * rather than allocating space. + */ +Index: git/pseudo_client.c +=================================================================== +--- git.orig/pseudo_client.c ++++ git/pseudo_client.c +@@ -1919,6 +1919,10 @@ pseudo_client_op(pseudo_op_t op, int acc + #endif + if (result) { + pseudo_debug(PDBGF_OP, "(%d) %s", getpid(), pseudo_res_name(result->result)); ++ if (result->result == RESULT_ABORT) { ++ pseudo_diag("abort()ing pseudi client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.\n"); ++ abort(); ++ } + if (op == OP_STAT || op == OP_FSTAT) { + pseudo_debug(PDBGF_OP, " mode 0%o uid %d:%d", + (int) result->mode, +Index: git/enums/res.in +=================================================================== +--- git.orig/enums/res.in ++++ git/enums/res.in +@@ -2,3 +2,4 @@ res: RESULT + succeed + fail + error ++abort
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -4,6 +4,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0001-configure-Prune-PIE-flags.patch \ file://delete_mismatches.patch \ file://add_ignore_paths.patch \ + file://abort_on_mismatch.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 8/47 psuedo: Add tracking of linked files for fds
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Where files are link()'d and one is unlink()'d, pseudo's fd mappings can become confused. Add a patch to try and improve this for the common usecases we see.
--- .../pseudo/files/track_link_fds.patch | 155 ++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 156 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/track_link_fds.patch
meta/recipes-devtools/pseudo/files/track_link_fds.patch new file +155 -0
@@ -0,0 +1,155 @@ +Consider what happens if a program does: + +fd = fopen("A") +link("A", "B") +unlink("A") +fchown(fd) + +Assuming we can't use the database, in order to handle this correctly, +we need to change the open fd to point at B when A us unlinked. + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/ports/unix/guts/linkat.c +=================================================================== +--- git.orig/ports/unix/guts/linkat.c ++++ git/ports/unix/guts/linkat.c +@@ -116,6 +116,7 @@ + * if the thing linked is a symlink. + */ + pseudo_client_op(OP_LINK, 0, -1, -1, newpath, &buf); ++ pseudo_client_linked_paths(oldpath, newpath); + + errno = save_errno; + +Index: git/pseudo_client.c +=================================================================== +--- git.orig/pseudo_client.c ++++ git/pseudo_client.c +@@ -70,6 +70,8 @@ int pseudo_umask = 022; + + static char **fd_paths = NULL; + static int nfds = 0; ++static char **linked_fd_paths = NULL; ++static int linked_nfds = 0; + static const char **passwd_paths = NULL; + static int npasswd_paths = 0; + #ifdef PSEUDO_PROFILING +@@ -889,39 +891,70 @@ fd_path(int fd) { + } + + static void +-pseudo_client_path(int fd, const char *path) { ++pseudo_client_path_set(int fd, const char *path, char ***patharray, int *len) { + if (fd < 0) + return; + +- if (fd >= nfds) { ++ if (fd >= *len) { + int i; + pseudo_debug(PDBGF_CLIENT, "expanding fds from %d to %d\n", +- nfds, fd + 1); +- fd_paths = realloc(fd_paths, (fd + 1) * sizeof(char *)); +- for (i = nfds; i < fd + 1; ++i) +- fd_paths[i] = 0; +- nfds = fd + 1; ++ *len, fd + 1); ++ (*patharray) = realloc((*patharray), (fd + 1) * sizeof(char *)); ++ for (i = *len; i < fd + 1; ++i) ++ (*patharray)[i] = 0; ++ *len = fd + 1; + } else { +- if (fd_paths[fd]) { ++ if ((*patharray)[fd]) { + pseudo_debug(PDBGF_CLIENT, "reopening fd %d [%s] -- didn't see close\n", +- fd, fd_paths[fd]); ++ fd, (*patharray)[fd]); + } + /* yes, it is safe to free null pointers. yay for C89 */ +- free(fd_paths[fd]); +- fd_paths[fd] = 0; ++ free((*patharray)[fd]); ++ (*patharray)[fd] = 0; + } + if (path) { +- fd_paths[fd] = strdup(path); ++ (*patharray)[fd] = strdup(path); ++ } ++} ++ ++static void ++pseudo_client_path(int fd, const char *path) { ++ pseudo_client_path_set(fd, path, &fd_paths, &nfds); ++} ++ ++void ++pseudo_client_linked_paths(const char *oldpath, const char *newpath) { ++ int fd; ++ for (fd = 3; fd < nfds; ++fd) { ++ if (fd_paths[fd] && !strcmp(oldpath, fd_paths[fd])) { ++ pseudo_client_path_set(fd, newpath, &linked_fd_paths, &linked_nfds); ++ } + } + } + + static void ++pseudo_client_unlinked_path(const char *path) { ++ int fd; ++ for (fd = 0; fd < linked_nfds; ++fd) { ++ if (linked_fd_paths[fd] && fd_paths[fd] && !strcmp(path, fd_paths[fd])) { ++ pseudo_client_path(fd, linked_fd_paths[fd]); ++ } ++ } ++} ++ ++ ++static void + pseudo_client_close(int fd) { + if (fd < 0 || fd >= nfds) + return; + + free(fd_paths[fd]); + fd_paths[fd] = 0; ++ ++ if (fd < linked_nfds) { ++ free(linked_fd_paths[fd]); ++ linked_fd_paths[fd] = 0; ++ } + } + + /* spawn server */ +@@ -1860,6 +1893,12 @@ pseudo_client_op(pseudo_op_t op, int acc + dirfd); + pseudo_client_path(dirfd, fd_path(fd)); + break; ++ case OP_UNLINK: ++ case OP_DID_UNLINK: ++ if (path) ++ pseudo_client_unlinked_path(path); ++ do_request = 1; ++ break; + /* operations for which we should use the magic uid/gid */ + case OP_CHMOD: + case OP_CREAT: +@@ -1882,8 +1921,6 @@ pseudo_client_op(pseudo_op_t op, int acc + case OP_LINK: + case OP_RENAME: + case OP_STAT: +- case OP_UNLINK: +- case OP_DID_UNLINK: + case OP_CANCEL_UNLINK: + case OP_MAY_UNLINK: + case OP_GET_XATTR: +Index: git/pseudo_client.h +=================================================================== +--- git.orig/pseudo_client.h ++++ git/pseudo_client.h +@@ -9,6 +9,7 @@ + extern pseudo_msg_t *pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...); + extern int pseudo_client_ignore_path(const char *path); + extern int pseudo_client_ignore_fd(int fd); ++extern void pseudo_client_linked_paths(const char *oldpath, const char *newpath); + #if PSEUDO_STATBUF_64 + #define base_lstat real_lstat64 + #define base_fstat real_fstat64
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -5,6 +5,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://delete_mismatches.patch \ file://add_ignore_paths.patch \ file://abort_on_mismatch.patch \ + file://track_link_fds.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 9/47 pseudo: Fix xattr segfault
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Fix a NULL pointer dereference exposed by the path ignore code in xattr handling.
--- .../pseudo/files/xattr_fix.patch | 40 +++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 41 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/xattr_fix.patch
meta/recipes-devtools/pseudo/files/xattr_fix.patch new file +40 -0
@@ -0,0 +1,40 @@ + +In the xattr handling functions, if result is NULL, which it can be +with the path ignore code, there is a NULL pointer dereference and +segfault. Everywhere else checks result first, this appears to just +be an omission. + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/ports/linux/xattr/pseudo_wrappers.c +=================================================================== +--- git.orig/ports/linux/xattr/pseudo_wrappers.c ++++ git/ports/linux/xattr/pseudo_wrappers.c +@@ -134,7 +134,7 @@ static ssize_t shared_getxattr(const cha + pseudo_debug(PDBGF_XATTR, "getxattr(%s [fd %d], %s)\n", + path ? path : "<no path>", fd, name); + pseudo_msg_t *result = pseudo_client_op(OP_GET_XATTR, 0, fd, -1, path, &buf, name); +- if (result->result != RESULT_SUCCEED) { ++ if (!result || result->result != RESULT_SUCCEED) { + errno = ENOATTR; + return -1; + } +@@ -254,7 +254,7 @@ static int shared_setxattr(const char *p + static ssize_t shared_listxattr(const char *path, int fd, char *list, size_t size) { + RC_AND_BUF + pseudo_msg_t *result = pseudo_client_op(OP_LIST_XATTR, 0, fd, -1, path, &buf); +- if (result->result != RESULT_SUCCEED) { ++ if (!result || result->result != RESULT_SUCCEED) { + pseudo_debug(PDBGF_XATTR, "listxattr: no success.\n"); + errno = ENOATTR; + return -1; +@@ -276,7 +276,7 @@ static int shared_removexattr(const char + RC_AND_BUF + pseudo_msg_t *result = pseudo_client_op(OP_REMOVE_XATTR, 0, fd, -1, path, &buf, name); + +- if (result->result != RESULT_SUCCEED) { ++ if (!result || result->result != RESULT_SUCCEED) { + /* docs say ENOATTR, but I don't have one */ + errno = ENOENT; + return -1;
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -6,6 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://add_ignore_paths.patch \ file://abort_on_mismatch.patch \ file://track_link_fds.patch \ + file://xattr_fix.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 10/47 pseudo: Add may unlink patch
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Mark files which are unlinked (nlink == 0) but open with fd's as "may-unlink" to avoid problematic database entries.
--- .../pseudo/files/mayunlink.patch | 37 +++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 38 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/mayunlink.patch
meta/recipes-devtools/pseudo/files/mayunlink.patch new file +37 -0
@@ -0,0 +1,37 @@ +Some operations may call unlink() on an open fd, then call fchown/fchmod/fstat +on that fd. This would currently readd its entry to the database, which +is necessary to preserve its permissions information however since that +file will be lost when it is closed, we don't want the DB entry to persist. +Marking it as may_unlink means the code will know its likely been deleted +and ignore the entry later, giving improved behaviour that simple path +mismatch warnings. We can use an nlink of zero to detect this. + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/pseudo.c +=================================================================== +--- git.orig/pseudo.c ++++ git/pseudo.c +@@ -1100,6 +1100,21 @@ pseudo_op(pseudo_msg_t *msg, const char + break; + } + ++ switch (msg->op) { ++ case OP_FCHOWN: /* FALLTHROUGH */ ++ case OP_FCHMOD: /* FALLTHROUGH */ ++ case OP_FSTAT: ++ if (!found_path && !found_ino && (msg->nlink == 0)) { ++ /* If nlink is 0 for an fchown/fchmod/fstat, we probably have an fd which is ++ * unlinked and we don't want to do inode/path matching against it. Marking it ++ * as may unlink gives the right hints in the database to ensure we ++ * handle correctly whilst maintaining the permissions whilst the ++ * file exists for the fd. */ ++ pdb_may_unlink_file(msg, msg->client); ++ } ++ break; ++ } ++ + op_exit: + /* in the case of an exact match, we just used the pointer + * rather than allocating space.
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -7,6 +7,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://abort_on_mismatch.patch \ file://track_link_fds.patch \ file://xattr_fix.patch \ + file://mayunlink.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 11/47 pseudo: Add pathfix patch
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Add a path to fix up handling of dirfd being passed as a full file and with path="".
--- .../pseudo/files/pathfix.patch | 25 +++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 26 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/pathfix.patch
meta/recipes-devtools/pseudo/files/pathfix.patch new file +25 -0
@@ -0,0 +1,25 @@ +We're seeing systems in the wild (e.g. ubuntu 20.04) which call +with a dirfd set to the full filename and path set to "". Since +this seems to be expected to work, handle it accordingly. + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/pseudo_client.c +=================================================================== +--- git.orig/pseudo_client.c ++++ git/pseudo_client.c +@@ -1549,8 +1549,12 @@ base_path(int dirfd, const char *path, i + + if (!path) + return NULL; +- if (!*path) ++ ++ if (!*path) { ++ if (dirfd != -1 && dirfd != AT_FDCWD) ++ return fd_path(dirfd); + return ""; ++ } + + if (path[0] != '/') { + if (dirfd != -1 && dirfd != AT_FDCWD) {
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -8,6 +8,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://track_link_fds.patch \ file://xattr_fix.patch \ file://mayunlink.patch \ + file://pathfix.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 12/47 bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
In reply to Steve SakomanFrom: Jacob Kroon <[email protected]>
Since BitBake commit 0d5cdd0c0d65f2f81c3af0f3767fee86c4142c3a TERM is no longer preserved in the environment by BitBake, so there should be no need to whitelist it by default.
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -875,7 +875,7 @@ BB_CONSOLELOG ?= "${LOG_DIR}/cooker/${MACHINE}/${DATETIME}.log" # Setup our default hash policy BB_SIGNATURE_HANDLER ?= "OEBasicHash" BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \ - SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ + SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ STAMPS_DIR PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
Signature · 1 line
2.25.1
PATCH 13/47 base/bitbake.conf: Enable pseudo path filtering
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
This is a pretty big change to the way pseudo operates when used in OpenEmbedded. Normally, pseudo monitors and logs (adds to its database) any file created or modified whilst in a fakeroot environment. There are large numbers of files we simply don't care about the permissions of whilst in that fakeroot context, for example ${S}, ${B}, ${T}, ${SSTATE_DIR}, the central sstate control directories,
This change uses new functionality in pseudo to ignore these directory trees, resulting in a cleaner database with less chance of "stray" mismatches if files are modified outside pseudo context. It also should reduce some overhead from pseudo as the interprocess round trip to the server is avoided.
There is a possible complication where some existing recipe may break, for example, we found a recipe which was writing to "${B}/install" for "make install" in do_install and since we listed ${B} as not to be tracked, there were errors trying to chown root for files in this location.
This patch fixes a few corner cases in OE-Core when used with this new ignore list:
* The archiver directory matched a "${WORKDIR}/deploy*" pattern so was renamed to something else since that directory does need its root permissions
* The ${S} and ${B} ignoring is conditional on them being different to ${WORKDIR}
* package_write_* task output (the debs/rpms/ipks) are now owned by the build user so we don't want the file ownership information in the hashequiv outhash calculation even if they are built under pseudo.
* The fontcache postinstall intercept is run under qemu outside of pseudo context so delete files it may delete up front where pseudo can see this.
* SSTATE_DIR is in PSEUDO_PATHS_IGNORE, which is in FAKEROOTENV which is cached by bitbake. We therefore need to trigger reparsing if this changes, which means SSTATE_DIR can be in BB_HASHBASE_WHITELIST but not BB_HASHCONFIG_WHITELIST. Rework the variables to handle this. This otherwise breaks some of our sstate tests in oe-selftest.
* Ignore the temp directory wic uses for rebuilding rootfs.
--- meta/classes/archiver.bbclass | 2 +- meta/classes/base.bbclass | 5 +++++ meta/classes/image_types_wic.bbclass | 2 ++ meta/classes/populate_sdk_base.bbclass | 2 ++ meta/conf/bitbake.conf | 13 ++++++++----- meta/lib/oe/sstatesig.py | 2 ++ scripts/postinst-intercepts/update_font_cache | 2 ++ 7 files changed, 22 insertions(+), 6 deletions(-)
meta/classes/archiver.bbclass +1 -1
@@ -53,7 +53,7 @@ ARCHIVER_MODE[recipe] ?= "0" ARCHIVER_MODE[mirror] ?= "split" DEPLOY_DIR_SRC ?= "${DEPLOY_DIR}/sources" -ARCHIVER_TOPDIR ?= "${WORKDIR}/deploy-sources" +ARCHIVER_TOPDIR ?= "${WORKDIR}/archiver-sources" ARCHIVER_OUTDIR = "${ARCHIVER_TOPDIR}/${TARGET_SYS}/${PF}/" ARCHIVER_RPMTOPDIR ?= "${WORKDIR}/deploy-sources-rpm" ARCHIVER_RPMOUTDIR = "${ARCHIVER_RPMTOPDIR}/${TARGET_SYS}/${PF}/"
meta/classes/base.bbclass +5 -0
@@ -388,6 +388,11 @@ python () { oe.utils.features_backfill("DISTRO_FEATURES", d) oe.utils.features_backfill("MACHINE_FEATURES", d) + if d.getVar("WORKDIR") != d.getVar("S"): + d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}") + if d.getVar("WORKDIR") != d.getVar("B"): + d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}") + # Handle PACKAGECONFIG # # These take the form:
meta/classes/image_types_wic.bbclass +2 -0
@@ -40,6 +40,8 @@ IMAGE_CMD_wic () { IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" do_image_wic[cleandirs] = "${WORKDIR}/build-wic" +PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic" + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
meta/classes/populate_sdk_base.bbclass +2 -0
@@ -178,6 +178,8 @@ do_populate_sdk[sstate-inputdirs] = "${SDKDEPLOYDIR}" do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}" do_populate_sdk[stamp-extra-info] = "${MACHINE_ARCH}${SDKMACHINE}" +PSEUDO_IGNORE_PATHS .= ",${SDKDEPLOYDIR}" + fakeroot create_sdk_files() { cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
meta/conf/bitbake.conf +8 -5
@@ -686,13 +686,15 @@ SRC_URI = "" PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/" PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}" PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native" +PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts" + export PSEUDO_DISABLED = "1" #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}" #export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}" #export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib -FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_DISABLED=1" +FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=1" FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo" -FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_DISABLED=0" +FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=0" FAKEROOTNOENV = "PSEUDO_UNLOAD=1" FAKEROOTDIRS = "${PSEUDO_LOCALSTATEDIR}" PREFERRED_PROVIDER_virtual/fakeroot-native ?= "pseudo-native"
@@ -874,8 +876,8 @@ BB_CONSOLELOG ?= "${LOG_DIR}/cooker/${MACHINE}/${DATETIME}.log" # Setup our default hash policy BB_SIGNATURE_HANDLER ?= "OEBasicHash" -BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \ - SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ +BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \ + THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ STAMPS_DIR PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
@@ -883,7 +885,8 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES" -BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \ +BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR SSTATE_DIR " +BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \ SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \ PARALLEL_MAKE BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \ GIT_PROXY_COMMAND ALL_PROXY all_proxy NO_PROXY no_proxy FTP_PROXY ftp_proxy \
meta/lib/oe/sstatesig.py +2 -0
@@ -477,6 +477,8 @@ def OEOuthashBasic(path, sigfile, task, d): h = hashlib.sha256() prev_dir = os.getcwd() include_owners = os.environ.get('PSEUDO_DISABLED') == '0' + if "package_write_" in task or task == "package_qa": + include_owners = False include_timestamps = False if task == "package": include_timestamps = d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1'
scripts/postinst-intercepts/update_font_cache +2 -0
@@ -5,6 +5,8 @@ set -e +rm -f $D${fontconfigcachedir}/CACHEDIR.TAG + PSEUDO_UNLOAD=1 ${binprefix}qemuwrapper -L $D -E ${fontconfigcacheenv} $D${libexecdir}/${binprefix}fc-cache --sysroot=$D --system-only ${fontconfigcacheparams} chown -R root:root $D${fontconfigcachedir}
Signature · 1 line
2.25.1
PATCH 14/47 wic: Fix permissions when using exclude or include path
In reply to Steve SakomanFrom: Ricardo Ribalda Delgado <[email protected]>
When parameters include_path or exclude_path are passed to the rootfs plugin, it will copy the partition content into a folder and make all the modifications there.
This is done using copyhardlinktree(), which does not take into consideration the content of the pseudo folder, which contains the information about the right permissions and ownership of the folders.
This results in a rootfs owned by the user that is running the wic command (usually UID 1000), which makes some rootfs unbootable.
This bug can be easily triggerd with the following .wks
part / --source rootfs --fstype=ext4 --exclude-path=home
And this sequence:
$ wic create test-permissions -e core-image-minimal -o test/ $ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt $ ls -la /mnt/etc/shadow
To fix this we copy the content of the pseudo folders to the new folder and modify the pseudo database using the "pseudo -B" command.
If the rootfs is not a rootfs generated by bitbake a warning is shown making the user aware that the permissions on the target might not match what he expects.
WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions will be invalid
--- scripts/lib/wic/partition.py | 7 +++-- scripts/lib/wic/plugins/source/rootfs.py | 37 ++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 6 deletions(-)
scripts/lib/wic/partition.py +4 -3
@@ -191,7 +191,7 @@ class Partition(): (self.mountpoint, self.size, self.fixed_size)) def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir, - native_sysroot, real_rootfs = True): + native_sysroot, real_rootfs = True, pseudo_dir = None): """ Prepare content for a rootfs partition i.e. create a partition and fill it from a /rootfs dir.
@@ -199,8 +199,9 @@ class Partition(): Currently handles ext2/3/4, btrfs, vfat and squashfs. """ p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot) - p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", - "%s/../pseudo" % rootfs_dir) + if (pseudo_dir == None): + pseudo_dir = "%s/../pseudo" % rootfs_dir + p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir) p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir) p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1") pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
scripts/lib/wic/plugins/source/rootfs.py +34 -3
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree from wic import WicError from wic.pluginbase import SourcePlugin -from wic.misc import get_bitbake_var +from wic.misc import get_bitbake_var, exec_native_cmd logger = logging.getLogger('wic')
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin): return os.path.realpath(image_rootfs_dir) + @staticmethod + def __get_pseudo(native_sysroot, rootfs, pseudo_dir): + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") + return pseudo + @classmethod def do_prepare_partition(cls, part, source_params, cr, cr_workdir, oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin): "it is not a valid path, exiting" % part.rootfs_dir) part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir) + pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo") + if not os.path.lexists(pseudo_dir): + logger.warn("%s folder does not exist. " + "Usernames and permissions will be invalid " % pseudo_dir) + pseudo_dir = None new_rootfs = None + new_pseudo = None # Handle excluded paths. if part.exclude_path or part.include_path: # We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,24 @@ class RootfsPlugin(SourcePlugin): if os.path.lexists(new_rootfs): shutil.rmtree(os.path.join(new_rootfs)) - copyhardlinktree(part.rootfs_dir, new_rootfs) + # Convert the pseudo directory to its new location + if (pseudo_dir): + new_pseudo = os.path.realpath( + os.path.join(cr_workdir, "pseudo%d" % part.lineno)) + if os.path.lexists(new_pseudo): + shutil.rmtree(new_pseudo) + os.mkdir(new_pseudo) + shutil.copy(os.path.join(pseudo_dir, "files.db"), + os.path.join(new_pseudo, "files.db")) + + pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot, + new_rootfs, + new_pseudo), + part.rootfs_dir, new_rootfs) + exec_native_cmd(pseudo_cmd, native_sysroot) + for path in part.include_path or []: copyhardlinktree(path, new_rootfs)
@@ -112,4 +142,5 @@ class RootfsPlugin(SourcePlugin): shutil.rmtree(full_path) part.prepare_rootfs(cr_workdir, oe_builddir, - new_rootfs or part.rootfs_dir, native_sysroot) + new_rootfs or part.rootfs_dir, native_sysroot, + pseudo_dir = new_pseudo or pseudo_dir)
Signature · 1 line
2.25.1
PATCH 15/47 wic: Fix multi images .wks with bitbake
In reply to Steve SakomanFrom: Ricardo Ribalda Delgado <[email protected]>
In order to support .wks files with multiple images inside bitbake we need to explicitly set the pseudo database in use.
Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
The username for all the files under /export will be set to the runner of bitbake (usually UID 1000).
Before we run wic, we need to make sure that the pseudo database will be flushed, and contains all the data needed.
--- meta/classes/image_types_wic.bbclass | 8 ++++++-- scripts/lib/wic/partition.py | 9 +++------ 2 files changed, 9 insertions(+), 8 deletions(-)
meta/classes/image_types_wic.bbclass +6 -2
@@ -33,8 +33,7 @@ IMAGE_CMD_wic () { if [ -z "$wks" ]; then bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately." fi - - BUILDDIR="${TOPDIR}" wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS} + BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS} mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic" } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -89,6 +88,10 @@ python do_write_wks_template () { bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file))) } +do_flush_pseudodb() { + ${FAKEROOTENV} ${FAKEROOTCMD} -S +} + python () { if d.getVar('USING_WIC'): wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -142,6 +145,7 @@ python do_rootfs_wicenv () { depdir = d.getVar('IMGDEPLOYDIR') bb.utils.copyfile(os.path.join(outdir, basename) + '.env', os.path.join(depdir, basename) + '.env') } +addtask do_flush_pseudodb after do_image before do_image_wic addtask do_rootfs_wicenv after do_image before do_image_wic do_rootfs_wicenv[vardeps] += "${WICVARS}" do_rootfs_wicenv[prefuncs] = 'set_image_size'
scripts/lib/wic/partition.py +3 -6
@@ -201,13 +201,10 @@ class Partition(): p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot) if (pseudo_dir == None): pseudo_dir = "%s/../pseudo" % rootfs_dir - p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir) - p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir) - p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1") pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix - pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir - pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd - pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
Signature · 1 line
2.25.1
PATCH 16/47 wic: Avoid creating invalid pseudo directory
In reply to Steve SakomanFrom: Ricardo Ribalda Delgado <[email protected]>
If the source of the rootfs is not a bitbake cooked image, or it is not pointing to the root of one, we call pseudo again, which will produce a new pseudo folder at rootfs/../pseudo
Eg:
part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4
--- scripts/lib/wic/partition.py | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
scripts/lib/wic/partition.py +8 -7
@@ -199,13 +199,14 @@ class Partition(): Currently handles ext2/3/4, btrfs, vfat and squashfs. """ p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot) - if (pseudo_dir == None): - pseudo_dir = "%s/../pseudo" % rootfs_dir - pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix - pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir - pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir - pseudo += "export PSEUDO_NOSYMLINKEXP=1;" - pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") + if (pseudo_dir): + pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") + else: + pseudo = None rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label, self.lineno, self.fstype)
Signature · 1 line
2.25.1
PATCH 17/47 wic: Handle new PSEUDO_IGNORE_PATHS variable
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Adjust wic to correctly handle the new PSEUDO_IGNORE_PATH variable and avoid inode corruption issues.
--- meta/classes/image_types_wic.bbclass | 2 +- scripts/lib/wic/partition.py | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-)
meta/classes/image_types_wic.bbclass +1 -1
@@ -5,7 +5,7 @@ WICVARS ?= "\ IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \ ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS \ KERNEL_IMAGETYPE MACHINE INITRAMFS_IMAGE INITRAMFS_IMAGE_BUNDLE INITRAMFS_LINK_NAME APPEND \ - ASSUME_PROVIDED" + ASSUME_PROVIDED PSEUDO_IGNORE_PATHS" inherit ${@bb.utils.contains('INITRAMFS_IMAGE_BUNDLE', '1', 'kernel-artifact-names', '', d)}
scripts/lib/wic/partition.py +7 -5
@@ -198,21 +198,23 @@ class Partition(): Currently handles ext2/3/4, btrfs, vfat and squashfs. """ + + rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label, + self.lineno, self.fstype) + if os.path.isfile(rootfs): + os.remove(rootfs) + p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot) if (pseudo_dir): pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % (rootfs + "," + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "")) pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") else: pseudo = None - rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label, - self.lineno, self.fstype) - if os.path.isfile(rootfs): - os.remove(rootfs) - if not self.size and real_rootfs: # The rootfs size is not set in .ks file so try to get it # from bitbake variable
Signature · 1 line
2.25.1
PATCH 18/47 pseudo: Fix statx function usage
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
There is magic in the posts where specific variable names have specific magic. For that magic to work, "path" needs to be used not "pathname" as is currently there. Fix this, which fixes path issues on systems using statx (Ubuntu 20.04 in particular).
--- .../pseudo/files/statxfix.patch | 56 +++++++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 1 + 2 files changed, 57 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/statxfix.patch
meta/recipes-devtools/pseudo/files/statxfix.patch new file +56 -0
@@ -0,0 +1,56 @@ +There is magic in the posts where specific variable names have specific +magic. For that magic to work, "path" needs to be used not "pathname" as +is currently there. Fix this, which fixes path issues on systems using +statx (Ubuntu 20.04 in particular). + +Signed-off-by: Richard Purdie <[email protected]> +Upstream-Status: Pending + +Index: git/ports/linux/statx/guts/statx.c +=================================================================== +--- git.orig/ports/linux/statx/guts/statx.c ++++ git/ports/linux/statx/guts/statx.c +@@ -5,14 +5,14 @@ + * SPDX-License-Identifier: LGPL-2.1-only + * + * int +- * statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf) { ++ * statx(int dirfd, const char *path, int flags, unsigned int mask, struct statx *statxbuf) { + * int rc = -1; + */ + pseudo_msg_t *msg; + PSEUDO_STATBUF buf; + int save_errno; + +- rc = real_statx(dirfd, pathname, flags, mask, statxbuf); ++ rc = real_statx(dirfd, path, flags, mask, statxbuf); + save_errno = errno; + if (rc == -1) { + return rc; +@@ -25,16 +25,16 @@ + buf.st_mode = statxbuf->stx_mode; + buf.st_rdev = makedev(statxbuf->stx_rdev_major, statxbuf->stx_rdev_minor); + buf.st_nlink = statxbuf->stx_nlink; +- msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, pathname, &buf); ++ msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, path, &buf); + if (msg && msg->result == RESULT_SUCCEED) { +- pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid); ++ pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", path, flags, rc, statxbuf->stx_uid); + statxbuf->stx_uid = msg->uid; + statxbuf->stx_gid = msg->gid; + statxbuf->stx_mode = msg->mode; + statxbuf->stx_rdev_major = major(msg->rdev); + statxbuf->stx_rdev_minor = minor(msg->rdev); + } else { +- pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid); ++ pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", path, flags, rc, statxbuf->stx_uid); + } + errno = save_errno; + /* return rc; +Index: git/ports/linux/statx/wrapfuncs.in +=================================================================== +--- git.orig/ports/linux/statx/wrapfuncs.in ++++ git/ports/linux/statx/wrapfuncs.in +@@ -1 +1 @@ +-int statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf); ++int statx(int dirfd, const char *path, int flags, unsigned int mask, struct statx *statxbuf);
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -0
@@ -9,6 +9,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://xattr_fix.patch \ file://mayunlink.patch \ file://pathfix.patch \ + file://statxfix.patch \ file://fallback-passwd \ file://fallback-group \ "
Signature · 1 line
2.25.1
PATCH 19/47 bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Unfortunately, .pyc files can be generated in meta/lib/oe which corrupt the pseudo database so we need to extend the ignore list to cover this as well.
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -686,7 +686,7 @@ SRC_URI = "" PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/" PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}" PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native" -PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts" +PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta" export PSEUDO_DISABLED = "1" #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
Signature · 1 line
2.25.1
PATCH 20/47 abi_version,sanity: Tell users TMPDIR must be clean after pseudo changes
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
After the recent path ignore changes, there are invalid entries in exiting pseudo databases which will trip up users with the new abort() (proving how common the pseudo potential corrpution is!). Inform them a clean TMPDIR is needed.
--- meta/classes/sanity.bbclass | 3 +++ meta/conf/abi_version.conf | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
meta/classes/sanity.bbclass +3 -0
@@ -619,6 +619,9 @@ def sanity_handle_abichanges(status, d): f.write(current_abi) elif int(abi) <= 11 and current_abi == "12": status.addresult("The layout of TMPDIR changed for Recipe Specific Sysroots.\nConversion doesn't make sense and this change will rebuild everything so please delete TMPDIR (%s).\n" % d.getVar("TMPDIR")) + elif int(abi) <= 13 and current_abi == "14": + status.addresult("TMPDIR changed to include path filtering from the pseudo database.\nIt is recommended to use a clean TMPDIR with the new pseudo path filtering so TMPDIR (%s) would need to be removed to continue.\n" % d.getVar("TMPDIR")) + elif (abi != current_abi): # Code to convert from one ABI to another could go here if possible. status.addresult("Error, TMPDIR has changed its layout version number (%s to %s) and you need to either rebuild, revert or adjust it at your own risk.\n" % (abi, current_abi))
meta/conf/abi_version.conf +1 -1
@@ -4,7 +4,7 @@ # that breaks the format and have been previously discussed on the mailing list # with general agreement from the core team. # -OELAYOUT_ABI = "12" +OELAYOUT_ABI = "14" # # HASHEQUIV_HASH_VERSION is injected into the output hash calculation used by
Signature · 1 line
2.25.1
PATCH 21/47 pseudo: Update to account for patches merged on branch
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
--- .../pseudo/files/abort_on_mismatch.patch | 64 ---- .../pseudo/files/add_ignore_paths.patch | 298 ------------------ .../pseudo/files/delete_mismatches.patch | 51 --- .../pseudo/files/mayunlink.patch | 37 --- .../pseudo/files/pathfix.patch | 25 -- .../pseudo/files/statxfix.patch | 56 ---- .../pseudo/files/track_link_fds.patch | 155 --------- .../pseudo/files/xattr_fix.patch | 40 --- meta/recipes-devtools/pseudo/pseudo_git.bb | 10 +- 9 files changed, 1 insertion(+), 735 deletions(-) delete mode 100644 meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch delete mode 100644 meta/recipes-devtools/pseudo/files/add_ignore_paths.patch delete mode 100644 meta/recipes-devtools/pseudo/files/delete_mismatches.patch delete mode 100644 meta/recipes-devtools/pseudo/files/mayunlink.patch delete mode 100644 meta/recipes-devtools/pseudo/files/pathfix.patch delete mode 100644 meta/recipes-devtools/pseudo/files/statxfix.patch delete mode 100644 meta/recipes-devtools/pseudo/files/track_link_fds.patch delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_fix.patch
meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch deleted +0 -64
@@ -1,64 +0,0 @@ -Rather than mapping mismatched inode entries to paths, thrown an abort() -instead. Add a new result type to allow the server to pass back -this instruction to the client. - -Signed-off-by: Richard Purdie <[email protected]> -Upstream-Status: Pending - -Index: git/pseudo.c -=================================================================== ---- git.orig/pseudo.c -+++ git/pseudo.c -@@ -695,17 +695,15 @@ pseudo_op(pseudo_msg_t *msg, const char - msg->path); - pdb_did_unlink_file(path_by_ino, &by_ino, by_ino.deleting); - } else { -- int flags = 0; -- if (msg->nlink > 1) { -- flags = PDBGF_FILE | PDBGF_VERBOSE; -- } -- pseudo_debug(flags, "path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n", -+ pseudo_diag("path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n", - msg->nlink, - msg->nlink == 1 ? "" : "s", - (unsigned long long) msg_header.ino, - path_by_ino ? path_by_ino : "no path", - msg->path); - found_ino = 0; -+ msg->result = RESULT_ABORT; -+ goto op_exit; - } - } - } else { -@@ -1025,6 +1023,7 @@ pseudo_op(pseudo_msg_t *msg, const char - break; - } - -+op_exit: - /* in the case of an exact match, we just used the pointer - * rather than allocating space. - */ -Index: git/pseudo_client.c -=================================================================== ---- git.orig/pseudo_client.c -+++ git/pseudo_client.c -@@ -1919,6 +1919,10 @@ pseudo_client_op(pseudo_op_t op, int acc - #endif - if (result) { - pseudo_debug(PDBGF_OP, "(%d) %s", getpid(), pseudo_res_name(result->result)); -+ if (result->result == RESULT_ABORT) { -+ pseudo_diag("abort()ing pseudi client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.\n"); -+ abort(); -+ } - if (op == OP_STAT || op == OP_FSTAT) { - pseudo_debug(PDBGF_OP, " mode 0%o uid %d:%d", - (int) result->mode, -Index: git/enums/res.in -=================================================================== ---- git.orig/enums/res.in -+++ git/enums/res.in -@@ -2,3 +2,4 @@ res: RESULT - succeed - fail - error -+abort
meta/recipes-devtools/pseudo/files/add_ignore_paths.patch deleted +0 -298
298 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/delete_mismatches.patch deleted +0 -51
51 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/mayunlink.patch deleted +0 -37
@@ -1,37 +0,0 @@ -Some operations may call unlink() on an open fd, then call fchown/fchmod/fstat -on that fd. This would currently readd its entry to the database, which -is necessary to preserve its permissions information however since that -file will be lost when it is closed, we don't want the DB entry to persist. -Marking it as may_unlink means the code will know its likely been deleted -and ignore the entry later, giving improved behaviour that simple path -mismatch warnings. We can use an nlink of zero to detect this. - -Signed-off-by: Richard Purdie <[email protected]> -Upstream-Status: Pending - -Index: git/pseudo.c -=================================================================== ---- git.orig/pseudo.c -+++ git/pseudo.c -@@ -1100,6 +1100,21 @@ pseudo_op(pseudo_msg_t *msg, const char - break; - } - -+ switch (msg->op) { -+ case OP_FCHOWN: /* FALLTHROUGH */ -+ case OP_FCHMOD: /* FALLTHROUGH */ -+ case OP_FSTAT: -+ if (!found_path && !found_ino && (msg->nlink == 0)) { -+ /* If nlink is 0 for an fchown/fchmod/fstat, we probably have an fd which is -+ * unlinked and we don't want to do inode/path matching against it. Marking it -+ * as may unlink gives the right hints in the database to ensure we -+ * handle correctly whilst maintaining the permissions whilst the -+ * file exists for the fd. */ -+ pdb_may_unlink_file(msg, msg->client); -+ } -+ break; -+ } -+ - op_exit: - /* in the case of an exact match, we just used the pointer - * rather than allocating space.
meta/recipes-devtools/pseudo/files/pathfix.patch deleted +0 -25
@@ -1,25 +0,0 @@ -We're seeing systems in the wild (e.g. ubuntu 20.04) which call -with a dirfd set to the full filename and path set to "". Since -this seems to be expected to work, handle it accordingly. - -Signed-off-by: Richard Purdie <[email protected]> -Upstream-Status: Pending - -Index: git/pseudo_client.c -=================================================================== ---- git.orig/pseudo_client.c -+++ git/pseudo_client.c -@@ -1549,8 +1549,12 @@ base_path(int dirfd, const char *path, i - - if (!path) - return NULL; -- if (!*path) -+ -+ if (!*path) { -+ if (dirfd != -1 && dirfd != AT_FDCWD) -+ return fd_path(dirfd); - return ""; -+ } - - if (path[0] != '/') { - if (dirfd != -1 && dirfd != AT_FDCWD) {
meta/recipes-devtools/pseudo/files/statxfix.patch deleted +0 -56
56 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/track_link_fds.patch deleted +0 -155
155 lines not on this page · raw message
meta/recipes-devtools/pseudo/files/xattr_fix.patch deleted +0 -40
@@ -1,40 +0,0 @@ - -In the xattr handling functions, if result is NULL, which it can be -with the path ignore code, there is a NULL pointer dereference and -segfault. Everywhere else checks result first, this appears to just -be an omission. - -Signed-off-by: Richard Purdie <[email protected]> -Upstream-Status: Pending - -Index: git/ports/linux/xattr/pseudo_wrappers.c -=================================================================== ---- git.orig/ports/linux/xattr/pseudo_wrappers.c -+++ git/ports/linux/xattr/pseudo_wrappers.c -@@ -134,7 +134,7 @@ static ssize_t shared_getxattr(const cha - pseudo_debug(PDBGF_XATTR, "getxattr(%s [fd %d], %s)\n", - path ? path : "<no path>", fd, name); - pseudo_msg_t *result = pseudo_client_op(OP_GET_XATTR, 0, fd, -1, path, &buf, name); -- if (result->result != RESULT_SUCCEED) { -+ if (!result || result->result != RESULT_SUCCEED) { - errno = ENOATTR; - return -1; - } -@@ -254,7 +254,7 @@ static int shared_setxattr(const char *p - static ssize_t shared_listxattr(const char *path, int fd, char *list, size_t size) { - RC_AND_BUF - pseudo_msg_t *result = pseudo_client_op(OP_LIST_XATTR, 0, fd, -1, path, &buf); -- if (result->result != RESULT_SUCCEED) { -+ if (!result || result->result != RESULT_SUCCEED) { - pseudo_debug(PDBGF_XATTR, "listxattr: no success.\n"); - errno = ENOATTR; - return -1; -@@ -276,7 +276,7 @@ static int shared_removexattr(const char - RC_AND_BUF - pseudo_msg_t *result = pseudo_client_op(OP_REMOVE_XATTR, 0, fd, -1, path, &buf, name); - -- if (result->result != RESULT_SUCCEED) { -+ if (!result || result->result != RESULT_SUCCEED) { - /* docs say ENOATTR, but I don't have one */ - errno = ENOENT; - return -1;
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -9
@@ -2,19 +2,11 @@ require pseudo.inc SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0001-configure-Prune-PIE-flags.patch \ - file://delete_mismatches.patch \ - file://add_ignore_paths.patch \ - file://abort_on_mismatch.patch \ - file://track_link_fds.patch \ - file://xattr_fix.patch \ - file://mayunlink.patch \ - file://pathfix.patch \ - file://statxfix.patch \ file://fallback-passwd \ file://fallback-group \ " -SRCREV = "d6b1b13c268d7246f0288d32d6b5eccc658cff4e" +SRCREV = "07141f919dc8582460f984ed1d2fb73e05f82ef3" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 22/47 pseudo: Upgrade to include mkostemp64 wrapper
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Newer systems are using this currently unwrapped glibc function, add it.
[YOCTO #14080]
--- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://fallback-group \ " -SRCREV = "07141f919dc8582460f984ed1d2fb73e05f82ef3" +SRCREV = "cca0d7f15b7197095cd587420d31b187620c3093" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 23/47 base.bbclass: use os.path.normpath instead of just comparing WORKDIR and S as strings
In reply to Steve SakomanFrom: Martin Jansa <[email protected]>
* cannot use os.path.samefile, because S/B might not exist at this time yet
* there is issue with PSEUDO_IGNORE_PATHS when some recipe sets e.g. S = "${WORKDIR}/"
whole WORKDIR gets added to PSEUDO_IGNORE_PATHS and then the build can fail with various strange errors, in my case do_package was failing when do_package calls:
fix_perms(.../1.0-r0/package/etc, 755, 0, 0, /etc)
and fails with "[Errno 1] Operation not permitted:"
--- meta/classes/base.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/base.bbclass +2 -2
@@ -388,9 +388,9 @@ python () { oe.utils.features_backfill("DISTRO_FEATURES", d) oe.utils.features_backfill("MACHINE_FEATURES", d) - if d.getVar("WORKDIR") != d.getVar("S"): + if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("S")): d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}") - if d.getVar("WORKDIR") != d.getVar("B"): + if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("B")): d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}") # Handle PACKAGECONFIG
Signature · 1 line
2.25.1
PATCH 24/47 oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Avoid pseudo abort()s like:
path mismatch [1 link]: ino 6295376 db '/tmp/oeqa-feed-sign-2mw7z81v/S.gpg-agent.yocto-native' req '/tmp/jwkivmu6'.
--- meta/lib/oeqa/selftest/cases/runtime_test.py | 1 + 1 file changed, 1 insertion(+)
meta/lib/oeqa/selftest/cases/runtime_test.py +1 -0
@@ -161,6 +161,7 @@ class TestImage(OESelftestTestCase): features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') features += 'GPG_PATH = "%s"\n' % self.gpg_home + features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home self.write_config(features) # Build core-image-sato and testimage
Signature · 1 line
2.25.1
PATCH 25/47 tcl: adapt to potential pseudo changes
In reply to Steve SakomanFrom: Mingli Yu <[email protected]>
It failed as below when rerun do_install for tcl:
$ bitbake tcl $ bitbake tcl -f -cinstall [snip] | NOTE: make -j 72 DESTDIR=/build/tmp/work/core2-64-wrs-linux/tcl/8.6.10-r0/image install | abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this. | /build/tmp/work/core2-64-wrs-linux/tcl/8.6.10-r0/temp/run.do_install.3490694: line 184: 3499214 Aborted (core dumped) make -j 72 "$@"
By default the S is ${WORKDIR}/${BPN}-${PV}, but after unpack, the tcl source [1] unpacked to ${WORKDIR}/${BPN}${PV} and all the files under ${WORKDIR}/${BPN}${PV} are acutally the source files.
But the the main Makefile.in is under ${WORKDIR}/${BPN}${PV}/unix for tcl, so there is below logic in tcl recipe:
S = "${WORKDIR}/${BPN}${PV}/unix"
To adapt the potential pseudo changes, there is a general logic to exclude ${S} from pseudo database in base.bbclass in [2]. That's to say, just the dir ${WORKDIR}/${BPN}${PV}/unix is excluded from the pseudo database.
But it's not enough for tcl, we need to exclude the actual source dir ${WORKDIR}/${BPN}${PV} from pseudo database specifically to fix the above do_install failure.
[1] https://downloads.sourceforge.net/tcl/tcl8.6.0-src.tar.gz
[2] https://git.openembedded.org/openembedded-core/tree/meta/classes/base.bbclass#n396
--- meta/recipes-devtools/tcltk/tcl_8.6.10.bb | 1 + 1 file changed, 1 insertion(+)
meta/recipes-devtools/tcltk/tcl_8.6.10.bb +1 -0
@@ -32,6 +32,7 @@ SRC_URI_class-native = "${BASE_SRC_URI}" S = "${WORKDIR}/${BPN}${PV}/unix" +PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/${BPN}${PV}" VER = "${PV}" inherit autotools ptest binconfig update-alternatives
Signature · 1 line
2.25.1
PATCH 26/47 bitbake.conf: Exclude ${CCACHE_DIR} from pseudo database
In reply to Steve SakomanFrom: Mingli Yu <[email protected]>
When ccache enabled as below:
INHERIT += "ccache"CCACHE_DIR = "/path/build/ccache"
There comes do_install failure for some recipes randomly, take linux-libc-headerswhen as example.
$ cat /path/build/tmp/work/corei7-64-wrs-linux/linux-libc-headers/5.8-r0/pseudo/pseudo.log [snip] path mismatch [1 link]: ino 243004209 db '/path/build/ccache/6/stats' req '/path/build/ccache/7/stats.lock'. [snip]
Exclude ${CCACHE_DIR} from pseudo database to fix the potential do_install failure.
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -686,7 +686,7 @@ SRC_URI = "" PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/" PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}" PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native" -PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta" +PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta,${CCACHE_DIR}" export PSEUDO_DISABLED = "1" #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
Signature · 1 line
2.25.1
PATCH 27/47 uninative: Don't use single sstate for pseudo-native
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
pseudo-native is a bit special. It conditionally compiles in support for xattr, statx and statvfs amongst other options. If a pseudo-native binary is used on a system where these functions are present but it wasn't compiled in we see hard to debug permissions problems.
An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target oe-selftest which shows a cryptic error:
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target
self.assertEqual(filelist1, filelist2)
File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual
self.assertSequenceEqual(list1, list2, msg, seq_type=list)
File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual
self.fail(msg)
File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail
raise self.failureException(msg)
AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8']
First differing element 0:
'-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor'
'-rwxr-xr-x root root /etc/init.d/mdmonitor'
This is due to a version of pseudo without statx being used on a system where ls uses statx, hence the files are displayed as 6000.6000 instead of root.root.
Avoid this by always building pseudo-native for the specific distro in question rather than using a universal sstate feed.
This hopefully fixes one of the mysterious AB-INT issues.
--- meta/classes/base.bbclass | 1 + meta/classes/sstate.bbclass | 4 ++++ meta/lib/oe/sstatesig.py | 2 +- 3 files changed, 6 insertions(+), 1 deletion(-)
meta/classes/base.bbclass +1 -0
@@ -231,6 +231,7 @@ python base_eventhandler() { if isinstance(e, bb.event.ConfigParsed): if not d.getVar("NATIVELSBSTRING", False): d.setVar("NATIVELSBSTRING", lsb_distro_identifier(d)) + d.setVar("ORIGNATIVELSBSTRING", d.getVar("NATIVELSBSTRING", False)) d.setVar('BB_VERSION', bb.__version__) # There might be no bb.event.ConfigParsed event if bitbake server is
meta/classes/sstate.bbclass +4 -0
@@ -72,6 +72,7 @@ BB_HASHFILENAME = "False ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}" SSTATE_ARCHS = " \ ${BUILD_ARCH} \ + ${BUILD_ARCH}_${ORIGNATIVELSBSTRING} \ ${BUILD_ARCH}_${SDK_ARCH}_${SDK_OS} \ ${BUILD_ARCH}_${TARGET_ARCH} \ ${SDK_ARCH}_${SDK_OS} \
@@ -80,6 +81,7 @@ SSTATE_ARCHS = " \ ${PACKAGE_ARCH} \ ${PACKAGE_EXTRA_ARCHS} \ ${MACHINE_ARCH}" +SSTATE_ARCHS[vardepsexclude] = "ORIGNATIVELSBSTRING" SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
@@ -121,6 +123,8 @@ SSTATE_HASHEQUIV_REPORT_TASKDATA[doc] = "Report additional useful data to the \ python () { if bb.data.inherits_class('native', d): d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH', False)) + if d.getVar("PN") == "pseudo-native": + d.appendVar('SSTATE_PKGARCH', '_${ORIGNATIVELSBSTRING}') elif bb.data.inherits_class('crosssdk', d): d.setVar('SSTATE_PKGARCH', d.expand("${BUILD_ARCH}_${SDK_ARCH}_${SDK_OS}")) elif bb.data.inherits_class('cross', d):
meta/lib/oe/sstatesig.py +1 -1
@@ -434,7 +434,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache): d2 = multilibcache[variant] if taskdata.endswith("-native"): - pkgarchs = ["${BUILD_ARCH}"] + pkgarchs = ["${BUILD_ARCH}", "${BUILD_ARCH}_${ORIGNATIVELSBSTRING}"] elif taskdata.startswith("nativesdk-"): pkgarchs = ["${SDK_ARCH}_${SDK_OS}", "allarch"] elif "-cross-canadian" in taskdata:
Signature · 1 line
2.25.1
PATCH 28/47 pseudo: Simplify pseudo_client_ignore_path_chroot()
In reply to Steve SakomanFrom: Peter Kjellerstedt <[email protected]>
This also plugs a memory leak in pseudo_client_ignore_path_chroot().
--- ...ssen-indentation-of-pseudo_client_ig.patch | 69 +++++++++++++++++++ ...mplify-pseudo_client_ignore_path_chr.patch | 50 ++++++++++++++ meta/recipes-devtools/pseudo/pseudo_git.bb | 2 + 3 files changed, 121 insertions(+) create mode 100644 meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch create mode 100644 meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch new file +69 -0
@@ -0,0 +1,69 @@ +From 28c760542eecd7c5b35ea88aa2b14d62afbda961 Mon Sep 17 00:00:00 2001 +From: Peter Kjellerstedt <[email protected]> +Date: Sat, 21 Nov 2020 17:22:38 +0100 +Subject: [PATCH] pseudo_client: Lessen indentation of + pseudo_client_ignore_path_chroot() + +Change-Id: I739b18efb7a95ce2d2d907d0faf7df539ab9af1c +--- + pseudo_client.c | 45 +++++++++++++++++++++++++-------------------- + 1 file changed, 25 insertions(+), 20 deletions(-) + +diff --git a/pseudo_client.c b/pseudo_client.c +index 116d926..a8bc3dc 100644 +--- a/pseudo_client.c ++++ b/pseudo_client.c +@@ -1527,28 +1527,33 @@ int pseudo_client_ignore_fd(int fd) { + + int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) { + char *env; +- if (path) { +- if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0) +- return 0; +- env = pseudo_get_value("PSEUDO_IGNORE_PATHS"); +- if (env) { +- char *p = env; +- while (*p) { +- char *next = strchr(p, ','); +- if (!next) +- next = strchr(p, '\0'); +- if ((next - p) && !strncmp(path, p, next - p)) { +- pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path); +- return 1; +- } +- if (next && *next != '\0') +- p = next+1; +- else +- break; +- } +- free(env); ++ ++ if (!path) ++ return 0; ++ ++ if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0) ++ return 0; ++ ++ env = pseudo_get_value("PSEUDO_IGNORE_PATHS"); ++ if (!env) ++ return 0; ++ ++ char *p = env; ++ while (*p) { ++ char *next = strchr(p, ','); ++ if (!next) ++ next = strchr(p, '\0'); ++ if ((next - p) && !strncmp(path, p, next - p)) { ++ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path); ++ return 1; + } ++ if (next && *next != '\0') ++ p = next+1; ++ else ++ break; + } ++ free(env); ++ + return 0; + } +
meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch new file +50 -0
50 lines not on this page · raw message
meta/recipes-devtools/pseudo/pseudo_git.bb +2 -0
@@ -4,6 +4,8 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0001-configure-Prune-PIE-flags.patch \ file://fallback-passwd \ file://fallback-group \ + file://0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch \ + file://0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch \ " SRCREV = "cca0d7f15b7197095cd587420d31b187620c3093"
Signature · 1 line
2.25.1
PATCH 29/47 pseudo: Update to print PSEUDO_LOGFILE in abort message on path mismatches
In reply to Steve SakomanFrom: Tomasz Dziendzielski <[email protected]>
--- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
@@ -8,7 +8,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch \ " -SRCREV = "cca0d7f15b7197095cd587420d31b187620c3093" +SRCREV = "69f205c41902e17933b81b1450636848e8da2126" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 30/47 distutils3: allow setup.py to be run from a different directory to ${S}
In reply to Steve SakomanFrom: Jack Mitchell <[email protected]>
Sometimes setup.py can be buried deep in a source tree. This has traditionally been solved with setting S to the subdirectory in the source. However with the new pseudo changes, some python modules make changes to files beneath ${S}, for example:
S = "${WORKDIR}/git/python/pythonmodule"
then in setup.py it works with source code in a relative fashion, such as:
../../src
This causes pseudo to abort as it isn't tracking the paths. Therefore implement the variable DISTUTILS_SETUP_PATH so that recipes can use:
S = "${WORKDIR}/git"DISTUTILS_SETUP_PATH = "${S}/python/pythonmodule"
inherit distutils3
This allows the full source tree to be monitored, while distutils can run setup.py from a location other than ${S}.
--- meta/classes/distutils3.bbclass | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
meta/classes/distutils3.bbclass +6 -4
@@ -12,28 +12,30 @@ DISTUTILS_INSTALL_ARGS ?= "--root=${D} \ DISTUTILS_PYTHON = "python3" DISTUTILS_PYTHON_class-native = "nativepython3" +DISTUTILS_SETUP_PATH ?= "${S}" + distutils3_do_configure() { : } distutils3_do_compile() { - cd ${S} + cd ${DISTUTILS_SETUP_PATH} NO_FETCH_BUILD=1 \ STAGING_INCDIR=${STAGING_INCDIR} \ STAGING_LIBDIR=${STAGING_LIBDIR} \ - ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} ${S}/setup.py \ + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ build --build-base=${B} ${DISTUTILS_BUILD_ARGS} || \ bbfatal_log "'${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS}' execution failed." } distutils3_do_compile[vardepsexclude] = "MACHINE" distutils3_do_install() { - cd ${S} + cd ${DISTUTILS_SETUP_PATH} install -d ${D}${PYTHON_SITEPACKAGES_DIR} STAGING_INCDIR=${STAGING_INCDIR} \ STAGING_LIBDIR=${STAGING_LIBDIR} \ PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \ - ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} ${S}/setup.py \ + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ build --build-base=${B} install --skip-build ${DISTUTILS_INSTALL_ARGS} || \ bbfatal_log "'${PYTHON_PN} setup.py install ${DISTUTILS_INSTALL_ARGS}' execution failed."
Signature · 1 line
2.25.1
PATCH 31/47 pseudo: Drop patches merged into upstream branch
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
--- ...ssen-indentation-of-pseudo_client_ig.patch | 69 ------------------- ...mplify-pseudo_client_ignore_path_chr.patch | 50 -------------- meta/recipes-devtools/pseudo/pseudo_git.bb | 4 +- 3 files changed, 1 insertion(+), 122 deletions(-) delete mode 100644 meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch deleted +0 -69
@@ -1,69 +0,0 @@ -From 28c760542eecd7c5b35ea88aa2b14d62afbda961 Mon Sep 17 00:00:00 2001 -From: Peter Kjellerstedt <[email protected]> -Date: Sat, 21 Nov 2020 17:22:38 +0100 -Subject: [PATCH] pseudo_client: Lessen indentation of - pseudo_client_ignore_path_chroot() - -Change-Id: I739b18efb7a95ce2d2d907d0faf7df539ab9af1c ---- - pseudo_client.c | 45 +++++++++++++++++++++++++-------------------- - 1 file changed, 25 insertions(+), 20 deletions(-) - -diff --git a/pseudo_client.c b/pseudo_client.c -index 116d926..a8bc3dc 100644 ---- a/pseudo_client.c -+++ b/pseudo_client.c -@@ -1527,28 +1527,33 @@ int pseudo_client_ignore_fd(int fd) { - - int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) { - char *env; -- if (path) { -- if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0) -- return 0; -- env = pseudo_get_value("PSEUDO_IGNORE_PATHS"); -- if (env) { -- char *p = env; -- while (*p) { -- char *next = strchr(p, ','); -- if (!next) -- next = strchr(p, '\0'); -- if ((next - p) && !strncmp(path, p, next - p)) { -- pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path); -- return 1; -- } -- if (next && *next != '\0') -- p = next+1; -- else -- break; -- } -- free(env); -+ -+ if (!path) -+ return 0; -+ -+ if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0) -+ return 0; -+ -+ env = pseudo_get_value("PSEUDO_IGNORE_PATHS"); -+ if (!env) -+ return 0; -+ -+ char *p = env; -+ while (*p) { -+ char *next = strchr(p, ','); -+ if (!next) -+ next = strchr(p, '\0'); -+ if ((next - p) && !strncmp(path, p, next - p)) { -+ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path); -+ return 1; - } -+ if (next && *next != '\0') -+ p = next+1; -+ else -+ break; - } -+ free(env); -+ - return 0; - } -
meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch deleted +0 -50
50 lines not on this page · raw message
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -3
@@ -4,11 +4,9 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://0001-configure-Prune-PIE-flags.patch \ file://fallback-passwd \ file://fallback-group \ - file://0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch \ - file://0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch \ " -SRCREV = "69f205c41902e17933b81b1450636848e8da2126" +SRCREV = "6fd57da7b1de1a2b6cf530e336d58bb5f8bdd015" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 32/47 lib/oe/path: Add canonicalize()
In reply to Steve SakomanFrom: Peter Kjellerstedt <[email protected]>
oe.path.canonicalize() is used to canonicalize paths (i.e., remove symbolic links and "..", and make them absolute). It takes a string with paths separated by commas, and returns the canonicalized path in the same format.
--- meta/lib/oe/path.py | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
meta/lib/oe/path.py +21 -0
@@ -320,3 +320,24 @@ def which_wild(pathname, path=None, mode=os.F_OK, *, reverse=False, candidates=F return files +def canonicalize(paths, sep=','): + """Given a string with paths (separated by commas by default), expand + each path using os.path.realpath() and return the resulting paths as a + string (separated using the same separator a the original string). + """ + # Ignore paths containing "$" as they are assumed to be unexpanded bitbake + # variables. Normally they would be ignored, e.g., when passing the paths + # through the shell they would expand to empty strings. However, when they + # are passed through os.path.realpath(), it will cause them to be prefixed + # with the absolute path to the current directory and thus not be empty + # anymore. + # + # Also maintain trailing slashes, as the paths may actually be used as + # prefixes in sting compares later on, where the slashes then are important. + canonical_paths = [] + for path in (paths or '').split(sep): + if '$' not in path: + trailing_slash = path.endswith('/') and '/' or '' + canonical_paths.append(os.path.realpath(path) + trailing_slash) + + return sep.join(canonical_paths)
Signature · 1 line
2.25.1
PATCH 33/47 bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
In reply to Steve SakomanFrom: Peter Kjellerstedt <[email protected]>
Use oe.path.canonicalize() to canonicalize the paths in PSEUDO_IGNORE_PATHS before passing them to pseudo. This is needed since pseudo will compare them to paths that are canonicalized.
--- meta/conf/bitbake.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +2 -2
@@ -692,9 +692,9 @@ export PSEUDO_DISABLED = "1" #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}" #export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}" #export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib -FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=1" +FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1" FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo" -FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=0" +FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0" FAKEROOTNOENV = "PSEUDO_UNLOAD=1" FAKEROOTDIRS = "${PSEUDO_LOCALSTATEDIR}" PREFERRED_PROVIDER_virtual/fakeroot-native ?= "pseudo-native"
Signature · 1 line
2.25.1
PATCH 34/47 wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
In reply to Steve SakomanFrom: Peter Kjellerstedt <[email protected]>
--- scripts/lib/wic/partition.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
scripts/lib/wic/partition.py +11 -1
@@ -206,11 +206,21 @@ class Partition(): p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot) if (pseudo_dir): + # Canonicalize the ignore paths. This corresponds to + # calling oe.path.canonicalize(), which is used in bitbake.conf. + ignore_paths = [rootfs] + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",") + canonical_paths = [] + for path in ignore_paths: + if "$" not in path: + trailing_slash = path.endswith("/") and "/" or "" + canonical_paths.append(os.path.realpath(path) + trailing_slash) + ignore_paths = ",".join(canonical_paths) + pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir pseudo += "export PSEUDO_NOSYMLINKEXP=1;" - pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % (rootfs + "," + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "")) + pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") else: pseudo = None
Signature · 1 line
2.25.1
PATCH 35/47 bitbake.conf: Prevent pyc file generation in pseudo context
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
This also effectively reverts commit b6d30c21b0:
bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/metaThe contents of ${COREBASE}/meta were ignored as pyc files could be generated for the contents of the lib subdirectory if python modules were imported within a pseudo context. However this doesn't protect us from pyc files being generated in the lib directories for other layers. It's far better to tell python not to produce pyc files when running under pseudo (by setting the PYTHONDONTWRITEBYTECODE variable) as this will cover any location where pyc files could possibly be created. This variable is set in FAKEROOTBASEENV so that it applies to the bitbake-worker instance for fakeroot tasks, preventing pyc files from being generated for imports in python tasks themselves.
Also add a test case to ensure that pyc files are not created in tasks which are executed under pseudo.
--- meta-selftest/lib/pseudo_pyc_test1.py | 1 + meta-selftest/lib/pseudo_pyc_test2.py | 1 + .../pseudo-pyc-test/pseudo-pyc-test.bb | 15 +++++++++++ meta/conf/bitbake.conf | 4 +-- meta/lib/oeqa/selftest/cases/pseudo.py | 27 +++++++++++++++++++ 5 files changed, 46 insertions(+), 2 deletions(-) create mode 100644 meta-selftest/lib/pseudo_pyc_test1.py create mode 100644 meta-selftest/lib/pseudo_pyc_test2.py create mode 100644 meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb create mode 100644 meta/lib/oeqa/selftest/cases/pseudo.py
meta-selftest/lib/pseudo_pyc_test1.py new file +1 -0
@@ -0,0 +1 @@ +STRING = "pseudo_pyc_test1"
meta-selftest/lib/pseudo_pyc_test2.py new file +1 -0
@@ -0,0 +1 @@ +STRING = "pseudo_pyc_test2"
meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb new file +15 -0
@@ -0,0 +1,15 @@ +SUMMARY = "pseudo env test" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" + +INHIBIT_DEFAULT_DEPS = "1" + +python do_compile() { + import pseudo_pyc_test1 + print(pseudo_pyc_test1.STRING) +} + +python do_install() { + import pseudo_pyc_test2 + print(pseudo_pyc_test2.STRING) +}
meta/conf/bitbake.conf +2 -2
@@ -686,13 +686,13 @@ SRC_URI = "" PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/" PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}" PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native" -PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta,${CCACHE_DIR}" +PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${CCACHE_DIR}" export PSEUDO_DISABLED = "1" #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}" #export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}" #export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib -FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1" +FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1 PYTHONDONTWRITEBYTECODE=1" FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo" FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0" FAKEROOTNOENV = "PSEUDO_UNLOAD=1"
meta/lib/oeqa/selftest/cases/pseudo.py new file +27 -0
@@ -0,0 +1,27 @@ +# +# SPDX-License-Identifier: MIT +# + +import glob +import os +import shutil +from oeqa.utils.commands import bitbake, get_test_layer +from oeqa.selftest.case import OESelftestTestCase + +class Pseudo(OESelftestTestCase): + + def test_pseudo_pyc_creation(self): + self.write_config("") + + metaselftestpath = get_test_layer() + pycache_path = os.path.join(metaselftestpath, 'lib/__pycache__') + if os.path.exists(pycache_path): + shutil.rmtree(pycache_path) + + bitbake('pseudo-pyc-test -c install') + + test1_pyc_present = len(glob.glob(os.path.join(pycache_path, 'pseudo_pyc_test1.*.pyc'))) + self.assertTrue(test1_pyc_present, 'test1 pyc file missing, should be created outside of pseudo context.') + + test2_pyc_present = len(glob.glob(os.path.join(pycache_path, 'pseudo_pyc_test2.*.pyc'))) + self.assertFalse(test2_pyc_present, 'test2 pyc file present, should not be created in pseudo context.')
Signature · 1 line
2.25.1
PATCH 36/47 license.bbclass: Add COMMON_LICENSE_DIR and LICENSE_PATH dirs to PSEUDO_IGNORE_PATHS
In reply to Steve SakomanFrom: Martin Jansa <[email protected]>
* now without ${COREBASE}/meta in PSEUDO_IGNORE_PATHS after:
73d538f207 bitbake.conf: Prevent pyc file generation in pseudo context
the do_package tasks are failing when LICENSE_CREATE_PACKAGE is enabled
pseudo.log shows that it's because of license texts copied from
${COREBASE}/meta:
path mismatch [46 links]: ino 96733640 db '/OE/build/tmp-glibc/work/all-oe-linux/foo/1.0/image/usr/share/licenses/foo/generic_Apache-2.0' req '/OE/build/openembedded-core/meta/files/common-licenses/Apache-2.0'.
--- meta/classes/license.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes/license.bbclass +1 -0
@@ -31,6 +31,7 @@ python do_populate_lic() { f.write("%s: %s\n" % (key, info[key])) } +PSEUDO_IGNORE_PATHS .= ",${@','.join(((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' + (d.getVar('LICENSE_PATH') or '')).split())}" # it would be better to copy them in do_install_append, but find_license_filesa is python python perform_packagecopy_prepend () { enabled = oe.data.typed_value('LICENSE_CREATE_PACKAGE', d)
Signature · 1 line
2.25.1
PATCH 37/47 bitbake.conf: Add /run/ to PSEUDO_IGNORE_PATHS
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Builds were failing on WSL2 which turns out to be due to accesses to /run/ on those systems. Add this to PSEUDO_IGNORE_PATHS to fix builds on WSL2.
[YOCTO #14175]
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -686,7 +686,7 @@ SRC_URI = "" PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/" PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}" PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native" -PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${CCACHE_DIR}" +PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,/run/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${CCACHE_DIR}" export PSEUDO_DISABLED = "1" #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
Signature · 1 line
2.25.1
PATCH 38/47 pseudo: Add lchmod wrapper
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
New versions of glibc have an lchmod function so we need to wrap it.
Identified through a reproducibility issue in initramfs-base where /dev/console created by mknod from coreutils changed permissions depending on the host distro (mknod used the gnulib wrapper on most hosts but newer ones used the libc call).
[YOCTO #14162]
--- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://fallback-group \ " -SRCREV = "6fd57da7b1de1a2b6cf530e336d58bb5f8bdd015" +SRCREV = "1ee9a1e05e200f189f6644411ba9304e8a479b0b" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 39/47 pseudo: Update for arm host and memleak fixes/cleanup
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Pulls in:
makewrappers: support architecture-overrides in wrapper modifiers
makewrappers: fix Python 2 hangover
Fix some memory leaks
Disable deprecated function warnings
Silence switch block warningspseudo_util: don't overrun strings when looking for keys
--- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/pseudo/pseudo_git.bb +1 -1
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \ file://fallback-group \ " -SRCREV = "1ee9a1e05e200f189f6644411ba9304e8a479b0b" +SRCREV = "f9754ac14672c4af19b77bc698a1a808b0828265" S = "${WORKDIR}/git" PV = "1.9.0+git${SRCPV}"
Signature · 1 line
2.25.1
PATCH 40/47 wic: Add workdir argument
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
This allows the path for the temporary workdir used by wic to be set when running wic from bitbake or directly from the command line.
--- scripts/lib/wic/plugins/imager/direct.py | 2 +- scripts/wic | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/lib/wic/plugins/imager/direct.py +1 -1
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin): self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0], strftime("%Y%m%d%H%M")) - self.workdir = tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.') + self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.') self._image = None self.ptable_format = self.ks.bootloader.ptable self.parts = self.ks.partitions
scripts/wic +2 -0
@@ -312,6 +312,8 @@ def wic_init_parser_create(subparser): subparser.add_argument("-o", "--outdir", dest="outdir", default='.', help="name of directory to create image in") + subparser.add_argument("-w", "--workdir", + help="temporary workdir to use for intermediate files") subparser.add_argument("-e", "--image-name", dest="image_name", help="name of the image to use the artifacts from " "e.g. core-image-sato")
Signature · 1 line
2.25.1
PATCH 41/47 wic: Allow exec_native_cmd to run HOSTTOOLS
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to be more easily executed by wic. Without this change only programs from an actual *-native recipe built by bitbake can be executed by wic.
--- meta/classes/image_types_wic.bbclass | 2 +- scripts/lib/wic/misc.py | 7 +++++-- 2 files changed, 6 insertions(+), 3 deletions(-)
meta/classes/image_types_wic.bbclass +1 -1
@@ -3,7 +3,7 @@ WICVARS ?= "\ BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME IMAGE_BOOT_FILES \ IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \ - ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS \ + ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS HOSTTOOLS_DIR \ KERNEL_IMAGETYPE MACHINE INITRAMFS_IMAGE INITRAMFS_IMAGE_BUNDLE INITRAMFS_LINK_NAME APPEND \ ASSUME_PROVIDED PSEUDO_IGNORE_PATHS"
scripts/lib/wic/misc.py +5 -2
@@ -128,9 +128,12 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): if pseudo: cmd_and_args = pseudo + cmd_and_args - native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \ + hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR") + + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \ (native_sysroot, native_sysroot, - native_sysroot, native_sysroot) + native_sysroot, native_sysroot, + hosttools_dir) native_cmd_and_args = "export PATH=%s:$PATH;%s" % \ (native_paths, cmd_and_args)
Signature · 1 line
2.25.1
PATCH 42/47 wic: Ensure internal workdir is not reused
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
If a path is specified for the internal wic working directory using the -w/--workdir argument then it must not already exist. Re-using a previous workdir could easily result in rootfs and intermediate files from a previous build being added to the current image.
--- scripts/lib/wic/plugins/imager/direct.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
scripts/lib/wic/plugins/imager/direct.py +11 -1
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin): self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0], strftime("%Y%m%d%H%M")) - self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.') + self.workdir = self.setup_workdir(options.workdir) self._image = None self.ptable_format = self.ks.bootloader.ptable self.parts = self.ks.partitions
@@ -78,6 +78,16 @@ class DirectPlugin(ImagerPlugin): self._image = PartitionedImage(image_path, self.ptable_format, self.parts, self.native_sysroot) + def setup_workdir(self, workdir): + if workdir: + if os.path.exists(workdir): + raise WicError("Internal workdir '%s' specified in wic arguments already exists!" % (workdir)) + + os.makedirs(workdir) + return workdir + else: + return tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.') + def do_create(self): """ Plugin entry point.
Signature · 1 line
2.25.1
PATCH 43/47 image_types_wic: Move wic working directory
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
By default the wic working directory is placed under the output directory. When invoking wic under bitbake, the wic output directory is added to PSEUDO_PATHS_IGNORE to avoid issues with files being removed from outside a pseudo environment (see oe-core commit ad8f5532ff).
However, wic will copy the rootfs directory into its working directory if it needs to add or remove content before creating a filesystem image. This copy of the rootfs directory must be tracked by pseudo in order to keep the permissions correct in the resulting image. So we can't have the wic working directory under a path in PSEUDO_PATHS_IGNORE unless we like broken permissions.
To fix this the new '-w' argument to wic is used to move the working directory away from the output directory.
Note that wic deletes the temporary working directory automatically when it finishes creating an image so users won't normally see the 'tmp-wic' directory under WORKDIR.
Fixes [Yocto #14129]
--- meta/classes/image_types_wic.bbclass | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
meta/classes/image_types_wic.bbclass +7 -1
@@ -29,11 +29,17 @@ WIC_CREATE_EXTRA_ARGS ?= "" IMAGE_CMD_wic () { out="${IMGDEPLOYDIR}/${IMAGE_NAME}" build_wic="${WORKDIR}/build-wic" + tmp_wic="${WORKDIR}/tmp-wic" wks="${WKS_FULL_PATH}" + if [ -e "$tmp_wic" ]; then + # Ensure we don't have any junk leftover from a previously interrupted + # do_image_wic execution + rm -rf "$tmp_wic" + fi if [ -z "$wks" ]; then bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately." fi - BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS} + BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS} mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic" } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
Signature · 1 line
2.25.1
PATCH 44/47 wic: Update pseudo db when excluding content from rootfs
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
To exclude content from the rootfs, wic makes a copy (using hardlinks if possible) of the rootfs directory and associated pseudo db, then removes files & directories as needed. However if these files and directories are removed using the python functions os.remove and shutil.rmtree, the copied pseudo db will not be updated correctly. For files copied from the original rootfs, if hardlinks were used successfully when copying the rootfs this should mean that the relevant inodes can't be reused and so the risk of pseudo aborts should be avoided. However, this logic doesn't apply for directories (as they can't be hardlinked) or for files added via the '--include-path' argument (as they weren't present in the original rootfs) and so there remains some risk of inodes being reused and the pseudo db becoming corrupted.
To fix this, use the 'rm' command under pseudo when removing files & directories from the copied rootfs to ensure that the copied pseudo db is updated.
--- scripts/lib/wic/plugins/source/rootfs.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
scripts/lib/wic/plugins/source/rootfs.py +8 -5
@@ -129,17 +129,20 @@ class RootfsPlugin(SourcePlugin): logger.error("'%s' points to a path outside the rootfs" % orig_path) sys.exit(1) + if new_pseudo: + pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) + else: + pseudo = None if path.endswith(os.sep): # Delete content only. for entry in os.listdir(full_path): full_entry = os.path.join(full_path, entry) - if os.path.isdir(full_entry) and not os.path.islink(full_entry): - shutil.rmtree(full_entry) - else: - os.remove(full_entry) + rm_cmd = "rm -rf %s" % (full_entry) + exec_native_cmd(rm_cmd, native_sysroot, pseudo) else: # Delete whole directory. - shutil.rmtree(full_path) + rm_cmd = "rm -rf %s" % (full_path) + exec_native_cmd(rm_cmd, native_sysroot, pseudo) part.prepare_rootfs(cr_workdir, oe_builddir, new_rootfs or part.rootfs_dir, native_sysroot,
Signature · 1 line
2.25.1
PATCH 45/47 wic: Add --change-directory argument
In reply to Steve SakomanFrom: Ricardo Ribalda Delgado <[email protected]>
This option allows to specify which part of a rootfs is going to be included, the same way the -C argument on tar.
Thanks to this option we can make sure the permissions and usernames on the target partition are respected, and also simplify the creation of splitted partitons, not neeting to invoke external vars or using .wks.in files. Eg:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/ part /etc --source rootfs --fstype=ext4 --change-directory=etc
--- scripts/lib/wic/help.py | 6 ++++++ scripts/lib/wic/ksparser.py | 1 + scripts/lib/wic/partition.py | 1 + scripts/lib/wic/plugins/source/rootfs.py | 21 ++++++++++++++++++--- 4 files changed, 26 insertions(+), 3 deletions(-)
scripts/lib/wic/help.py +6 -0
@@ -980,6 +980,12 @@ DESCRIPTION copies. This option only has an effect with the rootfs source plugin. + --change-directory: This option is specific to wic. It changes to the + given directory before copying the files. This + option is useful when we want to split a rootfs in + multiple partitions and we want to keep the right + permissions and usernames in all the partitions. + --extra-space: This option is specific to wic. It adds extra space after the space filled by the content of the partition. The final size can go
scripts/lib/wic/ksparser.py +1 -0
@@ -152,6 +152,7 @@ class KickStart(): part.add_argument('--offset', type=sizetype("K", True)) part.add_argument('--exclude-path', nargs='+') part.add_argument('--include-path', nargs='+') + part.add_argument('--change-directory') part.add_argument("--extra-space", type=sizetype("M")) part.add_argument('--fsoptions', dest='fsopts') part.add_argument('--fstype', default='vfat',
scripts/lib/wic/partition.py +1 -0
@@ -31,6 +31,7 @@ class Partition(): self.extra_space = args.extra_space self.exclude_path = args.exclude_path self.include_path = args.include_path + self.change_directory = args.change_directory self.fsopts = args.fsopts self.fstype = args.fstype self.label = args.label
scripts/lib/wic/plugins/source/rootfs.py +18 -3
@@ -86,14 +86,29 @@ class RootfsPlugin(SourcePlugin): new_rootfs = None new_pseudo = None # Handle excluded paths. - if part.exclude_path or part.include_path: + if part.exclude_path or part.include_path or part.change_directory: # We need a new rootfs directory we can delete files from. Copy to # workdir. new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno)) if os.path.lexists(new_rootfs): shutil.rmtree(os.path.join(new_rootfs)) - copyhardlinktree(part.rootfs_dir, new_rootfs) + + if part.change_directory: + cd = part.change_directory + if cd[-1] == '/': + cd = cd[:-1] + if os.path.isabs(cd): + logger.error("Must be relative: --change-directory=%s" % cd) + sys.exit(1) + orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd)) + if not orig_dir.startswith(part.rootfs_dir): + logger.error("'%s' points to a path outside the rootfs" % orig_dir) + sys.exit(1) + + else: + orig_dir = part.rootfs_dir + copyhardlinktree(orig_dir, new_rootfs) # Convert the pseudo directory to its new location if (pseudo_dir):
@@ -108,7 +123,7 @@ class RootfsPlugin(SourcePlugin): pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo), - part.rootfs_dir, new_rootfs) + orig_dir, new_rootfs) exec_native_cmd(pseudo_cmd, native_sysroot) for path in part.include_path or []:
Signature · 1 line
2.25.1
PATCH 46/47 wic: Copy rootfs dir if fstab needs updating
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
By default, wic updates the /etc/fstab in the rootfs to include details of additional partitions described in the selected wks file. If this modification is performed in place, other tasks which create an image file from the rootfs directory (e.g. do_image_tar and do_image_ext4) will pick up the modified fstab file which would not be appropriate for those images as they do not include the additional partitions described in the wks file. wic does undo modifications to the fstab file once it has finished creating the filesystem image, however this leaves open a race condition if one of the other tasks reads the contents of the fstab file from the rootfs directory between the point where wic modifies the fstab file and the point where wic restores the files original content.
This could be solved by adding a lockfile for tasks which use the rootfs directory to ensure that no other such task is reading the rootfs directory while do_image_wic is running. This would serialize several do_image_* tasks and result in slower builds, especially for large images. Another drawback of this solution is that it is hard to selectively optimise - adding lockfiles to do_image_* tasks would result in these tasks always being serialized even if no fstab modification will take place.
An alternative solution is to copy the rootfs directory when fstab needs to be modified. The code to do this in wic already exists as it is needed when including or excluding content in the rootfs. This still results in an impact on build times but the copy uses hardlinks if possible (so little data is actually copied) and we can make selective optimisations to improve things. The rootfs copy will only take place if fstab modification is required (or if it was already needed to include or exclude rootfs content). We can also follow up with further optimisations after this commit. So this second solution is chosen.
Fixes [Yocto #13994]
--- scripts/lib/wic/partition.py | 5 +++- scripts/lib/wic/plugins/imager/direct.py | 36 ++++++++---------------- scripts/lib/wic/plugins/source/rootfs.py | 17 +++++++++-- 3 files changed, 29 insertions(+), 29 deletions(-)
scripts/lib/wic/partition.py +4 -1
@@ -54,6 +54,7 @@ class Partition(): self.uuid = args.uuid self.fsuuid = args.fsuuid self.type = args.type + self.updated_fstab_path = None self.lineno = lineno self.source_file = ""
@@ -118,11 +119,13 @@ class Partition(): return self.fixed_size if self.fixed_size else self.size def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir, - bootimg_dir, kernel_dir, native_sysroot): + bootimg_dir, kernel_dir, native_sysroot, updated_fstab_path): """ Prepare content for individual partitions, depending on partition command parameters. """ + self.updated_fstab_path = updated_fstab_path + if not self.source: if not self.size and not self.fixed_size: raise WicError("The %s partition has a size of zero. Please "
scripts/lib/wic/plugins/imager/direct.py +11 -25
@@ -58,7 +58,7 @@ class DirectPlugin(ImagerPlugin): self.compressor = options.compressor self.bmap = options.bmap self.no_fstab_update = options.no_fstab_update - self.original_fstab = None + self.updated_fstab_path = None self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0], strftime("%Y%m%d%H%M"))
@@ -100,11 +100,8 @@ class DirectPlugin(ImagerPlugin): finally: self.cleanup() - def _write_fstab(self, image_rootfs): - """overriden to generate fstab (temporarily) in rootfs. This is called - from _create, make sure it doesn't get called from - BaseImage.create() - """ + def update_fstab(self, image_rootfs): + """Assume partition order same as in wks""" if not image_rootfs: return
@@ -114,18 +111,9 @@ class DirectPlugin(ImagerPlugin): with open(fstab_path) as fstab: fstab_lines = fstab.readlines() - self.original_fstab = fstab_lines.copy() - - if self._update_fstab(fstab_lines, self.parts): - with open(fstab_path, "w") as fstab: - fstab.writelines(fstab_lines) - else: - self.original_fstab = None - def _update_fstab(self, fstab_lines, parts): - """Assume partition order same as in wks""" updated = False - for part in parts: + for part in self.parts: if not part.realnum or not part.mountpoint \ or part.mountpoint == "/": continue
@@ -154,7 +142,10 @@ class DirectPlugin(ImagerPlugin): fstab_lines.append(line) updated = True - return updated + if updated: + self.updated_fstab_path = os.path.join(self.workdir, "fstab") + with open(self.updated_fstab_path, "w") as f: + f.writelines(fstab_lines) def _full_path(self, path, name, extention): """ Construct full file path to a file we generate. """
@@ -170,7 +161,7 @@ class DirectPlugin(ImagerPlugin): a partitioned image. """ if not self.no_fstab_update: - self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR")) + self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR")) for part in self.parts: # get rootfs size from bitbake variable if it's not set in .ks file
@@ -283,12 +274,6 @@ class DirectPlugin(ImagerPlugin): if os.path.isfile(path): shutil.move(path, os.path.join(self.outdir, fname)) - #Restore original fstab - if self.original_fstab: - fstab_path = self.rootfs_dir.get("ROOTFS_DIR") + "/etc/fstab" - with open(fstab_path, "w") as fstab: - fstab.writelines(self.original_fstab) - # remove work directory shutil.rmtree(self.workdir, ignore_errors=True)
@@ -368,7 +353,8 @@ class PartitionedImage(): # sizes before we can add them and do the layout. part.prepare(imager, imager.workdir, imager.oe_builddir, imager.rootfs_dir, imager.bootimg_dir, - imager.kernel_dir, imager.native_sysroot) + imager.kernel_dir, imager.native_sysroot, + imager.updated_fstab_path) # Converting kB to sectors for parted part.size_sec = part.disk_size * 1024 // self.sector_size
scripts/lib/wic/plugins/source/rootfs.py +14 -3
@@ -86,9 +86,9 @@ class RootfsPlugin(SourcePlugin): new_rootfs = None new_pseudo = None # Handle excluded paths. - if part.exclude_path or part.include_path or part.change_directory: - # We need a new rootfs directory we can delete files from. Copy to - # workdir. + if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path: + # We need a new rootfs directory we can safely modify without + # interfering with other tasks. Copy to workdir. new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno)) if os.path.lexists(new_rootfs):
@@ -159,6 +159,17 @@ class RootfsPlugin(SourcePlugin): rm_cmd = "rm -rf %s" % (full_path) exec_native_cmd(rm_cmd, native_sysroot, pseudo) + has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab")) + if part.updated_fstab_path and has_fstab: + fstab_path = os.path.join(new_rootfs, "etc/fstab") + # Assume that fstab should always be owned by root with fixed permissions + install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path) + if new_pseudo: + pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) + else: + pseudo = None + exec_native_cmd(install_cmd, native_sysroot, pseudo) + part.prepare_rootfs(cr_workdir, oe_builddir, new_rootfs or part.rootfs_dir, native_sysroot, pseudo_dir = new_pseudo or pseudo_dir)
Signature · 1 line
2.25.1
PATCH 47/47 wic: Optimise fstab modification for ext2/3/4 and msdos partitions
In reply to Steve SakomanFrom: Paul Barker <[email protected]>
The fix for [Yocto #13994] required the rootfs directory to be copied (using hardlinks if possible) when modifying the fstab file under wic.
We can optimise this copy away for filesystems where we have the tools to modify the contents of the partition image after it is created. For ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat filesystems we have the mcopy tool. So for any of these filesystems we skip the modification of the fstab file in the rootfs directory (and skip the associated copy unless it is otherwise necessary) and update the contents of fstab directly in the partition image.
--- scripts/lib/wic/partition.py | 27 +++++++++++++++++++----- scripts/lib/wic/plugins/source/rootfs.py | 9 +++++--- 2 files changed, 28 insertions(+), 8 deletions(-)
scripts/lib/wic/partition.py +22 -5
@@ -55,6 +55,8 @@ class Partition(): self.fsuuid = args.fsuuid self.type = args.type self.updated_fstab_path = None + self.has_fstab = False + self.update_fstab_in_rootfs = False self.lineno = lineno self.source_file = ""
@@ -125,6 +127,8 @@ class Partition(): partition command parameters. """ self.updated_fstab_path = updated_fstab_path + if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"): + self.update_fstab_in_rootfs = True if not self.source: if not self.size and not self.fixed_size:
@@ -250,7 +254,7 @@ class Partition(): prefix = "ext" if self.fstype.startswith("ext") else self.fstype method = getattr(self, "prepare_rootfs_" + prefix) - method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo) + method(rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo) self.source_file = rootfs # get the rootfs size in the right units for kickstart (kB)
@@ -258,7 +262,7 @@ class Partition(): out = exec_cmd(du_cmd) self.size = int(out.split()[0]) - def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir, + def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ Prepare content for an ext2/3/4 rootfs partition.
@@ -282,10 +286,19 @@ class Partition(): (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) + if self.updated_fstab_path and self.has_fstab: + debugfs_script_path = os.path.join(cr_workdir, "debugfs_script") + with open(debugfs_script_path, "w") as f: + f.write("cd etc\n") + f.write("rm fstab\n") + f.write("write %s fstab\n" % (self.updated_fstab_path)) + debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs) + exec_native_cmd(debugfs_cmd, native_sysroot) + mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir, + def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ Prepare content for a btrfs rootfs partition.
@@ -308,7 +321,7 @@ class Partition(): self.mkfs_extraopts, self.fsuuid, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir, + def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ Prepare content for a msdos/vfat rootfs partition.
@@ -337,12 +350,16 @@ class Partition(): mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) exec_native_cmd(mcopy_cmd, native_sysroot) + if self.updated_fstab_path and self.has_fstab: + mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) + exec_native_cmd(mcopy_cmd, native_sysroot) + chmod_cmd = "chmod 644 %s" % rootfs exec_cmd(chmod_cmd) prepare_rootfs_vfat = prepare_rootfs_msdos - def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir, + def prepare_rootfs_squashfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ Prepare content for a squashfs rootfs partition.
scripts/lib/wic/plugins/source/rootfs.py +6 -3
@@ -77,6 +77,7 @@ class RootfsPlugin(SourcePlugin): "it is not a valid path, exiting" % part.rootfs_dir) part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir) + part.has_fstab = os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab")) pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo") if not os.path.lexists(pseudo_dir): logger.warn("%s folder does not exist. "
@@ -86,7 +87,7 @@ class RootfsPlugin(SourcePlugin): new_rootfs = None new_pseudo = None # Handle excluded paths. - if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path: + if part.exclude_path or part.include_path or part.change_directory or part.update_fstab_in_rootfs: # We need a new rootfs directory we can safely modify without # interfering with other tasks. Copy to workdir. new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
@@ -159,8 +160,10 @@ class RootfsPlugin(SourcePlugin): rm_cmd = "rm -rf %s" % (full_path) exec_native_cmd(rm_cmd, native_sysroot, pseudo) - has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab")) - if part.updated_fstab_path and has_fstab: + # Update part.has_fstab here as fstab may have been added or + # removed by the above modifications. + part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab")) + if part.update_fstab_in_rootfs and part.has_fstab: fstab_path = os.path.join(new_rootfs, "etc/fstab") # Assume that fstab should always be owned by root with fixed permissions install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
Signature · 1 line
2.25.1
Quoted 20 lines · Steve Sakoman
Issues with undetected file mode corruption in pseudo have been identified.
Fixes have been merged into master and gatesgarth over the past several months and things seem to have stabilized enough that we can consider backporting these fixes to dunfell.
This is a somewhat more invasive change than normal. Specifically the user will be required to clean TMPDIR after this merge. It may also expose issues with other layers, though master and gategarth have probably helped to pave the way for this change.
Personally I think that this fixes a serious enough issue that we should consider this series for dunfell.
This second version includes the wic patches which have been the gating item for merging this series.
Note also that there is an associated bitbake patch for stable/1.46:
Quoted 7 lines
lib/bb/utils.py: Do not preserve TERM in the environmentThis is due to taking:
bitbake.conf: Remove TERM from default
BB_HASHBASE_WHITELISTwhich was needed to do clean cherry-picks of the pseudo changes.
If this is just brought in to allow clean cherry-picks I think it would be better left out and instead do the small bit of rebasing required in the patches. We don't know what downstream users are doing with TERM so minimising unnecessary changes in the stable branch is preferable.
Everything else looks good to me at a glance.
Signature · 2 lines
Paul Barker Konsulko Group
Quoted 28 lines · Paul Barker
Quoted 26 lines · Steve Sakoman
Issues with undetected file mode corruption in pseudo have been identified.
Fixes have been merged into master and gatesgarth over the past several months and things seem to have stabilized enough that we can consider backporting these fixes to dunfell.
This is a somewhat more invasive change than normal. Specifically the user will be required to clean TMPDIR after this merge. It may also expose issues with other layers, though master and gategarth have probably helped to pave the way for this change.
Personally I think that this fixes a serious enough issue that we should consider this series for dunfell.
This second version includes the wic patches which have been the gating item for merging this series.
Note also that there is an associated bitbake patch for stable/1.46:
lib/bb/utils.py: Do not preserve TERM in the environmentThis is due to taking:
bitbake.conf: Remove TERM from default
BB_HASHBASE_WHITELIST
Quoted 7 lines
Quoted 1 line
which was needed to do clean cherry-picks of the pseudo changes.
If this is just brought in to allow clean cherry-picks I think it would be better left out and instead do the small bit of rebasing required in the patches. We don't know what downstream users are doing with TERM so minimising unnecessary changes in the stable branch is preferable.
OK, will do.
Quoted 1 line
Everything else looks good to me at a glance.
Thanks for reviewing. I'll wait for more feedback and send a v3. This is one series we really want to get right!
Steve
Quoted 38 lines · Steve Sakoman
Quoted 28 lines · Paul Barker
Quoted 26 lines · Steve Sakoman
Issues with undetected file mode corruption in pseudo have been identified.
Fixes have been merged into master and gatesgarth over the past several months and things seem to have stabilized enough that we can consider backporting these fixes to dunfell.
This is a somewhat more invasive change than normal. Specifically the user will be required to clean TMPDIR after this merge. It may also expose issues with other layers, though master and gategarth have probably helped to pave the way for this change.
Personally I think that this fixes a serious enough issue that we should consider this series for dunfell.
This second version includes the wic patches which have been the gating item for merging this series.
Note also that there is an associated bitbake patch for stable/1.46:
lib/bb/utils.py: Do not preserve TERM in the environmentThis is due to taking:
bitbake.conf: Remove TERM from default
BB_HASHBASE_WHITELISTQuoted 7 lines
Quoted 1 line
which was needed to do clean cherry-picks of the pseudo changes.
If this is just brought in to allow clean cherry-picks I think it would be better left out and instead do the small bit of rebasing required in the patches. We don't know what downstream users are doing with TERM so minimising unnecessary changes in the stable branch is preferable.
Quoted 6 lines
OK, will do.
Quoted 1 line
Everything else looks good to me at a glance.
Thanks for reviewing. I'll wait for more feedback and send a v3. This is one series we really want to get right!
Agreed. It's a difficult balance. I want stable branches to be as conservative as possible and not risk making breaking changes but this series fixes an important issue (risk of silently creating images with incorrect permissions). On balance it does make sense to backport these pretty invasive changes.
Thanks,
Signature · 2 lines
Paul Barker Konsulko Group
Quoted 17 lines · Steve Sakoman
Issues with undetected file mode corruption in pseudo have been identified.
Fixes have been merged into master and gatesgarth over the past several months and
things seem to have stabilized enough that we can consider backporting these fixes
to dunfell.This is a somewhat more invasive change than normal. Specifically the user will
be required to clean TMPDIR after this merge. It may also expose issues with
other layers, though master and gategarth have probably helped to pave the way for
this change.
Quoted 3 lines
Personally I think that this fixes a serious enough issue that we should consider
this series for dunfell.
I think it might be better to wait for some more time before they are merged in LTS so they get wider testing by people who might be using gatesgarth. The wic patches have not been merged in gatesgarth yet (I just sent the pull request).
Thanks,
Anuj
Quoted 178 lines
This second version includes the wic patches which have been the gating item for
merging this series.Note also that there is an associated bitbake patch for stable/1.46:
lib/bb/utils.py: Do not preserve TERM in the environmentThis is due to taking:
bitbake.conf: Remove TERM from default
BB_HASHBASE_WHITELISTwhich was needed to do clean cherry-picks of the pseudo changes.
Let me know if you think this is an issue, or if you are aware of any other pseudo
issues which would necessitate delaying the merge of this series.Please have comments back by Wednesday evening.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1779
The following changes since commit
897822233faef0f8f35dc1d8a39e1c4bc0550f1e:ppp: Whitelist CVE-2020-15704 (2021-01-19 04:22:10 -1000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib
stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nutJack Mitchell (1):
distutils3: allowsetup.pyto be run from a different directory to ${S}Jacob Kroon (1):
bitbake.conf: Remove TERM from defaultBB_HASHBASE_WHITELISTJoe Slater (1):
pseudo: fix renaming to selfMartin Jansa (2):
base.bbclass: use os.path.normpath instead of just comparing WORKDIR
and S as strings
license.bbclass: AddCOMMON_LICENSE_DIRandLICENSE_PATHdirs toPSEUDO_IGNORE_PATHSMingli Yu (2):
tcl: adapt to potential pseudo changes bitbake.conf: Exclude ${CCACHE_DIR} from pseudo databasePaul Barker (8):
bitbake.conf: Prevent pyc file generation in pseudo context
wic: Add workdir argument
wic: Allowexec_native_cmdto run HOSTTOOLS
wic: Ensure internal workdir is not reused
image_types_wic: Move wic working directory
wic: Update pseudo db when excluding content from rootfs
wic: Copy rootfs dir if fstab needs updating
wic: Optimise fstab modification for ext2/3/4 and msdos partitionsPeter Kjellerstedt (4):
pseudo: Simplifypseudo_client_ignore_path_chroot()lib/oe/path: Addcanonicalize()
bitbake.conf: Canonicalize paths inPSEUDO_IGNORE_PATHS
wic: Pass canonicalized paths inPSEUDO_IGNORE_PATHSRicardo Ribalda Delgado (4):
wic: Fix permissions when using exclude or include path
wic: Fix multi images .wks with bitbake
wic: Avoid creating invalid pseudo directory
wic: Add --change-directory argumentRichard Purdie (23):
pseudo: Switch to oe-core branch in git repo
pseudo: merge in fixes for setfacl issue
pseudo: Update to add OFC fcntl lock updates
pseudo: Ignore mismatched inodes from the db
pseudo: Add support for ignoring paths from the pseudo DB
pseudo: Abort on mismatch patch
psuedo: Add tracking of linked files for fds
pseudo: Fix xattr segfault
pseudo: Add may unlink patch
pseudo: Add pathfix patch
base/bitbake.conf: Enable pseudo path filtering
wic: Handle newPSEUDO_IGNORE_PATHSvariable
pseudo: Fix statx function usage
bitbake.conf: ExtendPSEUDO_IGNORE_PATHSto ${COREBASE}/meta abi_version,sanity: Tell users TMPDIR must be clean after pseudo changes
pseudo: Update to account for patches merged on branch
pseudo: Upgrade to include mkostemp64 wrapper oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database
uninative: Don't use single sstate for pseudo-native
pseudo: Drop patches merged into upstream branch bitbake.conf: Add/run/toPSEUDO_IGNORE_PATHS
pseudo: Add lchmod wrapper
pseudo: Update for arm host and memleak fixes/cleanupTomasz Dziendzielski (1):
pseudo: Update to printPSEUDO_LOGFILEin abort message on path mismatchesmeta-selftest/lib/pseudo_pyc_test1.py | 1 + meta-selftest/lib/pseudo_pyc_test2.py | 1 + .../pseudo-pyc-test/pseudo-pyc-test.bb | 15 ++ meta/classes/archiver.bbclass | 2 +- meta/classes/base.bbclass | 6 + meta/classes/distutils3.bbclass | 10 +- meta/classes/image_types_wic.bbclass | 20 ++- meta/classes/license.bbclass | 1 + meta/classes/populate_sdk_base.bbclass | 2 + meta/classes/sanity.bbclass | 3 + meta/classes/sstate.bbclass | 4 + meta/conf/abi_version.conf | 2 +- meta/conf/bitbake.conf | 13 +- meta/lib/oe/path.py | 21 +++ meta/lib/oe/sstatesig.py | 4 +- meta/lib/oeqa/selftest/cases/pseudo.py | 27 ++++ meta/lib/oeqa/selftest/cases/runtime_test.py | 1 + .../pseudo/files/0001-Add-statx.patch | 106 -------------- ...001-maketables-wrappers-use-Python-3.patch | 34 ----- ...ixup-remove-files-that-do-not-exist-.patch | 49 ------- .../0001-pseudo_ipc.h-Fix-enum-typedef.patch | 31 ---- ...1-realpath.c-Remove-trailing-slashes.patch | 57 -------- ...xattr-adjust-for-attr-2.4.48-release.patch | 48 ------ .../pseudo/files/moreretries.patch | 19 --- .../pseudo/files/seccomp.patch | 137 ---------------- --
.../pseudo/files/toomanyfiles.patch | 71 --------- .../pseudo/files/xattr_version.patch | 54 ------- meta/recipes-devtools/pseudo/pseudo_git.bb | 14 +- meta/recipes-devtools/tcltk/tcl_8.6.10.bb | 1 + scripts/lib/wic/help.py | 6 + scripts/lib/wic/ksparser.py | 1 + scripts/lib/wic/misc.py | 7 +- scripts/lib/wic/partition.py | 66 ++++++--- scripts/lib/wic/plugins/imager/direct.py | 48 +++--- scripts/lib/wic/plugins/source/rootfs.py | 85 +++++++++--scripts/postinst-intercepts/update_font_cache| 2 + scripts/wic | 2 +37 files changed, 281 insertions(+), 690 deletions(-)create mode 100644
meta-selftest/lib/pseudo_pyc_test1.pycreate mode 100644meta-selftest/lib/pseudo_pyc_test2.pycreate mode 100644 meta-selftest/recipes-test/pseudo-pyc- test/pseudo-pyc-test.bb
create mode 100644meta/lib/oeqa/selftest/cases/pseudo.pydelete mode 100644 meta/recipes-devtools/pseudo/files/0001-Add- statx.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0001- maketables-wrappers-use-Python-3.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo- On-a-DB-fixup-remove-files-that-do-not-exist-.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001- pseudo_ipc.h-Fix-enum-typedef.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0001- realpath.c-Remove-trailing-slashes.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0006-xattr- adjust-for-attr-2.4.48-release.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/moreretries.patch
delete mode 100644meta/recipes-devtools/pseudo/files/seccomp.patchdelete mode 100644 meta/recipes-devtools/pseudo/files/toomanyfiles.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_version.patch
Quoted 19 lines · Anuj
Quoted 17 lines · Steve Sakoman
Issues with undetected file mode corruption in pseudo have been identified.
Fixes have been merged into master and gatesgarth over the past several months and
things seem to have stabilized enough that we can consider backporting these fixes
to dunfell.This is a somewhat more invasive change than normal. Specifically the user will
be required to clean TMPDIR after this merge. It may also expose issues with
other layers, though master and gategarth have probably helped to pave the way for
this change.
Quoted 8 lines
Quoted 3 lines
Personally I think that this fixes a serious enough issue that we should consider
this series for dunfell.I think it might be better to wait for some more time before they are merged in LTS so they get wider testing by people who might be using gatesgarth. The wic patches have not been merged in gatesgarth yet (I just sent the pull request).
How long would you suggest that we wait?
I felt that this was a good time since we've just released 3.1.5 and it would give layer maintainers a good chunk of time to react before we release 3.1.6
I'm open to suggestions, but if we don't do it now I feel we ought to wait till just after 3.1.6 then.
Steve
Quoted 178 lines
Quoted 177 lines
This second version includes the wic patches which have been the gating item for
merging this series.Note also that there is an associated bitbake patch for stable/1.46:
lib/bb/utils.py: Do not preserve TERM in the environmentThis is due to taking:
bitbake.conf: Remove TERM from default
BB_HASHBASE_WHITELISTwhich was needed to do clean cherry-picks of the pseudo changes.
Let me know if you think this is an issue, or if you are aware of any other pseudo
issues which would necessitate delaying the merge of this series.Please have comments back by Wednesday evening.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1779
The following changes since commit
897822233faef0f8f35dc1d8a39e1c4bc0550f1e:ppp: Whitelist CVE-2020-15704 (2021-01-19 04:22:10 -1000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib
stable/dunfell-nuthttp://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut
Jack Mitchell (1):
distutils3: allowsetup.pyto be run from a different directory to${S}Jacob Kroon (1):
bitbake.conf: Remove TERM from defaultBB_HASHBASE_WHITELISTJoe Slater (1):
pseudo: fix renaming to selfMartin Jansa (2):
base.bbclass: use os.path.normpath instead of just comparing
WORKDIR
and S as strings
license.bbclass: AddCOMMON_LICENSE_DIRandLICENSE_PATHdirs to
PSEUDO_IGNORE_PATHSMingli Yu (2):
tcl: adapt to potential pseudo changes bitbake.conf: Exclude ${CCACHE_DIR} from pseudo databasePaul Barker (8):
bitbake.conf: Prevent pyc file generation in pseudo context
wic: Add workdir argument
wic: Allowexec_native_cmdto run HOSTTOOLS
wic: Ensure internal workdir is not reused
image_types_wic: Move wic working directory
wic: Update pseudo db when excluding content from rootfs
wic: Copy rootfs dir if fstab needs updating
wic: Optimise fstab modification for ext2/3/4 and msdos partitionsPeter Kjellerstedt (4):
pseudo: Simplifypseudo_client_ignore_path_chroot()lib/oe/path: Addcanonicalize()
bitbake.conf: Canonicalize paths inPSEUDO_IGNORE_PATHS
wic: Pass canonicalized paths inPSEUDO_IGNORE_PATHSRicardo Ribalda Delgado (4):
wic: Fix permissions when using exclude or include path
wic: Fix multi images .wks with bitbake
wic: Avoid creating invalid pseudo directory
wic: Add --change-directory argumentRichard Purdie (23):
pseudo: Switch to oe-core branch in git repo
pseudo: merge in fixes for setfacl issue
pseudo: Update to add OFC fcntl lock updates
pseudo: Ignore mismatched inodes from the db
pseudo: Add support for ignoring paths from the pseudo DB
pseudo: Abort on mismatch patch
psuedo: Add tracking of linked files for fds
pseudo: Fix xattr segfault
pseudo: Add may unlink patch
pseudo: Add pathfix patch
base/bitbake.conf: Enable pseudo path filtering
wic: Handle newPSEUDO_IGNORE_PATHSvariable
pseudo: Fix statx function usage
bitbake.conf: ExtendPSEUDO_IGNORE_PATHSto ${COREBASE}/meta abi_version,sanity: Tell users TMPDIR must be clean after pseudo
changes
pseudo: Update to account for patches merged on branch
pseudo: Upgrade to include mkostemp64 wrapper oeqa/selftest/runtime_test: Exclude gpg directory from pseudo
database
uninative: Don't use single sstate for pseudo-native
pseudo: Drop patches merged into upstream branch bitbake.conf: Add/run/toPSEUDO_IGNORE_PATHS
pseudo: Add lchmod wrapper
pseudo: Update for arm host and memleak fixes/cleanupTomasz Dziendzielski (1):
pseudo: Update to printPSEUDO_LOGFILEin abort message on path
mismatchesmeta-selftest/lib/pseudo_pyc_test1.py | 1 + meta-selftest/lib/pseudo_pyc_test2.py | 1 + .../pseudo-pyc-test/pseudo-pyc-test.bb | 15 ++ meta/classes/archiver.bbclass | 2 +- meta/classes/base.bbclass | 6 + meta/classes/distutils3.bbclass | 10 +- meta/classes/image_types_wic.bbclass | 20 ++- meta/classes/license.bbclass | 1 + meta/classes/populate_sdk_base.bbclass | 2 + meta/classes/sanity.bbclass | 3 + meta/classes/sstate.bbclass | 4 + meta/conf/abi_version.conf | 2 +- meta/conf/bitbake.conf | 13 +- meta/lib/oe/path.py | 21 +++ meta/lib/oe/sstatesig.py | 4 +- meta/lib/oeqa/selftest/cases/pseudo.py | 27 ++++ meta/lib/oeqa/selftest/cases/runtime_test.py | 1 + .../pseudo/files/0001-Add-statx.patch | 106 -------------- ...001-maketables-wrappers-use-Python-3.patch | 34 ----- ...ixup-remove-files-that-do-not-exist-.patch | 49 ------- .../0001-pseudo_ipc.h-Fix-enum-typedef.patch | 31 ---- ...1-realpath.c-Remove-trailing-slashes.patch | 57 -------- ...xattr-adjust-for-attr-2.4.48-release.patch | 48 ------ .../pseudo/files/moreretries.patch | 19 --- .../pseudo/files/seccomp.patch | 137 ---------------- -- .../pseudo/files/toomanyfiles.patch | 71 --------- .../pseudo/files/xattr_version.patch | 54 ------- meta/recipes-devtools/pseudo/pseudo_git.bb | 14 +- meta/recipes-devtools/tcltk/tcl_8.6.10.bb | 1 + scripts/lib/wic/help.py | 6 + scripts/lib/wic/ksparser.py | 1 + scripts/lib/wic/misc.py | 7 +- scripts/lib/wic/partition.py | 66 ++++++--- scripts/lib/wic/plugins/imager/direct.py | 48 +++--- scripts/lib/wic/plugins/source/rootfs.py | 85 +++++++++-- scripts/postinst-intercepts/update_font_cache | 2 + scripts/wic | 2 + 37 files changed, 281 insertions(+), 690 deletions(-) create mode 100644 meta-selftest/lib/pseudo_pyc_test1.py create mode 100644 meta-selftest/lib/pseudo_pyc_test2.py create mode 100644 meta-selftest/recipes-test/pseudo-pyc- test/pseudo-pyc-test.bb create mode 100644 meta/lib/oeqa/selftest/cases/pseudo.py delete mode 100644 meta/recipes-devtools/pseudo/files/0001-Add- statx.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001- maketables-wrappers-use-Python-3.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo- On-a-DB-fixup-remove-files-that-do-not-exist-.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001- pseudo_ipc.h-Fix-enum-typedef.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0001- realpath.c-Remove-trailing-slashes.patch delete mode 100644 meta/recipes-devtools/pseudo/files/0006-xattr- adjust-for-attr-2.4.48-release.patch delete mode 100644 meta/recipes- devtools/pseudo/files/moreretries.patch delete mode 100644 meta/recipes-devtools/pseudo/files/seccomp.patch delete mode 100644 meta/recipes- devtools/pseudo/files/toomanyfiles.patch delete mode 100644 meta/recipes-
devtools/pseudo/files/xattr_version.patch
Quoted 29 lines · Steve Sakoman
Quoted 19 lines · Anuj
Quoted 17 lines · Steve Sakoman
Issues with undetected file mode corruption in pseudo have been identified.
Fixes have been merged into master and gatesgarth over the past several months and
things seem to have stabilized enough that we can consider backporting these fixes
to dunfell.This is a somewhat more invasive change than normal. Specifically the user will
be required to clean TMPDIR after this merge. It may also expose issues with
other layers, though master and gategarth have probably helped to pave the way for
this change.Quoted 8 lines
Quoted 3 lines
Personally I think that this fixes a serious enough issue that we should consider
this series for dunfell.I think it might be better to wait for some more time before they are merged in LTS so they get wider testing by people who might be using gatesgarth. The wic patches have not been merged in gatesgarth yet (I just sent the pull request).
Quoted 8 lines
How long would you suggest that we wait?
I felt that this was a good time since we've just released 3.1.5 and it would give layer maintainers a good chunk of time to react before we release 3.1.6
I'm open to suggestions, but if we don't do it now I feel we ought to wait till just after 3.1.6 then.
It is a tricky balance but FWIW I'm leaning towards having these as one of the 3.1.6 changes.
Cheers,
Richard