pseudo: Outdated records for newly-ignored paths in database cause mismatches
Our CI Dunfell builds started failing during image creation with pseudo aborts like:
path mismatch [2 links]: ino 123107550 db '/.../build/tmp-glibc/work/mymachine-oe-linux/myimage/1.0-r2/oe-rootfs-repo/mymachine/mypackage-dbg_1.0-r7_mymachine.ipk' req '/.../build/mymachine-root/usr/bin'.
Inode 123107550 is the second of the two paths.
We're using the latest pseudo (b988b0a6b8afd8d459bc9a2528e834f63a3d59b2) because we ran into problems sharing sstate cache between different build OS versions prior to oe-core:d7e87a5851d717da047f552be394d5712efa0402.
The mismatches started happening just after we took oe-core:9463be2292b942a1072eea88881b9644e55aadb9 (as b04d7a7aed5b05e8561029c5e570206ac9b9fa4e for Dunfell):
index 459d872b4a..244f5bb8ff 100644
meta/classes/image.bbclass +2 -0
@@ -180,6 +180,8 @@ LINGUAS_INSTALL ?= "${@" ".join(map(lambda s: "locale-base-%s" % s, d.getVar('IM # aren't yet available. PSEUDO_PASSWD = "${IMAGE_ROOTFS}:${STAGING_DIR_NATIVE}" +PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/intercept_scripts,${WORKDIR}/oe-rootfs-repo" +
I was able to reproduce a similar problem by commenting out the above PSEUDO_IGNORE_PATHS line, building and image, putting it back and forcing do_rootfs for the image to run again without any intervening cleaning. It didn't happen every time though.
I believe that the pseudo database was populated with many paths in oe-rootfs-repo before this change. After the change, the files in oe-rootfs-repo were replaced which freed up their inodes, but because the paths were ignored the database wasn't updated. Those inodes were then used for files and directories in during rootfs creation. Pseudo incorrectly believed that these inodes were already associated with files it knew about based on the out-of-date database records.
Cleaning the work directory makes the problem go away because that deletes the pseudo databases.
Does the above make sense as an explanation for these errors? If so, is there a good way to avoid these errors?
Could pseudo check whether mismatched paths are now ignored and if so not treat the mismatch as fatal?
Should changing PSEUDO_IGNORE_PATHS cause all tasks for the recipe to be re-run so that the out-of-date database is removed?
Even if it's not worth employing some technical measure, perhaps this is worth mentioning as a potential false alarm at
https://wiki.yoctoproject.org/wiki/Pseudo_Abort ?
Thanks.
Mike.
Quoted 5 lines · On Mon, 9 Aug 2021 13:19:51 +0100 "Mike Crowe via lists.openembedded.org"…
Cleaning the work directory makes the problem go away because that deletes the pseudo databases.
Does the above make sense as an explanation for these errors? If so, is there a good way to avoid these errors?
Good diagnostic work, makes sense to me. It would make some sense for pseudo to ignore mismatches involving ignored paths, but it wasn't originally designed with the ignored paths concept, so it currently doesn't.
-s
Quoted 7 lines · Seebs
Quoted 3 lines · On Mon, 9 Aug 2021 13:19:51 +0100 "Mike Crowe via lists.openembedded.org"…
Cleaning the work directory makes the problem go away because that deletes the pseudo databases.
Quoted 7 lines
Quoted 2 lines
Does the above make sense as an explanation for these errors? If so, is there a good way to avoid these errors?
Good diagnostic work, makes sense to me. It would make some sense for pseudo to ignore mismatches involving ignored paths, but it wasn't originally designed with the ignored paths concept, so it currently doesn't.
Thanks for the review.
I have a test case and patch for pseudo (attached) to detect newly-ignored paths and warn rather than abort on them, but I'm not really convinced that it is the right solution. Ideally the errant entry would be removed from the database too in order to avoid having to continue to consult the ignore list.
It's not even clear to me that oe-core continuing to use an existing pseudo database after the value of PSEUDO_CLIENT_IGNORE_PATH changes is a sane thing to expect to work. Perhaps we could just arrange to force a whole new work directory in that case?
Thanks.
Mike.
From e81aeff391148280d76609e5782bf7f0a115f72e Mon Sep 17 00:00:00 2001
From: Mike Crowe <[email protected]>
Date: Wed, 11 Aug 2021 15:55:55 +0100
Subject: [PATCH] pseudo: Path mismatch on now-ignored path should not be fatal
If a database survives from before a change to PSEUDO_CLIENT_IGNORE_PATH then there's a risk that the now-ignored files have been deleted and their inodes re-used without pseudo noticing. Such files are reported as path mismatches which provoke aborts.
Let's check to see whether the database filename would now be ignored, and if so just warn about the mismatch rather than aborting.
Unfortunately the test case for this doesn't fit into the existing infrastructure since the server must be restarted during the test.
--- pseudo.c | 2 + run_tests.sh | 16 ++++++++ .../standalone-test-newly-ignored-mismatch.sh | 41 +++++++++++++++++++ 3 files changed, 59 insertions(+) create mode 100755 test/standalone-test-newly-ignored-mismatch.sh
pseudo.c +2 -0
@@ -695,6 +695,8 @@ pseudo_op(pseudo_msg_t *msg, const char *program, const char *tag, char **respon*/ pseudo_debug (PDBGF_FILE ,"inode mismatch for '%s' -- old one was marked for deletion.\n" , msg->path); + }else if (path_by_ino &&pseudo_client_ignore_path (path_by_ino)) { +pseudo_diag ("path mismatch on now-ignored '%s'" , path_by_ino); }else { pseudo_diag("path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n", msg->nlink,
run_tests.sh +16 -0
@@ -48,5 +48,21 @@ do fi rm -rf var/pseudo/* done +for file in test/standalone-test*.sh +do + filename=${file#test/} + let num_tests++ + mkdir -p var/pseudo + $file ${opt_verbose} + if [ "$?" -eq "0" ]; then + let num_passed_tests++ + if [ "${opt_verbose}" == "-v" ]; then + echo "${filename%.sh}: Passed." + fi + else + echo "${filename/%.sh}: Failed." + fi + rm -rf var/pseudo/* +done echo "${num_passed_tests}/${num_tests} test(s) passed."
test/standalone-test-newly-ignored-mismatch.sh new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.30.2