Skip to contentSkip to conversation

Ongoing pseudo issues and request for help

DISCUSSION Richard Purdie 1 participant 1 message Sep 30, 2020

lore
Share & tools
lore
#0 lore raw

As people have probably gathered we have some concerns about rare file mode corruption with pseudo. This is really really rare but its not right and we need to do something about it.

As discussed on list recently, we need to make pseudo abort() when it encounters such an issue rather than hacking around it.

Patches to make pseudo abort in such a case are in master-next, along with a set of other changes. These changes make pseudo ignore a set of directories from a permissions/ownership perspective.

The current setting is (line wrapped to make it more readable):

+PSEUDO_IGNORE_PATHS = " \
/usr/,
/etc/,
/lib,
/dev/,
${T},
${WORKDIR}/recipe-sysroot,
${SSTATE_DIR},
${STAMPS_DIR},
${WORKDIR}/pkgdata-sysroot,
${TMPDIR}/sstate-control,
${DEPLOY_DIR},
${WORKDIR}/deploy-,
${TMPDIR}/buildstats,
${WORKDIR}/sstate-build-package_,
${WORKDIR}/sstate-install-package_,
${TMPDIR}/sysroots-components"

which seems to give us a good start. In particular, this makes packaging during do_package_write_* mostly happen under pseudo but the final files are not written out in pseudo active paths. This is deliberate and necessary but also slightly complicates things.

Also note there is conditional code to add ${S} and ${B} to this list if they're not ${WORKDIR}. Path matching happens by the start of the path matching one of these fragments.

If you run with these patches you'll probably end up with one of a number of issues:

ERROR: Task (/home/pokybuild/yocto-worker/non-gpl3/build/meta/recipes-core/glibc/glibc_2.32.bb:do_package) failed with exit code '134'

means the task aborted. To debug, take a look at the pseudo log file:

pokybuild@tumbleweed-ty-3:~/yocto-worker/non-gpl3/build/build-renamed/tmp/work/core2-32-poky-linux/glibc/2.32-r0/pseudo> cat pseudo.log
debug_logfile: fd 2
pid 1997 [parent 1996], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 1996.
path mismatch [2 links]: ino 236066483 db '/home/pokybuild/yocto-worker/non-gpl3/build/build/tmp/work/core2-32-poky-linux/glibc/2.32-r0/image/usr/share/locale/cs' req '/home/pokybuild/yocto-worker/non-gpl3/build/build/tmp/work/core2-32-poky-linux/glibc/2.32-r0/sstate-build-package/package'. db cleanup for server shutdown, 14:15:45.384
memory-to-file backup complete, 14:15:45.385.
db cleanup finished, 14:15:45.385

The abort is from the path mismatch.

It means that a lookup was seen for:

/home/pokybuild/yocto-worker/non-gpl3/build/build/tmp/work/core2-32-poky-linux/glibc/2.32-r0/sstate-build-package/package

but it wasn't in the database. There was a matching inode entry in the database of:

/home/pokybuild/yocto-worker/non-gpl3/build/build/tmp/work/core2-32-poky-linux/glibc/2.32-r0/image/usr/share/locale/cs

This is puzzling as files in WIORKDIR/image/ would come from do_install, so something has removed a file added during do_install but has done so outside of pseudo's "sight" (a task without fakeroot context?).

At this point the work is in figuring out what happened to that file and ensuring pseudo does see what happens to anything in WORKDIR/image.

You may also see user/group number lookup issues, usually where its tried to resolve the build user ID. Usually something like:

Exception: KeyError: 'getpwuid(): uid not found: 1000'

If this is from the sstatesig code, the file being accessed should be listed before the traceback. Its then a question of debugging why that file isn't being seen by pseudo (and whether it should be or not).

We need to start to look into the failing cases and figure out what the patterns are, why they're not being seen by pseudo and work through them.

For one case for locale-archive in do_rootfs, I had to add new functionality to pseudo.

In that case it was an access pattern of:

fd = fopen("A")
link("A", "B")
unlink("A")
fchown(fd)

pseudo maintains a list of paths for the open fds and this breaks with the above scenario.

I've put the patches in master-next on the autobuilder and will start working through the issues it highlights but help would be appreciated. Its hard to tell how big this problem even is right now.

Cheers,

Richard