cooker: warn when sstate is outside of build dir, but hash equiv database is inside it
PATCH 1/1 cooker: warn when sstate is outside of build dir, but hash equiv database is inside it
From: Alexander Kanavin <[email protected]>
This should help with the long-standing usability problem: when someone tweaks the configuration to put sstate somewhere else than the default (so that it can be shared between local builds, or over NFS), they should also share the hash equivalency database, but there would not be any indication to do so.
This will issue a warning and recommend to start a dedicated hash equivalency server (if sstate is on NFS), or set BB_HASHSERVE_DB_DIR (if it isn't).
--- lib/bb/cooker.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
lib/bb/cooker.py +19 -0
@@ -336,6 +336,25 @@ cannot be on a NFS mount due to potential NFS locking issues between sqlite clie If you need to share the database between several computers, set up a permanently running hash equivalency server according to https://docs.yoctoproject.org/dev-manual/hashequivserver.html""".format(bb_hashserve_db_dir)) dbdir = bb_hashserve_db_dir or self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE") + + topdir = self.data.getVar("TOPDIR") + sstatedir = self.data.getVar("SSTATE_DIR") + if (dbdir.startswith(topdir) and not sstatedir.startswith(topdir)): + if utils.is_path_on_nfs(sstatedir): + bb.warn("""Sstate directory is on a shared NFS (it is set via SSTATE_DIR to {}), +but hash equivalency database is inside this particular build directory {}. + +This will prevent sstate reuse, and it is recommended to set up a permanently running hash equivalency server +according to https://docs.yoctoproject.org/dev-manual/hashequivserver.html""".format(sstatedir, topdir)) + else: + bb.warn("""Sstate directory is shared between several builds (it is set via SSTATE_DIR to {}), +but hash equivalency database is inside this particular build directory {}. + +This will prevent sstate reuse, and it is recommended to set the location for the database to a common path +via BB_HASHSERVE_DB_DIR, for example: + +BB_HASHSERVE_DB_DIR = \"${{SSTATE_DIR}}\"""".format(sstatedir, topdir)) + os.makedirs(dbdir, exist_ok=True) dbfile = dbdir + "/hashserv.db" upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") or None
Signature · 1 line
2.47.3
Quoted 8 lines · Alexander Kanavin via lists.openembedded.org
From: Alexander Kanavin <[email protected]>
This should help with the long-standing usability problem: when someone tweaks the configuration to put sstate somewhere else than the default (so that it can be shared between local builds, or over NFS), they should also share the hash equivalency database, but there would not be any indication to do so.
Quoted 5 lines
This will issue a warning and recommend to start a dedicated hash equivalency server (if sstate is on NFS), or set
BB_HASHSERVE_DB_DIR(if it isn't).Signed-off-by Alexander Kanavin <[email protected]>---
Hi Alex,
Thanks for your patch.
It looks like this is breaking the bb.tests.runqueue.RunQueueTests.test_hashserv_double test:
FAIL: test_hashserv_double (bb.tests.runqueue.RunQueueTests.test_hashserv_double)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/lib/bb/tests/runqueue.py", line 41, in run_bitbakecmd
output = subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bitbake', 'a1', 'b1', 'e1']' returned non-zero exit status 1.
The bb.tests.runqueue.RunQueueTests.test_hashserv_multiple_setscene test is also failing, with a similar error.
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3266
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3144
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3385
Can you have a look at the issue?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com