Skip to contentSkip to conversation

Add bblock helper scripts

PATCH In review Julien Stephan 5 participants 19 messages Sep 25, 2023 – Oct 6, 2023

lore
Share & tools
lore
#0 lore raw

v5 cover Add bblock helper scripts

Hi all,

This is v5 for bblock script.

Improvement from v4:
* Fix self test error reported by autobuilder
* Allow to specify task with/without the "do_" prefix:
example: bblock bc -t compile,do_install
* Fix test_lock_single_recipe_all_tasks
* Fix getTaskSignatures: now retrieve all tasks and filter out the
* setscene and nostamps ones

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  31 ++++
 bitbake/lib/bb/event.py                |   8 +
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  18 ++-
 meta/lib/oeqa/selftest/cases/bblock.py | 201 +++++++++++++++++++++++++
 scripts/bblock                         | 184 ++++++++++++++++++++++
 7 files changed, 448 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

#1 lore raw

PATCH v5 1/5 bitbake.conf: include bblock.conf

include conf/bblock.conf. This file is generated by the bblock tool. It locks some package tasks by fixing their signatures. See bblock -h for more details

---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)
meta/conf/bitbake.conf +1 -0
index 67577c8fe7e..6a8deee4f72 100644
@@ -829,6 +829,7 @@ include conf/documentation.conf
 include conf/licenses.conf
 require conf/sanity.conf
 require conf/cve-check-map.conf
+include conf/bblock.conf
 
 ##################################################################
 # Weak variables (usually to retain backwards compatibility)
Signature · 1 line
2.41.0
#2 lore raw

PATCH v5 2/5 bitbake: cooker: add a new function to retrieve task signatures

In reply to Julien Stephan

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the signatures using tinfoil

---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 31 +++++++++++++++++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 45 insertions(+)
bitbake/lib/bb/command.py +6 -0
index 8663eed9331..f2ee5871616 100644
@@ -781,3 +781,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +31 -0
index 87aa71bb657..599c7ddaa28 100644
@@ -1442,6 +1442,37 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
+    def getTaskSignatures(self, target, tasks):
+        sig = []
+        getAllTaskSignatures = False
+
+        if not tasks:
+            tasks = ["do_build"]
+            getAllTaskSignatures = True
+
+        for task in tasks:
+            taskdata, runlist = self.buildTaskData(target, task, self.configuration.halt)
+            rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+            rq.rqdata.prepare()
+
+            for l in runlist:
+                mc, pn, taskname, fn = l
+
+                taskdep = rq.rqdata.dataCaches[mc].task_deps[fn]
+                for t in taskdep['tasks']:
+                    if t in taskdep['nostamp'] or "setscene" in t:
+                        continue
+                    tid = bb.runqueue.build_tid(mc, fn, t)
+
+                    if t in task or getAllTaskSignatures:
+                        try:
+                            rq.rqdata.prepare_task_hash(tid)
+                            sig.append([pn, t, rq.rqdata.get_task_unihash(tid)])
+                        except KeyError:
+                            sig.append(self.getTaskSignatures(target, [t])[0])
+
+        return sig
+
     def buildTargets(self, targets, task):
         """
         Attempt to build the targets specified
bitbake/lib/bb/event.py +8 -0
index 0d0e0a68aac..f8acacd80d1 100644
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
         Event.__init__(self)
         self.result = result
 
+class GetTaskSignatureResult(Event):
+    """
+    Event to return results from GetTaskSignatures command
+    """
+    def __init__(self, sig):
+        Event.__init__(self)
+        self.sig = sig
+
 class ParseError(Event):
     """
     Event to indicate parse failed
Signature · 1 line
2.41.0
#3 lore raw

PATCH v5 3/5 sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK

In reply to Julien Stephan

as of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and "error", displaying respectively a warning or a fatal error message only when a task is locked and the task signature is different from the locked one.

The "info" level is introduced to add a "note" message to remind the user that a recipe is locked even if the signature is equivalent to the locked one.

The "warn" and "error" level display the warn/error message for each task having a mismatch of the signature. Doing this with the "info" level would result in very verbose output if there are several tasks locked, so the info level will only print once the list of recipes that have locked signature.

---
 meta/lib/oe/sstatesig.py | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +17 -1
index 633a0fd4502..5bf1697e727 100644
@@ -104,6 +104,8 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         self.lockedhashfn = {}
         self.machine = data.getVar("MACHINE")
         self.mismatch_msgs = []
+        self.mismatch_number = 0
+        self.lockedsigs_msgs = ""
         self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or
                                 "").split()
         self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
@@ -187,6 +189,7 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                 #bb.warn("Using %s %s %s" % (recipename, task, h))
 
                 if h != h_locked and h_locked != unihash:
+                    self.mismatch_number += 1
                     self.mismatch_msgs.append('The %s:%s sig is computed to be %s, but the sig is locked to %s in %s'
                                           % (recipename, task, h, h_locked, var))
 
@@ -267,6 +270,15 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         warn_msgs = []
         error_msgs = []
         sstate_missing_msgs = []
+        info_msgs = None
+
+        if self.lockedsigs:
+            if len(self.lockedsigs) > 10:
+                self.lockedsigs_msgs = "There are %s recipes with locked tasks (%s task(s) have non matching signature)" % (len(self.lockedsigs), self.mismatch_number)
+            else:
+                self.lockedsigs_msgs = "The following recipes have locked tasks:"
+                for pn in self.lockedsigs:
+                    self.lockedsigs_msgs += " %s" % (pn)
 
         for tid in sq_data['hash']:
             if tid not in found:
@@ -279,7 +291,9 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                                                % (pn, taskname, sq_data['hash'][tid]))
 
         checklevel = d.getVar("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK")
-        if checklevel == 'warn':
+        if checklevel == 'info':
+            info_msgs = self.lockedsigs_msgs
+        if checklevel == 'warn' or checklevel == 'info':
             warn_msgs += self.mismatch_msgs
         elif checklevel == 'error':
             error_msgs += self.mismatch_msgs
@@ -290,6 +304,8 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         elif checklevel == 'error':
             error_msgs += sstate_missing_msgs
 
+        if info_msgs:
+            bb.note(info_msgs)
         if warn_msgs:
             bb.warn("\n".join(warn_msgs))
         if error_msgs:
Signature · 1 line
2.41.0
#4 lore raw

PATCH v5 4/5 scripts/bblock: add a script to lock/unlock recipes

In reply to Julien Stephan

bblock script allows to lock/unlock recipes to latest task signatures. The idea is to prevent some recipes to be rebuilt during development. For example when working on rust recipe, one may not want rust-native to be rebuilt.

This tool can be used, with proper environment set up, using the following command:

bblock <recipe_name>

See help for more details

if a <recipe_name>'s task signature change, this task will not be built again and sstate cache will be used.

[YOCTO #13425]

---
 scripts/bblock | 184 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 184 insertions(+)
 create mode 100755 scripts/bblock
scripts/bblock new file +184 -0
new file mode 100755
index 00000000000..0082059af81
@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <[email protected]>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def getTaskSignatures(tinfoil, pn, tasks):
+    tinfoil.set_event_mask(
+        [
+            "bb.event.GetTaskSignatureResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("getTaskSignatures", pn, tasks)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.GetTaskSignatureResult):
+                    sig = event.sig
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from getTaskSignatures command")
+        sys.exit(2)
+    return sig
+
+
+def parseRecipe(tinfoil, recipe):
+    try:
+        tinfoil.parse_recipes()
+        d = tinfoil.parse_recipe(recipe)
+    except Exception:
+        logger.error("Failed to get recipe info for: %s" % recipe)
+        sys.exit(1)
+    return d
+
+
+def bblockDump(lockfile):
+    try:
+        with open(lockfile, "r") as lockfile:
+            for line in lockfile:
+                print(line.strip())
+    except IOError:
+        return 1
+    return 0
+
+
+def bblockReset(lockfile, pns, package_archs, tasks):
+    if not pns:
+        logger.info("Unlocking all recipes")
+        try:
+            os.remove(lockfile)
+        except FileNotFoundError:
+            pass
+    else:
+        logger.info("Unlocking {pns}".format(pns=pns))
+        tmp_lockfile = lockfile + ".tmp"
+        with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+            for line in infile:
+                if not (
+                    any(element in line for element in pns)
+                    and any(element in line for element in package_archs.split())
+                ):
+                    outfile.write(line)
+                else:
+                    if tasks and not any(element in line for element in tasks):
+                        outfile.write(line)
+        os.remove(lockfile)
+        os.rename(tmp_lockfile, lockfile)
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+    parser.add_argument(
+        "-t",
+        "--tasks",
+        help="Comma separated list of tasks",
+        type=lambda s: [
+            task if task.startswith("do_") else "do_" + task for task in s.split(",")
+        ],
+    )
+    parser.add_argument(
+        "-r",
+        "--reset",
+        action="store_true",
+        help="Unlock pn recipes, or all recipes if pn is empty",
+    )
+    parser.add_argument(
+        "-d",
+        "--dump",
+        action="store_true",
+        help="Dump generated bblock.conf file",
+    )
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+
+        package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+        if global_args.dump:
+            bblockDump(lockfile)
+            return 0
+
+        if global_args.reset:
+            bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks)
+            return 0
+
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+                s += "\n"
+
+            for pn in global_args.pn:
+                d = parseRecipe(tinfoil, pn)
+                package_arch = d.getVar("PACKAGE_ARCH")
+                siggen_locked_sigs_package_arch = d.getVar(
+                    "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch)
+                )
+                sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks)
+                for sig in sigs:
+                    new_entry = "{pn}:{taskname}:{sig}".format(
+                        pn=sig[0], taskname=sig[1], sig=sig[2]
+                    )
+                    if (
+                        siggen_locked_sigs_package_arch
+                        and not new_entry in siggen_locked_sigs_package_arch
+                    ) or not siggen_locked_sigs_package_arch:
+                        s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format(
+                            package_arch=package_arch, new_entry=new_entry
+                        )
+            lockfile.write(s)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
Signature · 1 line
2.41.0
#5 lore raw

PATCH v5 5/5 oeqa/selftest/bblock: add self test for bblock tool

In reply to Julien Stephan

it implements various combination of locking single/multiple recipe(s)/task(s)

it also tests that locked sig are architecture dependant

---
 meta/lib/oeqa/selftest/cases/bblock.py | 201 +++++++++++++++++++++++++
 1 file changed, 201 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
meta/lib/oeqa/selftest/cases/bblock.py new file +201 -0
new file mode 100644
index 00000000000..07143c59676
@@ -0,0 +1,201 @@
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <[email protected]>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import re
+import bb.tinfoil
+
+import oeqa.utils.ftools as ftools
+from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake
+
+from oeqa.selftest.case import OESelftestTestCase
+
+
+class BBLock(OESelftestTestCase):
+    @classmethod
+    def setUpClass(cls):
+        super(BBLock, cls).setUpClass()
+        cls.lockfile = cls.builddir + "/conf/bblock.conf"
+
+    def unlock_recipes(self, recipes=None, tasks=None):
+        cmd = "bblock -r "
+        if recipes:
+            cmd += " ".join(recipes)
+        if tasks:
+            cmd += " -t " + ",".join(tasks)
+        result = runCmd(cmd)
+
+        if recipes:
+            # ensure all signatures are removed from lockfile
+            contents = ftools.read_file(self.lockfile)
+            for recipe in recipes:
+                for task in tasks:
+                    find_in_contents = re.search(
+                        'SIGGEN_LOCKEDSIGS_.+\s\+=\s"%s:%s:.*"' % (recipe, task),
+                        contents,
+                    )
+                    self.assertFalse(
+                        find_in_contents,
+                        msg="%s:%s should not be present into bblock.conf anymore"
+                        % (recipe, task),
+                    )
+                self.assertExists(self.lockfile)
+        else:
+            self.assertNotExists(self.lockfile)
+
+    def lock_recipes(self, recipes, tasks=None):
+        cmd = "bblock " + " ".join(recipes)
+        if tasks:
+            cmd += " -t " + ",".join(tasks)
+
+        result = runCmd(cmd)
+
+        self.assertExists(self.lockfile)
+
+        # ensure all signatures are added to lockfile
+        contents = ftools.read_file(self.lockfile)
+        for recipe in recipes:
+            if tasks:
+                for task in tasks:
+                    find_in_contents = re.search(
+                        'SIGGEN_LOCKEDSIGS_.+\s\+=\s"%s:%s:.*"' % (recipe, task),
+                        contents,
+                    )
+                    self.assertTrue(
+                        find_in_contents,
+                        msg="%s:%s was not added into bblock.conf. bblock output: %s"
+                        % (recipe, task, result.output),
+                    )
+
+    def modify_tasks(self, recipes, tasks):
+        task_append = ""
+        for recipe in recipes:
+            bb_vars = get_bb_vars(["PV"], recipe)
+            recipe_pv = bb_vars["PV"]
+            recipe_append_file = recipe + "_" + recipe_pv + ".bbappend"
+
+            os.mkdir(os.path.join(self.testlayer_path, "recipes-test", recipe))
+            recipe_append_path = os.path.join(
+                self.testlayer_path, "recipes-test", recipe, recipe_append_file
+            )
+
+            for task in tasks:
+                task_append += "%s:append() {\n#modify task hash \n}\n" % task
+            ftools.write_file(recipe_append_path, task_append)
+            self.add_command_to_tearDown(
+                "rm -rf %s" % os.path.join(self.testlayer_path, "recipes-test", recipe)
+            )
+
+    def test_lock_single_recipe_single_task(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_single_recipe_multiple_tasks(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile", "do_install"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_single_recipe_all_tasks(self):
+        recipes = ["quilt"]
+        self._run_test(recipes, None)
+
+    def test_lock_multiple_recipe_single_task(self):
+        recipes = ["quilt", "bc"]
+        tasks = ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_architecture_specific(self):
+        # unlock all recipes and ensure no bblock.conf file exist
+        self.unlock_recipes()
+
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+
+        # lock quilt's do_compile task for another machine
+        if self.td["MACHINE"] == "qemux86-64":
+            machine = "qemuarm"
+        else:
+            machine = "qemux86-64"
+
+        self.write_config('MACHINE = "%s"\n' % machine)
+
+        self.lock_recipes(recipes, tasks)
+
+        self.write_config('MACHINE = "%s"\n' % self.td["MACHINE"])
+        # modify quilt's do_compile task
+        self.modify_tasks(recipes, tasks)
+
+        # build quilt using the default machine
+        # No Note/Warning should be emitted since sig is locked for another machine
+        # (quilt package is architecture dependant)
+        info_message = "NOTE: The following recipes have locked tasks: " + recipes[0]
+        warn_message = "The %s:%s sig is computed to be" % (recipes[0], tasks[0])
+        result = bitbake(recipes[0] + " -n")
+        self.assertNotIn(info_message, result.output)
+        self.assertNotIn(warn_message, result.output)
+
+        # unlock all recipes
+        self.unlock_recipes()
+
+    def _run_test(self, recipes, tasks=None):
+        # unlock all recipes and ensure no bblock.conf file exist
+        self.unlock_recipes()
+
+        # lock tasks for recipes
+        result = self.lock_recipes(recipes, tasks)
+
+        if not tasks:
+            tasks = []
+            result = bitbake("-c listtasks " + recipes[0])
+            with bb.tinfoil.Tinfoil() as tinfoil:
+                tinfoil.prepare(config_only=False, quiet=2)
+                d = tinfoil.parse_recipe(recipes[0])
+
+                for line in result.output.splitlines():
+                    if line.startswith("do_"):
+                        task = line.split()[0]
+                        if "setscene" in task:
+                            continue
+                        if d.getVarFlag(task, "nostamp"):
+                            continue
+                        tasks.append(task)
+
+        # build recipes. At this stage we should have a Note about recipes
+        # having locked task's sig, but no warning since sig still match
+        info_message = "NOTE: The following recipes have locked tasks: " + " ".join(
+            recipes
+        )
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
+
+        # modify all tasks that are locked to trigger a sig change then build the recipes
+        # at this stage we should have a Note as before, but also a Warning for all
+        # locked tasks indicating the sig mismatch
+        self.modify_tasks(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertIn(warn_message, result.output)
+
+        # unlock all tasks and rebuild, no more Note/Warning should remain
+        self.unlock_recipes(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertNotIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
+
+        # unlock all recipes
+        self.unlock_recipes()
Signature · 1 line
2.41.0
#6 lore raw
In reply to Julien Stephan

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Quoted 100 lines · Julien Stephan

Hi all,

This is v5 for bblock script.

Improvement from v4:
* Fix self test error reported by autobuilder
* Allow to specify task with/without the "do_" prefix:
example: bblock bc -t compile,do_install
* Fix test_lock_single_recipe_all_tasks
* Fix getTaskSignatures: now retrieve all tasks and filter out the
* setscene and nostamps ones

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  31 ++++
 bitbake/lib/bb/event.py                |   8 +
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  18 ++-
 meta/lib/oeqa/selftest/cases/bblock.py | 201 +++++++++++++++++++++++++
 scripts/bblock                         | 184 ++++++++++++++++++++++
 7 files changed, 448 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188174): https://lists.openembedded.org/g/openembedded-core/message/188174
Mute This Topic: https://lists.openembedded.org/mt/101570279/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw
Quoted 7 lines · Alexandre Belloni

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

Cheers
Julien

Quoted 6 lines

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#8 lore raw
Quoted 10 lines · Julien Stephan

Le mer. 27 sept. 2023 � 12:16, Alexandre Belloni <[email protected]> a �crit :

Quoted 7 lines

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Quoted 7 lines

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

You can scroll up or click on the looking glass on the top right.

Alternatively, you can download the log here:
https://autobuilder.yoctoproject.org/typhoon/api/v2/logs/8276762/raw
(the button is also on the top right)

The assertion error is fairly large, that's why I didn't copy it.

Quoted 9 lines

Cheers
Julien

Quoted 6 lines

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#9 lore raw
In reply to Julien Stephan

You need to scroll up. The failure is in lines 557 to 609.

Alex

Quoted 36 lines · Julien Stephan
Quoted 7 lines · Alexandre Belloni

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

Cheers
Julien

Quoted 6 lines

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188313): https://lists.openembedded.org/g/openembedded-core/message/188313
Mute This Topic: https://lists.openembedded.org/mt/101570279/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#10 lore raw
Quoted 2 lines · Alexander Kanavin

You need to scroll up. The failure is in lines 557 to 609.

Hmm stange ctrl-f didn't work. I can see it now. Thank you, I'll look at it

Julien

Quoted 41 lines

Alex

Quoted 36 lines · Julien Stephan
Quoted 7 lines · Alexandre Belloni

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

Cheers
Julien

Quoted 6 lines

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188313): https://lists.openembedded.org/g/openembedded-core/message/188313
Mute This Topic: https://lists.openembedded.org/mt/101570279/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#11 lore raw
Quoted 7 lines · Julien Stephan

Le mer. 27 sept. 2023 � 14:59, Alexander Kanavin <[email protected]> a �crit :

Quoted 2 lines

You need to scroll up. The failure is in lines 557 to 609.

Hmm stange ctrl-f didn't work. I can see it now. Thank you, I'll look at it

By default the log is not fully downloaded, you have to click on the looking glass before ctrl-f.

Quoted 42 lines

Julien

Quoted 41 lines

Alex

Quoted 36 lines · Julien Stephan

Le mer. 27 sept. 2023 � 12:16, Alexandre Belloni <[email protected]> a �crit :

Quoted 7 lines

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

Cheers
Julien

Quoted 6 lines

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188313): https://lists.openembedded.org/g/openembedded-core/message/188313
Mute This Topic: https://lists.openembedded.org/mt/101570279/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#12 lore raw
In reply to Julien Stephan
Quoted 10 lines · Julien Stephan
Quoted 7 lines · Alexandre Belloni

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Quoted 6 lines

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

I am a bit worried this will be hard to reproduce/debug so I had a look at things as I do really want to get things resolved.

The error log says:

WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64

and I couldn't reproduce those stamps in a local build or on the builder where it failed (once I got access which took a while).

The "while" is important as I think in the meantime things changed which effected the results.

On the autobuilder, "bitbake quilt -S none" generated stamp files for quilt. I did this for "build" and "build-st-3046552", the latter being the one where the test failed. There were two different do_build stamps generated. The diff:

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
NOTE: Reconnecting to bitbake server...
Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce

    Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
    Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec

i.e. a couple of dependencies were different.

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2

showed no output, i.e. they appear to be the same but different.

My best guess at what is going on here is hash equivalence causing problems.

So in summary, I think the best solution will be to disable hash equivalence for these tests.

Cheers,

Richard

#13 lore raw
Quoted 47 lines · Richard Purdie
Quoted 10 lines · Julien Stephan
Quoted 7 lines · Alexandre Belloni

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Quoted 6 lines

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

I am a bit worried this will be hard to reproduce/debug so I had a look at things as I do really want to get things resolved.

The error log says:

WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64

and I couldn't reproduce those stamps in a local build or on the builder where it failed (once I got access which took a while).

The "while" is important as I think in the meantime things changed which effected the results.

On the autobuilder, "bitbake quilt -S none" generated stamp files for quilt. I did this for "build" and "build-st-3046552", the latter being the one where the test failed. There were two different do_build stamps generated. The diff:

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
NOTE: Reconnecting to bitbake server...
Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce

    Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
    Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec

i.e. a couple of dependencies were different.

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2

Quoted 9 lines

showed no output, i.e. they appear to be the same but different.

My best guess at what is going on here is hash equivalence causing problems.

So in summary, I think the best solution will be to disable hash equivalence for these tests.

Hi Richard,

Thank you a lot for digging into this!
How can I disable the hash equivalence for these tests?

Should I use BB_BASEHASH_IGNORE_VARS for that?

I tried the following:
self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")

But this doesn't work, I get a lot of errors like the following:
ERROR: When reparsing /home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:do_packagedata, the basehash value changed from 8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to 9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake quilt -cdo_packagedata -Snone
ERROR: Then:
ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff

Cheers
Julien

Quoted 4 lines

Cheers,

Richard

#14 lore raw

Hi Julien,

Julien Stephan <[email protected]> escreveu no dia sábado, 30/09/2023 à(s) 15:13:

Quoted 74 lines
Quoted 10 lines · Richard Purdie
Quoted 8 lines · Julien Stephan
Quoted 6 lines · Alexandre Belloni

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it

somewhere?

Quoted 7 lines

I am a bit worried this will be hard to reproduce/debug so I had a look at things as I do really want to get things resolved.

The error log says:

WARNING: The quilt:do_build sig is computed to be

255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64

Quoted 7 lines

and I couldn't reproduce those stamps in a local build or on the builder where it failed (once I got access which took a while).

The "while" is important as I think in the meantime things changed which effected the results.

Quoted 6 lines

On the autobuilder, "bitbake quilt -S none" generated stamp files for quilt. I did this for "build" and "build-st-3046552", the latter being the one where the test failed. There were two different do_build stamps generated. The diff:

bitbake-diffsigs

tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407

Quoted 2 lines

NOTE: Reconnecting to bitbake server...
Hash for task dependency diffstat:do_package_write_rpm changed from

41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce

Quoted 1 line

Hash for task dependency diffstat:do_package changed from

f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec

Quoted 1 line

Unable to find matching sigdata for diffstat:do_package with hash

e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec

Quoted 4 lines

i.e. a couple of dependencies were different.

bitbake-diffsigs

tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2

Quoted 10 lines

showed no output, i.e. they appear to be the same but different.

My best guess at what is going on here is hash equivalence causing problems.

So in summary, I think the best solution will be to disable hash equivalence for these tests.

Quoted 5 lines

Hi Richard,

Thank you a lot for digging into this!
How can I disable the hash equivalence for these tests?

Can you try with BB_SIGNATURE_HANDLER using the bibake default config

self.write_config('BB_SIGNATURE_HANDLER = "OEBasicHash"')

Jose

Quoted 40 lines

Should I use BB_BASEHASH_IGNORE_VARS for that?

I tried the following:
self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")

But this doesn't work, I get a lot of errors like the following:
ERROR: When reparsing

/home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:
do_packagedata,
the basehash value changed from 8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to 9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake quilt -cdo_packagedata -Snone
ERROR: Then:
ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff

Cheers
Julien

Quoted 4 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#188459):
https://lists.openembedded.org/g/openembedded-core/message/188459
Mute This Topic: https://lists.openembedded.org/mt/101570279/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Best regards,

José Quaresma
#15 lore raw

Hi Jose,

I just pushed my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
with the fix you just suggested.

I do not reproduce the issue locally, so I am not sure this fix the problem.. Richard, Alexandre what do you think? Can you restart a build on autobuidler?

Cheers
Julien

Quoted 118 lines · Jose Quaresma

Hi Julien,

Julien Stephan <[email protected]> escreveu no dia sábado, 30/09/2023 à(s) 15:13:

Quoted 60 lines
Quoted 47 lines · Richard Purdie
Quoted 10 lines · Julien Stephan
Quoted 7 lines · Alexandre Belloni

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Quoted 6 lines

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

I am a bit worried this will be hard to reproduce/debug so I had a look at things as I do really want to get things resolved.

The error log says:

WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64

and I couldn't reproduce those stamps in a local build or on the builder where it failed (once I got access which took a while).

The "while" is important as I think in the meantime things changed which effected the results.

On the autobuilder, "bitbake quilt -S none" generated stamp files for quilt. I did this for "build" and "build-st-3046552", the latter being the one where the test failed. There were two different do_build stamps generated. The diff:

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
NOTE: Reconnecting to bitbake server...
Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce

    Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
    Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec

i.e. a couple of dependencies were different.

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2

Quoted 9 lines

showed no output, i.e. they appear to be the same but different.

My best guess at what is going on here is hash equivalence causing problems.

So in summary, I think the best solution will be to disable hash equivalence for these tests.

Quoted 4 lines

Hi Richard,

Thank you a lot for digging into this!
How can I disable the hash equivalence for these tests?

Can you try with BB_SIGNATURE_HANDLER using the bibake default config

self.write_config('BB_SIGNATURE_HANDLER = "OEBasicHash"')

Jose

Quoted 28 lines

Should I use BB_BASEHASH_IGNORE_VARS for that?

I tried the following:
self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")

But this doesn't work, I get a lot of errors like the following:
ERROR: When reparsing /home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:do_packagedata, the basehash value changed from 8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to 9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake quilt -cdo_packagedata -Snone
ERROR: Then:
ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff

Cheers
Julien

Quoted 4 lines

Cheers,

Richard

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188459): https://lists.openembedded.org/g/openembedded-core/message/188459
Mute This Topic: https://lists.openembedded.org/mt/101570279/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Best regards,

José Quaresma

#16 lore raw

I started an oe-selftest that passed, I'm going to carry that in my branch to see whether there are other issues.

Quoted 135 lines · Julien Stephan

Hi Jose,

I just pushed my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
with the fix you just suggested.

I do not reproduce the issue locally, so I am not sure this fix the problem.. Richard, Alexandre what do you think? Can you restart a build on autobuidler?

Cheers
Julien

Le lun. 2 oct. 2023 � 12:25, Jose Quaresma <[email protected]> a �crit :

Quoted 108 lines

Hi Julien,

Julien Stephan <[email protected]> escreveu no dia s�bado, 30/09/2023 �(s) 15:13:

Quoted 60 lines

Le mer. 27 sept. 2023 � 23:17, Richard Purdie
<[email protected]> a �crit :

Quoted 47 lines · Julien Stephan
Quoted 10 lines

Le mer. 27 sept. 2023 � 12:16, Alexandre Belloni <[email protected]> a �crit :

Quoted 7 lines

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

Quoted 6 lines

Hi Alexandre,
I only see

bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

I am a bit worried this will be hard to reproduce/debug so I had a look at things as I do really want to get things resolved.

The error log says:

WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64

and I couldn't reproduce those stamps in a local build or on the builder where it failed (once I got access which took a while).

The "while" is important as I think in the meantime things changed which effected the results.

On the autobuilder, "bitbake quilt -S none" generated stamp files for quilt. I did this for "build" and "build-st-3046552", the latter being the one where the test failed. There were two different do_build stamps generated. The diff:

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
NOTE: Reconnecting to bitbake server...
Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce

    Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
    Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec

i.e. a couple of dependencies were different.

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2

Quoted 9 lines

showed no output, i.e. they appear to be the same but different.

My best guess at what is going on here is hash equivalence causing problems.

So in summary, I think the best solution will be to disable hash equivalence for these tests.

Quoted 4 lines

Hi Richard,

Thank you a lot for digging into this!
How can I disable the hash equivalence for these tests?

Can you try with BB_SIGNATURE_HANDLER using the bibake default config

self.write_config('BB_SIGNATURE_HANDLER = "OEBasicHash"')

Jose

Quoted 20 lines

Should I use BB_BASEHASH_IGNORE_VARS for that?

I tried the following:
self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")

But this doesn't work, I get a lot of errors like the following:
ERROR: When reparsing /home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:do_packagedata, the basehash value changed from 8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to 9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake quilt -cdo_packagedata -Snone
ERROR: Then:
ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff

Quoted 10 lines

Cheers
Julien

Quoted 4 lines

Cheers,

Richard

Quoted 4 lines

--
Best regards,

Jos� Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188705): https://lists.openembedded.org/g/openembedded-core/message/188705
Mute This Topic: https://lists.openembedded.org/mt/101570279/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#17 lore raw
Quoted 2 lines · Alexandre Belloni

I started an oe-selftest that passed, I'm going to carry that in my branch to see whether there are other issues.

I've had a couple of runs in my branch and not seen any issue from that as yet FWIW.

I am leaning towards merging this at this point.

Cheers,

Richard

#18 lore raw
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Alexandre Belloni

I started an oe-selftest that passed, I'm going to carry that in my branch to see whether there are other issues.

Quoted 8 lines

I've had a couple of runs in my branch and not seen any issue from that as yet FWIW.

I am leaning towards merging this at this point.

Cheers,

Richard

Hi Richard and Alexandre,

That is very good news :)

Cheers

Julien