Skip to contentSkip to conversation

Add bblock helper scripts

PATCH Superseded Julien Stephan 2 participants 6 messages Jul 27, 2023 – Aug 2, 2023

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Add bblock helper scripts

Hi all,

This is v3 from bblock script.

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (3):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures scripts/bblock: add a script to lock/unlock recipes

 bitbake/lib/bb/command.py |   6 ++
 bitbake/lib/bb/cooker.py  |  16 ++++
 bitbake/lib/bb/event.py   |   8 ++
 meta/conf/bitbake.conf    |   1 +
 scripts/bblock            | 182 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 213 insertions(+)
 create mode 100755 scripts/bblock

--
2.41.0

#1 lore raw

PATCH v3 1/3 bitbake.conf: include bblock.conf

include conf/bblock.conf. This file is generated by the bblock tool. It locks some package tasks by fixing their signatures. See bblock -h for more details

---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)
meta/conf/bitbake.conf +1 -0
index 9625a6fef4c..a4b2b4b0380 100644
@@ -831,6 +831,7 @@ include conf/distro/defaultsetup.conf
 include conf/documentation.conf
 include conf/licenses.conf
 require conf/sanity.conf
+include conf/bblock.conf
 
 ##################################################################
 # Weak variables (usually to retain backwards compatibility)
Signature · 1 line
2.41.0
#2 lore raw

PATCH v3 2/3 bitbake: cooker: add a new function to retrieve task signatures

In reply to Julien Stephan

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the signatures using tinfoil

---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
index a355f56c60c..12202779ac0 100644
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +16 -0
index 11c9fa2c40d..687cdde5e6d 100644
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+
     def buildTargets(self, targets, task):
         """
         Attempt to build the targets specified
bitbake/lib/bb/event.py +8 -0
index 0d0e0a68aac..f8acacd80d1 100644
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
         Event.__init__(self)
         self.result = result
 
+class GetTaskSignatureResult(Event):
+    """
+    Event to return results from GetTaskSignatures command
+    """
+    def __init__(self, sig):
+        Event.__init__(self)
+        self.sig = sig
+
 class ParseError(Event):
     """
     Event to indicate parse failed
Signature · 1 line
2.41.0
#3 lore raw

PATCH v3 3/3 scripts/bblock: add a script to lock/unlock recipes

In reply to Julien Stephan

bblock script allows to lock/unlock recipes to latest task signatures. The idea is to prevent some recipes to be rebuilt during development. For example when working on rust recipe, one may not want rust-native to be rebuilt.

This tool can be used, with proper environment set up, using the following command:

bblock <recipe_name>

if a <recipe_name>'s task signature change, this task will not be built again and sstate cache will be used.

[YOCTO #13425]

---
 scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100755 scripts/bblock
scripts/bblock new file +182 -0
new file mode 100755
index 00000000000..a9c5583127e
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <[email protected]>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def getTaskSignatures(tinfoil, pn, tasks):
+    tinfoil.set_event_mask(
+        [
+            "bb.event.GetTaskSignatureResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("getTaskSignatures", pn, tasks)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.GetTaskSignatureResult):
+                    sig = event.sig
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from getTaskSignatures command")
+        sys.exit(2)
+    return sig
+
+
+def parseRecipe(tinfoil, recipe):
+    try:
+        tinfoil.parse_recipes()
+        d = tinfoil.parse_recipe(recipe)
+    except Exception:
+        logger.error("Failed to get recipe info for: %s" % recipe)
+        sys.exit(1)
+    return d
+
+
+def bblockDump(lockfile):
+    try:
+        with open(lockfile, "r") as lockfile:
+            for line in lockfile:
+                print(line.strip())
+    except IOError:
+        return 1
+    return 0
+
+
+def bblockReset(lockfile, pns, package_archs, tasks):
+    if not pns:
+        logger.info("Unlocking all recipes")
+        try:
+            os.remove(lockfile)
+        except FileNotFoundError:
+            pass
+    else:
+        logger.info("Unlocking {pns}".format(pns=pns))
+        tmp_lockfile = lockfile + ".tmp"
+        with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+            for line in infile:
+                if not (
+                    any(element in line for element in pns)
+                    and any(element in line for element in package_archs.split())
+                ):
+                    outfile.write(line)
+                else:
+                    if tasks and not any(element in line for element in tasks):
+                        outfile.write(line)
+        os.remove(lockfile)
+        os.rename(tmp_lockfile, lockfile)
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+    parser.add_argument(
+        "-t",
+        "--tasks",
+        help="Comma separated list of tasks",
+        type=lambda s: [task for task in s.split(",")],
+    )
+    parser.add_argument(
+        "-r",
+        "--reset",
+        action="store_true",
+        help="Unlock pn recipes, or all recipes if pn is empty",
+    )
+    parser.add_argument(
+        "-d",
+        "--dump",
+        action="store_true",
+        help="Dump generated bblock.conf file",
+    )
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+
+        package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+        if global_args.dump:
+            bblockDump(lockfile)
+            return 0
+
+        if global_args.reset:
+            bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks)
+            return 0
+
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+                s += "\n"
+
+            for pn in global_args.pn:
+                d = parseRecipe(tinfoil, pn)
+                package_arch = d.getVar("PACKAGE_ARCH")
+                siggen_locked_sigs_package_arch = d.getVar(
+                    "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch)
+                )
+                sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks)
+                for sig in sigs:
+                    new_entry = "{pn}:{taskname}:{sig}".format(
+                        pn=sig[0], taskname=sig[1], sig=sig[2]
+                    )
+                    if (
+                        siggen_locked_sigs_package_arch
+                        and not new_entry in siggen_locked_sigs_package_arch
+                    ) or not siggen_locked_sigs_package_arch:
+                        s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format(
+                            package_arch=package_arch, new_entry=new_entry
+                        )
+            lockfile.write(s)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
Signature · 1 line
2.41.0
#4 lore raw
In reply to Julien Stephan

Hello,

Quoted 4 lines · Julien Stephan

Hi all,

This is v3 from bblock script.

This doesn't apply cleanly on master, can you rebase?

Thanks!

Quoted 72 lines

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (3):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures scripts/bblock: add a script to lock/unlock recipes

 bitbake/lib/bb/command.py |   6 ++
 bitbake/lib/bb/cooker.py  |  16 ++++
 bitbake/lib/bb/event.py   |   8 ++
 meta/conf/bitbake.conf    |   1 +
 scripts/bblock            | 182 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 213 insertions(+)
 create mode 100755 scripts/bblock

--
2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184932): https://lists.openembedded.org/g/openembedded-core/message/184932
Mute This Topic: https://lists.openembedded.org/mt/100390731/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw
Quoted 4 lines · Alexandre Belloni

Hello,

On 27/07/2023 15:56:09+0200, Julien Stephan wrote:

Quoted 8 lines
Quoted 4 lines

Hi all,

This is v3 from bblock script.

This doesn't apply cleanly on master, can you rebase?

Thanks!

Hi Alexandre,

done, just pushed v4 here:
https://lists.openembedded.org/g/openembedded-core/message/185403

Best
Julien

Quoted 79 lines
Quoted 65 lines

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (3):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures scripts/bblock: add a script to lock/unlock recipes

 bitbake/lib/bb/command.py |   6 ++
 bitbake/lib/bb/cooker.py  |  16 ++++
 bitbake/lib/bb/event.py   |   8 ++
 meta/conf/bitbake.conf    |   1 +
 scripts/bblock            | 182 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 213 insertions(+)
 create mode 100755 scripts/bblock

--
2.41.0

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184932): https://lists.openembedded.org/g/openembedded-core/message/184932
Mute This Topic: https://lists.openembedded.org/mt/100390731/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com