Add bblock helper scripts
v3 cover Add bblock helper scripts
Hi all,
This is v3 from bblock script.
Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf
Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist
I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock
Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584
My branch is available here [1]
Cheers
Julien
[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (3):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures scripts/bblock: add a script to lock/unlock recipes
bitbake/lib/bb/command.py | 6 ++ bitbake/lib/bb/cooker.py | 16 ++++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + scripts/bblock | 182 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 213 insertions(+) create mode 100755 scripts/bblock
--
2.41.0
PATCH v3 1/3 bitbake.conf: include bblock.conf
include conf/bblock.conf. This file is generated by the bblock tool. It locks some package tasks by fixing their signatures. See bblock -h for more details
--- meta/conf/bitbake.conf | 1 + 1 file changed, 1 insertion(+)
meta/conf/bitbake.conf +1 -0
@@ -831,6 +831,7 @@ include conf/distro/defaultsetup.conf include conf/documentation.conf include conf/licenses.conf require conf/sanity.conf +include conf/bblock.conf ################################################################## # Weak variables (usually to retain backwards compatibility)
Signature · 1 line
2.41.0
PATCH v3 2/3 bitbake: cooker: add a new function to retrieve task signatures
In reply to Julien Stephanadding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the signatures using tinfoil
--- bitbake/lib/bb/command.py | 6 ++++++ bitbake/lib/bb/cooker.py | 16 ++++++++++++++++ bitbake/lib/bb/event.py | 8 ++++++++ 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
@@ -776,3 +776,9 @@ class CommandsAsync: bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache = False + + def getTaskSignatures(self, command, params): + res = command.cooker.getTaskSignatures(params[0], params[1]) + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data) + command.finishAsyncCommand() + getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +16 -0
@@ -1542,6 +1542,22 @@ class BBCooker: self.idleCallBackRegister(buildFileIdle, rq) + def getTaskSignatures(self, target, task): + sig = [] + + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt) + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) + rq.rqdata.prepare() + + for key in rq.rqdata.runtaskentries: + pn = bb.parse.siggen.tidtopn[key] + taskname = bb.runqueue.taskname_from_tid(key) + if pn in target: + if (task and taskname in task) or (not task): + rq.rqdata.prepare_task_hash(key) + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)]) + return sig + def buildTargets(self, targets, task): """ Attempt to build the targets specified
bitbake/lib/bb/event.py +8 -0
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event): Event.__init__(self) self.result = result +class GetTaskSignatureResult(Event): + """ + Event to return results from GetTaskSignatures command + """ + def __init__(self, sig): + Event.__init__(self) + self.sig = sig + class ParseError(Event): """ Event to indicate parse failed
Signature · 1 line
2.41.0
PATCH v3 3/3 scripts/bblock: add a script to lock/unlock recipes
In reply to Julien Stephanbblock script allows to lock/unlock recipes to latest task signatures. The idea is to prevent some recipes to be rebuilt during development. For example when working on rust recipe, one may not want rust-native to be rebuilt.
This tool can be used, with proper environment set up, using the following command:
bblock <recipe_name>
if a <recipe_name>'s task signature change, this task will not be built again and sstate cache will be used.
[YOCTO #13425]
--- scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100755 scripts/bblock
scripts/bblock new file +182 -0
@@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +# bblock +# lock/unlock task to latest signature +# +# Copyright (c) 2023 BayLibre, SAS +# Author: Julien Stepahn <[email protected]> +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import os +import sys +import logging + +scripts_path = os.path.dirname(os.path.realpath(__file__)) +lib_path = scripts_path + "/lib" +sys.path = sys.path + [lib_path] + +import scriptpath + +scriptpath.add_bitbake_lib_path() + +import bb.tinfoil +import bb.msg + +import argparse_oe + +myname = os.path.basename(sys.argv[0]) +logger = bb.msg.logger_create(myname) + + +def getTaskSignatures(tinfoil, pn, tasks): + tinfoil.set_event_mask( + [ + "bb.event.GetTaskSignatureResult", + "logging.LogRecord", + "bb.command.CommandCompleted", + "bb.command.CommandFailed", + ] + ) + ret = tinfoil.run_command("getTaskSignatures", pn, tasks) + if ret: + while True: + event = tinfoil.wait_event(1) + if event: + if isinstance(event, bb.command.CommandCompleted): + break + elif isinstance(event, bb.command.CommandFailed): + logger.error(str(event)) + sys.exit(2) + elif isinstance(event, bb.event.GetTaskSignatureResult): + sig = event.sig + elif isinstance(event, logging.LogRecord): + logger.handle(event) + else: + logger.error("No result returned from getTaskSignatures command") + sys.exit(2) + return sig + + +def parseRecipe(tinfoil, recipe): + try: + tinfoil.parse_recipes() + d = tinfoil.parse_recipe(recipe) + except Exception: + logger.error("Failed to get recipe info for: %s" % recipe) + sys.exit(1) + return d + + +def bblockDump(lockfile): + try: + with open(lockfile, "r") as lockfile: + for line in lockfile: + print(line.strip()) + except IOError: + return 1 + return 0 + + +def bblockReset(lockfile, pns, package_archs, tasks): + if not pns: + logger.info("Unlocking all recipes") + try: + os.remove(lockfile) + except FileNotFoundError: + pass + else: + logger.info("Unlocking {pns}".format(pns=pns)) + tmp_lockfile = lockfile + ".tmp" + with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile: + for line in infile: + if not ( + any(element in line for element in pns) + and any(element in line for element in package_archs.split()) + ): + outfile.write(line) + else: + if tasks and not any(element in line for element in tasks): + outfile.write(line) + os.remove(lockfile) + os.rename(tmp_lockfile, lockfile) + + +def main(): + parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe") + parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock") + parser.add_argument( + "-t", + "--tasks", + help="Comma separated list of tasks", + type=lambda s: [task for task in s.split(",")], + ) + parser.add_argument( + "-r", + "--reset", + action="store_true", + help="Unlock pn recipes, or all recipes if pn is empty", + ) + parser.add_argument( + "-d", + "--dump", + action="store_true", + help="Dump generated bblock.conf file", + ) + + global_args, unparsed_args = parser.parse_known_args() + + with bb.tinfoil.Tinfoil() as tinfoil: + tinfoil.prepare(config_only=True) + + package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS") + builddir = tinfoil.config_data.getVar("TOPDIR") + lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir) + + if global_args.dump: + bblockDump(lockfile) + return 0 + + if global_args.reset: + bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks) + return 0 + + with open(lockfile, "a") as lockfile: + s = "" + if lockfile.tell() == 0: + s = "# Generated by bblock\n" + s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n' + s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n' + s += "\n" + + for pn in global_args.pn: + d = parseRecipe(tinfoil, pn) + package_arch = d.getVar("PACKAGE_ARCH") + siggen_locked_sigs_package_arch = d.getVar( + "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch) + ) + sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks) + for sig in sigs: + new_entry = "{pn}:{taskname}:{sig}".format( + pn=sig[0], taskname=sig[1], sig=sig[2] + ) + if ( + siggen_locked_sigs_package_arch + and not new_entry in siggen_locked_sigs_package_arch + ) or not siggen_locked_sigs_package_arch: + s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format( + package_arch=package_arch, new_entry=new_entry + ) + lockfile.write(s) + return 0 + + +if __name__ == "__main__": + try: + ret = main() + except Exception: + ret = 1 + import traceback + + traceback.print_exc() + sys.exit(ret)
Signature · 1 line
2.41.0
Hello,
Quoted 4 lines · Julien Stephan
Hi all,
This is v3 from bblock script.
This doesn't apply cleanly on master, can you rebase?
Thanks!
Quoted 72 lines
Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.confLimitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existI did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive forconf/bblock.confinside bitbake.conf
* Added -t option to specify the tasks to lock/unlockLimitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only thedo_compilefor now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missingV2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584My branch is available here [1]
Cheers
Julien[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (3):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signaturesscripts/bblock: add a script to lock/unlock recipesbitbake/lib/bb/command.py | 6 ++ bitbake/lib/bb/cooker.py | 16 ++++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + scripts/bblock | 182 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 213 insertions(+) create mode 100755 scripts/bblock--
2.41.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184932): https://lists.openembedded.org/g/openembedded-core/message/184932
Mute This Topic: https://lists.openembedded.org/mt/100390731/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 4 lines · Alexandre Belloni
Hello,
On 27/07/2023 15:56:09+0200, Julien Stephan wrote:
Quoted 8 lines
Quoted 4 lines
Hi all,
This is v3 from bblock script.
This doesn't apply cleanly on master, can you rebase?
Thanks!
Hi Alexandre,
done, just pushed v4 here:
https://lists.openembedded.org/g/openembedded-core/message/185403
Best
Julien
Quoted 79 lines
Quoted 65 lines
Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.confLimitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existI did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive forconf/bblock.confinside bitbake.conf
* Added -t option to specify the tasks to lock/unlockLimitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only thedo_compilefor now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missingV2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584My branch is available here [1]
Cheers
Julien[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (3):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signaturesscripts/bblock: add a script to lock/unlock recipesbitbake/lib/bb/command.py | 6 ++ bitbake/lib/bb/cooker.py | 16 ++++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + scripts/bblock | 182 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 213 insertions(+) create mode 100755 scripts/bblock--
2.41.0Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184932): https://lists.openembedded.org/g/openembedded-core/message/184932
Mute This Topic: https://lists.openembedded.org/mt/100390731/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com