Add bblock helper scripts
v4 cover Add bblock helper scripts
Hi all,
This is v4 for bblock script.
Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)
Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist
Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf
Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist
I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock
Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing
V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584
My branch is available here [1]
Cheers
Julien
[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool
bitbake/lib/bb/command.py | 6 + bitbake/lib/bb/cooker.py | 16 +++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + meta/lib/oe/sstatesig.py | 13 +- meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ scripts/bblock | 182 +++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py create mode 100755 scripts/bblock
--
2.41.0
PATCH v4 1/5 bitbake.conf: include bblock.conf
include conf/bblock.conf. This file is generated by the bblock tool. It locks some package tasks by fixing their signatures. See bblock -h for more details
--- meta/conf/bitbake.conf | 1 + 1 file changed, 1 insertion(+)
meta/conf/bitbake.conf +1 -0
@@ -832,6 +832,7 @@ include conf/documentation.conf include conf/licenses.conf require conf/sanity.conf require conf/cve-check-map.conf +include conf/bblock.conf ################################################################## # Weak variables (usually to retain backwards compatibility)
Signature · 1 line
2.41.0
PATCH v4 2/5 bitbake: cooker: add a new function to retrieve task signatures
In reply to Julien Stephanadding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the signatures using tinfoil
--- bitbake/lib/bb/command.py | 6 ++++++ bitbake/lib/bb/cooker.py | 16 ++++++++++++++++ bitbake/lib/bb/event.py | 8 ++++++++ 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
@@ -776,3 +776,9 @@ class CommandsAsync: bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache = False + + def getTaskSignatures(self, command, params): + res = command.cooker.getTaskSignatures(params[0], params[1]) + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data) + command.finishAsyncCommand() + getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +16 -0
@@ -1542,6 +1542,22 @@ class BBCooker: self.idleCallBackRegister(buildFileIdle, rq) + def getTaskSignatures(self, target, task): + sig = [] + + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt) + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) + rq.rqdata.prepare() + + for key in rq.rqdata.runtaskentries: + pn = bb.parse.siggen.tidtopn[key] + taskname = bb.runqueue.taskname_from_tid(key) + if pn in target: + if (task and taskname in task) or (not task): + rq.rqdata.prepare_task_hash(key) + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)]) + return sig + def buildTargets(self, targets, task): """ Attempt to build the targets specified
bitbake/lib/bb/event.py +8 -0
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event): Event.__init__(self) self.result = result +class GetTaskSignatureResult(Event): + """ + Event to return results from GetTaskSignatures command + """ + def __init__(self, sig): + Event.__init__(self) + self.sig = sig + class ParseError(Event): """ Event to indicate parse failed
Signature · 1 line
2.41.0
Re: [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures
Quoted 40 lines · Julien Stephan
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the signatures using tinfoil
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/command.py | 6 ++++++ bitbake/lib/bb/cooker.py | 16 ++++++++++++++++ bitbake/lib/bb/event.py | 8 ++++++++ 3 files changed, 30 insertions(+)bitbake/lib/bb/command.py +6
-0@@ -776,3 +776,9 @@ class CommandsAsync: bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache = False + + def getTaskSignatures(self, command, params): + res = command.cooker.getTaskSignatures(params[0], params[1]) + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data) + command.finishAsyncCommand() + getTaskSignatures.needcache = Truebitbake/lib/bb/cooker.py +6
-0@@ -1542,6 +1542,22 @@ class BBCooker: self.idleCallBackRegister(buildFileIdle, rq) + def getTaskSignatures(self, target, task): + sig = [] + + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt) + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) + rq.rqdata.prepare()
bitbake/lib/bb/cooker.py class BBCooker: · line 1560
Quoted 10 lines
bitbake/lib/bb/cooker.py +10
-0+ + for key in rq.rqdata.runtaskentries: + pn = bb.parse.siggen.tidtopn[key] + taskname = bb.runqueue.taskname_from_tid(key) + if pn in target: + if (task and taskname in task) or (not task): + rq.rqdata.prepare_task_hash(key) + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)]) + return sig +
Hi all,
I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks
Using buildTaskData with "do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"
At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks, but running bitbake -c listtasks gives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasks
So what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?
Cheers
Julien
[1]: https://docs.yoctoproject.org/ref-manual/tasks.html
Quoted 25 lines
bitbake/lib/bb/cooker.py +0
-0def buildTargets(self, targets, task): """ Attempt to build the targets specifiedbitbake/lib/bb/event.py +8
-1@@ -857,6 +857,14 @@ class FindSigInfoResult(Event): Event.__init__(self) self.result = result +class GetTaskSignatureResult(Event): + """ + Event to return results from GetTaskSignatures command + """ + def __init__(self, sig): + Event.__init__(self) + self.sig = sig + class ParseError(Event): """ Event to indicate parse failed --2.41.0
Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures
Quoted 35 lines · Julien Stephan
Quoted 34 lines · Julien Stephan
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the signatures using tinfoil
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/command.py | 6 ++++++ bitbake/lib/bb/cooker.py | 16 ++++++++++++++++ bitbake/lib/bb/event.py | 8 ++++++++ 3 files changed, 30 insertions(+)bitbake/lib/bb/command.py +6
-0@@ -776,3 +776,9 @@ class CommandsAsync: bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache = False + + def getTaskSignatures(self, command, params): + res = command.cooker.getTaskSignatures(params[0], params[1]) + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data) + command.finishAsyncCommand() + getTaskSignatures.needcache = Truebitbake/lib/bb/cooker.py +0
-0@@ -1542,6 +1542,22 @@ class BBCooker: self.idleCallBackRegister(buildFileIdle, rq)
Quoted 16 lines
Quoted 6 lines
bitbake/lib/bb/cooker.py +6
-0+ def getTaskSignatures(self, target, task): + sig = [] + + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt) + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) + rq.rqdata.prepare()bitbake/lib/bb/cooker.py class BBCooker: · line 1560
Quoted 10 lines
bitbake/lib/bb/cooker.py +10
-0+ + for key in rq.rqdata.runtaskentries: + pn = bb.parse.siggen.tidtopn[key] + taskname = bb.runqueue.taskname_from_tid(key) + if pn in target: + if (task and taskname in task) or (not task): + rq.rqdata.prepare_task_hash(key) + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)]) + return sig +
Firstly, thanks for working on this! These changes aren't simple and it means the review takes longer as we need to get this right.
Quoted 6 lines
I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks
Quoted 7 lines
Using buildTaskData with "
do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,
I think you need to query the tasks available for a recipe, remove the "nostamp" ones and then return the signatures for the rest.
Quoted 8 lines
but running
bitbake -c listtasksgives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasksSo what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?
I think bitbake should internally be able to work out the nostamp ones and return a list of all of them with the nostamp ones excluded.
Let me know if you need some pointers on how to do that in bitbake, it isn't entirely straightforward and I'd have to look it up. I *think* (without looking to know for sure) that the dataCache object does have the task data in it which can be looked up by target and that lists which ones are nostamp.
Cheers,
Richard
Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures
Quoted 83 lines · Richard Purdie
Quoted 35 lines · Julien Stephan
Quoted 34 lines · Julien Stephan
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the signatures using tinfoil
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/command.py | 6 ++++++ bitbake/lib/bb/cooker.py | 16 ++++++++++++++++ bitbake/lib/bb/event.py | 8 ++++++++ 3 files changed, 30 insertions(+)bitbake/lib/bb/command.py +6
-0@@ -776,3 +776,9 @@ class CommandsAsync: bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache = False + + def getTaskSignatures(self, command, params): + res = command.cooker.getTaskSignatures(params[0], params[1]) + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data) + command.finishAsyncCommand() + getTaskSignatures.needcache = Truebitbake/lib/bb/cooker.py +0
-0@@ -1542,6 +1542,22 @@ class BBCooker: self.idleCallBackRegister(buildFileIdle, rq)Quoted 16 lines
Quoted 6 lines
bitbake/lib/bb/cooker.py +6
-0+ def getTaskSignatures(self, target, task): + sig = [] + + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt) + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) + rq.rqdata.prepare()bitbake/lib/bb/cooker.py class BBCooker: · line 1560
Quoted 10 lines
bitbake/lib/bb/cooker.py +10
-0+ + for key in rq.rqdata.runtaskentries: + pn = bb.parse.siggen.tidtopn[key] + taskname = bb.runqueue.taskname_from_tid(key) + if pn in target: + if (task and taskname in task) or (not task): + rq.rqdata.prepare_task_hash(key) + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)]) + return sig +Firstly, thanks for working on this! These changes aren't simple and it means the review takes longer as we need to get this right.
Quoted 6 lines
I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasksQuoted 7 lines
Using buildTaskData with "
do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,I think you need to query the tasks available for a recipe, remove the "nostamp" ones and then return the signatures for the rest.
Quoted 8 lines
but running
bitbake -c listtasksgives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasksSo what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?
Quoted 8 lines
I think bitbake should internally be able to work out the nostamp ones and return a list of all of them with the nostamp ones excluded.
Let me know if you need some pointers on how to do that in bitbake, it isn't entirely straightforward and I'd have to look it up. I *think* (without looking to know for sure) that the dataCache object does have the task data in it which can be looked up by target and that lists which ones are nostamp.
Hi Richard,
Sorry for the late reply I was off last week.
Using rq.rqdata.dataCaches I am able to get the full list of tasks and filter out the no stamps.
But it is still unclear for me what to do :( Here is what I understood, please correct me if I am wrong:
* in getTaskSignatures (cooker.py): I should get the tasks list and filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ), then compute the signature for all
* in oeqa self test, I get the list of tasks using 'bitbake -c listtasks', but how to filter out the 'nostamps' ones?
Cheers
Julien
Quoted 6 lines
Cheers,
Richard
Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures
Hi,
Sorry, I meant to reply to this and somehow it got lost with everything else going on.
Quoted 94 lines · Julien Stephan
Quoted 83 lines · Richard Purdie
Quoted 35 lines · Julien Stephan
Quoted 34 lines · Julien Stephan
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the signatures using tinfoil
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/command.py | 6 ++++++ bitbake/lib/bb/cooker.py | 16 ++++++++++++++++ bitbake/lib/bb/event.py | 8 ++++++++ 3 files changed, 30 insertions(+)bitbake/lib/bb/command.py +6
-0@@ -776,3 +776,9 @@ class CommandsAsync: bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache = False + + def getTaskSignatures(self, command, params): + res = command.cooker.getTaskSignatures(params[0], params[1]) + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data) + command.finishAsyncCommand() + getTaskSignatures.needcache = Truebitbake/lib/bb/cooker.py +0
-0@@ -1542,6 +1542,22 @@ class BBCooker: self.idleCallBackRegister(buildFileIdle, rq)Quoted 16 lines
Quoted 6 lines
bitbake/lib/bb/cooker.py +6
-0+ def getTaskSignatures(self, target, task): + sig = [] + + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt) + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) + rq.rqdata.prepare()bitbake/lib/bb/cooker.py class BBCooker: · line 1560
Quoted 10 lines
bitbake/lib/bb/cooker.py +10
-0+ + for key in rq.rqdata.runtaskentries: + pn = bb.parse.siggen.tidtopn[key] + taskname = bb.runqueue.taskname_from_tid(key) + if pn in target: + if (task and taskname in task) or (not task): + rq.rqdata.prepare_task_hash(key) + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)]) + return sig +Firstly, thanks for working on this! These changes aren't simple and it means the review takes longer as we need to get this right.
Quoted 6 lines
I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasksQuoted 7 lines
Using buildTaskData with "
do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,I think you need to query the tasks available for a recipe, remove the "nostamp" ones and then return the signatures for the rest.
Quoted 8 lines
but running
bitbake -c listtasksgives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasksSo what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?
Quoted 8 lines
I think bitbake should internally be able to work out the nostamp ones and return a list of all of them with the nostamp ones excluded.
Let me know if you need some pointers on how to do that in bitbake, it isn't entirely straightforward and I'd have to look it up. I *think* (without looking to know for sure) that the dataCache object does have the task data in it which can be looked up by target and that lists which ones are nostamp.
Quoted 7 lines
Sorry for the late reply I was off last week.
Using rq.rqdata.dataCaches I am able to get the full list of tasks and filter out the no stamps.
But it is still unclear for me what to do :( Here is what I understood, please correct me if I am wrong:
* in getTaskSignatures (cooker.py): I should get the tasks list and filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ),
You can skip the setscene ones as they're not interesting or needed for what we need here, at least as I understand it as we never lock those.
Quoted 3 lines
then compute the signature for all
* in oeqa self test, I get the list of tasks using 'bitbake -c listtasks', but how to filter out the 'nostamps' ones?
Just thinking out loud, for a given recipe you could run d.getVarFlag("do_clean", "nostamp") and see that it was set?
"bitbake-getvar -r bash do_clean -f nostamp --value" also shows that.
Cheers,
Richard
PATCH v4 3/5 sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
In reply to Julien Stephanas of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and "error", displaying respectively a warning or a fatal error message only when a task is locked and the task signature is different from the locked one.
The "info" level is introduced to add a "note" message to remind the user that a recipe is locked even if the signature is equivalent to the locked one.
The "warn" and "error" level display the warn/error message for each task having a mismatch of the signature. Doing this with the "info" level would result in very verbose output if there are several tasks locked, so the info level will only print once the list of recipes that have locked signature.
--- meta/lib/oe/sstatesig.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +12 -1
@@ -104,6 +104,7 @@ class SignatureGeneratorOEBasicHashMixIn(object): self.lockedhashfn = {} self.machine = data.getVar("MACHINE") self.mismatch_msgs = [] + self.lockedsigs_msgs = "" self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or "").split() self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
@@ -264,6 +265,12 @@ class SignatureGeneratorOEBasicHashMixIn(object): warn_msgs = [] error_msgs = [] sstate_missing_msgs = [] + info_msgs = None + + if self.lockedsigs: + self.lockedsigs_msgs = "The following recipes have locked tasks:" + for pn in self.lockedsigs: + self.lockedsigs_msgs += " %s" % (pn) for tid in sq_data['hash']: if tid not in found:
@@ -276,7 +283,9 @@ class SignatureGeneratorOEBasicHashMixIn(object): % (pn, taskname, sq_data['hash'][tid])) checklevel = d.getVar("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK") - if checklevel == 'warn': + if checklevel == 'info': + info_msgs = self.lockedsigs_msgs + if checklevel == 'warn' or checklevel == 'info': warn_msgs += self.mismatch_msgs elif checklevel == 'error': error_msgs += self.mismatch_msgs
@@ -287,6 +296,8 @@ class SignatureGeneratorOEBasicHashMixIn(object): elif checklevel == 'error': error_msgs += sstate_missing_msgs + if info_msgs: + bb.note(info_msgs) if warn_msgs: bb.warn("\n".join(warn_msgs)) if error_msgs:
Signature · 1 line
2.41.0
Re: [OE-core] [PATCH v4 3/5] sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
Quoted 35 lines · Julien Stephan
as of now,
SIGGEN_LOCKEDSIGS_TASKSIG_CHECKcan take 2 values: "warn" and "error", displaying respectively a warning or a fatal error message only when a task is locked and the task signature is different from the locked one.The "info" level is introduced to add a "note" message to remind the user that a recipe is locked even if the signature is equivalent to the locked one.
The "warn" and "error" level display the warn/error message for each task having a mismatch of the signature. Doing this with the "info" level would result in very verbose output if there are several tasks locked, so the info level will only print once the list of recipes that have locked signature.
Signed-off-by Julien Stephan <[email protected]>--- meta/lib/oe/sstatesig.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)meta/lib/oe/sstatesig.py +1
-0@@ -104,6 +104,7 @@ class SignatureGeneratorOEBasicHashMixIn(object): self.lockedhashfn = {} self.machine = data.getVar("MACHINE") self.mismatch_msgs = [] + self.lockedsigs_msgs = "" self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or "").split() self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }@@ -264,6 +265,12 @@ class SignatureGeneratorOEBasicHashMixIn(object): warn_msgs = [] error_msgs = []
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHashMixIn(object): · line 276
Quoted 10 lines
meta/lib/oe/sstatesig.py +6
-0sstate_missing_msgs = [] + info_msgs = None + + if self.lockedsigs: + self.lockedsigs_msgs = "The following recipes have locked tasks:" + for pn in self.lockedsigs: + self.lockedsigs_msgs += " %s" % (pn) for tid in sq_data['hash']: if tid not in found:
How about something like:
if len(pn) > 10:
print "There are XX locked recipes (XX matching and XX not matching
parsing)"
so that in the case where lots of things are locked you get a summary instead showing the amount of match/mismatch?
Cheers,
Richard
PATCH v4 4/5 scripts/bblock: add a script to lock/unlock recipes
In reply to Julien Stephanbblock script allows to lock/unlock recipes to latest task signatures. The idea is to prevent some recipes to be rebuilt during development. For example when working on rust recipe, one may not want rust-native to be rebuilt.
This tool can be used, with proper environment set up, using the following command:
bblock <recipe_name>
if a <recipe_name>'s task signature change, this task will not be built again and sstate cache will be used.
[YOCTO #13425]
--- scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100755 scripts/bblock
scripts/bblock new file +182 -0
@@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +# bblock +# lock/unlock task to latest signature +# +# Copyright (c) 2023 BayLibre, SAS +# Author: Julien Stepahn <[email protected]> +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import os +import sys +import logging + +scripts_path = os.path.dirname(os.path.realpath(__file__)) +lib_path = scripts_path + "/lib" +sys.path = sys.path + [lib_path] + +import scriptpath + +scriptpath.add_bitbake_lib_path() + +import bb.tinfoil +import bb.msg + +import argparse_oe + +myname = os.path.basename(sys.argv[0]) +logger = bb.msg.logger_create(myname) + + +def getTaskSignatures(tinfoil, pn, tasks): + tinfoil.set_event_mask( + [ + "bb.event.GetTaskSignatureResult", + "logging.LogRecord", + "bb.command.CommandCompleted", + "bb.command.CommandFailed", + ] + ) + ret = tinfoil.run_command("getTaskSignatures", pn, tasks) + if ret: + while True: + event = tinfoil.wait_event(1) + if event: + if isinstance(event, bb.command.CommandCompleted): + break + elif isinstance(event, bb.command.CommandFailed): + logger.error(str(event)) + sys.exit(2) + elif isinstance(event, bb.event.GetTaskSignatureResult): + sig = event.sig + elif isinstance(event, logging.LogRecord): + logger.handle(event) + else: + logger.error("No result returned from getTaskSignatures command") + sys.exit(2) + return sig + + +def parseRecipe(tinfoil, recipe): + try: + tinfoil.parse_recipes() + d = tinfoil.parse_recipe(recipe) + except Exception: + logger.error("Failed to get recipe info for: %s" % recipe) + sys.exit(1) + return d + + +def bblockDump(lockfile): + try: + with open(lockfile, "r") as lockfile: + for line in lockfile: + print(line.strip()) + except IOError: + return 1 + return 0 + + +def bblockReset(lockfile, pns, package_archs, tasks): + if not pns: + logger.info("Unlocking all recipes") + try: + os.remove(lockfile) + except FileNotFoundError: + pass + else: + logger.info("Unlocking {pns}".format(pns=pns)) + tmp_lockfile = lockfile + ".tmp" + with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile: + for line in infile: + if not ( + any(element in line for element in pns) + and any(element in line for element in package_archs.split()) + ): + outfile.write(line) + else: + if tasks and not any(element in line for element in tasks): + outfile.write(line) + os.remove(lockfile) + os.rename(tmp_lockfile, lockfile) + + +def main(): + parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe") + parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock") + parser.add_argument( + "-t", + "--tasks", + help="Comma separated list of tasks", + type=lambda s: [task for task in s.split(",")], + ) + parser.add_argument( + "-r", + "--reset", + action="store_true", + help="Unlock pn recipes, or all recipes if pn is empty", + ) + parser.add_argument( + "-d", + "--dump", + action="store_true", + help="Dump generated bblock.conf file", + ) + + global_args, unparsed_args = parser.parse_known_args() + + with bb.tinfoil.Tinfoil() as tinfoil: + tinfoil.prepare(config_only=True) + + package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS") + builddir = tinfoil.config_data.getVar("TOPDIR") + lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir) + + if global_args.dump: + bblockDump(lockfile) + return 0 + + if global_args.reset: + bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks) + return 0 + + with open(lockfile, "a") as lockfile: + s = "" + if lockfile.tell() == 0: + s = "# Generated by bblock\n" + s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"\n' + s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n' + s += "\n" + + for pn in global_args.pn: + d = parseRecipe(tinfoil, pn) + package_arch = d.getVar("PACKAGE_ARCH") + siggen_locked_sigs_package_arch = d.getVar( + "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch) + ) + sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks) + for sig in sigs: + new_entry = "{pn}:{taskname}:{sig}".format( + pn=sig[0], taskname=sig[1], sig=sig[2] + ) + if ( + siggen_locked_sigs_package_arch + and not new_entry in siggen_locked_sigs_package_arch + ) or not siggen_locked_sigs_package_arch: + s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format( + package_arch=package_arch, new_entry=new_entry + ) + lockfile.write(s) + return 0 + + +if __name__ == "__main__": + try: + ret = main() + except Exception: + ret = 1 + import traceback + + traceback.print_exc() + sys.exit(ret)
Signature · 1 line
2.41.0
PATCH v4 5/5 oeqa/selftest/bblock: add self test for bblock tool
In reply to Julien Stephanit implements various combination of locking single/multiple recipe(s)/task(s)
it also tests that locked sig are architecture dependant
--- meta/lib/oeqa/selftest/cases/bblock.py | 146 +++++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
meta/lib/oeqa/selftest/cases/bblock.py new file +146 -0
@@ -0,0 +1,146 @@ +# +# Copyright (c) 2023 BayLibre, SAS +# Author: Julien Stepahn <[email protected]> +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import os +import re + +import oeqa.utils.ftools as ftools +from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake + +from oeqa.selftest.case import OESelftestTestCase + +class BBLock(OESelftestTestCase): + + @classmethod + def setUpClass(cls): + super(BBLock, cls).setUpClass() + cls.lockfile = cls.builddir+"/conf/bblock.conf" + + def unlock_recipes(self, recipes=None, tasks=None): + cmd = 'bblock -r ' + if recipes: + cmd += ' '.join(recipes) + if tasks: + cmd += ' -t ' + ','.join(tasks) + result = runCmd(cmd) + + if recipes: + # ensure all signatures are removed from lockfile + contents = ftools.read_file(self.lockfile) + for recipe in recipes: + for task in tasks: + find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents) + self.assertFalse(find_in_contents, msg = "%s:%s should not be present into bblock.conf anymore" % (recipe, task)) + self.assertExists(self.lockfile) + else: + self.assertNotExists(self.lockfile) + + def lock_recipes(self, recipes, tasks=None, machine=None): + cmd = 'bblock ' + ' '.join(recipes) + if tasks: + cmd += ' -t ' + ','.join(tasks) + + if machine: + cmd = "MACHINE=" + machine + " " + cmd + with open("/tmp/tmp.txt", "w") as file: + file.write(cmd) + result = runCmd(cmd) + + self.assertExists(self.lockfile) + + # ensure all signatures are added to lockfile + contents = ftools.read_file(self.lockfile) + for recipe in recipes: + for task in tasks: + find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents) + self.assertTrue(find_in_contents, msg = "%s:%s was not added into bblock.conf. bblock output: %s" % (recipe, task, result.output)) + + def modify_tasks(self, recipes, tasks): + task_append = "" + for recipe in recipes: + bb_vars = get_bb_vars(['PV'], recipe) + recipe_pv = bb_vars['PV'] + recipe_append_file = recipe + '_' + recipe_pv + '.bbappend' + + os.mkdir(os.path.join(self.testlayer_path, 'recipes-test', recipe)) + recipe_append_path = os.path.join(self.testlayer_path, 'recipes-test', recipe, recipe_append_file) + + for task in tasks: + task_append += "%s:append() {\necho \"modify task hash\"\n}\n" % task + ftools.write_file(recipe_append_path, task_append) + self.add_command_to_tearDown('rm -rf %s' % os.path.join(self.testlayer_path, 'recipes-test', recipe)) + + def test_lock_single_recipe_single_task(self): + recipes = ["quilt"] + tasks = ["do_compile"] + self._run_test(recipes, tasks) + + def test_lock_single_recipe_multiple_tasks(self): + recipes = ["quilt"] + tasks = ["do_compile" ,"do_install"] + self._run_test(recipes, tasks) + + def test_lock_architecture_specific(self): + recipes = ["quilt"] + tasks = ["do_compile"] + + # lock quilt's do_compile task for qemuarm machine + self.lock_recipes(recipes, tasks, "qemuarm") + + # modify quilt's do_compile task + self.modify_tasks(recipes, tasks) + + # build quilt using by default qemux86-64. + # No Note/Warning should be emitted since sig is locked for qemuarm + # (quilt package is architecture dependant) + info_message = "NOTE: The following recipes have locked tasks: " + recipes[0] + warn_message = "The %s:%s sig is computed to be" % (recipes[0], tasks[0]) + result = bitbake(recipes[0] + " -n") + self.assertNotIn(info_message, result.output) + self.assertNotIn(warn_message, result.output) + + def test_lock_multiple_recipe_single_task(self): + recipes = ["quilt", "bc"] + tasks= ["do_compile"] + self._run_test(recipes, tasks) + + def _run_test(self, recipes, tasks=None): + # unlock all recipes and ensure no bblock.conf file exist + self.unlock_recipes() + + # lock tasks for recipes + result = self.lock_recipes(recipes, tasks) + + # build recipes. At this stage we should have a Note about recipes + # having locked task's sig, but no warning since sig still match + info_message = "NOTE: The following recipes have locked tasks: " + " ".join(recipes) + for recipe in recipes: + result = bitbake(recipe + " -n") + self.assertIn(info_message, result.output) + for task in tasks: + warn_message = "The %s:%s sig is computed to be" % (recipe, task) + self.assertNotIn(warn_message, result.output) + + ## modify all tasks that are locked to trigger a sig change then build the recipes + ## at this stage we should have a Note as before, but also a warning for all + ## locked stage indicating the sig mismatch + self.modify_tasks(recipes, tasks) + for recipe in recipes: + result = bitbake(recipe + " -n") + self.assertIn(info_message, result.output) + for task in tasks: + warn_message = "The %s:%s sig is computed to be" % (recipe, task) + self.assertIn(warn_message, result.output) + + # unlock all tasks and rebuild, no more Note/Warning should remain + self.unlock_recipes(recipes, tasks) + for recipe in recipes: + result = bitbake(recipe + " -n") + self.assertNotIn(info_message, result.output) + for task in tasks: + warn_message = "The %s:%s sig is computed to be" % (recipe, task) + self.assertNotIn(warn_message, result.output)
Signature · 1 line
2.41.0
When ' include conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default?
Regards,
Qi
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Julien Stephan
Sent: Wednesday, August 2, 2023 10:24 PM
To: [email protected]
Cc: Julien Stephan <[email protected]>
Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scripts
Hi all,
This is v4 for bblock script.
Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)
Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist
Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf
Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist
I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock
Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing
V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584
My branch is available here [1]
Cheers
Julien
[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool
bitbake/lib/bb/command.py | 6 + bitbake/lib/bb/cooker.py | 16 +++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + meta/lib/oe/sstatesig.py | 13 +- meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ scripts/bblock | 182 +++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py create mode 100755 scripts/bblock
--
2.41.0
Quoted 6 lines · Qi
When ' include
conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default?Regards,
Qi
Hi Qi,
This was a request from Richard on the first submission [1] :
Quoted 4 lines
To simplify things, I think it would be reasonable to add a bblock.inc file unconditionally in our core configuration so that it is always included if present, just to make the tool easier since I think this will be a common use.
Cheers
Julien
[1): https://lists.openembedded.org/g/openembedded-core/message/184586
Quoted 85 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Julien Stephan
Sent: Wednesday, August 2, 2023 10:24 PM
To: [email protected]
Cc: Julien Stephan <[email protected]>
Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scriptsHi all,
This is v4 for bblock script.
Improvement from v3:
* Add self test
* Add a new "info" level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existImprovement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.confLimitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existI did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive forconf/bblock.confinside bitbake.conf
* Added -t option to specify the tasks to lock/unlockLimitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only thedo_compilefor now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missingV3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584My branch is available here [1]
Cheers
Julien[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECKscripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock toolbitbake/lib/bb/command.py | 6 + bitbake/lib/bb/cooker.py | 16 +++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + meta/lib/oe/sstatesig.py | 13 +- meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ scripts/bblock | 182 +++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py create mode 100755 scripts/bblock--
2.41.0
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
self.assertNotIn(info_message, result.output)
File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
self.fail(self._formatMessage(msg, standardMsg))
File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
raise self.failureException(msg)AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
Quoted 90 lines · Julien Stephan
Hi all,
This is v4 for bblock script.
Improvement from v3:
* Add self test
* Add a new "info" level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existImprovement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.confLimitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existI did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive forconf/bblock.confinside bitbake.conf
* Added -t option to specify the tasks to lock/unlockLimitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only thedo_compilefor now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missingV3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584My branch is available here [1]
Cheers
Julien[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECKscripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock toolbitbake/lib/bb/command.py | 6 + bitbake/lib/bb/cooker.py | 16 +++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + meta/lib/oe/sstatesig.py | 13 +- meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ scripts/bblock | 182 +++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py create mode 100755 scripts/bblock--
2.41.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 20 lines · Alexandre Belloni
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL Stderr: 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific self.assertNotIn(info_message, result.output) File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail raise self.failureException(msg)
Quoted 2 lines
AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
Hum, wait, I got this without the patch series. Let me investigate.
Quoted 99 lines · Julien Stephan
Quoted 90 lines
Hi all,
This is v4 for bblock script.
Improvement from v3:
* Add self test
* Add a new "info" level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existImprovement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.confLimitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existI did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive forconf/bblock.confinside bitbake.conf
* Added -t option to specify the tasks to lock/unlockLimitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only thedo_compilefor now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missingV3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584My branch is available here [1]
Cheers
Julien[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECKscripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock toolbitbake/lib/bb/command.py | 6 + bitbake/lib/bb/cooker.py | 16 +++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + meta/lib/oe/sstatesig.py | 13 +- meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ scripts/bblock | 182 +++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py create mode 100755 scripts/bblock--
2.41.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 24 lines · Alexandre Belloni
Quoted 20 lines · Alexandre Belloni
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL Stderr: 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific self.assertNotIn(info_message, result.output) File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail raise self.failureException(msg)Quoted 2 lines
AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
Quoted 1 line
Hum, wait, I got this without the patch series. Let me investigate.
Actually, I still had the series :)
Quoted 105 lines
Quoted 100 lines · Julien Stephan
Quoted 90 lines
Hi all,
This is v4 for bblock script.
Improvement from v3:
* Add self test
* Add a new "info" level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existImprovement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.confLimitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't existI did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive forconf/bblock.confinside bitbake.conf
* Added -t option to specify the tasks to lock/unlockLimitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only thedo_compilefor now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missingV3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584My branch is available here [1]
Cheers
Julien[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level forSIGGEN_LOCKEDSIGS_TASKSIG_CHECKscripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock toolbitbake/lib/bb/command.py | 6 + bitbake/lib/bb/cooker.py | 16 +++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + meta/lib/oe/sstatesig.py | 13 +- meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ scripts/bblock | 182 +++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py create mode 100755 scripts/bblock--
2.41.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.comSignature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Hi Alexandre,
sorry for the late reply.
I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.
Please let me know if I can do anything else
Cheers
Julien
Quoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines · Alexandre Belloni
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL Stderr: 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific self.assertNotIn(info_message, result.output) File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail raise self.failureException(msg)AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
Quoted 4 lines
Quoted 1 line
Hum, wait, I got this without the patch series. Let me investigate.
Actually, I still had the series :)
Hi Alexandre,
I am not able to reproduce it locally, so it is difficult to fix the issue.
Quoted 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Hello,
Quoted 11 lines · Julien Stephan
Hi Alexandre,
sorry for the late reply.I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.
This was sent to the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio
Quoted 47 lines
Please let me know if I can do anything else
Cheers
JulienLe ven. 11 ao�t 2023 � 10:45, Alexandre Belloni <[email protected]> a �crit :
Quoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL Stderr: 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific self.assertNotIn(info_message, result.output) File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail raise self.failureException(msg)AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
Quoted 4 lines
Quoted 1 line
Hum, wait, I got this without the patch series. Let me investigate.
Actually, I still had the series :)
Hi Alexandre,
I am not able to reproduce it locally, so it is difficult to fix the issue.
Quoted 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 16 lines · Alexandre Belloni
Hello,
Quoted 11 lines · Julien Stephan
Hi Alexandre,
sorry for the late reply.I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.
Quoted 4 lines
This was sent to the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio
Hi Alexandre,
Thank you! Unfortunately I can see that it is still failing on the same test :( I may need your help here.
Here is what my test is supposed to do:
From the autobuilder log I can see that the machine is set to qemux86-64 so :
$ bblock -d # unlock all recipes and remove build/conf/bblock.conf file $ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for qemuarm arch so block.conf file should look like this:
# Generated by bblockSIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"
SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>"
# modify the quilt's do_compile task by adding an empty do_compile:append function $ bitbake -n quilt # for qemux86-64 machine
Because quilt is supposed to be locked for qemuarm, I am not supposed to see any warning, but the log on the autobuilder says:
"WARNING: The quilt:do_compile sig is computed to be d5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but the sig is locked to c9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fe in SIGGEN_LOCKEDSIGS_core2-64\""
meaning bblock.conf contains:
SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig>
So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??
* bblock.conf gets polluted by another test that also tries to lock quilt's do_compile function for qemux86-64 because of parallelism (I was not able to reproduce it locally)
Do you have an idea why this test is failing?
My next move would be to place all the tests in a single test to remove the parallelism issue (if that is the issue) but I would prefer to avoid wasting your time starting builds for me several times..
Also, I didn't manage to print log messages on my test, how can I do that?
Cheers
Julien
Quoted 52 lines
Quoted 33 lines
Please let me know if I can do anything else
Cheers
JulienQuoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines · Alexandre Belloni
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL Stderr: 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific self.assertNotIn(info_message, result.output) File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail raise self.failureException(msg)AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
Quoted 14 lines
Quoted 4 lines
Quoted 1 line
Hum, wait, I got this without the patch series. Let me investigate.
Actually, I still had the series :)
Hi Alexandre,
I am not able to reproduce it locally, so it is difficult to fix the issue.
Quoted 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Quoted 53 lines · Julien Stephan
Le jeu. 21 sept. 2023 � 15:11, Alexandre Belloni <[email protected]> a �crit :
Quoted 16 lines
Hello,
Quoted 11 lines · Julien Stephan
Hi Alexandre,
sorry for the late reply.I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.Quoted 4 lines
This was sent to the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio
Hi Alexandre,
Thank you! Unfortunately I can see that it is still failing on the same test :( I may need your help here.
Here is what my test is supposed to do:
From the autobuilder log I can see that the machine is set to qemux86-64 so :$ bblock -d # unlock all recipes and remove build/conf/bblock.conf file $ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for qemuarm arch so block.conf file should look like this:# Generated by bblock
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK= "info"SIGGEN_LOCKEDSIGS_TYPES+= "${PACKAGE_ARCHS}"SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>"
# modify the quilt's do_compile task by adding an empty do_compile:append function $ bitbake -n quilt # for qemux86-64 machineBecause quilt is supposed to be locked for qemuarm, I am not supposed to see any warning, but the log on the autobuilder says:
"WARNING: The quilt:do_compile sig is computed to bed5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but the sig is locked toc9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fein SIGGEN_LOCKEDSIGS_core2-64\""
Quoted 8 lines
meaning bblock.conf contains:
SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig>
So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??
As I said in the call, look at this log from the build:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio
It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is then used by the build so overriding it from the command line doesn't work.
Quoted 65 lines
* bblock.conf gets polluted by another test that also tries to lock quilt's
do_compilefunction for qemux86-64 because of parallelism (I was not able to reproduce it locally)Do you have an idea why this test is failing?
My next move would be to place all the tests in a single test to remove the parallelism issue (if that is the issue) but I would prefer to avoid wasting your time starting builds for me several times..Also, I didn't manage to print log messages on my test, how can I do that?
Cheers
JulienQuoted 52 lines
Quoted 33 lines
Please let me know if I can do anything else
Cheers
JulienLe ven. 11 ao�t 2023 � 10:45, Alexandre Belloni <[email protected]> a �crit :
Quoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL Stderr: 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific self.assertNotIn(info_message, result.output) File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail raise self.failureException(msg)AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
Quoted 14 lines
Quoted 4 lines
Quoted 1 line
Hum, wait, I got this without the patch series. Let me investigate.
Actually, I still had the series :)
Hi Alexandre,
I am not able to reproduce it locally, so it is difficult to fix the issue.
Quoted 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 5 lines
Quoted 4 lines
So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??
Quoted 8 lines
As I said in the call, look at this log from the build:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio
It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is then used by the build so overriding it from the command line doesn't work.
Hi Alexandre,
Thank you, I managed to reproduce it locally and fixed it properly. I just force pushed my poky-contrib branch
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can you start a build? If successful, I'll send the v5 today.
Cheers
Julien
This is started:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5792
Quoted 23 lines · Julien Stephan
Quoted 5 lines
Quoted 4 lines
So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??Quoted 8 lines
As I said in the call, look at this log from the build:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio
It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is then used by the build so overriding it from the command line doesn't work.
Hi Alexandre,
Thank you, I managed to reproduce it locally and fixed it properly. I just force pushed my poky-contrib branch
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can you start a build? If successful, I'll send the v5 today.Cheers
Julien
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com