Skip to contentSkip to conversation

Add bblock helper scripts

PATCH Superseded Julien Stephan 4 participants 22 messages Aug 2, 2023 – Sep 22, 2023

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover Add bblock helper scripts

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

#1 lore raw

PATCH v4 1/5 bitbake.conf: include bblock.conf

include conf/bblock.conf. This file is generated by the bblock tool. It locks some package tasks by fixing their signatures. See bblock -h for more details

---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)
meta/conf/bitbake.conf +1 -0
index 475d6523bb8..e4a6bdef0ea 100644
@@ -832,6 +832,7 @@ include conf/documentation.conf
 include conf/licenses.conf
 require conf/sanity.conf
 require conf/cve-check-map.conf
+include conf/bblock.conf
 
 ##################################################################
 # Weak variables (usually to retain backwards compatibility)
Signature · 1 line
2.41.0
#2 lore raw

PATCH v4 2/5 bitbake: cooker: add a new function to retrieve task signatures

In reply to Julien Stephan

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the signatures using tinfoil

---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
index a355f56c60c..12202779ac0 100644
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +16 -0
index 11c9fa2c40d..687cdde5e6d 100644
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+
     def buildTargets(self, targets, task):
         """
         Attempt to build the targets specified
bitbake/lib/bb/event.py +8 -0
index 0d0e0a68aac..f8acacd80d1 100644
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
         Event.__init__(self)
         self.result = result
 
+class GetTaskSignatureResult(Event):
+    """
+    Event to return results from GetTaskSignatures command
+    """
+    def __init__(self, sig):
+        Event.__init__(self)
+        self.sig = sig
+
 class ParseError(Event):
     """
     Event to indicate parse failed
Signature · 1 line
2.41.0
#3 lore raw

Re: [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

Quoted 40 lines · Julien Stephan

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the signatures using tinfoil

---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
index a355f56c60c..12202779ac0 100644
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +6 -0
index 11c9fa2c40d..687cdde5e6d 100644
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()

bitbake/lib/bb/cooker.py class BBCooker: · line 1560

Quoted 10 lines
bitbake/lib/bb/cooker.py +10 -0
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+

Hi all,

I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks

Using buildTaskData with "do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"

At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks, but running bitbake -c listtasks gives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasks

So what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?

Cheers
Julien
[1]: https://docs.yoctoproject.org/ref-manual/tasks.html

Quoted 25 lines
bitbake/lib/bb/cooker.py +0 -0
     def buildTargets(self, targets, task):
         """
         Attempt to build the targets specified
bitbake/lib/bb/event.py +8 -1
index 0d0e0a68aac..f8acacd80d1 100644
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
         Event.__init__(self)
         self.result = result
 
+class GetTaskSignatureResult(Event):
+    """
+    Event to return results from GetTaskSignatures command
+    """
+    def __init__(self, sig):
+        Event.__init__(self)
+        self.sig = sig
+
 class ParseError(Event):
     """
     Event to indicate parse failed
--

2.41.0

#4 lore raw

Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

Quoted 35 lines · Julien Stephan
Quoted 34 lines · Julien Stephan

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the signatures using tinfoil

---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
index a355f56c60c..12202779ac0 100644
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +0 -0
index 11c9fa2c40d..687cdde5e6d 100644
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
Quoted 16 lines
Quoted 6 lines
bitbake/lib/bb/cooker.py +6 -0
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()

bitbake/lib/bb/cooker.py class BBCooker: · line 1560

Quoted 10 lines
bitbake/lib/bb/cooker.py +10 -0
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+

Firstly, thanks for working on this! These changes aren't simple and it means the review takes longer as we need to get this right.

Quoted 6 lines

I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks

Quoted 7 lines

Using buildTaskData with "do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"

At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,

I think you need to query the tasks available for a recipe, remove the "nostamp" ones and then return the signatures for the rest.

Quoted 8 lines

but running bitbake -c listtasks gives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasks

So what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?

I think bitbake should internally be able to work out the nostamp ones and return a list of all of them with the nostamp ones excluded.

Let me know if you need some pointers on how to do that in bitbake, it isn't entirely straightforward and I'd have to look it up. I *think* (without looking to know for sure) that the dataCache object does have the task data in it which can be looked up by target and that lists which ones are nostamp.

Cheers,

Richard

#5 lore raw

Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

Quoted 83 lines · Richard Purdie
Quoted 35 lines · Julien Stephan
Quoted 34 lines · Julien Stephan

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the signatures using tinfoil

---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
index a355f56c60c..12202779ac0 100644
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +0 -0
index 11c9fa2c40d..687cdde5e6d 100644
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
Quoted 16 lines
Quoted 6 lines
bitbake/lib/bb/cooker.py +6 -0
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()

bitbake/lib/bb/cooker.py class BBCooker: · line 1560

Quoted 10 lines
bitbake/lib/bb/cooker.py +10 -0
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+

Firstly, thanks for working on this! These changes aren't simple and it means the review takes longer as we need to get this right.

Quoted 6 lines

I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks

Quoted 7 lines

Using buildTaskData with "do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"

At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,

I think you need to query the tasks available for a recipe, remove the "nostamp" ones and then return the signatures for the rest.

Quoted 8 lines

but running bitbake -c listtasks gives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasks

So what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?

Quoted 8 lines

I think bitbake should internally be able to work out the nostamp ones and return a list of all of them with the nostamp ones excluded.

Let me know if you need some pointers on how to do that in bitbake, it isn't entirely straightforward and I'd have to look it up. I *think* (without looking to know for sure) that the dataCache object does have the task data in it which can be looked up by target and that lists which ones are nostamp.

Hi Richard,

Sorry for the late reply I was off last week.
Using rq.rqdata.dataCaches I am able to get the full list of tasks and filter out the no stamps.
But it is still unclear for me what to do :( Here is what I understood, please correct me if I am wrong:
* in getTaskSignatures (cooker.py): I should get the tasks list and filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ), then compute the signature for all
* in oeqa self test, I get the list of tasks using 'bitbake -c listtasks', but how to filter out the 'nostamps' ones?

Cheers
Julien

Quoted 6 lines

Cheers,

Richard

#6 lore raw

Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

Hi,

Sorry, I meant to reply to this and somehow it got lost with everything else going on.

Quoted 94 lines · Julien Stephan
Quoted 83 lines · Richard Purdie
Quoted 35 lines · Julien Stephan
Quoted 34 lines · Julien Stephan

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the signatures using tinfoil

---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)
bitbake/lib/bb/command.py +6 -0
index a355f56c60c..12202779ac0 100644
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
bitbake/lib/bb/cooker.py +0 -0
index 11c9fa2c40d..687cdde5e6d 100644
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
Quoted 16 lines
Quoted 6 lines
bitbake/lib/bb/cooker.py +6 -0
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()

bitbake/lib/bb/cooker.py class BBCooker: · line 1560

Quoted 10 lines
bitbake/lib/bb/cooker.py +10 -0
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+

Firstly, thanks for working on this! These changes aren't simple and it means the review takes longer as we need to get this right.

Quoted 6 lines

I would like some feedback on this function. The goal of this function was to be able to get a task signature (for bblock to use it, see bblock series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks

Quoted 7 lines

Using buildTaskData with "do_build" task gives me almost all tasks signatures, at least, the "normal recipe build tasks" according to the doc [1]. But we don't get the signatures for the "Manually called Tasks"

At first I thought it would be good enough for bblock use case, but now I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,

I think you need to query the tasks available for a recipe, remove the "nostamp" ones and then return the signatures for the rest.

Quoted 8 lines

but running bitbake -c listtasks gives *all* the tasks
* this function as is is not generic enough: we cannot get the task signature for all the "manually called tasks" (but is that really a problem?)
* bblock cannot lock those tasks

So what do you think? Should I make this function more generic and called it several times to get *all* the tasks signature?

Quoted 8 lines

I think bitbake should internally be able to work out the nostamp ones and return a list of all of them with the nostamp ones excluded.

Let me know if you need some pointers on how to do that in bitbake, it isn't entirely straightforward and I'd have to look it up. I *think* (without looking to know for sure) that the dataCache object does have the task data in it which can be looked up by target and that lists which ones are nostamp.

Quoted 7 lines

Sorry for the late reply I was off last week.
Using rq.rqdata.dataCaches I am able to get the full list of tasks and filter out the no stamps.
But it is still unclear for me what to do :( Here is what I understood, please correct me if I am wrong:
* in getTaskSignatures (cooker.py): I should get the tasks list and filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ),

You can skip the setscene ones as they're not interesting or needed for what we need here, at least as I understand it as we never lock those.

Quoted 3 lines

then compute the signature for all
* in oeqa self test, I get the list of tasks using 'bitbake -c listtasks', but how to filter out the 'nostamps' ones?

Just thinking out loud, for a given recipe you could run d.getVarFlag("do_clean", "nostamp") and see that it was set?

"bitbake-getvar -r bash do_clean -f nostamp --value" also shows that.

Cheers,

Richard

#7 lore raw

PATCH v4 3/5 sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK

In reply to Julien Stephan

as of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and "error", displaying respectively a warning or a fatal error message only when a task is locked and the task signature is different from the locked one.

The "info" level is introduced to add a "note" message to remind the user that a recipe is locked even if the signature is equivalent to the locked one.

The "warn" and "error" level display the warn/error message for each task having a mismatch of the signature. Doing this with the "info" level would result in very verbose output if there are several tasks locked, so the info level will only print once the list of recipes that have locked signature.

---
 meta/lib/oe/sstatesig.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +12 -1
index f943df181e6..90002f67550 100644
@@ -104,6 +104,7 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         self.lockedhashfn = {}
         self.machine = data.getVar("MACHINE")
         self.mismatch_msgs = []
+        self.lockedsigs_msgs = ""
         self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or
                                 "").split()
         self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
@@ -264,6 +265,12 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         warn_msgs = []
         error_msgs = []
         sstate_missing_msgs = []
+        info_msgs = None
+
+        if self.lockedsigs:
+            self.lockedsigs_msgs = "The following recipes have locked tasks:"
+            for pn in self.lockedsigs:
+                self.lockedsigs_msgs += " %s" % (pn)
 
         for tid in sq_data['hash']:
             if tid not in found:
@@ -276,7 +283,9 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                                                % (pn, taskname, sq_data['hash'][tid]))
 
         checklevel = d.getVar("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK")
-        if checklevel == 'warn':
+        if checklevel == 'info':
+            info_msgs = self.lockedsigs_msgs
+        if checklevel == 'warn' or checklevel == 'info':
             warn_msgs += self.mismatch_msgs
         elif checklevel == 'error':
             error_msgs += self.mismatch_msgs
@@ -287,6 +296,8 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         elif checklevel == 'error':
             error_msgs += sstate_missing_msgs
 
+        if info_msgs:
+            bb.note(info_msgs)
         if warn_msgs:
             bb.warn("\n".join(warn_msgs))
         if error_msgs:
Signature · 1 line
2.41.0
#8 lore raw

Re: [OE-core] [PATCH v4 3/5] sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK

Quoted 35 lines · Julien Stephan

as of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and "error", displaying respectively a warning or a fatal error message only when a task is locked and the task signature is different from the locked one.

The "info" level is introduced to add a "note" message to remind the user that a recipe is locked even if the signature is equivalent to the locked one.

The "warn" and "error" level display the warn/error message for each task having a mismatch of the signature. Doing this with the "info" level would result in very verbose output if there are several tasks locked, so the info level will only print once the list of recipes that have locked signature.

---
 meta/lib/oe/sstatesig.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -0
index f943df181e6..90002f67550 100644
@@ -104,6 +104,7 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         self.lockedhashfn = {}
         self.machine = data.getVar("MACHINE")
         self.mismatch_msgs = []
+        self.lockedsigs_msgs = ""
         self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or
                                 "").split()
         self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
@@ -264,6 +265,12 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         warn_msgs = []
         error_msgs = []

meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHashMixIn(object): · line 276

Quoted 10 lines
meta/lib/oe/sstatesig.py +6 -0
         sstate_missing_msgs = []
+        info_msgs = None
+
+        if self.lockedsigs:
+            self.lockedsigs_msgs = "The following recipes have locked tasks:"
+            for pn in self.lockedsigs:
+                self.lockedsigs_msgs += " %s" % (pn)
 
         for tid in sq_data['hash']:
             if tid not in found:

How about something like:

if len(pn) > 10:
print "There are XX locked recipes (XX matching and XX not matching
parsing)"

so that in the case where lots of things are locked you get a summary instead showing the amount of match/mismatch?

Cheers,

Richard

#9 lore raw

PATCH v4 4/5 scripts/bblock: add a script to lock/unlock recipes

In reply to Julien Stephan

bblock script allows to lock/unlock recipes to latest task signatures. The idea is to prevent some recipes to be rebuilt during development. For example when working on rust recipe, one may not want rust-native to be rebuilt.

This tool can be used, with proper environment set up, using the following command:

bblock <recipe_name>

if a <recipe_name>'s task signature change, this task will not be built again and sstate cache will be used.

[YOCTO #13425]

---
 scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100755 scripts/bblock
scripts/bblock new file +182 -0
new file mode 100755
index 00000000000..e5cf78722c7
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <[email protected]>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def getTaskSignatures(tinfoil, pn, tasks):
+    tinfoil.set_event_mask(
+        [
+            "bb.event.GetTaskSignatureResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("getTaskSignatures", pn, tasks)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.GetTaskSignatureResult):
+                    sig = event.sig
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from getTaskSignatures command")
+        sys.exit(2)
+    return sig
+
+
+def parseRecipe(tinfoil, recipe):
+    try:
+        tinfoil.parse_recipes()
+        d = tinfoil.parse_recipe(recipe)
+    except Exception:
+        logger.error("Failed to get recipe info for: %s" % recipe)
+        sys.exit(1)
+    return d
+
+
+def bblockDump(lockfile):
+    try:
+        with open(lockfile, "r") as lockfile:
+            for line in lockfile:
+                print(line.strip())
+    except IOError:
+        return 1
+    return 0
+
+
+def bblockReset(lockfile, pns, package_archs, tasks):
+    if not pns:
+        logger.info("Unlocking all recipes")
+        try:
+            os.remove(lockfile)
+        except FileNotFoundError:
+            pass
+    else:
+        logger.info("Unlocking {pns}".format(pns=pns))
+        tmp_lockfile = lockfile + ".tmp"
+        with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+            for line in infile:
+                if not (
+                    any(element in line for element in pns)
+                    and any(element in line for element in package_archs.split())
+                ):
+                    outfile.write(line)
+                else:
+                    if tasks and not any(element in line for element in tasks):
+                        outfile.write(line)
+        os.remove(lockfile)
+        os.rename(tmp_lockfile, lockfile)
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+    parser.add_argument(
+        "-t",
+        "--tasks",
+        help="Comma separated list of tasks",
+        type=lambda s: [task for task in s.split(",")],
+    )
+    parser.add_argument(
+        "-r",
+        "--reset",
+        action="store_true",
+        help="Unlock pn recipes, or all recipes if pn is empty",
+    )
+    parser.add_argument(
+        "-d",
+        "--dump",
+        action="store_true",
+        help="Dump generated bblock.conf file",
+    )
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+
+        package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+        if global_args.dump:
+            bblockDump(lockfile)
+            return 0
+
+        if global_args.reset:
+            bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks)
+            return 0
+
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+                s += "\n"
+
+            for pn in global_args.pn:
+                d = parseRecipe(tinfoil, pn)
+                package_arch = d.getVar("PACKAGE_ARCH")
+                siggen_locked_sigs_package_arch = d.getVar(
+                    "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch)
+                )
+                sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks)
+                for sig in sigs:
+                    new_entry = "{pn}:{taskname}:{sig}".format(
+                        pn=sig[0], taskname=sig[1], sig=sig[2]
+                    )
+                    if (
+                        siggen_locked_sigs_package_arch
+                        and not new_entry in siggen_locked_sigs_package_arch
+                    ) or not siggen_locked_sigs_package_arch:
+                        s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format(
+                            package_arch=package_arch, new_entry=new_entry
+                        )
+            lockfile.write(s)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
Signature · 1 line
2.41.0
#10 lore raw

PATCH v4 5/5 oeqa/selftest/bblock: add self test for bblock tool

In reply to Julien Stephan

it implements various combination of locking single/multiple recipe(s)/task(s)

it also tests that locked sig are architecture dependant

---
 meta/lib/oeqa/selftest/cases/bblock.py | 146 +++++++++++++++++++++++++
 1 file changed, 146 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
meta/lib/oeqa/selftest/cases/bblock.py new file +146 -0
new file mode 100644
index 00000000000..803ea3d64c8
@@ -0,0 +1,146 @@
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <[email protected]>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import re
+
+import oeqa.utils.ftools as ftools
+from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake
+
+from oeqa.selftest.case import OESelftestTestCase
+
+class BBLock(OESelftestTestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        super(BBLock, cls).setUpClass()
+        cls.lockfile = cls.builddir+"/conf/bblock.conf"
+
+    def unlock_recipes(self, recipes=None, tasks=None):
+        cmd = 'bblock -r '
+        if recipes:
+            cmd += ' '.join(recipes)
+        if tasks:
+            cmd += ' -t ' + ','.join(tasks)
+        result = runCmd(cmd)
+
+        if recipes:
+            # ensure all signatures are removed from lockfile
+            contents = ftools.read_file(self.lockfile)
+            for recipe in recipes:
+                for task in tasks:
+                    find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents)
+                    self.assertFalse(find_in_contents, msg = "%s:%s should not be present into bblock.conf anymore" % (recipe, task))
+                self.assertExists(self.lockfile)
+        else:
+            self.assertNotExists(self.lockfile)
+
+    def lock_recipes(self, recipes, tasks=None, machine=None):
+        cmd = 'bblock ' + ' '.join(recipes)
+        if tasks:
+            cmd += ' -t ' + ','.join(tasks)
+
+        if machine:
+            cmd = "MACHINE=" + machine + " " + cmd
+        with open("/tmp/tmp.txt", "w") as file:
+            file.write(cmd)
+        result = runCmd(cmd)
+
+        self.assertExists(self.lockfile)
+
+        # ensure all signatures are added to lockfile
+        contents = ftools.read_file(self.lockfile)
+        for recipe in recipes:
+            for task in tasks:
+                find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents)
+                self.assertTrue(find_in_contents, msg = "%s:%s was not added into bblock.conf. bblock output: %s" % (recipe, task, result.output))
+
+    def modify_tasks(self, recipes, tasks):
+        task_append = ""
+        for recipe in recipes:
+            bb_vars = get_bb_vars(['PV'], recipe)
+            recipe_pv = bb_vars['PV']
+            recipe_append_file = recipe + '_' + recipe_pv + '.bbappend'
+
+            os.mkdir(os.path.join(self.testlayer_path, 'recipes-test', recipe))
+            recipe_append_path = os.path.join(self.testlayer_path, 'recipes-test', recipe, recipe_append_file)
+
+            for task in tasks:
+                task_append += "%s:append() {\necho \"modify task hash\"\n}\n" % task
+            ftools.write_file(recipe_append_path, task_append)
+            self.add_command_to_tearDown('rm -rf %s' % os.path.join(self.testlayer_path, 'recipes-test', recipe))
+
+    def test_lock_single_recipe_single_task(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_single_recipe_multiple_tasks(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile" ,"do_install"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_architecture_specific(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+
+        # lock quilt's do_compile task for qemuarm machine
+        self.lock_recipes(recipes, tasks, "qemuarm")
+
+        # modify quilt's do_compile task
+        self.modify_tasks(recipes, tasks)
+
+        # build quilt using by default qemux86-64.
+        # No Note/Warning should be emitted since sig is locked for qemuarm
+        # (quilt package is architecture dependant)
+        info_message = "NOTE: The following recipes have locked tasks: " + recipes[0]
+        warn_message = "The %s:%s sig is computed to be" % (recipes[0], tasks[0])
+        result = bitbake(recipes[0] + " -n")
+        self.assertNotIn(info_message, result.output)
+        self.assertNotIn(warn_message, result.output)
+
+    def test_lock_multiple_recipe_single_task(self):
+        recipes = ["quilt", "bc"]
+        tasks= ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def _run_test(self, recipes, tasks=None):
+        # unlock all recipes and ensure no bblock.conf file exist
+        self.unlock_recipes()
+
+        # lock tasks for recipes
+        result = self.lock_recipes(recipes, tasks)
+
+        # build recipes. At this stage we should have a Note about recipes
+        # having locked task's sig, but no warning since sig still match
+        info_message = "NOTE: The following recipes have locked tasks: " + " ".join(recipes)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
+
+        ## modify all tasks that are locked to trigger a sig change then build the recipes
+        ## at this stage we should have a Note as before, but also a warning for all
+        ## locked stage indicating the sig mismatch
+        self.modify_tasks(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertIn(warn_message, result.output)
+
+        # unlock all tasks and rebuild, no more Note/Warning should remain
+        self.unlock_recipes(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertNotIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
Signature · 1 line
2.41.0
#11 lore raw
In reply to Julien Stephan

When ' include conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default?

Regards,
Qi

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Julien Stephan
Sent: Wednesday, August 2, 2023 10:24 PM
To: [email protected]
Cc: Julien Stephan <[email protected]>
Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

#12 lore raw
Quoted 6 lines · Qi

When ' include conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default?

Regards,
Qi

Hi Qi,

This was a request from Richard on the first submission [1] :

Quoted 4 lines

To simplify things, I think it would be reasonable to add a bblock.inc file unconditionally in our core configuration so that it is always included if present, just to make the tool easier since I think this will be a common use.

Cheers
Julien

[1): https://lists.openembedded.org/g/openembedded-core/message/184586

Quoted 85 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Julien Stephan
Sent: Wednesday, August 2, 2023 10:24 PM
To: [email protected]
Cc: Julien Stephan <[email protected]>
Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

#13 lore raw
In reply to Julien Stephan

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)

AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...

Quoted 90 lines · Julien Stephan

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#14 lore raw
Quoted 20 lines · Alexandre Belloni

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)
Quoted 2 lines

AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...

Hum, wait, I got this without the patch series. Let me investigate.

Quoted 99 lines · Julien Stephan
Quoted 90 lines

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#15 lore raw
Quoted 24 lines · Alexandre Belloni
Quoted 20 lines · Alexandre Belloni

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)
Quoted 2 lines

AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...

Quoted 1 line

Hum, wait, I got this without the patch series. Let me investigate.

Actually, I still had the series :)

Quoted 105 lines
Quoted 100 lines · Julien Stephan
Quoted 90 lines

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the following advantages:
* findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given recipe
* Check if a given task is already locked. If so, don't duplicate entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#16 lore raw

Hi Alexandre,
sorry for the late reply.

I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.

Please let me know if I can do anything else
Cheers
Julien

Quoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines · Alexandre Belloni

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)

AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...

Quoted 4 lines
Quoted 1 line

Hum, wait, I got this without the patch series. Let me investigate.

Actually, I still had the series :)

Hi Alexandre,

I am not able to reproduce it locally, so it is difficult to fix the issue.

Quoted 3 lines

Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#17 lore raw

Hello,

Quoted 11 lines · Julien Stephan

Hi Alexandre,
sorry for the late reply.

I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.

This was sent to the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio

Quoted 47 lines

Please let me know if I can do anything else
Cheers
Julien

Le ven. 11 ao�t 2023 � 10:45, Alexandre Belloni <[email protected]> a �crit :

Quoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)

AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...

Quoted 4 lines
Quoted 1 line

Hum, wait, I got this without the patch series. Let me investigate.

Actually, I still had the series :)

Hi Alexandre,

I am not able to reproduce it locally, so it is difficult to fix the issue.

Quoted 3 lines

Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#18 lore raw
Quoted 16 lines · Alexandre Belloni

Hello,

Quoted 11 lines · Julien Stephan

Hi Alexandre,
sorry for the late reply.

I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.

Quoted 4 lines

This was sent to the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio

Hi Alexandre,

Thank you! Unfortunately I can see that it is still failing on the same test :( I may need your help here.

Here is what my test is supposed to do:
From the autobuilder log I can see that the machine is set to qemux86-64 so :

$ bblock -d # unlock all recipes and remove build/conf/bblock.conf file
$ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for
qemuarm arch so block.conf file should look like this:

# Generated by bblock
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"

SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>"

# modify the quilt's do_compile task by adding an empty
do_compile:append function
$ bitbake -n quilt # for qemux86-64 machine

Because quilt is supposed to be locked for qemuarm, I am not supposed to see any warning, but the log on the autobuilder says:
"WARNING: The quilt:do_compile sig is computed to be d5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but the sig is locked to c9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fe in SIGGEN_LOCKEDSIGS_core2-64\""

meaning bblock.conf contains:

SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig>

So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??
* bblock.conf gets polluted by another test that also tries to lock quilt's do_compile function for qemux86-64 because of parallelism (I was not able to reproduce it locally)

Do you have an idea why this test is failing?
My next move would be to place all the tests in a single test to remove the parallelism issue (if that is the issue) but I would prefer to avoid wasting your time starting builds for me several times..

Also, I didn't manage to print log messages on my test, how can I do that?

Cheers
Julien

Quoted 52 lines
Quoted 33 lines

Please let me know if I can do anything else
Cheers
Julien

Quoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines · Alexandre Belloni

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)

AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...

Quoted 14 lines
Quoted 4 lines
Quoted 1 line

Hum, wait, I got this without the patch series. Let me investigate.

Actually, I still had the series :)

Hi Alexandre,

I am not able to reproduce it locally, so it is difficult to fix the issue.

Quoted 3 lines

Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#19 lore raw
Quoted 53 lines · Julien Stephan

Le jeu. 21 sept. 2023 � 15:11, Alexandre Belloni <[email protected]> a �crit :

Quoted 16 lines

Hello,

Quoted 11 lines · Julien Stephan

Hi Alexandre,
sorry for the late reply.

I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there.

Quoted 4 lines

This was sent to the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio

Hi Alexandre,

Thank you! Unfortunately I can see that it is still failing on the same test :( I may need your help here.

Here is what my test is supposed to do:
From the autobuilder log I can see that the machine is set to qemux86-64 so :

$ bblock -d # unlock all recipes and remove build/conf/bblock.conf file
$ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for
qemuarm arch so block.conf file should look like this:

# Generated by bblock
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"

SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>"

# modify the quilt's do_compile task by adding an empty
do_compile:append function
$ bitbake -n quilt # for qemux86-64 machine

Because quilt is supposed to be locked for qemuarm, I am not supposed to see any warning, but the log on the autobuilder says:
"WARNING: The quilt:do_compile sig is computed to be d5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but the sig is locked to c9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fe in SIGGEN_LOCKEDSIGS_core2-64\""

Quoted 8 lines

meaning bblock.conf contains:

SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig>

So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??

As I said in the call, look at this log from the build:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio

It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is then used by the build so overriding it from the command line doesn't work.

Quoted 65 lines

* bblock.conf gets polluted by another test that also tries to lock quilt's do_compile function for qemux86-64 because of parallelism (I was not able to reproduce it locally)

Do you have an idea why this test is failing?
My next move would be to place all the tests in a single test to remove the parallelism issue (if that is the issue) but I would prefer to avoid wasting your time starting builds for me several times..

Also, I didn't manage to print log messages on my test, how can I do that?

Cheers
Julien

Quoted 52 lines
Quoted 33 lines

Please let me know if I can do anything else
Cheers
Julien

Le ven. 11 ao�t 2023 � 10:45, Alexandre Belloni <[email protected]> a �crit :

Quoted 26 lines · Alexandre Belloni
Quoted 24 lines · Alexandre Belloni
Quoted 22 lines

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)

AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...

Quoted 14 lines
Quoted 4 lines
Quoted 1 line

Hum, wait, I got this without the patch series. Let me investigate.

Actually, I still had the series :)

Hi Alexandre,

I am not able to reproduce it locally, so it is difficult to fix the issue.

Quoted 3 lines

Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#20 lore raw
Quoted 5 lines
Quoted 4 lines

So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??

Quoted 8 lines

As I said in the call, look at this log from the build:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio

It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is then used by the build so overriding it from the command line doesn't work.

Hi Alexandre,

Thank you, I managed to reproduce it locally and fixed it properly. I just force pushed my poky-contrib branch
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can you start a build? If successful, I'll send the v5 today.

Cheers
Julien

#21 lore raw

This is started:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5792

Quoted 23 lines · Julien Stephan
Quoted 5 lines
Quoted 4 lines

So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why??

Quoted 8 lines

As I said in the call, look at this log from the build:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio

It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is then used by the build so overriding it from the command line doesn't work.

Hi Alexandre,

Thank you, I managed to reproduce it locally and fixed it properly. I just force pushed my poky-contrib branch
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can you start a build? If successful, I'll send the v5 today.

Cheers
Julien

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com