Skip to contentSkip to conversation

webhob related changes in bitbake, version 3

PATCH In review Alex DAMIAN 2 participants 17 messages Sep 16, 2013

lore
Share & tools
lore
#0 lore raw

v3 cover webhob related changes in bitbake, version 3

From: Alexandru DAMIAN <[email protected]>

I'm sending in version 3 of the patches that bring in changes that prepare bitbake for supporting WebHob-related tasks.

The changes focus on improving and standardizing the information sent through the event system, and cleaning up code in bitbake.

The following changes since commit 5fca4d286eec240ad57d90defa43cd0ff24f88ed:

linux-yocto/3.10: genericx86 meta data factoring (2013-09-14 08:36:26 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib adamian/bitbake-core-changes-1609-2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/bitbake-core-changes-1602-2

Alexandru DAMIAN (7):
bitbake: cooker: clean up code and avoid duplication
bitbake: event: send the task dependency tree to UI
bitbake: runqueue: add runQueueTaskSkipped event
bitbake: cooker: add extra recipe information
bitbake: cooker, command: add a command to return global data
bitbake: event: adding events to allow bbclasses to push custom info
bitbake: runqueue: add task hash to Queue events

 bitbake/lib/bb/build.py     |  3 ++
 bitbake/lib/bb/command.py   |  8 +++++
 bitbake/lib/bb/cooker.py    | 79 +++++++++++++++++++++++++++++----------------
 bitbake/lib/bb/event.py     | 16 +++++++++
 bitbake/lib/bb/runqueue.py  | 35 ++++++++++++++++++--
 bitbake/lib/bb/ui/knotty.py |  5 +++
 6 files changed, 115 insertions(+), 31 deletions(-)
Signature · 1 line
1.8.1.2
#2 lore raw

PATCH v3 2/7 bitbake: event: send the task dependency tree to UI

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We add an event that will send the dependency tree graph to the UI client once it is computed.

This will allow the clients to display dependency data in an efficient manner, and not redo the runqueue computation specifically to get the dependency data.

---
 bitbake/lib/bb/cooker.py    | 3 +++
 bitbake/lib/bb/runqueue.py  | 3 +++
 bitbake/lib/bb/ui/knotty.py | 3 +++
 3 files changed, 9 insertions(+)
bitbake/lib/bb/cooker.py +3 -0
index 3bfeefb..68a3c01 100644
@@ -415,7 +415,10 @@ class BBCooker:
         runlist, taskdata = self.prepareTreeData(pkgs_to_build, task)
         rq = bb.runqueue.RunQueue(self, self.data, self.recipecache, taskdata, runlist)
         rq.rqdata.prepare()
+        return self.buildDependTree(rq, taskdata)
 
+
+    def buildDependTree(self, rq, taskdata):
         seen_fnids = []
         depend_tree = {}
         depend_tree["depends"] = {}
bitbake/lib/bb/runqueue.py +3 -0
index 25f1ab5..aa2f147 100644
@@ -999,6 +999,9 @@ class RunQueue:
             else:
                 self.state = runQueueSceneInit
 
+                depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
+                bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
+
         if self.state is runQueueSceneInit:
             if self.cooker.configuration.dump_signatures:
                 self.dump_signatures()
bitbake/lib/bb/ui/knotty.py +3 -0
index 35590a2..0211b50 100644
@@ -472,6 +472,9 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                              event.taskid, event.taskstring, event.exitcode)
                 continue
 
+            if isinstance(event, bb.event.DepTreeGenerated):
+                continue
+
             # ignore
             if isinstance(event, (bb.event.BuildBase,
                                   bb.event.StampUpdate,
Signature · 1 line
1.8.1.2
#4 lore raw

PATCH v3 3/7 bitbake: runqueue: add runQueueTaskSkipped event

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding a runQueueTaskSkipped to notify all listeners about the tasks that are not run either because they are set-scened or they don't need an update (timestamps are ok)

Adds RunQueueData functions to get the task name and task file for usage with the runQueue* events.

---
 bitbake/lib/bb/build.py    |  3 +++
 bitbake/lib/bb/runqueue.py | 27 ++++++++++++++++++++++++---
 2 files changed, 27 insertions(+), 3 deletions(-)
bitbake/lib/bb/build.py +3 -0
index a53aba9..1265bdc 100644
@@ -72,6 +72,9 @@ class TaskBase(event.Event):
     def __init__(self, t, d ):
         self._task = t
         self._package = d.getVar("PF", True)
+        self._file = d.getVar("FILE", True)
+        self.taskfile = self._file
+        self.taskname = self._task
         event.Event.__init__(self)
         self._message = "recipe %s: task %s: %s" % (d.getVar("PF", True), t, self.getDisplayName())
 
bitbake/lib/bb/runqueue.py +24 -3
index aa2f147..2ab4405 100644
@@ -217,6 +217,12 @@ class RunQueueData:
             ret.extend([nam])
         return ret
 
+    def get_task_name(self, task):
+        return self.runq_task[task]
+
+    def get_task_file(self, task):
+        return self.taskData.fn_index[self.runq_fnid[task]]
+
     def get_user_idstring(self, task, task_name_suffix = ""):
         fn = self.taskData.fn_index[self.runq_fnid[task]]
         taskname = self.runq_task[task] + task_name_suffix
@@ -1328,9 +1334,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
         if self.rqdata.taskData.abort:
             self.rq.state = runQueueCleanUp
 
-    def task_skip(self, task):
+    def task_skip(self, task, skip_reason = ""):
         self.runq_running[task] = 1
         self.runq_buildable[task] = 1
+        bb.event.fire(runQueueTaskSkipped(task, self.stats, self.rq, skip_reason), self.cfgData)
         self.task_completeoutright(task)
         self.stats.taskCompleted()
         self.stats.taskSkipped()
@@ -1355,13 +1362,13 @@ class RunQueueExecuteTasks(RunQueueExecute):
             if task in self.rq.scenequeue_covered:
                 logger.debug(2, "Setscene covered task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "covered")
                 return True
 
             if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
                 logger.debug(2, "Stamp current task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "existing")
                 return True
 
             taskdep = self.rqdata.dataCache.task_deps[fn]
@@ -1783,6 +1790,8 @@ class runQueueEvent(bb.event.Event):
     def __init__(self, task, stats, rq):
         self.taskid = task
         self.taskstring = rq.rqdata.get_user_idstring(task)
+        self.taskname = rq.rqdata.get_task_name(task)
+        self.taskfile = rq.rqdata.get_task_file(task)
         self.stats = stats.copy()
         bb.event.Event.__init__(self)
 
@@ -1794,6 +1803,8 @@ class sceneQueueEvent(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         realtask = rq.rqdata.runq_setscene[task]
         self.taskstring = rq.rqdata.get_user_idstring(realtask, "_setscene")
+        self.taskname = rq.rqdata.get_task_name(realtask) + "_setscene"
+        self.taskfile = rq.rqdata.get_task_file(realtask)
 
 class runQueueTaskStarted(runQueueEvent):
     """
@@ -1837,6 +1848,16 @@ class sceneQueueTaskCompleted(sceneQueueEvent):
     Event notifing a setscene task completed
     """
 
+class runQueueTaskSkipped(runQueueEvent):
+    """
+    Event notifing a task was skipped
+    """
+    def __init__(self, task, stats, rq, skip_reason):
+        runQueueEvent.__init__(self, task, stats, rq)
+        self._file = rq.rqdata.get_task_file(task)
+        self._task = rq.rqdata.get_task_name(task)
+        self._skip = skip_reason
+
 class runQueuePipe():
     """
     Abstraction for a pipe between a worker thread and the server
Signature · 1 line
1.8.1.2
#8 lore raw

PATCH v3 4/7 bitbake: cooker: add extra recipe information

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding in the extra recipe information when creating the dependency tree information.

This works in server-mode because the HOB extra_cache is already enabled. In normal mode, it will do nothing with no performance impact.

---
 bitbake/lib/bb/cooker.py | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
bitbake/lib/bb/cooker.py +23 -0
index 68a3c01..cb0e3e5 100644
@@ -439,6 +439,29 @@ class BBCooker:
                 depend_tree["pn"][pn] = {}
                 depend_tree["pn"][pn]["filename"] = fn
                 depend_tree["pn"][pn]["version"] = version
+
+                # This data is needed for webhob;
+                # it's here if the bitbake runs in server mode
+                try:
+                    summary = self.recipecache.summary[fn]
+                    lic = self.recipecache.license[fn]
+                    section = self.recipecache.section[fn]
+                    description = self.recipecache.description[fn]
+                    homepage = self.recipecache.homepage[fn]
+                    bugtracker = self.recipecache.bugtracker[fn]
+                    inherits = self.recipecache.inherits.get(fn, None)
+                    depend_tree["pn"][pn]["filename"] = fn
+                    depend_tree["pn"][pn]["version"] = version
+                    depend_tree["pn"][pn]["summary"] = summary
+                    depend_tree["pn"][pn]["license"] = lic
+                    depend_tree["pn"][pn]["section"] = section
+                    depend_tree["pn"][pn]["description"] = description
+                    depend_tree["pn"][pn]["inherits"] = inherits
+                    depend_tree["pn"][pn]["homepage"] = homepage
+                    depend_tree["pn"][pn]["bugtracker"] = bugtracker
+                except:
+                    pass
+
             for dep in rq.rqdata.runq_depends[task]:
                 depfn = taskdata.fn_index[rq.rqdata.runq_fnid[dep]]
                 deppn = self.recipecache.pkg_fn[depfn]
Signature · 1 line
1.8.1.2
#16 lore raw

PATCH v3 7/7 bitbake: runqueue: add task hash to Queue events

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding the sstate-related hash for all runqueue and scenequeue tasks, as it's needed in the WebHob data.

---
 bitbake/lib/bb/runqueue.py | 5 +++++
 1 file changed, 5 insertions(+)
bitbake/lib/bb/runqueue.py +5 -0
index 2ab4405..18ebce0 100644
@@ -223,6 +223,9 @@ class RunQueueData:
     def get_task_file(self, task):
         return self.taskData.fn_index[self.runq_fnid[task]]
 
+    def get_task_hash(self, task):
+        return self.runq_hash[task]
+
     def get_user_idstring(self, task, task_name_suffix = ""):
         fn = self.taskData.fn_index[self.runq_fnid[task]]
         taskname = self.runq_task[task] + task_name_suffix
@@ -1792,6 +1795,7 @@ class runQueueEvent(bb.event.Event):
         self.taskstring = rq.rqdata.get_user_idstring(task)
         self.taskname = rq.rqdata.get_task_name(task)
         self.taskfile = rq.rqdata.get_task_file(task)
+        self.taskhash = rq.rqdata.get_task_hash(task)
         self.stats = stats.copy()
         bb.event.Event.__init__(self)
 
@@ -1805,6 +1809,7 @@ class sceneQueueEvent(runQueueEvent):
         self.taskstring = rq.rqdata.get_user_idstring(realtask, "_setscene")
         self.taskname = rq.rqdata.get_task_name(realtask) + "_setscene"
         self.taskfile = rq.rqdata.get_task_file(realtask)
+        self.taskhash = rq.rqdata.get_task_hash(task)
 
 class runQueueTaskStarted(runQueueEvent):
     """
Signature · 1 line
1.8.1.2
#10 lore raw

PATCH v3 5/7 bitbake: cooker, command: add a command to return global data

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding the 'getAllKeysWithFlags' read-only command that will return a dump of the global data state, together with specified flags for each key. The flag list is passed in as the first parameter to the command.

This will be used by UI clients to get the build configuration.

---
 bitbake/lib/bb/command.py |  8 ++++++++
 bitbake/lib/bb/cooker.py  | 15 +++++++++++++++
 2 files changed, 23 insertions(+)
bitbake/lib/bb/command.py +8 -0
index f1abaf7..5d359e7 100644
@@ -145,6 +145,14 @@ class CommandsSync:
         """
         command.cooker.shutdown(True)
 
+    def getAllKeysWithFlags(self, command, params):
+        """
+        Returns a dump of the global state. Call with
+        variable flags to be retrieved as params.
+        """
+        return command.cooker.getAllKeysWithFlags(params[0])
+    getAllKeysWithFlags.readonly = True
+
     def getVariable(self, command, params):
         """
         Read the value of a variable from data
bitbake/lib/bb/cooker.py +15 -0
index cb0e3e5..981379b 100644
@@ -1134,6 +1134,21 @@ class BBCooker:
 
         self.configuration.server_register_idlecallback(buildTargetsIdle, rq)
 
+
+    def getAllKeysWithFlags(self, flaglist):
+        dump = {}
+        for k in self.data.keys():
+            try:
+                v = self.data.getVar(k, True)
+                if not k.startswith("__") and not bool(self.data.getVarFlag(k, 'func')) and not isinstance(v, bb.data_smart.DataSmart):
+                    dump[k] = { 'v' : v }
+                    for d in flaglist:
+                        dump[k][d] = self.data.getVarFlag(k, d)
+            except:
+                pass
+        return dump
+
+
     def generateNewImage(self, image, base_image, package_queue, timestamp, description):
         '''
         Create a new image with a "require"/"inherit" base_image statement
Signature · 1 line
1.8.1.2
#12 lore raw

PATCH v3 6/7 bitbake: event: adding events to allow bbclasses to push custom info

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We add a new event that will be triggered by the package-building class, containing the information about the currently built package.

Adding an event that will passi information about the content of each package file that gets built.

Knotty UI will ignore these events by default.

Maybe a mechanism for using a single generic event is in order ?

---
 bitbake/lib/bb/event.py     | 16 ++++++++++++++++
 bitbake/lib/bb/ui/knotty.py |  2 ++
 2 files changed, 18 insertions(+)
bitbake/lib/bb/event.py +16 -0
index 67cfcea..7dd9236 100644
@@ -589,6 +589,22 @@ class PackageInfo(Event):
         Event.__init__(self)
         self._pkginfolist = pkginfolist
 
+class SinglePackageInfo(Event):
+    """
+    Single Package Information sent on emit_pkgdata
+    """
+    def __init__(self, data):
+        Event.__init__(self)
+        self._data = data
+
+class PackageFileSizes(Event):
+    """
+    Event that contains information about the file sizes contained in a package.
+    """
+    def __init__(self, data):
+        Event.__init__(self)
+        self._data = data
+
 class SanityCheck(Event):
     """
     Event to issue sanity check
bitbake/lib/bb/ui/knotty.py +2 -0
index 0211b50..03baa75 100644
@@ -477,6 +477,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
 
             # ignore
             if isinstance(event, (bb.event.BuildBase,
+                                  bb.event.PackageFileSizes,
+                                  bb.event.SinglePackageInfo,
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
                                   bb.event.RecipeParsed,
Signature · 1 line
1.8.1.2
#1 lore raw

PATCH v3 1/7 bitbake: cooker: clean up code and avoid duplication

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Clean-up to avoid duplication and promote code reuse. We're using the prepareTreeData in all instances where this needs to be done.

bb.event.BuildStarted now contains unexpanded package names, since the UI will show the original target names.

When we're only testing target image contents, we do not abort with an error if the target is unbuildable.

---
 bitbake/lib/bb/cooker.py | 38 ++++++++++----------------------------
 1 file changed, 10 insertions(+), 28 deletions(-)
bitbake/lib/bb/cooker.py +10 -28
index a07615b..3bfeefb 100644
@@ -345,13 +345,7 @@ class BBCooker:
             if pkgs_to_build[0] in set(ignore.split()):
                 bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0])
 
-            localdata = data.createCopy(self.data)
-            bb.data.update_data(localdata)
-            bb.data.expandKeys(localdata)
-
-            taskdata = bb.taskdata.TaskData(self.configuration.abort)
-            taskdata.add_provider(localdata, self.recipecache, pkgs_to_build[0])
-            taskdata.add_unresolved(localdata, self.recipecache)
+            runlist, taskdata = self.prepareTreeData(pkgs_to_build, None)
 
             targetid = taskdata.getbuild_id(pkgs_to_build[0])
             fnid = taskdata.build_targets[targetid][0]
@@ -398,9 +392,7 @@ class BBCooker:
         localdata = data.createCopy(self.data)
         bb.data.update_data(localdata)
         bb.data.expandKeys(localdata)
-        # We set abort to False here to prevent unbuildable targets raising
-        # an exception when we're just generating data
-        taskdata = bb.taskdata.TaskData(False, skiplist=self.skiplist)
+        taskdata = bb.taskdata.TaskData(self.configuration.abort, skiplist=self.skiplist)
 
         runlist = []
         current = 0
@@ -492,7 +484,12 @@ class BBCooker:
         """
         Create a dependency tree of pkgs_to_build, returning the data.
         """
+
+        # We set abort to False here to prevent unbuildable targets raising
+        # an exception when we're just generating data
+        self.configuration.abort = False
         _, taskdata = self.prepareTreeData(pkgs_to_build, task)
+
         tasks_fnid = []
         if len(taskdata.tasks_name) != 0:
             for task in xrange(len(taskdata.tasks_name)):
@@ -1073,13 +1070,6 @@ class BBCooker:
         Attempt to build the targets specified
         """
 
-        # If we are told to do the NULL task then query the default task
-        if (task == None):
-            task = self.configuration.cmd
-
-        universe = ('universe' in targets)
-        targets = self.checkPackages(targets)
-
         def buildTargetsIdle(server, rq, abort):
             if abort or self.state == state.forceshutdown:
                 rq.finish_runqueue(True)
@@ -1106,21 +1096,13 @@ class BBCooker:
         self.buildSetVars()
 
         buildname = self.data.getVar("BUILDNAME")
-        bb.event.fire(bb.event.BuildStarted(buildname, targets), self.data)
-
-        localdata = data.createCopy(self.data)
-        bb.data.update_data(localdata)
-        bb.data.expandKeys(localdata)
 
-        taskdata = bb.taskdata.TaskData(self.configuration.abort, skiplist=self.skiplist)
+        bb.event.fire(bb.event.BuildStarted(buildname, targets), self.data)
 
-        runlist = []
-        for k in targets:
-            taskdata.add_provider(localdata, self.recipecache, k)
-            runlist.append([k, "do_%s" % task])
-        taskdata.add_unresolved(localdata, self.recipecache)
+        runlist, taskdata = self.prepareTreeData(targets, task)
 
         rq = bb.runqueue.RunQueue(self, self.data, self.recipecache, taskdata, runlist)
+        universe = ('universe' in targets)
         if universe:
             rq.rqdata.warn_multi_bb = True
 
Signature · 1 line
1.8.1.2
#13 lore raw

Re: [PATCH 6/7] bitbake: event: adding events to allow bbclasses to push custom info

In reply to Alex DAMIAN
Quoted 35 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We add a new event that will be triggered by the package-building class, containing the information about the currently built package.

Adding an event that will passi information about the content of each package file that gets built.

Knotty UI will ignore these events by default.

Maybe a mechanism for using a single generic event is in order ?

---
 bitbake/lib/bb/event.py     | 16 ++++++++++++++++
 bitbake/lib/bb/ui/knotty.py |  2 ++
 2 files changed, 18 insertions(+)
bitbake/lib/bb/event.py +8 -0
index 67cfcea..7dd9236 100644
@@ -589,6 +589,22 @@ class PackageInfo(Event):
         Event.__init__(self)
         self._pkginfolist = pkginfolist
 
+class SinglePackageInfo(Event):
+    """
+    Single Package Information sent on emit_pkgdata
+    """
+    def __init__(self, data):
+        Event.__init__(self)
+        self._data = data
+

bitbake/lib/bb/event.py class PackageInfo(Event): · line 609

Quoted 10 lines
bitbake/lib/bb/event.py +8 -0
+class PackageFileSizes(Event):
+    """
+    Event that contains information about the file sizes contained in a package.
+    """
+    def __init__(self, data):
+        Event.__init__(self)
+        self._data = data
+
 class SanityCheck(Event):
     """

Your explanation in the commit message isn't as detailed as it could be.

I'm guessing but I suspect the problem we have is that the event cannot be rebuilt on the other side of an XMLRPC connection without having the event class in a common namespace?

This is therefore a hack to allow the event to exist in all the client side namespaces.

Ideally we should really have an API which allows the classes themselves to declare their event formats and not require adding to event.py every time a new piece of code/event is added.

Presumably there is some change required to OE-Core in order for this get used to?

Is the PackageFileSizes not contained in the other event data?

I appreciate we allowed PackageInfo in but I didn't like it at the time, I like it even less now I see we're just going to keep doing this :(.

Quoted 6 lines
bitbake/lib/bb/event.py +0 -0
     Event to issue sanity check
bitbake/lib/bb/ui/knotty.py +0 -0
index 0211b50..03baa75 100644

bitbake/lib/bb/ui/knotty.py def main(server, eventHandler, params, tf = TerminalFilter): · line 484

Quoted 8 lines
bitbake/lib/bb/ui/knotty.py +2 -0
@@ -477,6 +477,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
 
             # ignore
             if isinstance(event, (bb.event.BuildBase,
+                                  bb.event.PackageFileSizes,
+                                  bb.event.SinglePackageInfo,
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
                                   bb.event.RecipeParsed,

Given the event masking that is now in knotty, do we need to do this? Can't we remove this block?

Cheers,

Richard

#11 lore raw

Re: [PATCH 5/7] bitbake: cooker, command: add a command to return global data

In reply to Alex DAMIAN
Quoted 20 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding the 'getAllKeysWithFlags' read-only command that will return a dump of the global data state, together with specified flags for each key. The flag list is passed in as the first parameter to the command.

This will be used by UI clients to get the build configuration.

---
 bitbake/lib/bb/command.py |  8 ++++++++
 bitbake/lib/bb/cooker.py  | 15 +++++++++++++++
 2 files changed, 23 insertions(+)
bitbake/lib/bb/command.py +0 -0
index f1abaf7..5d359e7 100644

bitbake/lib/bb/command.py class CommandsSync: · line 154

Quoted 10 lines
bitbake/lib/bb/command.py +7 -0
@@ -145,6 +145,14 @@ class CommandsSync:
         """
         command.cooker.shutdown(True)
 
+    def getAllKeysWithFlags(self, command, params):
+        """
+        Returns a dump of the global state. Call with
+        variable flags to be retrieved as params.
+        """
+        return command.cooker.getAllKeysWithFlags(params[0])
+    getAllKeysWithFlags.readonly = True

Please match the style of the other commands, i.e.:

+ flaglist = params[0]
+
+ return command.cooker.getAllKeysWithFlags(flaglist)

which then indicates what params[0] actually is. We spell this out in the functions so people can look at the commands and get some idea of how to use them without having to look at the code itself.

Quoted 8 lines
bitbake/lib/bb/command.py +0 -0
     def getVariable(self, command, params):
         """
         Read the value of a variable from data
bitbake/lib/bb/cooker.py +0 -0
index cb0e3e5..981379b 100644

bitbake/lib/bb/cooker.py class BBCooker: · line 1143

Quoted 10 lines
bitbake/lib/bb/cooker.py +7 -0
@@ -1134,6 +1134,21 @@ class BBCooker:
 
         self.configuration.server_register_idlecallback(buildTargetsIdle, rq)
 
+
+    def getAllKeysWithFlags(self, flaglist):
+        dump = {}
+        for k in self.data.keys():
+            try:
+                v = self.data.getVar(k, True)
+                if not k.startswith("__") and not bool(self.data.getVarFlag(k, 'func')) and not isinstance(v, bb.data_smart.DataSmart):

Please get rid of the "not bool(self.data.getVarFlag(k, 'func')" and filter your results on the other side of the connection if you need to do that (you can request the func flag too).

bitbake/lib/bb/cooker.py class BBCooker: · line 1147

Quoted 4 lines
bitbake/lib/bb/cooker.py +4 -0
+                    dump[k] = { 'v' : v }
+                    for d in flaglist:
+                        dump[k][d] = self.data.getVarFlag(k, d)
+            except:

which exceptions are we worried about here? Lets be specific please.

Quoted 7 lines
bitbake/lib/bb/cooker.py +4 -0
+                pass
+        return dump
+
+
     def generateNewImage(self, image, base_image, package_queue, timestamp, description):
         '''
         Create a new image with a "require"/"inherit" base_image statement
#14 lore raw

Re: [PATCH 6/7] bitbake: event: adding events to allow bbclasses to push custom info

In reply to Richard Purdie
Quoted 51 lines · Richard Purdie
Quoted 28 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We add a new event that will be triggered by the package-building class, containing the information about the currently built package.

Adding an event that will passi information about the content of each package file that gets built.

Knotty UI will ignore these events by default.

Maybe a mechanism for using a single generic event is in order ?

---
 bitbake/lib/bb/event.py     | 16 ++++++++++++++++
 bitbake/lib/bb/ui/knotty.py |  2 ++
 2 files changed, 18 insertions(+)
bitbake/lib/bb/event.py +1 -0
index 67cfcea..7dd9236 100644
@@ -589,6 +589,22 @@ class PackageInfo(Event):
         Event.__init__(self)
         self._pkginfolist = pkginfolist
 
+class SinglePackageInfo(Event):

bitbake/lib/bb/event.py class PackageInfo(Event): · line 602

Quoted 10 lines
bitbake/lib/bb/event.py +10 -0
+    """
+    Single Package Information sent on emit_pkgdata
+    """
+    def __init__(self, data):
+        Event.__init__(self)
+        self._data = data
+
+class PackageFileSizes(Event):
+    """
+    Event that contains information about the file sizes contained in a

package.

bitbake/lib/bb/event.py class PackageInfo(Event): · line 609

Quoted 7 lines
bitbake/lib/bb/event.py +5 -0
+    """
+    def __init__(self, data):
+        Event.__init__(self)
+        self._data = data
+
 class SanityCheck(Event):
     """

Your explanation in the commit message isn't as detailed as it could be.

Quoted 7 lines

I'm guessing but I suspect the problem we have is that the event cannot be rebuilt on the other side of an XMLRPC connection without having the event class in a common namespace?

This is therefore a hack to allow the event to exist in all the client side namespaces.

[Alex] Yep, this is an ugly hack.

Quoted 5 lines

Ideally we should really have an API which allows the classes themselves to declare their event formats and not require adding to event.py every time a new piece of code/event is added.

[Alex] I would favor a single "GenericEvent" that will take a set of defined parameters,
that would allow "subclassing" at runtime - sort of introspection if you will.

Something like this:

class GenericInfo(Event):
def __init__(self, type, data):
Event.__init__(self)
self._type = type
self._data = data

with a call like

bb.event.fire(bb.event.GenericInfo("PackageInfo", pkginfolist), e.data)

the UI handlers / anybody would subscribe to GenericInfo events, and the processing code would look like:

if isinstance(event, bb.event.GenericInfo):

    if event._type == "PackageInfo":
         savePackageInfo(event)
    elif event._type == "PackageFileSizes":
        ....

What do you think ?

Quoted 4 lines

Presumably there is some change required to OE-Core in order for this get used to?

The code using the e

Quoted 3 lines

Is the PackageFileSizes not contained in the other event data?

[Alex] Not that I could find.

Quoted 21 lines

I appreciate we allowed PackageInfo in but I didn't like it at the time, I like it even less now I see we're just going to keep doing this :(.

bitbake/lib/bb/ui/knotty.py def main(server, eventHandler, params, tf =

Quoted 6 lines
bitbake/lib/bb/event.py +0 -0
     Event to issue sanity check
bitbake/lib/bb/ui/knotty.py +0 -0
index 0211b50..03baa75 100644
@@ -477,6 +477,8 @@ def main(server, eventHandler, params, tf =

TerminalFilter):

bitbake/lib/bb/ui/knotty.py def main(server, eventHandler, params, tf = TerminalFilter): · line 484

Quoted 8 lines
bitbake/lib/bb/ui/knotty.py +2 -0
             # ignore
             if isinstance(event, (bb.event.BuildBase,
+                                  bb.event.PackageFileSizes,
+                                  bb.event.SinglePackageInfo,
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
                                   bb.event.RecipeParsed,
Quoted 3 lines

Given the event masking that is now in knotty, do we need to do this? Can't we remove this block?

[Alex] Yep, it should get removed. I refrained from doing so since it wasn't deleted
when the subscription code got merged in.

Quoted 6 lines

Cheers,

Richard

Signature · 3 lines
Alex Damian
Yocto Project
SSG / OTC
#9 lore raw

Re: [PATCH 4/7] bitbake: cooker: add extra recipe information

In reply to Alex DAMIAN
Quoted 38 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding in the extra recipe information when creating the dependency tree information.

This works in server-mode because the HOB extra_cache is already enabled. In normal mode, it will do nothing with no performance impact.

---
 bitbake/lib/bb/cooker.py | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
bitbake/lib/bb/cooker.py +16 -0
index 68a3c01..cb0e3e5 100644
@@ -439,6 +439,29 @@ class BBCooker:
                 depend_tree["pn"][pn] = {}
                 depend_tree["pn"][pn]["filename"] = fn
                 depend_tree["pn"][pn]["version"] = version
+
+                # This data is needed for webhob;
+                # it's here if the bitbake runs in server mode
+                try:
+                    summary = self.recipecache.summary[fn]
+                    lic = self.recipecache.license[fn]
+                    section = self.recipecache.section[fn]
+                    description = self.recipecache.description[fn]
+                    homepage = self.recipecache.homepage[fn]
+                    bugtracker = self.recipecache.bugtracker[fn]
+                    inherits = self.recipecache.inherits.get(fn, None)
+                    depend_tree["pn"][pn]["filename"] = fn
+                    depend_tree["pn"][pn]["version"] = version
+                    depend_tree["pn"][pn]["summary"] = summary
+                    depend_tree["pn"][pn]["license"] = lic
+                    depend_tree["pn"][pn]["section"] = section

bitbake/lib/bb/cooker.py class BBCooker: · line 467

Quoted 10 lines
bitbake/lib/bb/cooker.py +7 -0
+                    depend_tree["pn"][pn]["description"] = description
+                    depend_tree["pn"][pn]["inherits"] = inherits
+                    depend_tree["pn"][pn]["homepage"] = homepage
+                    depend_tree["pn"][pn]["bugtracker"] = bugtracker
+                except:
+                    pass
+
             for dep in rq.rqdata.runq_depends[task]:
                 depfn = taskdata.fn_index[rq.rqdata.runq_fnid[dep]]
                 deppn = self.recipecache.pkg_fn[depfn]

This is horrible, please try and do something like:

bitbake/lib/bb/cache_extra.py +3 -0
index 9e38a43..041ba0e 100644
@@ -67,3 +67,6 @@ class HobRecipeInfo(RecipeInfoCommon):
         cachedata.bugtracker[fn] = self.bugtracker
         cachedata.prevision[fn] = self.prevision
         cachedata.files_info[fn] = self.files_info
+
+    @staticmethod
+    def addDepTreePackageInfo(xxx)
bitbake/lib/bb/cooker.py +4 -0
index 2c8d4dc..84d62e5 100644
@@ -485,6 +485,10 @@ class BBCooker:
                         depend_tree["packages"][package]["filename"] = fn
                         depend_tree["packages"][package]["version"] = version
 
+                        for extracache in self.cooker.caches_array:
+                            if <executable function>:
+                                 <function>(depend_tree["packages"][package])
+
         return depend_tree
 

so the extra cache info is added as a staticmethod function in the extra cache info file which generated the data. You might as well fix the hardcoding in the following function marked with:

######## WARNING : this function requires cache_extra to be enabled ########

to use this approach too whilst you're doing this.

Thanks,

Richard

#5 lore raw

Re: [PATCH 3/7] bitbake: runqueue: add runQueueTaskSkipped event

In reply to Alex DAMIAN
Quoted 100 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding a runQueueTaskSkipped to notify all listeners about the tasks that are not run either because they are set-scened or they don't need an update (timestamps are ok)

Adds RunQueueData functions to get the task name and task file for usage with the runQueue* events.

---
 bitbake/lib/bb/build.py    |  3 +++
 bitbake/lib/bb/runqueue.py | 27 ++++++++++++++++++++++++---
 2 files changed, 27 insertions(+), 3 deletions(-)
bitbake/lib/bb/build.py +3 -0
index a53aba9..1265bdc 100644
@@ -72,6 +72,9 @@ class TaskBase(event.Event):
     def __init__(self, t, d ):
         self._task = t
         self._package = d.getVar("PF", True)
+        self._file = d.getVar("FILE", True)
+        self.taskfile = self._file
+        self.taskname = self._task
         event.Event.__init__(self)
         self._message = "recipe %s: task %s: %s" % (d.getVar("PF", True), t, self.getDisplayName())
 
bitbake/lib/bb/runqueue.py +17 -3
index aa2f147..2ab4405 100644
@@ -217,6 +217,12 @@ class RunQueueData:
             ret.extend([nam])
         return ret
 
+    def get_task_name(self, task):
+        return self.runq_task[task]
+
+    def get_task_file(self, task):
+        return self.taskData.fn_index[self.runq_fnid[task]]
+
     def get_user_idstring(self, task, task_name_suffix = ""):
         fn = self.taskData.fn_index[self.runq_fnid[task]]
         taskname = self.runq_task[task] + task_name_suffix
@@ -1328,9 +1334,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
         if self.rqdata.taskData.abort:
             self.rq.state = runQueueCleanUp
 
-    def task_skip(self, task):
+    def task_skip(self, task, skip_reason = ""):
         self.runq_running[task] = 1
         self.runq_buildable[task] = 1
+        bb.event.fire(runQueueTaskSkipped(task, self.stats, self.rq, skip_reason), self.cfgData)
         self.task_completeoutright(task)
         self.stats.taskCompleted()
         self.stats.taskSkipped()
@@ -1355,13 +1362,13 @@ class RunQueueExecuteTasks(RunQueueExecute):
             if task in self.rq.scenequeue_covered:
                 logger.debug(2, "Setscene covered task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "covered")
                 return True
 
             if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
                 logger.debug(2, "Stamp current task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "existing")
                 return True
 
             taskdep = self.rqdata.dataCache.task_deps[fn]
@@ -1783,6 +1790,8 @@ class runQueueEvent(bb.event.Event):
     def __init__(self, task, stats, rq):
         self.taskid = task
         self.taskstring = rq.rqdata.get_user_idstring(task)
+        self.taskname = rq.rqdata.get_task_name(task)
+        self.taskfile = rq.rqdata.get_task_file(task)
         self.stats = stats.copy()
         bb.event.Event.__init__(self)
 
@@ -1794,6 +1803,8 @@ class sceneQueueEvent(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         realtask = rq.rqdata.runq_setscene[task]
         self.taskstring = rq.rqdata.get_user_idstring(realtask, "_setscene")
+        self.taskname = rq.rqdata.get_task_name(realtask) + "_setscene"
+        self.taskfile = rq.rqdata.get_task_file(realtask)
 
 class runQueueTaskStarted(runQueueEvent):
     """
@@ -1837,6 +1848,16 @@ class sceneQueueTaskCompleted(sceneQueueEvent):
     Event notifing a setscene task completed
     """
 
+class runQueueTaskSkipped(runQueueEvent):
+    """
+    Event notifing a task was skipped

bitbake/lib/bb/runqueue.py class sceneQueueTaskCompleted(sceneQueueEvent): · line 1863

Quoted 10 lines
bitbake/lib/bb/runqueue.py +7 -0
+    """
+    def __init__(self, task, stats, rq, skip_reason):
+        runQueueEvent.__init__(self, task, stats, rq)
+        self._file = rq.rqdata.get_task_file(task)
+        self._task = rq.rqdata.get_task_name(task)
+        self._skip = skip_reason
+
 class runQueuePipe():
     """
     Abstraction for a pipe between a worker thread and the server

Please rebase this one off
http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=39c70d0ad00afa05d343dc19aa557a092db7c963

since we should do one thing in a commit, not three. I renamed _skip -> reason since its not internal and its a skip event, we know this, the field is the reason.

You need to separate out the runQueueEvent changes from the TaskBase changes.

Why add self._file to TaskBase? Surely just taskfile would be enough? The _ is meant to indicate internal use only and aren't the above events inconsistent with each other?

Cheers,

Richard

#6 lore raw

Re: [PATCH 3/7] bitbake: runqueue: add runQueueTaskSkipped event

Quoted 125 lines · Richard Purdie
Quoted 21 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding a runQueueTaskSkipped to notify all listeners about the tasks that are not run either because they are set-scened or they don't need an update (timestamps are ok)

Adds RunQueueData functions to get the task name and task file for usage with the runQueue* events.

---
 bitbake/lib/bb/build.py    |  3 +++
 bitbake/lib/bb/runqueue.py | 27 ++++++++++++++++++++++++---
 2 files changed, 27 insertions(+), 3 deletions(-)
bitbake/lib/bb/build.py +0 -0
index a53aba9..1265bdc 100644

bitbake/lib/bb/build.py class TaskBase(event.Event): · line 79

Quoted 8 lines
bitbake/lib/bb/build.py +3 -0
@@ -72,6 +72,9 @@ class TaskBase(event.Event):
     def __init__(self, t, d ):
         self._task = t
         self._package = d.getVar("PF", True)
+        self._file = d.getVar("FILE", True)
+        self.taskfile = self._file
+        self.taskname = self._task
         event.Event.__init__(self)
         self._message = "recipe %s: task %s: %s" % (d.getVar("PF",

True), t, self.getDisplayName())

Quoted 19 lines
bitbake/lib/bb/runqueue.py +6 -0
index aa2f147..2ab4405 100644
@@ -217,6 +217,12 @@ class RunQueueData:
             ret.extend([nam])
         return ret
 
+    def get_task_name(self, task):
+        return self.runq_task[task]
+
+    def get_task_file(self, task):
+        return self.taskData.fn_index[self.runq_fnid[task]]
+
     def get_user_idstring(self, task, task_name_suffix = ""):
         fn = self.taskData.fn_index[self.runq_fnid[task]]
         taskname = self.runq_task[task] + task_name_suffix

bitbake/lib/bb/runqueue.py class RunQueueExecuteTasks(RunQueueExecute): · line 1340

Quoted 8 lines
bitbake/lib/bb/runqueue.py +2 -1
@@ -1328,9 +1334,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
         if self.rqdata.taskData.abort:
             self.rq.state = runQueueCleanUp
 
-    def task_skip(self, task):
+    def task_skip(self, task, skip_reason = ""):
         self.runq_running[task] = 1
         self.runq_buildable[task] = 1
+        bb.event.fire(runQueueTaskSkipped(task, self.stats, self.rq,

skip_reason), self.cfgData)

bitbake/lib/bb/runqueue.py class RunQueueExecuteTasks(RunQueueExecute): · line 1368

Quoted 12 lines
bitbake/lib/bb/runqueue.py +1 -1
         self.task_completeoutright(task)
         self.stats.taskCompleted()
         self.stats.taskSkipped()
@@ -1355,13 +1362,13 @@ class RunQueueExecuteTasks(RunQueueExecute):
             if task in self.rq.scenequeue_covered:
                 logger.debug(2, "Setscene covered task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "covered")
                 return True
 
             if self.rq.check_stamp_task(task, taskname,

cache=self.stampcache):

Quoted 17 lines
bitbake/lib/bb/runqueue.py +3 -1
                 logger.debug(2, "Stamp current task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "existing")
                 return True
 
             taskdep = self.rqdata.dataCache.task_deps[fn]
@@ -1783,6 +1790,8 @@ class runQueueEvent(bb.event.Event):
     def __init__(self, task, stats, rq):
         self.taskid = task
         self.taskstring = rq.rqdata.get_user_idstring(task)
+        self.taskname = rq.rqdata.get_task_name(task)
+        self.taskfile = rq.rqdata.get_task_file(task)
         self.stats = stats.copy()
         bb.event.Event.__init__(self)
 

bitbake/lib/bb/runqueue.py class sceneQueueEvent(runQueueEvent): · line 1805

Quoted 3 lines
bitbake/lib/bb/runqueue.py +0 -0
@@ -1794,6 +1803,8 @@ class sceneQueueEvent(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         realtask = rq.rqdata.runq_setscene[task]
         self.taskstring = rq.rqdata.get_user_idstring(realtask,

"_setscene")

Quoted 12 lines
bitbake/lib/bb/runqueue.py +5 -0
+        self.taskname = rq.rqdata.get_task_name(realtask) + "_setscene"
+        self.taskfile = rq.rqdata.get_task_file(realtask)
 
 class runQueueTaskStarted(runQueueEvent):
     """
@@ -1837,6 +1848,16 @@ class sceneQueueTaskCompleted(sceneQueueEvent):
     Event notifing a setscene task completed
     """
 
+class runQueueTaskSkipped(runQueueEvent):
+    """
+    Event notifing a task was skipped

bitbake/lib/bb/runqueue.py class sceneQueueTaskCompleted(sceneQueueEvent): · line 1863

Quoted 10 lines
bitbake/lib/bb/runqueue.py +7 -0
+    """
+    def __init__(self, task, stats, rq, skip_reason):
+        runQueueEvent.__init__(self, task, stats, rq)
+        self._file = rq.rqdata.get_task_file(task)
+        self._task = rq.rqdata.get_task_name(task)
+        self._skip = skip_reason
+
 class runQueuePipe():
     """
     Abstraction for a pipe between a worker thread and the server

Please rebase this one off

http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=39c70d0ad00afa05d343dc19aa557a092db7c963

since we should do one thing in a commit, not three. I renamed _skip -> reason since its not internal and its a skip event, we know this, the field is the reason.

Quoted 7 lines

You need to separate out the runQueueEvent changes from the TaskBase changes.

Why add self._file to TaskBase? Surely just taskfile would be enough? The _ is meant to indicate internal use only and aren't the above events inconsistent with each other?

[Alex] This is the idea - I actually tried to make TaskBase-derived events consistent with runQueueEvents in terms of class fields, so the code that reads them is easier to follow - I just have to look up for the same named field regardless of the event type.

Quoted 6 lines

Cheers,

Richard

Signature · 3 lines
Alex Damian
Yocto Project
SSG / OTC
#15 lore raw

Re: [PATCH 6/7] bitbake: event: adding events to allow bbclasses to push custom info

In reply to Damian, Alexandru
Quoted 12 lines · Alexandru

On Mon, Sep 16, 2013 at 2:46 PM, Richard Purdie
<[email protected]> wrote:

Your explanation in the commit message isn't as detailed as it could
be.

I'm guessing but I suspect the problem we have is that the event cannot
be rebuilt on the other side of an XMLRPC connection without having the
event class in a common namespace?

Quoted 6 lines

This is therefore a hack to allow the event to exist in all the client
side namespaces.

[Alex] Yep, this is an ugly hack.

In future *please* put this kind of information into commit messages. I'm having a lot of trouble with these patches as the commit messages don't give me all the information I need to review them.

Quoted 35 lines

Ideally we should really have an API which allows the classes themselves
to declare their event formats and not require adding to event.py every
time a new piece of code/event is added.

[Alex] I would favor a single "GenericEvent" that will take a set of defined parameters,
that would allow "subclassing" at runtime - sort of introspection if you will.

Something like this:

class GenericInfo(Event):

def __init__(self, type, data):
Event.__init__(self)
self._type = type
self._data = data

with a call like

bb.event.fire(bb.event.GenericInfo("PackageInfo", pkginfolist), e.data)

the UI handlers / anybody would subscribe to GenericInfo events, and the processing code would look like:

Quoted 6 lines

if isinstance(event, bb.event.GenericInfo):

    if event._type == "PackageInfo":
         savePackageInfo(event)
    elif event._type == "PackageFileSizes":
        ....

Lets do it but can you:

a) Stop using the underscores please. I know some older events have them but the data is not internal so lets not persist broken syntax going forward.
b) Call this MetadataEvent and put in a header comment explaining what it is and why/when you'd use it.
c) Convert the user of PackageInfo in OE-Core to use the new event if it is available, falling back to PackageInfo.
d) Update hob to accept the new event

c/d can come later but I do want to see patches for them ultimately. We'll remove PackageInfo in the 1.6 cycle.

Cheers,

Richard

#7 lore raw

Re: [PATCH 3/7] bitbake: runqueue: add runQueueTaskSkipped event

In reply to Damian, Alexandru
Quoted 155 lines · Alexandru

On Mon, Sep 16, 2013 at 3:24 PM, Richard Purdie <[email protected]> wrote:
On Mon, 2013-09-16 at 14:33 +0100, Alex DAMIAN wrote:
> From: Alexandru DAMIAN <[email protected]> >
> Adding a runQueueTaskSkipped to notify all listeners > about the tasks that are not run either because they > are set-scened or they don't need an update (timestamps > are ok)
>
> Adds RunQueueData functions to get the task name and task > file for usage with the runQueue* events.
>
> Signed-off-by: Alexandru DAMIAN <[email protected]> > ---
> bitbake/lib/bb/build.py | 3 +++
> bitbake/lib/bb/runqueue.py | 27 ++++++++++++++++++++++++--- > 2 files changed, 27 insertions(+), 3 deletions(-) >
> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> index a53aba9..1265bdc 100644
> --- a/bitbake/lib/bb/build.py
> +++ b/bitbake/lib/bb/build.py
> @@ -72,6 +72,9 @@ class TaskBase(event.Event):
> def __init__(self, t, d ):
> self._task = t
> self._package = d.getVar("PF", True)
> + self._file = d.getVar("FILE", True)
> + self.taskfile = self._file
> + self.taskname = self._task
> event.Event.__init__(self)
> self._message = "recipe %s: task %s: %s" %
(d.getVar("PF", True), t, self.getDisplayName())
>
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index aa2f147..2ab4405 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -217,6 +217,12 @@ class RunQueueData:
> ret.extend([nam])
> return ret
>
> + def get_task_name(self, task):
> + return self.runq_task[task]
> +
> + def get_task_file(self, task):
> + return self.taskData.fn_index[self.runq_fnid[task]]
> +
> def get_user_idstring(self, task, task_name_suffix =
""):
> fn = self.taskData.fn_index[self.runq_fnid[task]]
> taskname = self.runq_task[task] + task_name_suffix
> @@ -1328,9 +1334,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
> if self.rqdata.taskData.abort:
> self.rq.state = runQueueCleanUp
>
> - def task_skip(self, task):
> + def task_skip(self, task, skip_reason = ""):
> self.runq_running[task] = 1
> self.runq_buildable[task] = 1
> + bb.event.fire(runQueueTaskSkipped(task, self.stats,
self.rq, skip_reason), self.cfgData)
> self.task_completeoutright(task)
> self.stats.taskCompleted()
> self.stats.taskSkipped()
> @@ -1355,13 +1362,13 @@ class RunQueueExecuteTasks(RunQueueExecute):
> if task in self.rq.scenequeue_covered:
> logger.debug(2, "Setscene covered task %s
(%s)", task,
>
self.rqdata.get_user_idstring(task))
> - self.task_skip(task)
> + self.task_skip(task, "covered")
> return True
>
> if self.rq.check_stamp_task(task, taskname,
cache=self.stampcache):
> logger.debug(2, "Stamp current task %s (%
s)", task,
>
self.rqdata.get_user_idstring(task))
> - self.task_skip(task)
> + self.task_skip(task, "existing")
> return True
>
> taskdep = self.rqdata.dataCache.task_deps[fn]
> @@ -1783,6 +1790,8 @@ class runQueueEvent(bb.event.Event):
> def __init__(self, task, stats, rq):
> self.taskid = task
> self.taskstring = rq.rqdata.get_user_idstring(task)
> + self.taskname = rq.rqdata.get_task_name(task)
> + self.taskfile = rq.rqdata.get_task_file(task)
> self.stats = stats.copy()
> bb.event.Event.__init__(self)
>
> @@ -1794,6 +1803,8 @@ class sceneQueueEvent(runQueueEvent):
> runQueueEvent.__init__(self, task, stats, rq)
> realtask = rq.rqdata.runq_setscene[task]
> self.taskstring =
rq.rqdata.get_user_idstring(realtask, "_setscene")
> + self.taskname = rq.rqdata.get_task_name(realtask) +
"_setscene"
> + self.taskfile = rq.rqdata.get_task_file(realtask)
>
> class runQueueTaskStarted(runQueueEvent):
> """
> @@ -1837,6 +1848,16 @@ class sceneQueueTaskCompleted(sceneQueueEvent):
> Event notifing a setscene task completed
> """
>
> +class runQueueTaskSkipped(runQueueEvent):
> + """
> + Event notifing a task was skipped
> + """
> + def __init__(self, task, stats, rq, skip_reason):
> + runQueueEvent.__init__(self, task, stats, rq)
> + self._file = rq.rqdata.get_task_file(task)
> + self._task = rq.rqdata.get_task_name(task)
> + self._skip = skip_reason
> +
> class runQueuePipe():
> """
> Abstraction for a pipe between a worker thread and the
server

Please rebase this one off
http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=39c70d0ad00afa05d343dc19aa557a092db7c963

since we should do one thing in a commit, not three. I renamed _skip ->
reason since its not internal and its a skip event, we know this, the
field is the reason.

You need to separate out the runQueueEvent changes from the TaskBase
changes.

Why add self._file to TaskBase? Surely just taskfile would be enough?
The _ is meant to indicate internal use only and aren't the above events
inconsistent with each other?

Quoted 5 lines

[Alex] This is the idea - I actually tried to make TaskBase-derived events consistent with runQueueEvents in terms of class fields, so the code that reads them is easier to follow - I just have to look up for the same named field regardless of the event type.

No events have a _file currently though?

To be honest, I think you need to log the data from one or the other. The events are effectively the same, just one happens server side, the other lets us know the message reached the worker and it also did something. There are reasons for debugging it is useful to have events on both sides but for your purposes on webhob, I doubt you need both.

Cheers,

Richard

#3 lore raw

Re: [PATCH 2/7] bitbake: event: send the task dependency tree to UI

In reply to Alex DAMIAN
Quoted 9 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We add an event that will send the dependency tree graph to the UI client once it is computed.

This will allow the clients to display dependency data in an efficient manner, and not redo the runqueue computation specifically to get the dependency data.

Quoted 6 lines
---
 bitbake/lib/bb/cooker.py    | 3 +++
 bitbake/lib/bb/runqueue.py  | 3 +++
 bitbake/lib/bb/ui/knotty.py | 3 +++
 3 files changed, 9 insertions(+)

I continue to feel quite strongly that this should only happen when there are UIs connected that actually want this event. We actually have two problems, this one and the part in bin/bitbake which says:

# Collect all the caches we need
if configParams.server_only:
configuration.extra_caches = gather_extra_cache_data()
else:
configuration.extra_caches = getattr(ui_module, "extraCaches", [])

I was never happy when this went in and I've already had one complaint about the memory resident bitbake server reparsing compared to the normal knotty client. You asked how this could be fixed as you couldn't see how it could be done. I have two ideas:

The first is simpler but only fixes this problem, not the caches one. The basic idea there would be to query the event interface and see if anyone was listening for DepTreeGenerated events. If nobody is, no point in generating it.

The second idea was having "server features", the caches being a special case of this. When UI clients connect, they would merge their features into those of the server. I've shown some pseudo code below. The "features" could take immediate effect, the caches requested would take effect the next time the server reparses (i.e finish current command, then they take effect). The server features would be sent to the server via some special command which would be allowed in an otherwise readonly server.

Does that help?

Cheers,

Richard

bitbake/bin/bitbake +4 -0
@@ -327,6 +327,10 @@ def main():
             os.environ[k] = cleanedvars[k]
 
         try:
+            if configParams.remote_server:
+                # Request configuration.extra_caches
+                # Request server feature flags set in UI
+
             return ui_module.main(server_connection.connection, server_connection.events, configParams)
         finally:
             bb.event.ui_queue = []
bitbake/lib/bb/cooker.py +14 -0
index 2c8d4dc..89bf7c6 100644
@@ -63,6 +63,20 @@ class CollectionError(bb.BBHandledException):
 class state:
     initial, parsing, running, shutdown, forceshutdown, stopped = range(6)
 
+#
+# Cooker Features control certain functionality in the server. This allows
+# things to happen only when we know we have a UI that needs the particular
+# feature. This is usually used to control event generation
+#
+class CookerFeatures(object):
+    def __init__(self):
+        self._features = ["generateRunQueueDepTreeEvents"]
+        for f in self._features:
+            setattr(self, f, False)
+    def merge(self, features):
+        for f in self._features:
+            v = getattr(features, f)
+            setattr(self, f, v)
 
 class SkippedPackage:
     def __init__(self, info = None, reason = None):