Skip to contentSkip to conversation

Adding Bitbake support for WebHob

PATCH New Alex DAMIAN 1 participant 8 messages Sep 18, 2013 Arm STM32

lore
Share & tools
lore
#0 lore raw

cover Adding Bitbake support for WebHob

From: Alexandru DAMIAN <[email protected]>

This is a set of patches that prepare Bitbake to support WebHob.

Among the significant changes:

* The extra_caches mechanism is replaced with a CookerFeature mechanism that allows the UI clients to specify what features they need at connection time. This allows the server to start, for example, with a minimum set of caches, and load any additional caches on demand when the clients connect.

* Another CookerFeature is added that will make the server send the dependency tree to the clients, on request, when the server is ready to start processing the runqueue.

* There is a new server command that returns the contents of the data store.

* The data read from the the recipecache depends on what extra caches have been loaded, and it's discovered at runtime instead of hardcoding.

The following changes since commit eaf06bc2840cc39026033de0779858b1bf82f06f:

bitbake: cooker: return a copy checkPackages (2013-09-18 08:41:55 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib adamian/bitbake-core-changes-1809-2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/bitbake-core-changes-1809-2

Alexandru DAMIAN (7):
bitbake: cooker,xmlrpc,servers: implement CookerFeatures
bitbake: cooker,runqueue: send the task dependency tree
bitbake: build, runqueue: adds info to the *runQueue* events
bitbake: cooker: get extra information from recipe cache
bitbake: cooker, command: add a command to return global data
bitbake: event: adding generic event for metadata usage
bitbake: runqueue: add task hash to Queue events

 bitbake/bin/bitbake                      |  25 ++------
 bitbake/lib/bb/build.py                  |   2 +
 bitbake/lib/bb/cache_extra.py            |   6 ++
 bitbake/lib/bb/command.py                |   9 +++
 bitbake/lib/bb/cooker.py                 | 102 ++++++++++++++++++++++++-------
 bitbake/lib/bb/cookerdata.py             |   1 -
 bitbake/lib/bb/event.py                  |  10 +++
 bitbake/lib/bb/runqueue.py               |  20 ++++++
 bitbake/lib/bb/server/__init__.py        |   2 +-
 bitbake/lib/bb/server/process.py         |  23 +++++--
 bitbake/lib/bb/server/xmlrpc.py          |  29 ++++++---
 bitbake/lib/bb/ui/crumbs/hoblistmodel.py |   2 +-
 bitbake/lib/bb/ui/hob.py                 |   2 +-
 bitbake/lib/bb/ui/knotty.py              |   4 ++
 bitbake/lib/bb/ui/uievent.py             |   7 ++-
 15 files changed, 183 insertions(+), 61 deletions(-)
Signature · 1 line
1.8.1.2
#1 lore raw

PATCH 1/7 bitbake: cooker, xmlrpc, servers: implement CookerFeatures

From: Alexandru DAMIAN <[email protected]>

Implementing feature set selection that allows a client to enable specific features in the server at connection time.

Only enabling of features is supported, as there is no way to safely remove data loaded into the cooker. Once enabled, a feature will remain enabled for the life of the cooker.

Client-server connection now supports specifying the feature set required by the client. This is implemented in the Process server using a managed proxy list, so the server cooker will now load dynamically needed features based on what client connects to it.

In the XMLRPC server the feature set is requested by using a parameter for registerUIHandler function. This allows observer-only clients to also specify features for the server.

The server code configuration now is completly separated from the client code. All hardcoding of client knowledge is removed from the server.

The extra_caches is removed as the client can now specify the caches it needs using the feature. The UI modules now need to specify the desired featureSet. HOB is modified to conform to the featureSet specification.

The only feature available is CookerFeatures.HOB_EXTRA_CACHES which forces loading the bb.cache_extra:HobRecipeInfo class.

---
 bitbake/bin/bitbake               | 25 +++++--------------------
 bitbake/lib/bb/cooker.py          | 34 ++++++++++++++++++++++++++++++++--
 bitbake/lib/bb/cookerdata.py      |  1 -
 bitbake/lib/bb/server/__init__.py |  2 +-
 bitbake/lib/bb/server/process.py  | 23 ++++++++++++++++++-----
 bitbake/lib/bb/server/xmlrpc.py   | 29 ++++++++++++++++++++++-------
 bitbake/lib/bb/ui/hob.py          |  2 +-
 bitbake/lib/bb/ui/knotty.py       |  3 +++
 bitbake/lib/bb/ui/uievent.py      |  7 +++++--
 9 files changed, 87 insertions(+), 39 deletions(-)
bitbake/bin/bitbake +5 -20
index 60c4b96..b4506ef 100755
@@ -75,18 +75,6 @@ def get_ui(config):
         sys.exit("FATAL: Invalid user interface '%s' specified.\n"
                  "Valid interfaces: depexp, goggle, ncurses, hob, knotty [default]." % interface)
 
-def gather_extra_cache_data():
-    extra = []
-    interfaces = ['depexp', 'goggle', 'ncurses', 'hob', 'knotty']
-    for i in interfaces:
-        try:
-            ui = __import__("bb.ui." + i, fromlist = [i])
-            if hasattr(ui, "extraCaches"):
-                extra = extra + ui.extraCaches
-            del ui
-        except:
-            pass
-    return extra
 
 # Display bitbake/OE warnings via the BitBake.Warnings logger, ignoring others"""
 warnlog = logging.getLogger("BitBake.Warnings")
@@ -302,25 +290,22 @@ def main():
     # Clear away any spurious environment variables while we stoke up the cooker
     cleanedvars = bb.utils.clean_environment()
 
-    # Collect all the caches we need
-    if configParams.server_only:
-        configuration.extra_caches = gather_extra_cache_data()
-    else:
-        configuration.extra_caches = getattr(ui_module, "extraCaches", [])
-
     if not configParams.remote_server:
         # we start a server with a given configuration
         server = start_server(servermodule, configParams, configuration)
     else:
-        # we start a stub server that is actually a XMLRPClient to
+        # we start a stub server that is actually a XMLRPClient that connects to a real server
         server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
 
     logger.removeHandler(handler)
 
     if not configParams.server_only:
+        # Collect the feature set for the UI
+        featureset = getattr(ui_module, "featureSet", [])
+
         # Setup a connection to the server (cooker)
-        server_connection = server.establishConnection()
+        server_connection = server.establishConnection(featureset)
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:
bitbake/lib/bb/cooker.py +32 -2
index 1d5e5f6..8776e18 100644
@@ -79,6 +79,29 @@ class SkippedPackage:
         elif reason:
             self.skipreason = reason
 
+
+class CookerFeatures(object):
+    _feature_list = [HOB_EXTRA_CACHES] = range(1)
+
+    def __init__(self):
+        self._features=set()
+
+    def setFeature(self, f):
+        # validate we got a request for a feature we support
+        if f not in CookerFeatures._feature_list:
+            return
+        self._features.add(f)
+
+    def __contains__(self, f):
+        return f in self._features
+
+    def __iter__(self):
+        return self._features.__iter__()
+
+    def next(self):
+        return self._features.next()
+
+
 #============================================================================#
 # BBCooker
 #============================================================================#
@@ -90,6 +113,7 @@ class BBCooker:
     def __init__(self, configuration):
         self.recipecache = None
         self.skiplist = {}
+        self.featureset = CookerFeatures()
 
         self.configuration = configuration
 
@@ -122,7 +146,13 @@ class BBCooker:
         self.state = state.initial
 
         self.caches_array = []
-        caches_name_array = ['bb.cache:CoreRecipeInfo'] + self.configuration.extra_caches
+
+        all_extra_cache_names = []
+        # We hardcode all known cache types in a single place, here.
+        if CookerFeatures.HOB_EXTRA_CACHES in self.featureset:
+            all_extra_cache_names.append("bb.cache_extra:HobRecipeInfo")
+
+        caches_name_array = ['bb.cache:CoreRecipeInfo'] + all_extra_cache_names
 
         # At least CoreRecipeInfo will be loaded, so caches_array will never be empty!
         # This is the entry point, no further check needed!
@@ -130,7 +160,7 @@ class BBCooker:
             try:
                 module_name, cache_name = var.split(':')
                 module = __import__(module_name, fromlist=(cache_name,))
-                self.caches_array.append(getattr(module, cache_name)) 
+                self.caches_array.append(getattr(module, cache_name))
             except ImportError as exc:
                 logger.critical("Unable to import extra RecipeInfo '%s' from '%s': %s" % (cache_name, module_name, exc))
                 sys.exit("FATAL: Failed to import extra cache class '%s'." % cache_name)
bitbake/lib/bb/cookerdata.py +0 -1
index 0b278b1..e640ed0 100644
@@ -127,7 +127,6 @@ class CookerConfiguration(object):
         self.dump_signatures = False
         self.dry_run = False
         self.tracking = False
-        self.extra_caches = []
 
         self.env = {}
 
bitbake/lib/bb/server/__init__.py +1 -1
index 2e1c619..da5e480 100644
@@ -89,7 +89,7 @@ class BitBakeBaseServer(object):
     def detach(self):
         return
 
-    def establishConnection(self):
+    def establishConnection(self, featureset):
         raise   "Must redefine the %s.establishConnection()" % self.__class__.__name__
 
     def endSession(self):
bitbake/lib/bb/server/process.py +18 -5
index e45e0c2..aa07202 100644
@@ -31,7 +31,7 @@ import sys
 import time
 import select
 from Queue import Empty
-from multiprocessing import Event, Process, util, Queue, Pipe, queues
+from multiprocessing import Event, Process, util, Queue, Pipe, queues, Manager
 
 from . import BitBakeBaseServer, BitBakeBaseServerConnection, BaseImplServer
 
@@ -78,12 +78,13 @@ class ProcessServer(Process, BaseImplServer):
     profile_filename = "profile.log"
     profile_processed_filename = "profile.log.processed"
 
-    def __init__(self, command_channel, event_queue):
+    def __init__(self, command_channel, event_queue, featurelist):
         BaseImplServer.__init__(self)
-        Process.__init__(self)
+        Process.__init__(self, args=(featurelist))
         self.command_channel = command_channel
         self.event_queue = event_queue
         self.event = EventAdapter(event_queue)
+        self.featurelist = featurelist
         self.quit = False
 
         self.keep_running = Event()
@@ -94,6 +95,14 @@ class ProcessServer(Process, BaseImplServer):
         for event in bb.event.ui_queue:
             self.event_queue.put(event)
         self.event_handle.value = bb.event.register_UIHhandler(self)
+
+        # process any feature changes based on what UI requested
+        original_featureset = list(self.cooker.featureset)
+        while len(self.featurelist)> 0:
+            self.cooker.featureset.setFeature(self.featurelist.pop())
+        if (original_featureset != list(self.cooker.featureset)):
+            self.cooker.reset()
+
         bb.cooker.server_main(self.cooker, self.main)
 
     def main(self):
@@ -198,13 +207,17 @@ class BitBakeServer(BitBakeBaseServer):
         #
         self.ui_channel, self.server_channel = Pipe()
         self.event_queue = ProcessEventQueue(0)
-        self.serverImpl = ProcessServer(self.server_channel, self.event_queue)
+        manager = Manager()
+        self.featurelist = manager.list()
+        self.serverImpl = ProcessServer(self.server_channel, self.event_queue, self.featurelist)
 
     def detach(self):
         self.serverImpl.start()
         return
 
-    def establishConnection(self):
+    def establishConnection(self, featureset):
+        for f in featureset:
+            self.featurelist.append(f)
         self.connection = BitBakeProcessServerConnection(self.serverImpl, self.ui_channel, self.event_queue)
         signal.signal(signal.SIGTERM, lambda i, s: self.connection.terminate())
         return self.connection
bitbake/lib/bb/server/xmlrpc.py +22 -7
index 8326623..389327a 100644
@@ -89,12 +89,23 @@ class BitBakeServerCommands():
         self.server = server
         self.has_client = False
 
-    def registerEventHandler(self, host, port):
+    def registerEventHandler(self, host, port, featureset = []):
         """
         Register a remote UI Event Handler
         """
         s, t = _create_server(host, port)
 
+        # we don't allow connections if the cooker is running
+        if (self.cooker.state in [bb.cooker.state.parsing, bb.cooker.state.running]):
+            return None
+
+        original_featureset = list(self.cooker.featureset)
+        for f in featureset:
+            self.cooker.featureset.setFeature(f)
+
+        if (original_featureset != list(self.cooker.featureset)):
+            self.cooker.reset()
+
         self.event_handle = bb.event.register_UIHhandler(s)
         return self.event_handle
 
@@ -263,11 +274,12 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
         self.connection_token = token
 
 class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
-    def __init__(self, serverImpl, clientinfo=("localhost", 0), observer_only = False):
+    def __init__(self, serverImpl, clientinfo=("localhost", 0), observer_only = False, featureset = []):
         self.connection, self.transport = _create_server(serverImpl.host, serverImpl.port)
         self.clientinfo = clientinfo
         self.serverImpl = serverImpl
         self.observer_only = observer_only
+        self.featureset = featureset
 
     def connect(self):
         if not self.observer_only:
@@ -277,7 +289,8 @@ class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
         if token is None:
             return None
         self.transport.set_connection_token(token)
-        self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo)
+
+        self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo, self.featureset)
         for event in bb.event.ui_queue:
             self.events.queue_event(event)
         return self
@@ -301,14 +314,15 @@ class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
 
 class BitBakeServer(BitBakeBaseServer):
     def initServer(self, interface = ("localhost", 0)):
+        self.interface = interface
         self.serverImpl = XMLRPCServer(interface)
 
     def detach(self):
         daemonize.createDaemon(self.serverImpl.serve_forever, "bitbake-cookerdaemon.log")
         del self.cooker
 
-    def establishConnection(self):
-        self.connection = BitBakeXMLRPCServerConnection(self.serverImpl)
+    def establishConnection(self, featureset):
+        self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, self.interface, False, featureset)
         return self.connection.connect()
 
     def set_connection_token(self, token):
@@ -318,12 +332,13 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
 
     def __init__(self, observer_only = False):
         self.observer_only = observer_only
+        # if we need extra caches, just tell the server to load them all
         pass
 
     def saveConnectionDetails(self, remote):
         self.remote = remote
 
-    def establishConnection(self):
+    def establishConnection(self, featureset):
         # The format of "remote" must be "server:port"
         try:
             [host, port] = self.remote.split(":")
@@ -340,7 +355,7 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
         except:
             return None
         self.serverImpl = XMLRPCProxyServer(host, port)
-        self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only)
+        self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
         return self.connection.connect()
 
     def endSession(self):
bitbake/lib/bb/ui/hob.py +1 -1
index bc5ee14..154a3b3 100755
@@ -46,7 +46,7 @@ from bb.ui.crumbs.hoblistmodel import RecipeListModel, PackageListModel
 from bb.ui.crumbs.hobeventhandler import HobHandler
 from bb.ui.crumbs.builder import Builder
 
-extraCaches = ['bb.cache_extra:HobRecipeInfo']
+featureSet = [bb.cooker.CookerFeatures.HOB_EXTRA_CACHES]
 
 def event_handle_idle_func(eventHandler, hobHandler):
     # Consume as many messages as we can in the time available to us
bitbake/lib/bb/ui/knotty.py +3 -0
index 35590a2..0211b50 100644
@@ -472,6 +472,9 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                              event.taskid, event.taskstring, event.exitcode)
                 continue
 
+            if isinstance(event, bb.event.DepTreeGenerated):
+                continue
+
             # ignore
             if isinstance(event, (bb.event.BuildBase,
                                   bb.event.StampUpdate,
bitbake/lib/bb/ui/uievent.py +5 -2
index 038029f..2133b44 100644
@@ -28,7 +28,7 @@ import socket, threading, pickle
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 
 class BBUIEventQueue:
-    def __init__(self, BBServer, clientinfo=("localhost, 0")):
+    def __init__(self, BBServer, clientinfo=("localhost, 0"), featureset=[]):
 
         self.eventQueue = []
         self.eventQueueLock = threading.Lock()
@@ -44,7 +44,10 @@ class BBUIEventQueue:
         server.register_function( self.send_event, "event.sendpickle" )
         server.socket.settimeout(1)
 
-        self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
+        self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port, featureset)
+
+        if (self.EventHandle == None):
+            bb.fatal("Could not register UI event handler")
 
         self.server = server
 
Signature · 1 line
1.8.1.2
#2 lore raw

PATCH 2/7 bitbake: cooker, runqueue: send the task dependency tree

From: Alexandru DAMIAN <[email protected]>

Adding a CookerFeature that allows UIs to enable receving a dependency tree once the task data has been computed and the runQueue is ready to start.

This will allow the clients to display dependency data in an efficient manner, and not recompute the runqueue specifically to get the dependency data.

---
 bitbake/lib/bb/cooker.py   | 5 ++++-
 bitbake/lib/bb/runqueue.py | 5 +++++
 2 files changed, 9 insertions(+), 1 deletion(-)
bitbake/lib/bb/cooker.py +4 -1
index 8776e18..dd45ae8 100644
@@ -81,7 +81,7 @@ class SkippedPackage:
 
 
 class CookerFeatures(object):
-    _feature_list = [HOB_EXTRA_CACHES] = range(1)
+    _feature_list = [HOB_EXTRA_CACHES, SEND_DEPENDS_TREE] = range(2)
 
     def __init__(self):
         self._features=set()
@@ -457,7 +457,10 @@ class BBCooker:
         runlist, taskdata = self.prepareTreeData(pkgs_to_build, task)
         rq = bb.runqueue.RunQueue(self, self.data, self.recipecache, taskdata, runlist)
         rq.rqdata.prepare()
+        return self.buildDependTree(rq, taskdata)
 
+
+    def buildDependTree(self, rq, taskdata):
         seen_fnids = []
         depend_tree = {}
         depend_tree["depends"] = {}
bitbake/lib/bb/runqueue.py +5 -0
index 6346c77..472509f 100644
@@ -999,6 +999,11 @@ class RunQueue:
             else:
                 self.state = runQueueSceneInit
 
+                # we are ready to run, see if any UI client needs the dependency info
+                if bb.cooker.CookerFeatures.SEND_DEPENDS_TREE in self.cooker.featureset:
+                    depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
+                    bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
+
         if self.state is runQueueSceneInit:
             if self.cooker.configuration.dump_signatures:
                 self.dump_signatures()
Signature · 1 line
1.8.1.2
#3 lore raw

PATCH 3/7 bitbake: build, runqueue: adds info to the *runQueue* events

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

This patch adds task identifying information for all runQueue and sceneQueue events, and for bb.build.Task* events. This will allow matching event to specific tasks in the UI handlers processing these events.

Adds RunQueueData functions to get the task name and task file for usage with the runQueue* events.

Adds taskfile and taskname properties to bb.build.TaskBase.

Adds taskfile and taskname properties to the *runQueue* events

---
 bitbake/lib/bb/build.py    |  2 ++
 bitbake/lib/bb/runqueue.py | 10 ++++++++++
 2 files changed, 12 insertions(+)
bitbake/lib/bb/build.py +2 -0
index 7ee5007..8aec699 100644
@@ -72,6 +72,8 @@ class TaskBase(event.Event):
     def __init__(self, t, logfile, d):
         self._task = t
         self._package = d.getVar("PF", True)
+        self.taskfile = d.getVar("FILE", True)
+        self.taskname = self._task
         self.logfile = logfile
         event.Event.__init__(self)
         self._message = "recipe %s: task %s: %s" % (d.getVar("PF", True), t, self.getDisplayName())
bitbake/lib/bb/runqueue.py +10 -0
index 472509f..19e89e9 100644
@@ -217,6 +217,12 @@ class RunQueueData:
             ret.extend([nam])
         return ret
 
+    def get_task_name(self, task):
+        return self.runq_task[task]
+
+    def get_task_file(self, task):
+        return self.taskData.fn_index[self.runq_fnid[task]]
+
     def get_user_idstring(self, task, task_name_suffix = ""):
         fn = self.taskData.fn_index[self.runq_fnid[task]]
         taskname = self.runq_task[task] + task_name_suffix
@@ -1786,6 +1792,8 @@ class runQueueEvent(bb.event.Event):
     def __init__(self, task, stats, rq):
         self.taskid = task
         self.taskstring = rq.rqdata.get_user_idstring(task)
+        self.taskname = rq.rqdata.get_task_name(task)
+        self.taskfile = rq.rqdata.get_task_file(task)
         self.stats = stats.copy()
         bb.event.Event.__init__(self)
 
@@ -1797,6 +1805,8 @@ class sceneQueueEvent(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         realtask = rq.rqdata.runq_setscene[task]
         self.taskstring = rq.rqdata.get_user_idstring(realtask, "_setscene")
+        self.taskname = rq.rqdata.get_task_name(realtask) + "_setscene"
+        self.taskfile = rq.rqdata.get_task_file(realtask)
 
 class runQueueTaskStarted(runQueueEvent):
     """
Signature · 1 line
1.8.1.2
#7 lore raw

PATCH 7/7 bitbake: runqueue: add task hash to Queue events

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding the sstate-related hash for all runqueue and scenequeue tasks, as it's needed in the WebHob data.

---
 bitbake/lib/bb/runqueue.py | 5 +++++
 1 file changed, 5 insertions(+)
bitbake/lib/bb/runqueue.py +5 -0
index 19e89e9..c09cfd4 100644
@@ -223,6 +223,9 @@ class RunQueueData:
     def get_task_file(self, task):
         return self.taskData.fn_index[self.runq_fnid[task]]
 
+    def get_task_hash(self, task):
+        return self.runq_hash[task]
+
     def get_user_idstring(self, task, task_name_suffix = ""):
         fn = self.taskData.fn_index[self.runq_fnid[task]]
         taskname = self.runq_task[task] + task_name_suffix
@@ -1794,6 +1797,7 @@ class runQueueEvent(bb.event.Event):
         self.taskstring = rq.rqdata.get_user_idstring(task)
         self.taskname = rq.rqdata.get_task_name(task)
         self.taskfile = rq.rqdata.get_task_file(task)
+        self.taskhash = rq.rqdata.get_task_hash(task)
         self.stats = stats.copy()
         bb.event.Event.__init__(self)
 
@@ -1807,6 +1811,7 @@ class sceneQueueEvent(runQueueEvent):
         self.taskstring = rq.rqdata.get_user_idstring(realtask, "_setscene")
         self.taskname = rq.rqdata.get_task_name(realtask) + "_setscene"
         self.taskfile = rq.rqdata.get_task_file(realtask)
+        self.taskhash = rq.rqdata.get_task_hash(task)
 
 class runQueueTaskStarted(runQueueEvent):
     """
Signature · 1 line
1.8.1.2
#6 lore raw

PATCH 6/7 bitbake: event: adding generic event for metadata usage

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding the generic bb.event.MetadataEvent that is targeted specifically at metadata usage. This is needed in order to let the metadata code send and receive events during asynchrous execution without having to define each event specifically in Bitbake.

Metadata code should subscribe to and fire the MetadataEvent in order to communicate asynchronously, and identify the object using event.type field, and parse the data in the event.data field.

Knotty UI will ignore these event by default.

This deprecates RequestPackageInfo/PackageInfo, and that event pair will be removed in the future.

---
 bitbake/lib/bb/event.py     | 10 ++++++++++
 bitbake/lib/bb/ui/knotty.py |  1 +
 2 files changed, 11 insertions(+)
bitbake/lib/bb/event.py +10 -0
index 67cfcea..10eae5f 100644
@@ -589,6 +589,16 @@ class PackageInfo(Event):
         Event.__init__(self)
         self._pkginfolist = pkginfolist
 
+class MetadataEvent(Event):
+    """
+    Generic event that target for OE-Core classes
+    to report information during asynchrous execution
+    """
+    def __init__(self, eventtype, eventdata):
+        Event.__init__(self)
+        self.type = eventtype
+        self.data = eventdata
+
 class SanityCheck(Event):
     """
     Event to issue sanity check
bitbake/lib/bb/ui/knotty.py +1 -0
index 0211b50..c1ee9f52 100644
@@ -477,6 +477,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
 
             # ignore
             if isinstance(event, (bb.event.BuildBase,
+                                  bb.event.MetadataEvent,
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
                                   bb.event.RecipeParsed,
Signature · 1 line
1.8.1.2
#4 lore raw

PATCH 4/7 bitbake: cooker: get extra information from recipe cache

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

The loaded cache modules may add extra attributes to the recipecache, that will be populated by the cache classes required by the UI. These attributes will be used by the UI to display relevant information.

Adds cachefields cache class field to specify for each cache class which attributes will be set in the recipecache.

Adds code to automatically expand depends tree with the fields exported by the extra cache class.

Fixes a cache field name in the HOB UI.

---
 bitbake/lib/bb/cache_extra.py            |  6 ++++
 bitbake/lib/bb/cooker.py                 | 50 ++++++++++++++++++--------------
 bitbake/lib/bb/ui/crumbs/hoblistmodel.py |  2 +-
 3 files changed, 36 insertions(+), 22 deletions(-)
bitbake/lib/bb/cache_extra.py +6 -0
index 9e38a43..83f4959 100644
@@ -35,6 +35,12 @@ class HobRecipeInfo(RecipeInfoCommon):
     # such as (bb_cache.dat, bb_extracache_hob.dat) 
     cachefile = "bb_extracache_" + classname +".dat"        
 
+    # override this member with the list of extra cache fields
+    # that this class will provide
+    cachefields = ['summary', 'license', 'section',
+            'description', 'homepage', 'bugtracker',
+            'prevision', 'files_info']
+
     def __init__(self, filename, metadata):
 
         self.summary = self.getvar('SUMMARY', metadata)
bitbake/lib/bb/cooker.py +29 -21

70 lines not on this page · raw message

bitbake/lib/bb/ui/crumbs/hoblistmodel.py +1 -1
index 74d335c..b4d2a62 100644
@@ -690,7 +690,7 @@ class RecipeListModel(gtk.ListStore):
             inherits = event_model["pn"][item]["inherits"]
             summary = event_model["pn"][item]["summary"]
             version = event_model["pn"][item]["version"]
-            revision = event_model["pn"][item]["revision"]
+            revision = event_model["pn"][item]["prevision"]
             homepage = event_model["pn"][item]["homepage"]
             bugtracker = event_model["pn"][item]["bugtracker"]
             filename = event_model["pn"][item]["filename"]
Signature · 1 line
1.8.1.2
#5 lore raw

PATCH 5/7 bitbake: cooker, command: add a command to return global data

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Adding the 'getAllKeysWithFlags' read-only command that will return a dump of the global data state, together with specified flags for each key. The flag list is passed in as the first parameter to the command.

This will be used by UI clients to get the build configuration.

---
 bitbake/lib/bb/command.py |  9 +++++++++
 bitbake/lib/bb/cooker.py  | 15 +++++++++++++++
 2 files changed, 24 insertions(+)
bitbake/lib/bb/command.py +9 -0
index f1abaf7..21a6de0 100644
@@ -145,6 +145,15 @@ class CommandsSync:
         """
         command.cooker.shutdown(True)
 
+    def getAllKeysWithFlags(self, command, params):
+        """
+        Returns a dump of the global state. Call with
+        variable flags to be retrieved as params.
+        """
+        flaglist = params[0]
+        return command.cooker.getAllKeysWithFlags(flaglist)
+    getAllKeysWithFlags.readonly = True
+
     def getVariable(self, command, params):
         """
         Read the value of a variable from data
bitbake/lib/bb/cooker.py +15 -0
index fafa518..ff2af69 100644
@@ -1154,6 +1154,21 @@ class BBCooker:
 
         self.configuration.server_register_idlecallback(buildTargetsIdle, rq)
 
+
+    def getAllKeysWithFlags(self, flaglist):
+        dump = {}
+        for k in self.data.keys():
+            try:
+                v = self.data.getVar(k, True)
+                if not k.startswith("__") and not isinstance(v, bb.data_smart.DataSmart):
+                    dump[k] = { 'v' : v }
+                    for d in flaglist:
+                        dump[k][d] = self.data.getVarFlag(k, d)
+            except Exception as e:
+                print(e)
+        return dump
+
+
     def generateNewImage(self, image, base_image, package_queue, timestamp, description):
         '''
         Create a new image with a "require"/"inherit" base_image statement
Signature · 1 line
1.8.1.2