Skip to contentSkip to conversation

Hob2 related bitbake changes

GIT PULL In review Dongxiao Xu 2 participants 19 messages Jan 6, 2012 – Jan 7, 2012

lore
Share & tools
lore
#0 lore raw

Hi Richard,

This pull request is the hob2 related bitbake changes. Please help to review and pull.

Thanks,
Dongxiao

The following changes since commit bc954f74620959703a0e921fcac3a6892675fe0d:
Richard Purdie (1):
cooker.py: Add missing BB_WORKERCONTEXT conditionals

are available in the git repository at:

git://git.pokylinux.org/poky-contrib dxu4/hob-bitbake-changes
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/hob-bitbake-changes

Dongxiao Xu (15):
command.py: Modify needcache value for certain functions
cache: Use configuration's hash value to validate cache
cooker: user bb.configuration.data to inject events command.py: add initCooker API
command.py: add parseConfigurationFiles API command.py: add resolve option for generateTargetsTree API event.py: Add a new event PackageInfo
xmlrpc: Change BitbakeServerInfo init function
cooker: remove command import in cooker.py
bitbake: add a new option "--server-only"
bitbake: add -B option to bind with interface
bitbake: Add client socket info for BitBakeServerConnection command.py: add a new API findCoreBaseFiles command.py: add new command to get the CPU info
runqueue: fire sceneQueueTaskStarted event when a setscene queue
starts

Shane Wang (1):
Bitbake: change for adding progress bar in Hob2.

 bin/bitbake              |   34 +++++++++++----
 bitbake/lib/bb/helper.py |   39 +++++++++++++++++
 lib/bb/cache.py          |   34 ++++++---------
 lib/bb/command.py        |   54 ++++++++++++++++++++++--
 lib/bb/cooker.py         |  105 +++++++++++++++++++++++++++++++++++++--------
 lib/bb/data_smart.py     |   17 +++++++
 lib/bb/event.py          |  100 +++++++++++++++++++++++++++++++++----------
 lib/bb/runqueue.py       |   10 ++++
 lib/bb/server/xmlrpc.py  |   18 ++++----
 lib/bb/ui/knotty.py      |    5 ++-
 lib/bb/ui/uievent.py     |    7 ++-
 11 files changed, 334 insertions(+), 89 deletions(-)
 create mode 100644 bitbake/lib/bb/helper.py
#1 lore raw

PATCH 1/16 command.py: Modify needcache value for certain functions

for findConfigFiels() and findFilesMatchingInDir() functions, they don't need to parse all the bb files, thus setting the needcache value to be False.

---
 lib/bb/command.py |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
lib/bb/command.py +2 -2
index 83907f6..5dec6a9 100644
@@ -241,7 +241,7 @@ class CommandsAsync:
 
         command.cooker.findConfigFiles(varname)
         command.finishAsyncCommand()
-    findConfigFiles.needcache = True
+    findConfigFiles.needcache = False
 
     def findFilesMatchingInDir(self, command, params):
         """
@@ -253,7 +253,7 @@ class CommandsAsync:
 
         command.cooker.findFilesMatchingInDir(pattern, directory)
         command.finishAsyncCommand()
-    findFilesMatchingInDir.needcache = True
+    findFilesMatchingInDir.needcache = False
 
     def findConfigFilePath(self, command, params):
         """
Signature · 1 line
1.7.0.4
#2 lore raw

PATCH 2/16 cache: Use configuration's hash value to validate cache

In reply to Dongxiao Xu

Previously we use the file time stamp to judge if a cache is valid. Here this commit introduce a new method, which calculates the total hash value for a certain configuration's key/value paris, and tag it into cache filename, for example, bb_cache.dat.xxxyyyzzz.

This mechanism also ensures the cache's correctness if user dynamically setting variables from some frontend GUI, like HOB.

---
 lib/bb/cache.py      |   32 ++++++++++++--------------------
 lib/bb/cooker.py     |    4 +++-
 lib/bb/data_smart.py |   17 +++++++++++++++++
 3 files changed, 32 insertions(+), 21 deletions(-)
lib/bb/cache.py +12 -20
index 6b7fa6f..955b6df 100644
@@ -42,10 +42,10 @@ except ImportError:
     logger.info("Importing cPickle failed. "
                 "Falling back to a very slow implementation.")
 
-__cache_version__ = "142"
+__cache_version__ = "143"
 
-def getCacheFile(path, filename):
-    return os.path.join(path, filename)
+def getCacheFile(path, filename, data_hash):
+    return os.path.join(path, filename + "." + data_hash)
 
 # RecipeInfoCommon defines common data retrieving methods
 # from meta data for caches. CoreRecipeInfo as well as other
@@ -254,7 +254,7 @@ class Cache(object):
     BitBake Cache implementation
     """
 
-    def __init__(self, data, caches_array):
+    def __init__(self, data, data_hash, caches_array):
         # Pass caches_array information into Cache Constructor
         # It will be used in later for deciding whether we 
         # need extra cache file dump/load support 
@@ -266,6 +266,7 @@ class Cache(object):
         self.data = None
         self.data_fn = None
         self.cacheclean = True
+        self.data_hash = data_hash
 
         if self.cachedir in [None, '']:
             self.has_cache = False
@@ -274,26 +275,17 @@ class Cache(object):
             return
 
         self.has_cache = True
-        self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat")
+        self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash)
 
         logger.debug(1, "Using cache in '%s'", self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
-        # If any of configuration.data's dependencies are newer than the
-        # cache there isn't even any point in loading it...
-        newest_mtime = 0
-        deps = data.getVar("__base_depends")
-
-        old_mtimes = [old_mtime for _, old_mtime in deps]
-        old_mtimes.append(newest_mtime)
-        newest_mtime = max(old_mtimes)
-
         cache_ok = True
         if self.caches_array:
             for cache_class in self.caches_array:
                 if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                    cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
-                    cache_ok = cache_ok and (bb.parse.cached_mtime_noerror(cachefile) >= newest_mtime)
+                    cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+                    cache_ok = cache_ok and os.path.exists(cachefile)
                     cache_class.init_cacheData(self)
         if cache_ok:
             self.load_cachefile()
@@ -327,7 +319,7 @@ class Cache(object):
         # Calculate the correct cachesize of all those cache files
         for cache_class in self.caches_array:
             if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
                 with open(cachefile, "rb") as cachefile:
                     cachesize += os.fstat(cachefile.fileno()).st_size
 
@@ -335,7 +327,7 @@ class Cache(object):
         
         for cache_class in self.caches_array:
             if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
                 with open(cachefile, "rb") as cachefile:
                     pickled = pickle.Unpickler(cachefile)                    
                     while cachefile:
@@ -588,7 +580,7 @@ class Cache(object):
         for cache_class in self.caches_array:
             if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
                 cache_class_name = cache_class.__name__
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
                 file_dict[cache_class_name] = open(cachefile, "wb")
                 pickler_dict[cache_class_name] =  pickle.Pickler(file_dict[cache_class_name], pickle.HIGHEST_PROTOCOL)
                    
@@ -693,7 +685,7 @@ def init(cooker):
     Files causing parsing errors are evicted from the cache.
 
     """
-    return Cache(cooker.configuration.data)
+    return Cache(cooker.configuration.data, cooker.configuration.data_hash)
 
 
 class CacheData(object):
lib/bb/cooker.py +3 -1
index 15243f2..b33e9ff 100644
@@ -836,6 +836,7 @@ class BBCooker:
         bb.parse.init_parser(data)
         bb.event.fire(bb.event.ConfigParsed(), data)
         self.configuration.data = data
+        self.configuration.data_hash = data.get_hash()
 
     def handleCollections( self, collections ):
         """Handle collections"""
@@ -1403,6 +1404,7 @@ class CookerParser(object):
         self.filelist = filelist
         self.cooker = cooker
         self.cfgdata = cooker.configuration.data
+        self.cfghash = cooker.configuration.data_hash
 
         # Accounting statistics
         self.parsed = 0
@@ -1418,7 +1420,7 @@ class CookerParser(object):
         self.num_processes = int(self.cfgdata.getVar("BB_NUMBER_PARSE_THREADS", True) or
                                  multiprocessing.cpu_count())
 
-        self.bb_cache = bb.cache.Cache(self.cfgdata, cooker.caches_array)
+        self.bb_cache = bb.cache.Cache(self.cfgdata, self.cfghash, cooker.caches_array)
         self.fromcache = []
         self.willparse = []
         for filename in self.filelist:
lib/bb/data_smart.py +17 -0
index ea13478..06d8d0b 100644
@@ -31,6 +31,7 @@ BitBake build tools.
 import copy, re
 from collections import MutableMapping
 import logging
+import hashlib
 import bb, bb.codeparser
 from bb   import utils
 from bb.COW  import COWDictBase
@@ -459,3 +460,19 @@ class DataSmart(MutableMapping):
 
     def __delitem__(self, var):
         self.delVar(var)
+
+    def get_hash(self):
+        data = ""
+        keys = iter(self)
+        for key in keys:
+            if key == "TIME":
+                continue
+            if key == "__depends":
+                deps = list(self.getVar(key, False))
+                deps.sort()
+                value = [deps[i][0] for i in range(len(deps))]
+            else:
+                value = self.getVar(key, False) or ""
+            data = data + key + str(value)
+
+        return hashlib.md5(data).hexdigest()
Signature · 1 line
1.7.0.4
#4 lore raw

PATCH 4/16 command.py: add initCooker API

In reply to Dongxiao Xu

initCooker is to set the cooker to the initial state with nothing parsed.

---
 lib/bb/command.py |    6 ++++++
 lib/bb/cooker.py  |   13 +++++++++++++
 2 files changed, 19 insertions(+), 0 deletions(-)
lib/bb/command.py +6 -0
index 5dec6a9..701b286 100644
@@ -157,6 +157,12 @@ class CommandsSync:
         value = params[1]
         command.cooker.configuration.data.setVar(varname, value)
 
+    def initCooker(self, command, params):
+        """
+        Init the cooker to initial state with nothing parsed
+        """
+        command.cooker.initialize()
+
     def resetCooker(self, command, params):
         """
         Reset the cooker to its initial state, thus forcing a reparse for
lib/bb/cooker.py +13 -0
index fe44839..c45088c 100644
@@ -167,6 +167,15 @@ class BBCooker:
 
         self.parser = None
 
+    def initConfigurationData(self):
+        self.configuration.data = bb.data.init()
+
+        if not self.server_registration_cb:
+            bb.data.setVar("BB_WORKERCONTEXT", "1", self.configuration.data)
+
+        filtered_keys = bb.utils.approved_variables()
+        bb.data.inheritFromOS(self.configuration.data, self.savedenv, filtered_keys)
+
     def loadConfigurationData(self):
         self.configuration.data = bb.data.init()
 
@@ -1305,6 +1314,10 @@ class BBCooker:
     def reparseFiles(self):
         return
 
+    def initialize(self):
+        self.state = state.initial
+        self.initConfigurationData()
+
     def reset(self):
         self.state = state.initial
         self.loadConfigurationData()
Signature · 1 line
1.7.0.4
#3 lore raw

PATCH 3/16 cooker: user bb.configuration.data to inject events

In reply to Dongxiao Xu

For buildTargets function, we use bb.configuration.data as parameter to inject events, since in hob environment, some variables are modified dynamically and bb.configuration.event_data may out of date.

---
 lib/bb/cooker.py |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
lib/bb/cooker.py +2 -2
index b33e9ff..fe44839 100644
@@ -1089,7 +1089,7 @@ class BBCooker:
                 return False
 
             if not retval:
-                bb.event.fire(bb.event.BuildCompleted(buildname, targets, failures), self.configuration.event_data)
+                bb.event.fire(bb.event.BuildCompleted(buildname, targets, failures), self.configuration.data)
                 self.command.finishAsyncCommand()
                 return False
             if retval is True:
@@ -1099,7 +1099,7 @@ class BBCooker:
         self.buildSetVars()
 
         buildname = self.configuration.data.getVar("BUILDNAME")
-        bb.event.fire(bb.event.BuildStarted(buildname, targets), self.configuration.event_data)
+        bb.event.fire(bb.event.BuildStarted(buildname, targets), self.configuration.data)
 
         localdata = data.createCopy(self.configuration.data)
         bb.data.update_data(localdata)
Signature · 1 line
1.7.0.4
#6 lore raw

PATCH 6/16 command.py: add resolve option for generateTargetsTree API

In reply to Dongxiao Xu

Currently we have generateTargetsTree API, which is used to get dependency information. However in that tree, there will be "virtual/xxx" in depends fields. Therefore we add the resolve option to replace it with its real providers.

Besides, for packages that provided by multiple recipes, we will find their preverred provider.

---
 lib/bb/command.py |   11 ++++++++-
 lib/bb/cooker.py  |   58 ++++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 56 insertions(+), 13 deletions(-)
lib/bb/command.py +9 -2
index 6b4a598..e52041d 100644
@@ -227,14 +227,21 @@ class CommandsAsync:
         included in the package list.
         If pkg_list provided use that list (plus any extras brought in by
         klass) rather than generating a tree for all packages.
+
+        Add a new option "resolve" to indicate if we need to resolve the
+        replacement for "virtual/xxx" like pn.
         """
         klass = params[0]
-        if len(params) > 1:
+        resolve = False
+        if len(params) > 2:
+            pkg_list = params[1]
+            resolve = params[2]
+        elif len(params) > 1:
             pkg_list = params[1]
         else:
             pkg_list = []
 
-        command.cooker.generateTargetsTree(klass, pkg_list)
+        command.cooker.generateTargetsTree(klass, pkg_list, resolve)
         command.finishAsyncCommand()
     generateTargetsTree.needcache = True
 
lib/bb/cooker.py +47 -11

111 lines not on this page · raw message

Signature · 1 line
1.7.0.4
#5 lore raw

PATCH 5/16 command.py: add parseConfigurationFiles API

In reply to Dongxiao Xu

The parseConfigurationFiles API calls the related function in cooker.py to parse config files.

---
 lib/bb/command.py |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)
lib/bb/command.py +10 -0
index 701b286..6b4a598 100644
@@ -326,3 +326,13 @@ class CommandsAsync:
         else:
             command.finishAsyncCommand()
     compareRevisions.needcache = True
+
+    def parseConfigurationFiles(self, command, params):
+        """
+        Parse the configuration files
+        """
+        prefiles = params[0]
+        postfiles = params[1]
+        command.cooker.parseConfigurationFiles(prefiles, postfiles)
+        command.finishAsyncCommand()
+    parseConfigurationFiles.needcache = False
Signature · 1 line
1.7.0.4
#7 lore raw

PATCH 7/16 event.py: Add a new event PackageInfo

In reply to Dongxiao Xu

This event is to pass package information.

---
 lib/bb/event.py |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
lib/bb/event.py +9 -0
index 088cd7c..67543e7 100644
@@ -459,3 +459,12 @@ class LogHandler(logging.Handler):
     def filter(self, record):
         record.taskpid = worker_pid
         return True
+
+class PackageInfo(Event):
+    """
+    Package information for GUI
+    """
+    def __init__(self, recipe, pkginfolist):
+        Event.__init__(self)
+        self._recipe = recipe
+        self._pkginfolist = pkginfolist
Signature · 1 line
1.7.0.4
#8 lore raw

PATCH 8/16 Bitbake: change for adding progress bar in Hob2.

In reply to Dongxiao Xu

From: Shane Wang <[email protected]>

The changes include:
- Clean some events in event.py
- Fire essential events for Hob2 to handle with more information.
- knotty changes

---
 lib/bb/cache.py     |    2 +-
 lib/bb/cooker.py    |   12 +++++--
 lib/bb/event.py     |   83 ++++++++++++++++++++++++++++++++++++--------------
 lib/bb/ui/knotty.py |    5 ++-
 4 files changed, 73 insertions(+), 29 deletions(-)
lib/bb/cache.py +1 -1
index 955b6df..c268fe6 100644
@@ -345,7 +345,7 @@ class Cache(object):
                         current_percent = 100 * current_progress / cachesize
                         if current_percent > previous_percent:
                             previous_percent = current_percent
-                            bb.event.fire(bb.event.CacheLoadProgress(current_progress),
+                            bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize),
                                           self.data)
 
                     previous_progress += current_progress
lib/bb/cooker.py +8 -4
index 1e944b8..b495eef 100644
@@ -344,6 +344,7 @@ class BBCooker:
         """
         Prepare a runqueue and taskdata object for iteration over pkgs_to_build
         """
+        bb.event.fire(bb.event.TreeDataPreparationStarted(), self.configuration.data)
         # Need files parsed
         self.updateCache()
         # If we are told to do the None task then query the default task
@@ -360,11 +361,14 @@ class BBCooker:
         taskdata = bb.taskdata.TaskData(False, skiplist=self.skiplist)
 
         runlist = []
+        current = 0
         for k in pkgs_to_build:
             taskdata.add_provider(localdata, self.status, k)
             runlist.append([k, "do_%s" % task])
+            current += 1
+            bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(pkgs_to_build)), self.configuration.data)
         taskdata.add_unresolved(localdata, self.status)
-
+        bb.event.fire(bb.event.TreeDataPreparationCompleted(len(pkgs_to_build)), self.configuration.data)
         return runlist, taskdata
 
     def generateTaskDepTreeData(self, pkgs_to_build, task):
@@ -1093,7 +1097,7 @@ class BBCooker:
                 return False
 
             if not retval:
-                bb.event.fire(bb.event.BuildCompleted(buildname, item, failures), self.configuration.event_data)
+                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, item, failures), self.configuration.event_data)
                 self.command.finishAsyncCommand()
                 return False
             if retval is True:
@@ -1134,7 +1138,7 @@ class BBCooker:
                 return False
 
             if not retval:
-                bb.event.fire(bb.event.BuildCompleted(buildname, targets, failures), self.configuration.data)
+                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, targets, failures), self.configuration.data)
                 self.command.finishAsyncCommand()
                 return False
             if retval is True:
@@ -1550,7 +1554,7 @@ class CookerParser(object):
         if parsed:
             self.parsed += 1
             if self.parsed % self.progress_chunk == 0:
-                bb.event.fire(bb.event.ParseProgress(self.parsed),
+                bb.event.fire(bb.event.ParseProgress(self.parsed, self.toparse),
                               self.cfgdata)
         else:
             self.cached += 1
lib/bb/event.py +60 -23

130 lines not on this page · raw message

lib/bb/ui/knotty.py +4 -1
index 38acb47..fcb6311 100644
@@ -246,7 +246,10 @@ def main(server, eventHandler):
                                   bb.event.RecipeParsed,
                                   bb.event.RecipePreFinalise,
                                   bb.runqueue.runQueueEvent,
-                                  bb.runqueue.runQueueExitWait)):
+                                  bb.runqueue.runQueueExitWait,
+                                  bb.event.OperationStarted,
+                                  bb.event.OperationCompleted,
+                                  bb.event.OperationProgress)):
                 continue
 
             logger.error("Unknown event: %s", event)
Signature · 1 line
1.7.0.4
#9 lore raw

PATCH 9/16 xmlrpc: Change BitbakeServerInfo init function

In reply to Dongxiao Xu

Pass host and port to BitbakeServerInfo class instead of the "server" instance. With this change, GUI can connect with server individually by host address and port.

---
 lib/bb/server/xmlrpc.py |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
lib/bb/server/xmlrpc.py +4 -4
index a7ac969..b5980c6 100644
@@ -242,9 +242,9 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer):
         return
 
 class BitbakeServerInfo():
-    def __init__(self, server):
-        self.host = server.host
-        self.port = server.port
+    def __init__(self, host, port):
+        self.host = host
+        self.port = port
 
 class BitBakeServerConnection():
     def __init__(self, serverinfo):
@@ -278,7 +278,7 @@ class BitBakeServer(object):
         return self.server.register_idle_function
 
     def saveConnectionDetails(self): 
-        self.serverinfo = BitbakeServerInfo(self.server)
+        self.serverinfo = BitbakeServerInfo(self.server.host, self.server.port)
 
     def detach(self, cooker_logfile):
         daemonize.createDaemon(self.server.serve_forever, cooker_logfile)
Signature · 1 line
1.7.0.4
#11 lore raw

PATCH 11/16 bitbake: add a new option "--server-only"

In reply to Dongxiao Xu

Create a new option "--server-only" for bitbake command, which allows bitbake runs as a server, and let frontend connect the server itself.

"--server-only" should work with "-t xmlrpc", or bitbake will exit.

bitbake --server-only -t xmlrpc will print out the server address and port information.

---
 bin/bitbake |   23 ++++++++++++++++-------
 1 files changed, 16 insertions(+), 7 deletions(-)
bin/bitbake +16 -7
index 9eb558b..c2e6822 100755
@@ -165,6 +165,9 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("", "--revisions-changed", help = "Set the exit code depending on whether upstream floating revisions have changed or not",
                action = "store_true", dest = "revisions_changed", default = False)
 
+    parser.add_option("", "--server-only", help = "Run bitbake without UI,  the frontend can connect with bitbake server itself",
+               action = "store_true", dest = "server_only", default = False)
+
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options)
@@ -186,6 +189,9 @@ Default BBFILES are the .bb files in the current directory.""")
         sys.exit("FATAL: Invalid server type '%s' specified.\n"
                  "Valid interfaces: xmlrpc, process [default], none." % servertype)
 
+    if configuration.server_only and configuration.servertype != "xmlrpc":
+        sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+
     # Save a logfile for cooker into the current working directory. When the
     # server is daemonized this logfile will be truncated.
     cooker_logfile = os.path.join(os.getcwd(), "cooker.log")
@@ -222,14 +228,17 @@ Default BBFILES are the .bb files in the current directory.""")
 
     logger.removeHandler(handler)
 
-    # Setup a connection to the server (cooker)
-    server_connection = server.establishConnection()
+    if not configuration.server_only:
+        # Setup a connection to the server (cooker)
+        server_connection = server.establishConnection()
 
-    try:
-        return server.launchUI(ui_main, server_connection.connection, server_connection.events)
-    finally:
-        bb.event.ui_queue = []
-        server_connection.terminate()
+        try:
+            return server.launchUI(ui_main, server_connection.connection, server_connection.events)
+        finally:
+            bb.event.ui_queue = []
+            server_connection.terminate()
+    else:
+        print("server address: %s, server port: %s" % (server.serverinfo.host, server.serverinfo.port))
 
     return 1
 
Signature · 1 line
1.7.0.4
#10 lore raw

PATCH 10/16 cooker: remove command import in cooker.py

In reply to Dongxiao Xu

There is no direct use of command in cooker.py, and it is using bb.command instead. Remove command in the import list.

This fixes a problem of embedded import between command.py and cooker.py.

---
 lib/bb/cooker.py |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
lib/bb/cooker.py +2 -2
index b495eef..bfd61a9 100644
@@ -34,8 +34,8 @@ from cStringIO import StringIO
 from contextlib import closing
 from functools import wraps
 from collections import defaultdict
-import bb, bb.exceptions
-from bb import utils, data, parse, event, cache, providers, taskdata, command, runqueue
+import bb, bb.exceptions, bb.command
+from bb import utils, data, parse, event, cache, providers, taskdata, runqueue
 
 logger      = logging.getLogger("BitBake")
 collectlog  = logging.getLogger("BitBake.Collection")
Signature · 1 line
1.7.0.4
#12 lore raw

PATCH 12/16 bitbake: add -B option to bind with interface

In reply to Dongxiao Xu

When start bitbake as a server only process, we need to assign certain interface to it.

---
 bin/bitbake             |   17 ++++++++++++-----
 lib/bb/server/xmlrpc.py |    6 +++---
 2 files changed, 15 insertions(+), 8 deletions(-)
bin/bitbake +12 -5
index c2e6822..4bf1335 100755
@@ -168,6 +168,8 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("", "--server-only", help = "Run bitbake without UI,  the frontend can connect with bitbake server itself",
                action = "store_true", dest = "server_only", default = False)
 
+    parser.add_option("-B", "--bind", help = "Bind bitbake server with certain address",
+               action = "store", dest = "bind", default = False)
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options)
@@ -189,9 +191,6 @@ Default BBFILES are the .bb files in the current directory.""")
         sys.exit("FATAL: Invalid server type '%s' specified.\n"
                  "Valid interfaces: xmlrpc, process [default], none." % servertype)
 
-    if configuration.server_only and configuration.servertype != "xmlrpc":
-        sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
-
     # Save a logfile for cooker into the current working directory. When the
     # server is daemonized this logfile will be truncated.
     cooker_logfile = os.path.join(os.getcwd(), "cooker.log")
@@ -211,9 +210,17 @@ Default BBFILES are the .bb files in the current directory.""")
     # of the UIs (e.g. for DISPLAY, etc.)
     bb.utils.clean_environment()
 
-    server = server.BitBakeServer()
+    if configuration.server_only:
+        if configuration.servertype != "xmlrpc":
+            sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+        if not configuration.bind:
+            sys.exit("FATAL: If '--server-only' is defined, we need to bind the server with certain address'.\n")
+        server = server.BitBakeServer()
+        server.initServer((configuration.bind, 0))
+    else:
+        server = server.BitBakeServer()
+        server.initServer()
 
-    server.initServer()
     idle = server.getServerIdleCB()
 
     cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
lib/bb/server/xmlrpc.py +3 -3
index b5980c6..c53cee4 100644
@@ -163,7 +163,7 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer):
     # remove this when you're done with debugging
     # allow_reuse_address = True
 
-    def __init__(self, interface = ("localhost", 0)):
+    def __init__(self, interface):
         """
         Constructor
         """
@@ -267,8 +267,8 @@ class BitBakeServerConnection():
             pass
 
 class BitBakeServer(object):
-    def initServer(self):
-        self.server = BitBakeXMLRPCServer()
+    def initServer(self, interface = ("localhost", 0)):
+        self.server = BitBakeXMLRPCServer(interface)
 
     def addcooker(self, cooker):
         self.cooker = cooker
Signature · 1 line
1.7.0.4
#15 lore raw

PATCH 13/16 bitbake: Add client socket info for BitBakeServerConnection

In reply to Dongxiao Xu

In server/client split model, the client will bind to a specific address and port. We need to pass the values to BitBakeServerConnection().

---
 lib/bb/server/xmlrpc.py |    4 ++--
 lib/bb/ui/uievent.py    |    7 ++++---
 2 files changed, 6 insertions(+), 5 deletions(-)
lib/bb/server/xmlrpc.py +2 -2
index c53cee4..eff8009 100644
@@ -247,9 +247,9 @@ class BitbakeServerInfo():
         self.port = port
 
 class BitBakeServerConnection():
-    def __init__(self, serverinfo):
+    def __init__(self, serverinfo, clientinfo=("localhost", 0)):
         self.connection = _create_server(serverinfo.host, serverinfo.port)
-        self.events = uievent.BBUIEventQueue(self.connection)
+        self.events = uievent.BBUIEventQueue(self.connection, clientinfo)
         for event in bb.event.ui_queue:
             self.events.queue_event(event)
 
lib/bb/ui/uievent.py +4 -3
index 0e73817..28817a2 100644
@@ -28,13 +28,14 @@ import socket, threading, pickle
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 
 class BBUIEventQueue:
-    def __init__(self, BBServer):
+    def __init__(self, BBServer, clientinfo=("localhost, 0")):
 
         self.eventQueue = []
         self.eventQueueLock = threading.Lock()
         self.eventQueueNotify = threading.Event()
 
         self.BBServer = BBServer
+        self.clientinfo = clientinfo
 
         self.t = threading.Thread()
         self.t.setDaemon(True)
@@ -72,7 +73,7 @@ class BBUIEventQueue:
 
     def startCallbackHandler(self):
 
-        server = UIXMLRPCServer()
+        server = UIXMLRPCServer(self.clientinfo)
         self.host, self.port = server.socket.getsockname()
 
         server.register_function( self.system_quit, "event.quit" )
@@ -98,7 +99,7 @@ class BBUIEventQueue:
 
 class UIXMLRPCServer (SimpleXMLRPCServer):
 
-    def __init__( self, interface = ("localhost", 0) ):
+    def __init__( self, interface ):
         self.quit = False
         SimpleXMLRPCServer.__init__( self,
                                     interface,
Signature · 1 line
1.7.0.4
#16 lore raw

PATCH 14/16 command.py: add a new API findCoreBaseFiles

In reply to Dongxiao Xu

This new API is to find the available layers in server side.

---
 lib/bb/command.py |   11 +++++++++++
 lib/bb/cooker.py  |   12 ++++++++++++
 lib/bb/event.py   |    8 ++++++++
 3 files changed, 31 insertions(+), 0 deletions(-)
lib/bb/command.py +11 -0
index e52041d..141b51e 100644
@@ -245,6 +245,17 @@ class CommandsAsync:
         command.finishAsyncCommand()
     generateTargetsTree.needcache = True
 
+    def findCoreBaseFiles(self, command, params):
+        """
+        Find certain files in COREBASE directory. i.e. Layers
+        """
+        subdir = params[0]
+        filename = params[1]
+
+        command.cooker.findCoreBaseFiles(subdir, filename)
+        command.finishAsyncCommand()
+    findCoreBaseFiles.needcache = False
+
     def findConfigFiles(self, command, params):
         """
         Find config files which provide appropriate values
lib/bb/cooker.py +12 -0
index bfd61a9..b02fdd7 100644
@@ -681,6 +681,18 @@ class BBCooker:
             if regex in unmatched:
                 collectlog.warn("No bb files matched BBFILE_PATTERN_%s '%s'" % (collection, pattern))
 
+    def findCoreBaseFiles(self, subdir, configfile):
+        corebase = self.configuration.data.getVar('COREBASE', True) or ""
+        paths = []
+        for root, dirs, files in os.walk(corebase + '/' + subdir):
+            for d in dirs:
+                configfilepath = os.path.join(root, d, configfile)
+                if os.path.exists(configfilepath):
+                    paths.append(os.path.join(root, d))
+
+        if paths:
+            bb.event.fire(bb.event.CoreBaseFilesFound(paths), self.configuration.data)
+
     def findConfigFilePath(self, configfile):
         """
         Find the location on disk of configfile and if it exists and was parsed by BitBake
lib/bb/event.py +8 -0
index d120ba5..ffd4213 100644
@@ -439,6 +439,14 @@ class FilesMatchingFound(Event):
         self._pattern = pattern
         self._matches = matches
 
+class CoreBaseFilesFound(Event):
+    """
+    Event when a list of appropriate config files has been generated
+    """
+    def __init__(self, paths):
+        Event.__init__(self)
+        self._paths = paths
+
 class ConfigFilesFound(Event):
     """
     Event when a list of appropriate config files has been generated
Signature · 1 line
1.7.0.4
#17 lore raw

PATCH 15/16 command.py: add new command to get the CPU info

In reply to Dongxiao Xu

Add new API in command.py to get the CPU core and threads information in order to set the appropriate BB_NUMBER_THREADS and PARALLEL_MAKE variables.

---
 bitbake/lib/bb/helper.py |   39 +++++++++++++++++++++++++++++++++++++++
 lib/bb/command.py        |   12 ++++++++++++
 2 files changed, 51 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/helper.py
bitbake/lib/bb/helper.py new file +39 -0
new file mode 100644
index 0000000..291158b
@@ -0,0 +1,39 @@
+#
+# Helper for BitBake Graphical GTK User Interface
+#
+# Copyright (C) 2011        Intel Corporation
+#
+# Authored by Shane Wang <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+
+class CpuInfo(object):
+
+    coefficient = 4
+
+    @classmethod
+    def getNumOfCpus(cls):
+        pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores')
+        num = len(pfile.readlines())
+        return num
+
+    @classmethod
+    def getNumOfCpuCores(cls):
+        pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores | cut -d: -f2')
+        contents = pfile.readlines()
+        num = int(contents[0])
+        return num
+
lib/bb/command.py +12 -0
index 141b51e..ba7c753 100644
@@ -30,6 +30,7 @@ Commands are queued in a CommandQueue
 
 import bb.event
 import bb.cooker
+import bb.helper
 
 class CommandCompleted(bb.event.Event):
     pass
@@ -170,6 +171,17 @@ class CommandsSync:
         """
         command.cooker.reset()
 
+    def getDefaultNumOfThreads(self, command, params):
+        """
+        Get the default number of threads on the server = number of CPUs
+        """
+        return bb.helper.CpuInfo.getNumOfCpus()
+
+    def getMaxNumOfThreads(self, command, params):
+        """
+        Get the max number of threads that the server can tolerate
+        """
+        return bb.helper.CpuInfo.getNumOfCpus() * bb.helper.CpuInfo.coefficient
 
 class CommandsAsync:
     """
Signature · 1 line
1.7.0.4
#18 lore raw

PATCH 16/16 runqueue: fire sceneQueueTaskStarted event when a setscene queue starts

In reply to Dongxiao Xu

Fire a sceneQueueTaskStarted event to notify the setscene tasks progress.

---
 lib/bb/event.py    |    2 +-
 lib/bb/runqueue.py |   10 ++++++++++
 2 files changed, 11 insertions(+), 1 deletions(-)
lib/bb/event.py +1 -1
index ffd4213..10de984 100644
@@ -223,7 +223,7 @@ class OperationProgress(Event):
         Event.__init__(self)
         self.current = current
         self.total = total
-        self.msg = msg + ": %s/%s (%.0f%%)" % (current, total, (current*1.0/total)*100);
+        self.msg = msg + ": %s/%s" % (current, total);
 
 class ConfigParsed(Event):
     """Configuration Parsing Complete"""
lib/bb/runqueue.py +10 -0
index 0e83d05..0ba414f 100644
@@ -1619,6 +1619,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                 self.task_skip(task)
                 return True
 
+            startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
+            bb.event.fire(startevent, self.cfgData)
+
             pid, pipein, pipeout = self.fork_off_task(fn, realtask, taskname)
 
             self.build_pids[pid] = task
@@ -1686,6 +1689,13 @@ class runQueueTaskStarted(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         self.noexec = noexec
 
+class sceneQueueTaskStarted(runQueueTaskStarted):
+    """
+    Event notifing a setscene task was started
+    """
+    def __init__(self, task, stats, rq, noexec=False):
+        runQueueTaskStarted.__init__(self, task, stats, rq, noexec)
+
 class runQueueTaskFailed(runQueueEvent):
     """
     Event notifing a task failed
Signature · 1 line
1.7.0.4
#13 lore raw

Re: [PATCH 12/16] bitbake: add -B option to bind with interface

In reply to Dongxiao Xu
Quoted 14 lines · Dongxiao Xu

When start bitbake as a server only process, we need to assign certain interface to it.

---
 bin/bitbake             |   17 ++++++++++++-----
 lib/bb/server/xmlrpc.py |    6 +++---
 2 files changed, 15 insertions(+), 8 deletions(-)
bin/bitbake +0 -0
index c2e6822..4bf1335 100755

bin/bitbake Default BBFILES are the .bb files in the current directory.""") · line 171

Quoted 4 lines
bin/bitbake +1 -0
@@ -168,6 +168,8 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("", "--server-only", help = "Run bitbake without UI,  the frontend can connect with bitbake server itself",
                action = "store_true", dest = "server_only", default = False)
 
+    parser.add_option("-B", "--bind", help = "Bind bitbake server with certain address",

I find these help messages confusing. How about "The name/address for the bitbake server to bind to"?

Quoted 15 lines
bin/bitbake +1 -3
+               action = "store", dest = "bind", default = False)
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options)
@@ -189,9 +191,6 @@ Default BBFILES are the .bb files in the current directory.""")
         sys.exit("FATAL: Invalid server type '%s' specified.\n"
                  "Valid interfaces: xmlrpc, process [default], none." % servertype)
 
-    if configuration.server_only and configuration.servertype != "xmlrpc":
-        sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
-
     # Save a logfile for cooker into the current working directory. When the
     # server is daemonized this logfile will be truncated.
     cooker_logfile = os.path.join(os.getcwd(), "cooker.log")

bin/bitbake Default BBFILES are the .bb files in the current directory.""") · line 217

Quoted 9 lines
bin/bitbake +5 -1
@@ -211,9 +210,17 @@ Default BBFILES are the .bb files in the current directory.""")
     # of the UIs (e.g. for DISPLAY, etc.)
     bb.utils.clean_environment()
 
-    server = server.BitBakeServer()
+    if configuration.server_only:
+        if configuration.servertype != "xmlrpc":
+            sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+        if not configuration.bind:
+            sys.exit("FATAL: If '--server-only' is defined, we need to bind the server with certain address'.\n")

How about "FATAL: The '--server-only' option requires a name/address to bind to with the --bind option"?

To be honest, I'd also expect --bind to work without --server-only, defaulting to "localhost" if it wasn't set but using the address/name specified if it was. The usecase would be to attach a secondary UI to the bitbake server during the build process for monitoring/control purposes.

Cheers,

Richard

#14 lore raw

Re: [PATCH 12/16] bitbake: add -B option to bind with interface

Quoted 25 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Friday, January 06, 2012 11:51 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 12/16] bitbake: add -B option to bind with interface

bin/bitbake Default BBFILES are the .bb files in the current

Quoted 13 lines · Dongxiao Xu

When start bitbake as a server only process, we need to assign certain interface to it.

---
 bin/bitbake             |   17 ++++++++++++-----
 lib/bb/server/xmlrpc.py |    6 +++---
 2 files changed, 15 insertions(+), 8 deletions(-)
bin/bitbake +0 -0
@@ -168,6 +168,8 @@ Default BBFILES are the .bb files in the current

directory.""")

bin/bitbake Default BBFILES are the .bb files in the current · line 168

Quoted 1 line
bin/bitbake +0 -0
     parser.add_option("", "--server-only", help = "Run bitbake without UI,

the frontend can connect with bitbake server itself",

Quoted 8 lines

bin/bitbake Default BBFILES are the .bb files in the current · line 173

Quoted 5 lines
bin/bitbake +2 -0
                action = "store_true", dest = "server_only", default =
 False)
 
+    parser.add_option("-B", "--bind", help = "Bind bitbake server
+ with certain address",

I find these help messages confusing. How about "The name/address for the bitbake server to bind to"?

Yes, I think your suggestion is better.

Quoted 31 lines

bin/bitbake

Quoted 12 lines
bin/bitbake +1 -0
+               action = "store", dest = "bind", default = False)
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options) @@ -189,9 +191,6 @@

Default BBFILES are the .bb files in the current directory.""")
sys.exit("FATAL: Invalid server type '%s' specified.\n"
"Valid interfaces: xmlrpc, process [default], none."
% servertype)

bin/bitbake +0 -2
-    if configuration.server_only and configuration.servertype != "xmlrpc":
-        sys.exit("FATAL: If '--server-only' is defined, we must set the

servertype as 'xmlrpc'.\n")

Quoted 5 lines

-
# Save a logfile for cooker into the current working directory. When the # server is daemonized this logfile will be truncated. cooker_logfile = os.path.join(os.getcwd(), "cooker.log") @@
-211,9 +210,17 @@ Default BBFILES are the .bb files in the current

directory.""")

bin/bitbake Default BBFILES are the .bb files in the current directory.""")

Quoted 7 lines
     # of the UIs (e.g. for DISPLAY, etc.)
     bb.utils.clean_environment()
bin/bitbake +3 -1
-    server = server.BitBakeServer()
+    if configuration.server_only:
+        if configuration.servertype != "xmlrpc":
+            sys.exit("FATAL: If '--server-only' is defined, we must set the

servertype as 'xmlrpc'.\n")

bin/bitbake

Quoted 3 lines
bin/bitbake +3 -0
+        if not configuration.bind:
+            sys.exit("FATAL: If '--server-only' is defined, we need
+ to bind the server with certain address'.\n")
Quoted 7 lines

How about "FATAL: The '--server-only' option requires a name/address to bind to with the --bind option"?

To be honest, I'd also expect --bind to work without --server-only, defaulting to "localhost" if it wasn't set but using the address/name specified if it was. The usecase would be to attach a secondary UI to the bitbake server during the build process for monitoring/control purposes.

Yes, I can remove that limitation.

I Will revise this patch and send out again.

Thanks,
Dongxiao

Quoted 6 lines

Cheers,

Richard