Hob: bitbake related changes
Hi Richard,
This is the second pull request for Hob bitbake related changes.
Please help to reviewand pull.
Changes from v1:
1. Fixed a cache issue.
If we run "source oe-init-env-setup", it will force to add POKY/bitbake/bin/ and POKY/scripts into PATH variable, which changes the configuration hash value. We handled this situation in the new code.
2. event.py: Change the parameter of PackageInfo events.
We have a new approach to get all the available packages in the build repo,
and here we change the way to pass the information.
3. Remove the binding limitation of --server-only and -B.
Thanks,
Dongxiao
The following changes since commit eac9249b40ae1e3aa21e016010c862664e59a8d4:
Richard Purdie (1):
bitbake: Add BBHandledException exception class
are available in the git repository at:
git://git.pokylinux.org/poky-contrib dxu4/hob-bitbake-changes-v2
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/hob-bitbake-changes-v2
Dongxiao Xu (7):
cache: Use configuration's hash value to validate cache command.py: add resolve option for generateTargetsTree API event.py: Add a new event PackageInfo
bitbake: add -B option to bind with interface
bitbake: Add client socket info for BitBakeServerConnection command.py: add new command to get the CPU info
runqueue: fire sceneQueueTaskStarted event when a setscene queue
starts
Shane Wang (1):
Bitbake: change for adding progress bar in Hob2.
bin/bitbake | 15 +++++--- bitbake/lib/bb/helper.py | 39 ++++++++++++++++++++ lib/bb/cache.py | 34 +++++++----------- lib/bb/command.py | 23 +++++++++++- lib/bb/cooker.py | 74 +++++++++++++++++++++++++++++-------- lib/bb/data_smart.py | 21 +++++++++++ lib/bb/event.py | 91 ++++++++++++++++++++++++++++++++++------------ lib/bb/runqueue.py | 10 +++++ lib/bb/server/xmlrpc.py | 10 +++--- lib/bb/ui/knotty.py | 5 ++- lib/bb/ui/uievent.py | 7 ++-- 11 files changed, 253 insertions(+), 76 deletions(-) create mode 100644 bitbake/lib/bb/helper.py
PATCH 1/8 cache: Use configuration's hash value to validate cache
Previously we use the file time stamp to judge if a cache is valid. Here this commit introduce a new method, which calculates the total hash value for a certain configuration's key/value paris, and tag it into cache filename, for example, bb_cache.dat.xxxyyyzzz.
This mechanism also ensures the cache's correctness if user dynamically setting variables from some frontend GUI, like HOB.
--- lib/bb/cache.py | 32 ++++++++++++-------------------- lib/bb/cooker.py | 4 +++- lib/bb/data_smart.py | 21 +++++++++++++++++++++ 3 files changed, 36 insertions(+), 21 deletions(-)
lib/bb/cache.py +12 -20
@@ -42,10 +42,10 @@ except ImportError: logger.info("Importing cPickle failed. " "Falling back to a very slow implementation.") -__cache_version__ = "142" +__cache_version__ = "143" -def getCacheFile(path, filename): - return os.path.join(path, filename) +def getCacheFile(path, filename, data_hash): + return os.path.join(path, filename + "." + data_hash) # RecipeInfoCommon defines common data retrieving methods # from meta data for caches. CoreRecipeInfo as well as other
@@ -254,7 +254,7 @@ class Cache(object): BitBake Cache implementation """ - def __init__(self, data, caches_array): + def __init__(self, data, data_hash, caches_array): # Pass caches_array information into Cache Constructor # It will be used in later for deciding whether we # need extra cache file dump/load support
@@ -266,6 +266,7 @@ class Cache(object): self.data = None self.data_fn = None self.cacheclean = True + self.data_hash = data_hash if self.cachedir in [None, '']: self.has_cache = False
@@ -274,26 +275,17 @@ class Cache(object): return self.has_cache = True - self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat") + self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash) logger.debug(1, "Using cache in '%s'", self.cachedir) bb.utils.mkdirhier(self.cachedir) - # If any of configuration.data's dependencies are newer than the - # cache there isn't even any point in loading it... - newest_mtime = 0 - deps = data.getVar("__base_depends") - - old_mtimes = [old_mtime for _, old_mtime in deps] - old_mtimes.append(newest_mtime) - newest_mtime = max(old_mtimes) - cache_ok = True if self.caches_array: for cache_class in self.caches_array: if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon): - cachefile = getCacheFile(self.cachedir, cache_class.cachefile) - cache_ok = cache_ok and (bb.parse.cached_mtime_noerror(cachefile) >= newest_mtime) + cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash) + cache_ok = cache_ok and os.path.exists(cachefile) cache_class.init_cacheData(self) if cache_ok: self.load_cachefile()
@@ -327,7 +319,7 @@ class Cache(object): # Calculate the correct cachesize of all those cache files for cache_class in self.caches_array: if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon): - cachefile = getCacheFile(self.cachedir, cache_class.cachefile) + cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash) with open(cachefile, "rb") as cachefile: cachesize += os.fstat(cachefile.fileno()).st_size
@@ -335,7 +327,7 @@ class Cache(object): for cache_class in self.caches_array: if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon): - cachefile = getCacheFile(self.cachedir, cache_class.cachefile) + cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash) with open(cachefile, "rb") as cachefile: pickled = pickle.Unpickler(cachefile) while cachefile:
@@ -588,7 +580,7 @@ class Cache(object): for cache_class in self.caches_array: if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon): cache_class_name = cache_class.__name__ - cachefile = getCacheFile(self.cachedir, cache_class.cachefile) + cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash) file_dict[cache_class_name] = open(cachefile, "wb") pickler_dict[cache_class_name] = pickle.Pickler(file_dict[cache_class_name], pickle.HIGHEST_PROTOCOL)
@@ -693,7 +685,7 @@ def init(cooker): Files causing parsing errors are evicted from the cache. """ - return Cache(cooker.configuration.data) + return Cache(cooker.configuration.data, cooker.configuration.data_hash) class CacheData(object):
lib/bb/cooker.py +3 -1
@@ -857,6 +857,7 @@ class BBCooker: bb.parse.init_parser(data) bb.event.fire(bb.event.ConfigParsed(), data) self.configuration.data = data + self.configuration.data_hash = data.get_hash() def handleCollections( self, collections ): """Handle collections"""
@@ -1428,6 +1429,7 @@ class CookerParser(object): self.filelist = filelist self.cooker = cooker self.cfgdata = cooker.configuration.data + self.cfghash = cooker.configuration.data_hash # Accounting statistics self.parsed = 0
@@ -1443,7 +1445,7 @@ class CookerParser(object): self.num_processes = int(self.cfgdata.getVar("BB_NUMBER_PARSE_THREADS", True) or multiprocessing.cpu_count()) - self.bb_cache = bb.cache.Cache(self.cfgdata, cooker.caches_array) + self.bb_cache = bb.cache.Cache(self.cfgdata, self.cfghash, cooker.caches_array) self.fromcache = [] self.willparse = [] for filename in self.filelist:
lib/bb/data_smart.py +21 -0
@@ -31,6 +31,7 @@ BitBake build tools. import copy, re from collections import MutableMapping import logging +import hashlib import bb, bb.codeparser from bb import utils from bb.COW import COWDictBase
@@ -459,3 +460,23 @@ class DataSmart(MutableMapping): def __delitem__(self, var): self.delVar(var) + + def get_hash(self): + data = "" + keys = iter(self) + for key in keys: + if key == "TIME": + continue + if key == "__depends": + deps = list(self.getVar(key, False)) + deps.sort() + value = [deps[i][0] for i in range(len(deps))] + elif key == "PATH": + path = list(set(self.getVar(key, False).split(':'))) + path.sort() + value = " ".join(path) + else: + value = self.getVar(key, False) or "" + data = data + key + ': ' + str(value) + '\n' + + return hashlib.md5(data).hexdigest()
Signature · 1 line
1.7.0.4
PATCH 2/8 command.py: add resolve option for generateTargetsTree API
In reply to Dongxiao XuCurrently we have generateTargetsTree API, which is used to get dependency information. However in that tree, there will be "virtual/xxx" in depends fields. Therefore we add the resolve option to replace it with its real providers.
Besides, for packages that provided by multiple recipes, we will find their preverred provider.
--- lib/bb/command.py | 11 ++++++++- lib/bb/cooker.py | 58 ++++++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 56 insertions(+), 13 deletions(-)
lib/bb/command.py +9 -2
@@ -230,14 +230,21 @@ class CommandsAsync: included in the package list. If pkg_list provided use that list (plus any extras brought in by klass) rather than generating a tree for all packages. + + Add a new option "resolve" to indicate if we need to resolve the + replacement for "virtual/xxx" like pn. """ klass = params[0] - if len(params) > 1: + resolve = False + if len(params) > 2: + pkg_list = params[1] + resolve = params[2] + elif len(params) > 1: pkg_list = params[1] else: pkg_list = [] - command.cooker.generateTargetsTree(klass, pkg_list) + command.cooker.generateTargetsTree(klass, pkg_list, resolve) command.finishAsyncCommand() generateTargetsTree.needcache = True
lib/bb/cooker.py +47 -11
111 lines not on this page · raw message
Signature · 1 line
1.7.0.4
PATCH 3/8 event.py: Add a new event PackageInfo
In reply to Dongxiao XuThis event is to pass package information.
--- lib/bb/event.py | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
lib/bb/event.py +8 -0
@@ -467,3 +467,11 @@ class LogHandler(logging.Handler): def filter(self, record): record.taskpid = worker_pid return True + +class PackageInfo(Event): + """ + Package information for GUI + """ + def __init__(self, pkginfolist): + Event.__init__(self) + self._pkginfolist = pkginfolist
Signature · 1 line
1.7.0.4
PATCH 4/8 Bitbake: change for adding progress bar in Hob2.
In reply to Dongxiao XuFrom: Shane Wang <[email protected]>
The changes include:
- Clean some events in event.py
- Fire essential events for Hob2 to handle with more information.
- knotty changes
--- lib/bb/cache.py | 2 +- lib/bb/cooker.py | 12 +++++-- lib/bb/event.py | 83 ++++++++++++++++++++++++++++++++++++-------------- lib/bb/ui/knotty.py | 5 ++- 4 files changed, 73 insertions(+), 29 deletions(-)
lib/bb/cache.py +1 -1
@@ -345,7 +345,7 @@ class Cache(object): current_percent = 100 * current_progress / cachesize if current_percent > previous_percent: previous_percent = current_percent - bb.event.fire(bb.event.CacheLoadProgress(current_progress), + bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), self.data) previous_progress += current_progress
lib/bb/cooker.py +8 -4
@@ -344,6 +344,7 @@ class BBCooker: """ Prepare a runqueue and taskdata object for iteration over pkgs_to_build """ + bb.event.fire(bb.event.TreeDataPreparationStarted(), self.configuration.data) # Need files parsed self.updateCache() # If we are told to do the None task then query the default task
@@ -360,11 +361,14 @@ class BBCooker: taskdata = bb.taskdata.TaskData(False, skiplist=self.skiplist) runlist = [] + current = 0 for k in pkgs_to_build: taskdata.add_provider(localdata, self.status, k) runlist.append([k, "do_%s" % task]) + current += 1 + bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(pkgs_to_build)), self.configuration.data) taskdata.add_unresolved(localdata, self.status) - + bb.event.fire(bb.event.TreeDataPreparationCompleted(len(pkgs_to_build)), self.configuration.data) return runlist, taskdata def generateTaskDepTreeData(self, pkgs_to_build, task):
@@ -1105,7 +1109,7 @@ class BBCooker: return False if not retval: - bb.event.fire(bb.event.BuildCompleted(buildname, item, failures), self.configuration.event_data) + bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, item, failures), self.configuration.event_data) self.command.finishAsyncCommand() return False if retval is True:
@@ -1146,7 +1150,7 @@ class BBCooker: return False if not retval: - bb.event.fire(bb.event.BuildCompleted(buildname, targets, failures), self.configuration.data) + bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, targets, failures), self.configuration.data) self.command.finishAsyncCommand() return False if retval is True:
@@ -1562,7 +1566,7 @@ class CookerParser(object): if parsed: self.parsed += 1 if self.parsed % self.progress_chunk == 0: - bb.event.fire(bb.event.ParseProgress(self.parsed), + bb.event.fire(bb.event.ParseProgress(self.parsed, self.toparse), self.cfgdata) else: self.cached += 1
lib/bb/event.py +60 -23
130 lines not on this page · raw message
lib/bb/ui/knotty.py +4 -1
@@ -251,7 +251,10 @@ def main(server, eventHandler): bb.event.RecipeParsed, bb.event.RecipePreFinalise, bb.runqueue.runQueueEvent, - bb.runqueue.runQueueExitWait)): + bb.runqueue.runQueueExitWait, + bb.event.OperationStarted, + bb.event.OperationCompleted, + bb.event.OperationProgress)): continue logger.error("Unknown event: %s", event)
Signature · 1 line
1.7.0.4
PATCH 5/8 bitbake: add -B option to bind with interface
In reply to Dongxiao XuWhen start bitbake as a server only process, we need to assign certain interface to it.
--- bin/bitbake | 15 ++++++++++----- lib/bb/server/xmlrpc.py | 6 +++--- 2 files changed, 13 insertions(+), 8 deletions(-)
bin/bitbake +10 -5
@@ -168,6 +168,8 @@ Default BBFILES are the .bb files in the current directory.""") parser.add_option("", "--server-only", help = "Run bitbake without UI, the frontend can connect with bitbake server itself", action = "store_true", dest = "server_only", default = False) + parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to", + action = "store", dest = "bind", default = False) options, args = parser.parse_args(sys.argv) configuration = BBConfiguration(options)
@@ -189,9 +191,6 @@ Default BBFILES are the .bb files in the current directory.""") sys.exit("FATAL: Invalid server type '%s' specified.\n" "Valid interfaces: xmlrpc, process [default], none." % servertype) - if configuration.server_only and configuration.servertype != "xmlrpc": - sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n") - # Save a logfile for cooker into the current working directory. When the # server is daemonized this logfile will be truncated. cooker_logfile = os.path.join(os.getcwd(), "cooker.log")
@@ -211,9 +210,15 @@ Default BBFILES are the .bb files in the current directory.""") # of the UIs (e.g. for DISPLAY, etc.) bb.utils.clean_environment() - server = server.BitBakeServer() + if configuration.server_only: + if configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n") + server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + else: + server = server.BitBakeServer() + server.initServer() - server.initServer() idle = server.getServerIdleCB() cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
lib/bb/server/xmlrpc.py +3 -3
@@ -163,7 +163,7 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer): # remove this when you're done with debugging # allow_reuse_address = True - def __init__(self, interface = ("localhost", 0)): + def __init__(self, interface): """ Constructor """
@@ -267,8 +267,8 @@ class BitBakeServerConnection(): pass class BitBakeServer(object): - def initServer(self): - self.server = BitBakeXMLRPCServer() + def initServer(self, interface = ("localhost", 0)): + self.server = BitBakeXMLRPCServer(interface) def addcooker(self, cooker): self.cooker = cooker
Signature · 1 line
1.7.0.4
Re: [PATCH 5/8] bitbake: add -B option to bind with interface
Quoted 35 lines · Dongxiao Xu
When start bitbake as a server only process, we need to assign certain interface to it.
Signed-off-by Dongxiao Xu <[email protected]>--- bin/bitbake | 15 ++++++++++----- lib/bb/server/xmlrpc.py | 6 +++--- 2 files changed, 13 insertions(+), 8 deletions(-)bin/bitbake +2
-3@@ -168,6 +168,8 @@ Default BBFILES are the .bb files in the current directory.""") parser.add_option("", "--server-only", help = "Run bitbake without UI, the frontend can connect with bitbake server itself", action = "store_true", dest = "server_only", default = False) + parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to", + action = "store", dest = "bind", default = False) options, args = parser.parse_args(sys.argv) configuration = BBConfiguration(options)@@ -189,9 +191,6 @@ Default BBFILES are the .bb files in the current directory.""") sys.exit("FATAL: Invalid server type '%s' specified.\n" "Valid interfaces: xmlrpc, process [default], none." % servertype) - if configuration.server_only and configuration.servertype != "xmlrpc": - sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n") - # Save a logfile for cooker into the current working directory. When the # server is daemonized this logfile will be truncated. cooker_logfile = os.path.join(os.getcwd(), "cooker.log")@@ -211,9 +210,15 @@ Default BBFILES are the .bb files in the current directory.""") # of the UIs (e.g. for DISPLAY, etc.) bb.utils.clean_environment()
bin/bitbake Default BBFILES are the .bb files in the current directory.""") · line 220
Quoted 10 lines
bin/bitbake +8
-1- server = server.BitBakeServer() + if configuration.server_only: + if configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n") + server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + else: + server = server.BitBakeServer() + server.initServer()
This looks like it will silently ignore configuration.bind in the non server-only case. Don't you need something like:
+ if configuration.server_only:
+ if configuration.servertype != "xmlrpc":
+ sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+ server = server.BitBakeServer()
+ server.initServer((configuration.bind, 0))
+ elif configuration.bind:
+ server = server.BitBakeServer()
+ server.initServer((configuration.bind, 0))
+ else:
+ server = server.BitBakeServer()
+ server.initServer()
?
Cheers,
Richard
Re: [PATCH 5/8] bitbake: add -B option to bind with interface
Hi Richard,
Quoted 79 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Wednesday, January 11, 2012 7:38 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 5/8] bitbake: add -B option to bind with interfacebin/bitbake Default BBFILES are the .bb files in the current
Quoted 13 lines · Dongxiao Xu
When start bitbake as a server only process, we need to assign certain interface to it.
Signed-off-by Dongxiao Xu <[email protected]>--- bin/bitbake | 15 ++++++++++----- lib/bb/server/xmlrpc.py | 6 +++--- 2 files changed, 13 insertions(+), 8 deletions(-)bin/bitbake +0
-0@@ -168,6 +168,8 @@ Default BBFILES are the .bb files in the currentdirectory.""")
bin/bitbake Default BBFILES are the .bb files in the current · line 168
Quoted 1 line
bin/bitbake +0
-0parser.add_option("", "--server-only", help = "Run bitbake without UI,the frontend can connect with bitbake server itself",
bin/bitbake Default BBFILES are the .bb files in the current · line 172
Quoted 4 lines
bin/bitbake +1
-0action = "store_true", dest = "server_only", default = False) + parser.add_option("-B", "--bind", help = "The name/address for thebitbake server to bind to",
bin/bitbake
Quoted 11 lines
bin/bitbake +1
-0+ action = "store", dest = "bind", default = False) options, args = parser.parse_args(sys.argv)configuration = BBConfiguration(options) @@ -189,9 +191,6 @@Default BBFILES are the .bb files in the current directory.""")
sys.exit("FATAL: Invalid server type '%s' specified.\n"
"Valid interfaces: xmlrpc, process [default], none."
% servertype)bin/bitbake +0
-2- if configuration.server_only and configuration.servertype != "xmlrpc": - sys.exit("FATAL: If '--server-only' is defined, we must set theservertype as 'xmlrpc'.\n")
Quoted 5 lines
-
# Save a logfile for cooker into the current working directory. When the # server is daemonized this logfile will be truncated.cooker_logfile=os.path.join(os.getcwd(), "cooker.log")@@
-211,9 +210,15 @@ Default BBFILES are the .bb files in the currentdirectory.""")
bin/bitbake Default BBFILES are the .bb files in the current directory.""")
Quoted 7 lines
# of the UIs (e.g. for DISPLAY, etc.) bb.utils.clean_environment()bin/bitbake +3
-1- server = server.BitBakeServer() + if configuration.server_only: + if configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '--server-only' is defined, we must set theservertype as 'xmlrpc'.\n")
bin/bitbake Default BBFILES are the .bb files in the current directory.""") · line 220
Quoted 5 lines
bin/bitbake +5
-0+ server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + else: + server = server.BitBakeServer() + server.initServer()This looks like it will silently ignore configuration.bind in the non server-only case. Don't you need something like:
bin/bitbake +3
-0+ if configuration.server_only: + if configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '--server-only' is defined, we must set theservertype as 'xmlrpc'.\n")
bin/bitbake +8
-0+ server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + elif configuration.bind: + server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + else: + server = server.BitBakeServer() + server.initServer()
Quoted 1 line
?
I had a thought of this code piece, we may need the following code. For --server-only option, it requires --bind, while not vice versa.
@@ -189,8 +191,14 @@ Default BBFILES are the .bb files in the current directory.""")
sys.exit("FATAL: Invalid server type '%s' specified.\n"
"Valid interfaces: xmlrpc, process [default], none." % servertype)
- if configuration.server_only and configuration.servertype != "xmlrpc":
- sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+ if configuration.server_only:
+ if configuration.servertype != "xmlrpc":
+ sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+ if not configuration.bind:
+ sys.exit("FATAL: The '--server-only' option requires a name/address to bind to with the -B option.\n")
+
+ if configuration.bind and configuration.servertype != "xmlrpc":
+ sys.exit("FATAL: If '-B' or '--bind' is defined, we must set the servertype as 'xmlrpc'.\n")
# Save a logfile for cooker into the current working directory. When the # server is daemonized this logfile will be truncated.
@@ -212,8 +220,11 @@ Default BBFILES are the .bb files in the current directory.""")
bb.utils.clean_environment()
server = server.BitBakeServer()
+ if configuration.bind:
+ server.initServer((configuration.bind, 0))
+ else:
+ server.initServer()
- server.initServer()
idle = server.getServerIdleCB()
cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
Quoted 5 lines
Cheers,
Richard
Re: [PATCH 5/8] bitbake: add -B option to bind with interface
Quoted 44 lines · Dongxiao
Quoted 36 lines
Quoted 4 lines
# Save a logfile for cooker into the current working directory. When the # server is daemonized this logfile will be truncated.
cooker_logfile=os.path.join(os.getcwd(), "cooker.log")@@
-211,9 +210,15 @@ Default BBFILES are the .bb files in the currentdirectory.""")
bin/bitbake Default BBFILES are the .bb files in the current directory.""")
Quoted 7 lines
# of the UIs (e.g. for DISPLAY, etc.) bb.utils.clean_environment()bin/bitbake +3
-1- server = server.BitBakeServer() + if configuration.server_only: + if configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '--server-only' is defined, we must set theservertype as 'xmlrpc'.\n")
bin/bitbake Default BBFILES are the .bb files in the current directory.""") · line 220
Quoted 5 lines
bin/bitbake +5
-0+ server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + else: + server = server.BitBakeServer() + server.initServer()This looks like it will silently ignore configuration.bind in the non server-only case. Don't you need something like:
bin/bitbake +3
-0+ if configuration.server_only: + if configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '--server-only' is defined, we must set theservertype as 'xmlrpc'.\n")
bin/bitbake +8
-0+ server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + elif configuration.bind: + server = server.BitBakeServer() + server.initServer((configuration.bind, 0)) + else: + server = server.BitBakeServer() + server.initServer()Quoted 1 line
?
I had a thought of this code piece, we may need the following code. For --server-only option, it requires --bind, while not vice versa.
(quoted diff) +0
-0@@ -189,8 +191,14 @@ Default BBFILES are the .bb files in the current directory.""") sys.exit("FATAL: Invalid server type '%s' specified.\n" "Valid interfaces: xmlrpc, process [default], none." % servertype)
Review Default BBFILES are the .bb files in the current directory.""") · line 201
Quoted 10 lines
(quoted diff) +8
-2- if configuration.server_only and configuration.servertype != "xmlrpc": - sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n") + if configuration.server_only: + if configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n") + if not configuration.bind: + sys.exit("FATAL: The '--server-only' option requires a name/address to bind to with the -B option.\n") + + if configuration.bind and configuration.servertype != "xmlrpc": + sys.exit("FATAL: If '-B' or '--bind' is defined, we must set the servertype as 'xmlrpc'.\n")
That looks reasonable to me.
Cheers,
Richard
PATCH 6/8 bitbake: Add client socket info for BitBakeServerConnection
In reply to Dongxiao XuIn server/client split model, the client will bind to a specific address and port. We need to pass the values to BitBakeServerConnection().
--- lib/bb/server/xmlrpc.py | 4 ++-- lib/bb/ui/uievent.py | 7 ++++--- 2 files changed, 6 insertions(+), 5 deletions(-)
lib/bb/server/xmlrpc.py +2 -2
@@ -247,9 +247,9 @@ class BitbakeServerInfo(): self.port = port class BitBakeServerConnection(): - def __init__(self, serverinfo): + def __init__(self, serverinfo, clientinfo=("localhost", 0)): self.connection = _create_server(serverinfo.host, serverinfo.port) - self.events = uievent.BBUIEventQueue(self.connection) + self.events = uievent.BBUIEventQueue(self.connection, clientinfo) for event in bb.event.ui_queue: self.events.queue_event(event)
lib/bb/ui/uievent.py +4 -3
@@ -28,13 +28,14 @@ import socket, threading, pickle from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler class BBUIEventQueue: - def __init__(self, BBServer): + def __init__(self, BBServer, clientinfo=("localhost, 0")): self.eventQueue = [] self.eventQueueLock = threading.Lock() self.eventQueueNotify = threading.Event() self.BBServer = BBServer + self.clientinfo = clientinfo self.t = threading.Thread() self.t.setDaemon(True)
@@ -72,7 +73,7 @@ class BBUIEventQueue: def startCallbackHandler(self): - server = UIXMLRPCServer() + server = UIXMLRPCServer(self.clientinfo) self.host, self.port = server.socket.getsockname() server.register_function( self.system_quit, "event.quit" )
@@ -98,7 +99,7 @@ class BBUIEventQueue: class UIXMLRPCServer (SimpleXMLRPCServer): - def __init__( self, interface = ("localhost", 0) ): + def __init__( self, interface ): self.quit = False SimpleXMLRPCServer.__init__( self, interface,
Signature · 1 line
1.7.0.4
PATCH 7/8 command.py: add new command to get the CPU info
In reply to Dongxiao XuAdd new API in command.py to get the CPU core and threads information in order to set the appropriate BB_NUMBER_THREADS and PARALLEL_MAKE variables.
--- bitbake/lib/bb/helper.py | 39 +++++++++++++++++++++++++++++++++++++++ lib/bb/command.py | 12 ++++++++++++ 2 files changed, 51 insertions(+), 0 deletions(-) create mode 100644 bitbake/lib/bb/helper.py
bitbake/lib/bb/helper.py new file +39 -0
@@ -0,0 +1,39 @@ +# +# Helper for BitBake Graphical GTK User Interface +# +# Copyright (C) 2011 Intel Corporation +# +# Authored by Shane Wang <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import os + +class CpuInfo(object): + + coefficient = 4 + + @classmethod + def getNumOfCpus(cls): + pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores') + num = len(pfile.readlines()) + return num + + @classmethod + def getNumOfCpuCores(cls): + pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores | cut -d: -f2') + contents = pfile.readlines() + num = int(contents[0]) + return num +
lib/bb/command.py +12 -0
@@ -30,6 +30,7 @@ Commands are queued in a CommandQueue import bb.event import bb.cooker +import bb.helper class CommandCompleted(bb.event.Event): pass
@@ -173,6 +174,17 @@ class CommandsSync: """ command.cooker.reset() + def getDefaultNumOfThreads(self, command, params): + """ + Get the default number of threads on the server = number of CPUs + """ + return bb.helper.CpuInfo.getNumOfCpus() + + def getMaxNumOfThreads(self, command, params): + """ + Get the max number of threads that the server can tolerate + """ + return bb.helper.CpuInfo.getNumOfCpus() * bb.helper.CpuInfo.coefficient class CommandsAsync: """
Signature · 1 line
1.7.0.4
Re: [PATCH 7/8] command.py: add new command to get the CPU info
Quoted 32 lines · Dongxiao Xu
Add new API in
command.pyto get the CPU core and threads information in order to set the appropriateBB_NUMBER_THREADSandPARALLEL_MAKEvariables.Signed-off-by Shane Wang <[email protected]>--- bitbake/lib/bb/helper.py | 39 +++++++++++++++++++++++++++++++++++++++ lib/bb/command.py | 12 ++++++++++++ 2 files changed, 51 insertions(+), 0 deletions(-) create mode 100644 bitbake/lib/bb/helper.pybitbake/lib/bb/helper.py new file +15
-0@@ -0,0 +1,39 @@ +# +# Helper for BitBake Graphical GTK User Interface +# +# Copyright (C) 2011 Intel Corporation +# +# Authored by Shane Wang <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details.
bitbake/lib/bb/helper.py line 25
Quoted 10 lines
bitbake/lib/bb/helper.py new file +10
-0+# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import os + +class CpuInfo(object): + + coefficient = 4
This is setting off alarm bells...
bitbake/lib/bb/helper.py line 31
Quoted 5 lines
bitbake/lib/bb/helper.py new file +5
-0+ @classmethod + def getNumOfCpus(cls): + pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores') + num = len(pfile.readlines()) + return num
You can get this number with something like:
import multiprocessingmultiprocessing.cpu_count()
bitbake/lib/bb/helper.py line 38
Quoted 6 lines
bitbake/lib/bb/helper.py new file +6
-0+ @classmethod + def getNumOfCpuCores(cls): + pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores | cut -d: -f2') + contents = pfile.readlines() + num = int(contents[0]) + return num
I'm curious what you're using the number of cores to do? It doesn't seem used by your code?
Quoted 18 lines
lib/bb/command.py +3
-0@@ -30,6 +30,7 @@ Commands are queued in a CommandQueue import bb.event import bb.cooker +import bb.helper class CommandCompleted(bb.event.Event): pass@@ -173,6 +174,17 @@ class CommandsSync: """ command.cooker.reset() + def getDefaultNumOfThreads(self, command, params): + """
lib/bb/command.py class CommandsSync: · line 188
Quoted 10 lines
lib/bb/command.py +9
-0+ Get the default number of threads on the server = number of CPUs + """ + return bb.helper.CpuInfo.getNumOfCpus() + + def getMaxNumOfThreads(self, command, params): + """ + Get the max number of threads that the server can tolerate + """ + return bb.helper.CpuInfo.getNumOfCpus() * bb.helper.CpuInfo.coefficient
I can understand needing to query the number of cpus but this last function seems rather arbitrary. If you want to do bounds checking, I'd suggest just adding the factor of 4 into the UI. I'm still not convinced we should be setting any value for this though, or doing any bounds checking on the value.
My only other comment would be to use bb.utils instead of creating a bb.helper.
Cheers,
Richard
Re: [PATCH 7/8] command.py: add new command to get the CPU info
Richard Purdie wrote on 2012-01-11:
Quoted 36 lines · Dongxiao Xu
Quoted 18 lines
Add new API in
command.pyto get the CPU core and threads information in order to set the appropriateBB_NUMBER_THREADSandPARALLEL_MAKEvariables.Signed-off-by Shane Wang <[email protected]>--- bitbake/lib/bb/helper.py | 39 +++++++++++++++++++++++++++++++++++++++ lib/bb/command.py | 12 ++++++++++++ 2 files changed, 51 insertions(+), 0 deletions(-) create mode 100644 bitbake/lib/bb/helper.pybitbake/lib/bb/helper.py new renamed from bitbake/lib/bb/helper.py +0
-0file mode 100644 index 0000000..291158b ---
/dev/null+++b/bitbake/lib/bb/helper.py@@ -0,0 +1,39 @@ +# +# Helper for BitBake
Graphical GTK User Interface +# +# Copyright (C) 2011 Intel
Corporation +# +# Authored by Shane Wang <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program isQuoted 8 lines
distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import os + +class CpuInfo(object): + +
coefficient = 4This is setting off alarm bells...
bitbake/lib/bb/helper.py new
Quoted 5 lines
bitbake/lib/bb/helper.py new renamed from bitbake/lib/bb/helper.py +5
-0+ @classmethod + def getNumOfCpus(cls): + pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores') + num = len(pfile.readlines()) + return num
Quoted 4 lines
You can get this number with something like:
import multiprocessing
multiprocessing.cpu_count()
Good to see this works, will change that.
Quoted 29 lines
Quoted 5 lines
+ @classmethod + def getNumOfCpuCores(cls): + pfile =
os.popen('cat /proc/cpuinfo | grep cpu\ cores | cut -d: -f2')+
contents =pfile.readlines()+ num =int(contents[0])+
return numI'm curious what you're using the number of cores to do? It doesn't seem used by your code?
Quoted 19 lines
lib/bb/command.py +3
-0@@ -30,6 +30,7 @@ Commands are queued in a CommandQueue import bb.event import bb.cooker +import bb.helper class CommandCompleted(bb.event.Event): pass@@ -173,6 +174,17 @@ class CommandsSync: """ command.cooker.reset() + def getDefaultNumOfThreads(self, command, params): + """ + Get the default number of threads on the server = number of CPUs + """ + return bb.helper.CpuInfo.getNumOfCpus() + +def
getMaxNumOfThreads(self, command, params): + """ +
Quoted 8 lines
Quoted 4 lines
Get the max number of threads that the server can tolerate + """
+ returnbb.helper.CpuInfo.getNumOfCpus()*
bb.helper.CpuInfo.coefficientI can understand needing to query the number of cpus but this last function seems rather arbitrary. If you want to do bounds checking, I'd suggest just adding the factor of 4 into the UI.
OK
Quoted 3 lines
I'm still not convinced
we should be setting any value for this though, or doing any bounds checking on the value.
Which value? We hope when the UI starts up, the UI can take advantage of cpus for build and set it as default. But users can change between 0 ~ max num of threads.
Quoted 15 lines
My only other comment would be to use bb.utils instead of creating a bb.helper.
Cheers,
Richard
_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Re: [PATCH 7/8] command.py: add new command to get the CPU info
Wang, Shane wrote on 2012-01-11:
Quoted 6 lines
Quoted 3 lines
I'm still not convinced
we should be setting any value for this though, or doing any bounds checking on the value.Which value? We hope when the UI starts up, the UI can take advantage of cpus for build and set it as default.
But users can change between 0 ~ max num of threads.
Do you mean the max is not necessary and we can't get accurate max actually?
--
Shane
Re: [PATCH 7/8] command.py: add new command to get the CPU info
In reply to Wang, ShaneQuoted 81 lines · Shane
Richard Purdie wrote on 2012-01-11:
Quoted 36 lines · Dongxiao Xu
Quoted 18 lines
Add new API in
command.pyto get the CPU core and threads information in order to set the appropriateBB_NUMBER_THREADSandPARALLEL_MAKEvariables.Signed-off-by Shane Wang <[email protected]>--- bitbake/lib/bb/helper.py | 39 +++++++++++++++++++++++++++++++++++++++ lib/bb/command.py | 12 ++++++++++++ 2 files changed, 51 insertions(+), 0 deletions(-) create mode 100644 bitbake/lib/bb/helper.pybitbake/lib/bb/helper.py new renamed from bitbake/lib/bb/helper.py +0
-0file mode 100644 index 0000000..291158b ---
/dev/null+++b/bitbake/lib/bb/helper.py@@ -0,0 +1,39 @@ +# +# Helper for BitBake
Graphical GTK User Interface +# +# Copyright (C) 2011 Intel
Corporation +# +# Authored by Shane Wang <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program isQuoted 8 lines
distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import os + +class CpuInfo(object): + +
coefficient = 4This is setting off alarm bells...
bitbake/lib/bb/helper.py new
Quoted 5 lines
bitbake/lib/bb/helper.py new renamed from bitbake/lib/bb/helper.py +5
-0+ @classmethod + def getNumOfCpus(cls): + pfile = os.popen('cat /proc/cpuinfo | grep cpu\ cores') + num = len(pfile.readlines()) + return numQuoted 4 lines
You can get this number with something like:
import multiprocessing
multiprocessing.cpu_count()Good to see this works, will change that.
Quoted 29 lines
Quoted 5 lines
+ @classmethod + def getNumOfCpuCores(cls): + pfile =
os.popen('cat /proc/cpuinfo | grep cpu\ cores | cut -d: -f2')+
contents =pfile.readlines()+ num =int(contents[0])+
return numI'm curious what you're using the number of cores to do? It doesn't seem used by your code?
Quoted 19 lines
lib/bb/command.py +3
-0@@ -30,6 +30,7 @@ Commands are queued in a CommandQueue import bb.event import bb.cooker +import bb.helper class CommandCompleted(bb.event.Event): pass@@ -173,6 +174,17 @@ class CommandsSync: """ command.cooker.reset() + def getDefaultNumOfThreads(self, command, params): + """ + Get the default number of threads on the server = number of CPUs + """ + return bb.helper.CpuInfo.getNumOfCpus() + +def
getMaxNumOfThreads(self, command, params): + """ +Quoted 8 lines
Quoted 4 lines
Get the max number of threads that the server can tolerate + """
+ returnbb.helper.CpuInfo.getNumOfCpus()*
bb.helper.CpuInfo.coefficientI can understand needing to query the number of cpus but this last function seems rather arbitrary. If you want to do bounds checking, I'd suggest just adding the factor of 4 into the UI.
Quoted 7 lines
OK
Quoted 3 lines
I'm still not convinced
we should be setting any value for this though, or doing any bounds checking on the value.Which value? We hope when the UI starts up, the UI can take advantage of cpus for build and set it as default. But users can change between 0 ~ max num of threads.
You can either:
a) Make it a text input (and then attempt to convert to an int, error if its not an int). This means you don't need a maximum value.
b) In the UI, just show options up to 4 * cpu count.
I'd probably prefer a) but I can see why b) is easier. I don't see any value in encoding "4" in the bitbake server though.
Cheers,
Richard
PATCH 8/8 runqueue: fire sceneQueueTaskStarted event when a setscene queue starts
In reply to Dongxiao XuThe current code prints a log when a setscene task starts, therefore the progressbar in hob will not receive it. Use a sceneQueueTaskStarted event instead.
--- lib/bb/event.py | 2 +- lib/bb/runqueue.py | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletions(-)
lib/bb/event.py +1 -1
@@ -223,7 +223,7 @@ class OperationProgress(Event): Event.__init__(self) self.current = current self.total = total - self.msg = msg + ": %s/%s (%.0f%%)" % (current, total, (current*1.0/total)*100); + self.msg = msg + ": %s/%s" % (current, total); class ConfigParsed(Event): """Configuration Parsing Complete"""
lib/bb/runqueue.py +10 -0
@@ -1619,6 +1619,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute): self.task_skip(task) return True + startevent = sceneQueueTaskStarted(task, self.stats, self.rq) + bb.event.fire(startevent, self.cfgData) + pid, pipein, pipeout = self.fork_off_task(fn, realtask, taskname) self.build_pids[pid] = task
@@ -1686,6 +1689,13 @@ class runQueueTaskStarted(runQueueEvent): runQueueEvent.__init__(self, task, stats, rq) self.noexec = noexec +class sceneQueueTaskStarted(runQueueTaskStarted): + """ + Event notifing a setscene task was started + """ + def __init__(self, task, stats, rq, noexec=False): + runQueueTaskStarted.__init__(self, task, stats, rq, noexec) + class runQueueTaskFailed(runQueueEvent): """ Event notifing a task failed
Signature · 1 line
1.7.0.4