1.44 Patch review
cover 1.44 Patch review
This is a consolidate set which includes last weeks series.
Passes on AB. Please have comment back by Monday.
The following changes since commit cfa307aabf710d79c404a8571b4158b864a94727:
runqueue.py: not show warning for deferred multiconfig task (2019-11-29 11:26:07 +0000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.44-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.44-nut
Aníbal Limón (1):
lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on
directories
Chris Laplante via bitbake-devel (1):
bb.utils.fileslocked: don't leak files if yield throws
Joshua Watt (1):
runqueue: Batch scenequeue updates
Ola x Nilsson (1):
prserv/serv: Use with while reading pidfile
Richard Purdie (21):
hashserv: Add support for equivalent hash reporting runqueue/siggen: Allow handling of equivalent hashes
runqueue: Add extra debugging when locked sigs mismatches occur knotty/uihelper: Switch from pids to tids for Task event management
siggen: Avoid taskhash mismatch errors for nostamp tasks when
dependencies rehash
siggen: Ensure new unihash propagates through the system
siggen: Fix performance issue in get_unihash
runqueue: Rework process_possible_migrations() to improve performance
runqueue: Fix task mismatch failures from incorrect logic
siggen: Split get_tashhash for performance
runqueue: Fix sstate task iteration performance
runqueue: Optimise task migration code slightly
runqueue: Optimise out pointless loop iteration
runqueue: Optimise task filtering
runqueue: Only call into the migrations function if migrations active lib/bb: Optimise out debug messages from cooker
runqueue: Fix equiv hash handling build failures
runqueue: Ensure task dependencies are run correctly
runqueue: Fix task dependency corner case in sanity test
siggen: Test extra cross/native hashserv method
cache: Lower debug level for wold build messages
lib/bb/__init__.py | 5 ++ lib/bb/build.py | 25 +++---- lib/bb/cache.py | 6 +- lib/bb/checksum.py | 5 +- lib/bb/fetch2/__init__.py | 4 +- lib/bb/runqueue.py | 137 +++++++++++++++++++++++--------------- lib/bb/siggen.py | 104 +++++++++++++++++++++++------ lib/bb/ui/knotty.py | 12 ++-- lib/bb/ui/uihelper.py | 39 ++++++----- lib/bb/utils.py | 9 +-- lib/hashserv/client.py | 8 +++ lib/hashserv/server.py | 36 ++++++++++ lib/prserv/serv.py | 12 ++-- 13 files changed, 277 insertions(+), 125 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/25 hashserv: Add support for equivalent hash reporting
From: Richard Purdie <[email protected]>
The reason for this should be recorded in the commit logs. Imagine you have a target recipe (e.g. meta-extsdk-toolchain) which depends on gdb-cross. sstate in OE-Core allows gdb-cross to have the same hash regardless of whether its built on x86 or arm. The outhash will be different.
We need hashequiv to be able to adapt to the prescence of sstate artefacts for meta-extsdk-toolchain and allow the hashes to re-intersect, rather than trying to force a rebuild of meta-extsdk-toolchain. By this point in the build, it would have already been installed from sstate so the build needs to adapt.
Equivalent hashes should be reported to the server as a taskhash that needs to map to an specific unihash. This patch adds API to the hashserv client/server to allow this.
[Thanks to Joshua Watt for help with this patch]
--- lib/hashserv/client.py | 8 ++++++++ lib/hashserv/server.py | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+)
lib/hashserv/client.py +8 -0
@@ -148,6 +148,14 @@ class Client(object): m['unihash'] = unihash return self.send_message({'report': m}) + def report_unihash_equiv(self, taskhash, method, unihash, extra={}): + self._set_mode(self.MODE_NORMAL) + m = extra.copy() + m['taskhash'] = taskhash + m['method'] = method + m['unihash'] = unihash + return self.send_message({'report-equiv': m}) + def get_stats(self): self._set_mode(self.MODE_NORMAL) return self.send_message({'get-stats': None})
lib/hashserv/server.py +36 -0
@@ -143,6 +143,7 @@ class ServerClient(object): handlers = { 'get': self.handle_get, 'report': self.handle_report, + 'report-equiv': self.handle_equivreport, 'get-stream': self.handle_get_stream, 'get-stats': self.handle_get_stats, 'reset-stats': self.handle_reset_stats,
@@ -303,6 +304,41 @@ class ServerClient(object): self.write_message(d) + async def handle_equivreport(self, data): + with closing(self.db.cursor()) as cursor: + insert_data = { + 'method': data['method'], + 'outhash': "", + 'taskhash': data['taskhash'], + 'unihash': data['unihash'], + 'created': datetime.now() + } + + for k in ('owner', 'PN', 'PV', 'PR', 'task', 'outhash_siginfo'): + if k in data: + insert_data[k] = data[k] + + cursor.execute('''INSERT OR IGNORE INTO tasks_v2 (%s) VALUES (%s)''' % ( + ', '.join(sorted(insert_data.keys())), + ', '.join(':' + k for k in sorted(insert_data.keys()))), + insert_data) + + self.db.commit() + + # Fetch the unihash that will be reported for the taskhash. If the + # unihash matches, it means this row was inserted (or the mapping + # was already valid) + row = self.query_equivalent(data['method'], data['taskhash']) + + if row['unihash'] == data['unihash']: + logger.info('Adding taskhash equivalence for %s with unihash %s', + data['taskhash'], row['unihash']) + + d = {k: row[k] for k in ('taskhash', 'method', 'unihash')} + + self.write_message(d) + + async def handle_get_stats(self, request): d = { 'requests': self.request_stats.todict(),
Signature · 1 line
2.17.1
PATCH 2/25 runqueue/siggen: Allow handling of equivalent hashes
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
Based on the hashserv's new ability to accept hash mappings, update runqueue to use this through a helper function in siggen.
This addresses problems with meta-extsdk-toolchain and its dependency on gdb-cross which caused errors when building eSDK. See the previous commit for more details.
--- lib/bb/runqueue.py | 31 +++++++++++++++++++------------ lib/bb/siggen.py | 26 ++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 12 deletions(-)
lib/bb/runqueue.py +19 -12
@@ -2283,12 +2283,26 @@ class RunQueueExecute: for dep in self.rqdata.runtaskentries[tid].depends: procdep.append(dep) orighash = self.rqdata.runtaskentries[tid].hash - self.rqdata.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)]) + newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)]) origuni = self.rqdata.runtaskentries[tid].unihash - self.rqdata.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid) - logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, self.rqdata.runtaskentries[tid].hash, origuni, self.rqdata.runtaskentries[tid].unihash)) + newuni = bb.parse.siggen.get_unihash(tid) + # FIXME, need to check it can come from sstate at all for determinism? + remapped = False + if newuni == origuni: + # Nothing to do, we match, skip code below + remapped = True + elif tid in self.scenequeue_covered or tid in self.sq_live: + # Already ran this setscene task or it running. Report the new taskhash + remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) + logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) + + if not remapped: + logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni)) + self.rqdata.runtaskentries[tid].hash = newhash + self.rqdata.runtaskentries[tid].unihash = newuni + changed.add(tid) + next |= self.rqdata.runtaskentries[tid].revdeps - changed.add(tid) total.remove(tid) next.intersection_update(total)
@@ -2307,18 +2321,11 @@ class RunQueueExecute: self.pending_migrations.add(tid) for tid in self.pending_migrations.copy(): - if tid in self.runq_running: + if tid in self.runq_running or tid in self.sq_live: # Too late, task already running, not much we can do now self.pending_migrations.remove(tid) continue - if tid in self.scenequeue_covered or tid in self.sq_live: - # Already ran this setscene task or it running - # Potentially risky, should we report this hash as a match? - logger.info("Already covered setscene for %s so ignoring rehash" % (tid)) - self.pending_migrations.remove(tid) - continue - valid = True # Check no tasks this covers are running for dep in self.sqdata.sq_covered_tasks[tid]:
lib/bb/siggen.py +26 -0
@@ -525,6 +525,32 @@ class SignatureGeneratorUniHashMixIn(object): except OSError: pass + def report_unihash_equiv(self, tid, taskhash, wanted_unihash, current_unihash, datacaches): + try: + extra_data = {} + data = self.client().report_unihash_equiv(taskhash, self.method, wanted_unihash, extra_data) + bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) + + if data is None: + bb.warn("Server unable to handle unihash report") + return False + + finalunihash = data['unihash'] + + if finalunihash == current_unihash: + bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash)) + elif finalunihash == wanted_unihash: + bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) + self.set_unihash(tid, finalunihash) + return True + else: + # TODO: What to do here? + bb.note('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) + + except hashserv.client.HashConnectionError as e: + bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) + + return False # # Dummy class used for bitbake-selftest
Signature · 1 line
2.17.1
PATCH 3/25 runqueue: Add extra debugging when locked sigs mismatches occur
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
--- lib/bb/runqueue.py | 2 ++ 1 file changed, 2 insertions(+)
lib/bb/runqueue.py +2 -0
@@ -2524,6 +2524,8 @@ class RunQueueExecute: msg = 'Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname) else: msg = 'Task %s.%s attempted to execute unexpectedly' % (pn, taskname) + for t in self.scenequeue_notcovered: + msg = msg + "\nTask %s, unihash %s, taskhash %s" % (t, self.rqdata.runtaskentries[t].unihash, self.rqdata.runtaskentries[t].hash) logger.error(msg + '\nThis is usually due to missing setscene tasks. Those missing in this build were: %s' % pprint.pformat(self.scenequeue_notcovered)) return True return False
Signature · 1 line
2.17.1
PATCH 4/25 knotty/uihelper: Switch from pids to tids for Task event management
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
We've seen cases where a task can execute with a given pid, complete and a new task can start using the same pid before the UI handler has had time to adapt.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main
helper.eventHandler(event)
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler
del self.running_tasks[event.pid]
KeyError: 13490
This means using pids to match up events on the UI side is a bad idea. Change the code to use task ids instead. There is a small amount of fuzzy matching for the progress information since there is no task information there and we don't want the overhead of a task ID in every event, however since pid reuse is unlikely, we can live with a progress bar not quite working properly in a corner case like this.
[YOCTO #13667]
--- lib/bb/build.py | 25 +++++++++++++------------ lib/bb/ui/knotty.py | 12 ++++++------ lib/bb/ui/uihelper.py | 39 ++++++++++++++++++++++++--------------- 3 files changed, 43 insertions(+), 33 deletions(-)
lib/bb/build.py +13 -12
@@ -57,8 +57,9 @@ builtins['os'] = os class TaskBase(event.Event): """Base class for task events""" - def __init__(self, t, logfile, d): + def __init__(self, t, fn, logfile, d): self._task = t + self._fn = fn self._package = d.getVar("PF") self._mc = d.getVar("BB_CURRENT_MC") self.taskfile = d.getVar("FILE")
@@ -81,8 +82,8 @@ class TaskBase(event.Event): class TaskStarted(TaskBase): """Task execution started""" - def __init__(self, t, logfile, taskflags, d): - super(TaskStarted, self).__init__(t, logfile, d) + def __init__(self, t, fn, logfile, taskflags, d): + super(TaskStarted, self).__init__(t, fn, logfile, d) self.taskflags = taskflags class TaskSucceeded(TaskBase):
@@ -91,9 +92,9 @@ class TaskSucceeded(TaskBase): class TaskFailed(TaskBase): """Task execution failed""" - def __init__(self, task, logfile, metadata, errprinted = False): + def __init__(self, task, fn, logfile, metadata, errprinted = False): self.errprinted = errprinted - super(TaskFailed, self).__init__(task, logfile, metadata) + super(TaskFailed, self).__init__(task, fn, logfile, metadata) class TaskFailedSilent(TaskBase): """Task execution failed (silently)"""
@@ -103,8 +104,8 @@ class TaskFailedSilent(TaskBase): class TaskInvalid(TaskBase): - def __init__(self, task, metadata): - super(TaskInvalid, self).__init__(task, None, metadata) + def __init__(self, task, fn, metadata): + super(TaskInvalid, self).__init__(task, fn, None, metadata) self._message = "No such task '%s'" % task class TaskProgress(event.Event):
@@ -572,7 +573,7 @@ def _exec_task(fn, task, d, quieterr): try: try: - event.fire(TaskStarted(task, logfn, flags, localdata), localdata) + event.fire(TaskStarted(task, fn, logfn, flags, localdata), localdata) except (bb.BBHandledException, SystemExit): return 1
@@ -583,15 +584,15 @@ def _exec_task(fn, task, d, quieterr): for func in (postfuncs or '').split(): exec_func(func, localdata) except bb.BBHandledException: - event.fire(TaskFailed(task, logfn, localdata, True), localdata) + event.fire(TaskFailed(task, fn, logfn, localdata, True), localdata) return 1 except Exception as exc: if quieterr: - event.fire(TaskFailedSilent(task, logfn, localdata), localdata) + event.fire(TaskFailedSilent(task, fn, logfn, localdata), localdata) else: errprinted = errchk.triggered logger.error(str(exc)) - event.fire(TaskFailed(task, logfn, localdata, errprinted), localdata) + event.fire(TaskFailed(task, fn, logfn, localdata, errprinted), localdata) return 1 finally: sys.stdout.flush()
@@ -614,7 +615,7 @@ def _exec_task(fn, task, d, quieterr): logger.debug(2, "Zero size logfn %s, removing", logfn) bb.utils.remove(logfn) bb.utils.remove(loglink) - event.fire(TaskSucceeded(task, logfn, localdata), localdata) + event.fire(TaskSucceeded(task, fn, logfn, localdata), localdata) if not localdata.getVarFlag(task, 'nostamp', False) and not localdata.getVarFlag(task, 'selfstamp', False): make_stamp(task, localdata)
lib/bb/ui/knotty.py +6 -6
@@ -255,19 +255,19 @@ class TerminalFilter(object): start_time = activetasks[t].get("starttime", None) if not pbar or pbar.bouncing != (progress < 0): if progress < 0: - pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100, widgets=[progressbar.BouncingSlider(), ''], extrapos=2, resize_handler=self.sigwinch_handle) + pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.BouncingSlider(), ''], extrapos=2, resize_handler=self.sigwinch_handle) pbar.bouncing = True else: - pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100, widgets=[progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=4, resize_handler=self.sigwinch_handle) + pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=4, resize_handler=self.sigwinch_handle) pbar.bouncing = False activetasks[t]["progressbar"] = pbar tasks.append((pbar, progress, rate, start_time)) else: start_time = activetasks[t].get("starttime", None) if start_time: - tasks.append("%s - %s (pid %s)" % (activetasks[t]["title"], self.elapsed(currenttime - start_time), t)) + tasks.append("%s - %s (pid %s)" % (activetasks[t]["title"], self.elapsed(currenttime - start_time), activetasks[t]["pid"])) else: - tasks.append("%s (pid %s)" % (activetasks[t]["title"], t)) + tasks.append("%s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"])) if self.main.shutdown: content = "Waiting for %s running tasks to finish:" % len(activetasks)
@@ -517,8 +517,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): continue # Prefix task messages with recipe/task - if event.taskpid in helper.running_tasks and event.levelno != format.PLAIN: - taskinfo = helper.running_tasks[event.taskpid] + if event.taskpid in helper.pidmap and event.levelno != format.PLAIN: + taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]] event.msg = taskinfo['title'] + ': ' + event.msg if hasattr(event, 'fn'): event.msg = event.fn + ': ' + event.msg
lib/bb/ui/uihelper.py +24 -15
@@ -15,39 +15,48 @@ class BBUIHelper: # Running PIDs preserves the order tasks were executed in self.running_pids = [] self.failed_tasks = [] + self.pidmap = {} self.tasknumber_current = 0 self.tasknumber_total = 0 def eventHandler(self, event): + # PIDs are a bad idea as they can be reused before we process all UI events. + # We maintain a 'fuzzy' match for TaskProgress since there is no other way to match + def removetid(pid, tid): + self.running_pids.remove(tid) + del self.running_tasks[tid] + if self.pidmap[pid] == tid: + del self.pidmap[pid] + self.needUpdate = True + if isinstance(event, bb.build.TaskStarted): + tid = event._fn + ":" + event._task if event._mc != "default": - self.running_tasks[event.pid] = { 'title' : "mc:%s:%s %s" % (event._mc, event._package, event._task), 'starttime' : time.time() } + self.running_tasks[tid] = { 'title' : "mc:%s:%s %s" % (event._mc, event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid } else: - self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time() } - self.running_pids.append(event.pid) + self.running_tasks[tid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid } + self.running_pids.append(tid) + self.pidmap[event.pid] = tid self.needUpdate = True elif isinstance(event, bb.build.TaskSucceeded): - del self.running_tasks[event.pid] - self.running_pids.remove(event.pid) - self.needUpdate = True + tid = event._fn + ":" + event._task + removetid(event.pid, tid) elif isinstance(event, bb.build.TaskFailedSilent): - del self.running_tasks[event.pid] - self.running_pids.remove(event.pid) + tid = event._fn + ":" + event._task + removetid(event.pid, tid) # Don't add to the failed tasks list since this is e.g. a setscene task failure - self.needUpdate = True elif isinstance(event, bb.build.TaskFailed): - del self.running_tasks[event.pid] - self.running_pids.remove(event.pid) + tid = event._fn + ":" + event._task + removetid(event.pid, tid) self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)}) - self.needUpdate = True elif isinstance(event, bb.runqueue.runQueueTaskStarted): self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1 self.tasknumber_total = event.stats.total self.needUpdate = True elif isinstance(event, bb.build.TaskProgress): - if event.pid > 0: - self.running_tasks[event.pid]['progress'] = event.progress - self.running_tasks[event.pid]['rate'] = event.rate + if event.pid > 0 and event.pid in self.pidmap: + self.running_tasks[self.pidmap[event.pid]]['progress'] = event.progress + self.running_tasks[self.pidmap[event.pid]]['rate'] = event.rate self.needUpdate = True else: return False
Signature · 1 line
2.17.1
PATCH 5/25 siggen: Avoid taskhash mismatch errors for nostamp tasks when dependencies rehash
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
An example:
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started
ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded
Is caused by a rehash in a dependency happening somewhere earlier in the build and the taint being reset.
Change the code so that nostamp taints are preserved to avoid the issue.
--- lib/bb/siggen.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
lib/bb/siggen.py +8 -4
@@ -232,10 +232,14 @@ class SignatureGeneratorBasic(SignatureGenerator): taskdep = dataCache.task_deps[fn] if 'nostamp' in taskdep and task in taskdep['nostamp']: # Nostamp tasks need an implicit taint so that they force any dependent tasks to run - import uuid - taint = str(uuid.uuid4()) - data = data + taint - self.taints[tid] = "nostamp:" + taint + if tid in self.taints and self.taints[tid].startswith("nostamp:"): + # Don't reset taint value upon every call + data = data + self.taints[tid][8:] + else: + import uuid + taint = str(uuid.uuid4()) + data = data + taint + self.taints[tid] = "nostamp:" + taint taint = self.read_taint(fn, task, dataCache.stamp[fn]) if taint:
Signature · 1 line
2.17.1
PATCH 6/25 siggen: Ensure new unihash propagates through the system
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
Its possible the new unihash may not exist in sstate. Currently the code would create an sstate object with the old hash however this updates it to create the object with the new unihash.
--- lib/bb/siggen.py | 1 + 1 file changed, 1 insertion(+)
lib/bb/siggen.py +1 -0
@@ -513,6 +513,7 @@ class SignatureGeneratorUniHashMixIn(object): bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d) self.set_unihash(tid, new_unihash) + d.setVar('BB_UNIHASH', new_unihash) else: bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server)) except hashserv.client.HashConnectionError as e:
Signature · 1 line
2.17.1
PATCH 7/25 runqueue: Batch scenequeue updates
In reply to Armin KusterFrom: Joshua Watt <[email protected]>
Batch all updates to scenequeue data together in a single invocation instead of checking each task serially. This allows the checks for sstate object to happen in parallel, and also makes sure the log statement only happens once (per set of rehashes).
--- lib/bb/runqueue.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
lib/bb/runqueue.py +7 -4
@@ -2320,6 +2320,7 @@ class RunQueueExecute: if tid not in self.pending_migrations: self.pending_migrations.add(tid) + update_tasks = [] for tid in self.pending_migrations.copy(): if tid in self.runq_running or tid in self.sq_live: # Too late, task already running, not much we can do now
@@ -2379,11 +2380,13 @@ class RunQueueExecute: if tid in self.build_stamps: del self.build_stamps[tid] - origvalid = False - if tid in self.sqdata.valid: - origvalid = True + update_tasks.append((tid, harddepfail, tid in self.sqdata.valid)) + + if update_tasks: self.sqdone = False - update_scenequeue_data([tid], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) + update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) + + for (tid, harddepfail, origvalid) in update_tasks: if tid in self.sqdata.valid and not origvalid: logger.info("Setscene task %s became valid" % tid) if harddepfail:
Signature · 1 line
2.17.1
PATCH 8/25 siggen: Fix performance issue in get_unihash
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
There is a significant performance issue in get_unihash(). The issue turns out to be the lookups of setscene tasks. We can fix this by using a set() instead of the current list.
--- lib/bb/siggen.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
lib/bb/siggen.py +3 -3
@@ -44,7 +44,7 @@ class SignatureGenerator(object): self.file_checksum_values = {} self.taints = {} self.unitaskhashes = {} - self.setscenetasks = {} + self.setscenetasks = set() def finalise(self, fn, d, varient): return
@@ -110,7 +110,7 @@ class SignatureGeneratorBasic(SignatureGenerator): self.taints = {} self.gendeps = {} self.lookupcache = {} - self.setscenetasks = {} + self.setscenetasks = set() self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST") or "").split()) self.taskwhitelist = None self.init_rundepcheck(data)
@@ -157,7 +157,7 @@ class SignatureGeneratorBasic(SignatureGenerator): return taskdeps def set_setscene_tasks(self, setscene_tasks): - self.setscenetasks = setscene_tasks + self.setscenetasks = set(setscene_tasks) def finalise(self, fn, d, variant):
Signature · 1 line
2.17.1
PATCH 9/25 bb.utils.fileslocked: don't leak files if yield throws
In reply to Armin KusterFrom: Chris Laplante via bitbake-devel <[email protected]>
Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by externalsrc.bbclass) was leaked, giving a warning like:
WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'>
exec_func(task, localdata)
--- lib/bb/utils.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
lib/bb/utils.py +5 -4
@@ -428,10 +428,11 @@ def fileslocked(files): for lockfile in files: locks.append(bb.utils.lockfile(lockfile)) - yield - - for lock in locks: - bb.utils.unlockfile(lock) + try: + yield + finally: + for lock in locks: + bb.utils.unlockfile(lock) @contextmanager def timeout(seconds):
Signature · 1 line
2.17.1
PATCH 10/25 runqueue: Rework process_possible_migrations() to improve performance
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
The looping over multiple changed hashes causes many calls to get_taskhash and get_unihash which are potentially slow and then overwritten.
Instead, batch up all the tasks which have changed unihashes and then do one big loop over the changed tasks rather than each in turn.
This makes worlds of difference to the performance graphs and should speed up build where many tasks are being rehashed.
--- lib/bb/runqueue.py | 103 ++++++++++++++++++++++++--------------------- 1 file changed, 56 insertions(+), 47 deletions(-)
lib/bb/runqueue.py +56 -47
@@ -2248,6 +2248,7 @@ class RunQueueExecute: def process_possible_migrations(self): changed = set() + toprocess = set() for tid, unihash in self.updated_taskhash_queue.copy(): if tid in self.runq_running and tid not in self.runq_complete: continue
@@ -2258,53 +2259,61 @@ class RunQueueExecute: logger.info("Task %s unihash changed to %s" % (tid, unihash)) self.rqdata.runtaskentries[tid].unihash = unihash bb.parse.siggen.set_unihash(tid, unihash) - - # Work out all tasks which depend on this one - total = set() - next = set(self.rqdata.runtaskentries[tid].revdeps) - while next: - current = next.copy() - total = total |next - next = set() - for ntid in current: - next |= self.rqdata.runtaskentries[ntid].revdeps - next.difference_update(total) - - # Now iterate those tasks in dependency order to regenerate their taskhash/unihash - done = set() - next = set(self.rqdata.runtaskentries[tid].revdeps) - while next: - current = next.copy() - next = set() - for tid in current: - if not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): - continue - procdep = [] - for dep in self.rqdata.runtaskentries[tid].depends: - procdep.append(dep) - orighash = self.rqdata.runtaskentries[tid].hash - newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)]) - origuni = self.rqdata.runtaskentries[tid].unihash - newuni = bb.parse.siggen.get_unihash(tid) - # FIXME, need to check it can come from sstate at all for determinism? - remapped = False - if newuni == origuni: - # Nothing to do, we match, skip code below - remapped = True - elif tid in self.scenequeue_covered or tid in self.sq_live: - # Already ran this setscene task or it running. Report the new taskhash - remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) - logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) - - if not remapped: - logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni)) - self.rqdata.runtaskentries[tid].hash = newhash - self.rqdata.runtaskentries[tid].unihash = newuni - changed.add(tid) - - next |= self.rqdata.runtaskentries[tid].revdeps - total.remove(tid) - next.intersection_update(total) + toprocess.add(tid) + + # Work out all tasks which depend upon these + total = set() + for p in toprocess: + next = set(self.rqdata.runtaskentries[p].revdeps) + while next: + current = next.copy() + total = total | next + next = set() + for ntid in current: + next |= self.rqdata.runtaskentries[ntid].revdeps + next.difference_update(total) + + # Now iterate those tasks in dependency order to regenerate their taskhash/unihash + next = set() + for p in total: + if len(self.rqdata.runtaskentries[p].depends) == 0: + next.add(p) + elif self.rqdata.runtaskentries[p].depends.isdisjoint(total): + next.add(p) + + # When an item doesn't have dependencies in total, we can process it. Drop items from total when handled + while next: + current = next.copy() + next = set() + for tid in current: + if not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): + continue + procdep = [] + for dep in self.rqdata.runtaskentries[tid].depends: + procdep.append(dep) + orighash = self.rqdata.runtaskentries[tid].hash + newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)]) + origuni = self.rqdata.runtaskentries[tid].unihash + newuni = bb.parse.siggen.get_unihash(tid) + # FIXME, need to check it can come from sstate at all for determinism? + remapped = False + if newuni == origuni: + # Nothing to do, we match, skip code below + remapped = True + elif tid in self.scenequeue_covered or tid in self.sq_live: + # Already ran this setscene task or it running. Report the new taskhash + remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) + logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) + + if not remapped: + #logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni)) + self.rqdata.runtaskentries[tid].hash = newhash + self.rqdata.runtaskentries[tid].unihash = newuni + changed.add(tid) + + next |= self.rqdata.runtaskentries[tid].revdeps + total.remove(tid) + next.intersection_update(total) if changed: for mc in self.rq.worker:
Signature · 1 line
2.17.1
PATCH 11/25 runqueue: Fix task mismatch failures from incorrect logic
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
The "no dependencies" task case was not being correctly considered in this code and seemed to be the cause of occasionaly task hash mismatch errors that were being seen as the dependencies were never accounted for properly.
--- lib/bb/runqueue.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/runqueue.py +1 -1
@@ -2286,7 +2286,7 @@ class RunQueueExecute: current = next.copy() next = set() for tid in current: - if not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): + if len(self.rqdata.runtaskentries[p].depends) and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): continue procdep = [] for dep in self.rqdata.runtaskentries[tid].depends:
Signature · 1 line
2.17.1
PATCH 12/25 siggen: Split get_tashhash for performance
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash.
Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does.
This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing.
--- lib/bb/runqueue.py | 1 + lib/bb/siggen.py | 33 ++++++++++++++++++++++++--------- 2 files changed, 25 insertions(+), 9 deletions(-)
lib/bb/runqueue.py +1 -0
@@ -1185,6 +1185,7 @@ class RunQueueData: procdep = [] for dep in self.runtaskentries[tid].depends: procdep.append(dep) + bb.parse.siggen.prep_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)]) self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)]) self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
lib/bb/siggen.py +24 -9
@@ -52,6 +52,9 @@ class SignatureGenerator(object): def get_unihash(self, tid): return self.taskhash[tid] + def prep_taskhash(self, tid, deps, dataCache): + return + def get_taskhash(self, tid, deps, dataCache): self.taskhash[tid] = hashlib.sha256(tid.encode("utf-8")).hexdigest() return self.taskhash[tid]
@@ -198,12 +201,11 @@ class SignatureGeneratorBasic(SignatureGenerator): pass return taint - def get_taskhash(self, tid, deps, dataCache): + def prep_taskhash(self, tid, deps, dataCache): (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid) - data = dataCache.basetaskhash[tid] - self.basehash[tid] = data + self.basehash[tid] = dataCache.basetaskhash[tid] self.runtaskdeps[tid] = [] self.file_checksum_values[tid] = [] recipename = dataCache.pkg_fn[fn]
@@ -216,7 +218,6 @@ class SignatureGeneratorBasic(SignatureGenerator): continue if dep not in self.taskhash: bb.fatal("%s is not in taskhash, caller isn't calling in dependency order?" % dep) - data = data + self.get_unihash(dep) self.runtaskdeps[tid].append(dep) if task in dataCache.file_checksums[fn]:
@@ -226,27 +227,41 @@ class SignatureGeneratorBasic(SignatureGenerator): checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename) for (f,cs) in checksums: self.file_checksum_values[tid].append((f,cs)) - if cs: - data = data + cs taskdep = dataCache.task_deps[fn] if 'nostamp' in taskdep and task in taskdep['nostamp']: # Nostamp tasks need an implicit taint so that they force any dependent tasks to run if tid in self.taints and self.taints[tid].startswith("nostamp:"): # Don't reset taint value upon every call - data = data + self.taints[tid][8:] + pass else: import uuid taint = str(uuid.uuid4()) - data = data + taint self.taints[tid] = "nostamp:" + taint taint = self.read_taint(fn, task, dataCache.stamp[fn]) if taint: - data = data + taint self.taints[tid] = taint logger.warning("%s is tainted from a forced run" % tid) + return + + def get_taskhash(self, tid, deps, dataCache): + + data = self.basehash[tid] + for dep in self.runtaskdeps[tid]: + data = data + self.get_unihash(dep) + + for (f, cs) in self.file_checksum_values[tid]: + if cs: + data = data + cs + + if tid in self.taints: + if self.taints[tid].startswith("nostamp:"): + data = data + self.taints[tid][8:] + else: + data = data + self.taints[tid] + h = hashlib.sha256(data.encode("utf-8")).hexdigest() self.taskhash[tid] = h #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
Signature · 1 line
2.17.1
PATCH 13/25 runqueue: Fix sstate task iteration performance
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
Creating a new sorted list of sstate tasks each iteration through runqueue is extremely ineffecient and was compounded by the recent change from a list to set.
Create one sorted list instead of recreating it each time.
--- lib/bb/runqueue.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
lib/bb/runqueue.py +5 -1
@@ -1965,10 +1965,14 @@ class RunQueueExecute: self.rq.read_workers() self.process_possible_migrations() + if not hasattr(self, "sorted_setscene_tids"): + # Don't want to sort this set every execution + self.sorted_setscene_tids = sorted(self.rqdata.runq_setscene_tids) + task = None if not self.sqdone and self.can_start_task(): # Find the next setscene to run - for nexttask in sorted(self.rqdata.runq_setscene_tids): + for nexttask in self.sorted_setscene_tids: if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values(): if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]): if nexttask not in self.rqdata.target_tids:
Signature · 1 line
2.17.1
PATCH 14/25 runqueue: Optimise task migration code slightly
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
Move the calls to difference_update out a code level which improves efficiency significantly.
Also further combine the outer loop for efficiency too.
These two changes remove a bottleneck from the performance charts.
--- lib/bb/runqueue.py | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
lib/bb/runqueue.py +9 -8
@@ -2268,15 +2268,16 @@ class RunQueueExecute: # Work out all tasks which depend upon these total = set() + next = set() for p in toprocess: - next = set(self.rqdata.runtaskentries[p].revdeps) - while next: - current = next.copy() - total = total | next - next = set() - for ntid in current: - next |= self.rqdata.runtaskentries[ntid].revdeps - next.difference_update(total) + next |= self.rqdata.runtaskentries[p].revdeps + while next: + current = next.copy() + total = total | next + next = set() + for ntid in current: + next |= self.rqdata.runtaskentries[ntid].revdeps + next.difference_update(total) # Now iterate those tasks in dependency order to regenerate their taskhash/unihash next = set()
Signature · 1 line
2.17.1
PATCH 15/25 runqueue: Optimise out pointless loop iteration
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
--- lib/bb/runqueue.py | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)
lib/bb/runqueue.py +3 -9
@@ -1182,11 +1182,8 @@ class RunQueueData: return len(self.runtaskentries) def prepare_task_hash(self, tid): - procdep = [] - for dep in self.runtaskentries[tid].depends: - procdep.append(dep) - bb.parse.siggen.prep_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)]) - self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)]) + bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches[mc_from_tid(tid)]) + self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches[mc_from_tid(tid)]) self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid) def dump_data(self):
@@ -2294,11 +2291,8 @@ class RunQueueExecute: for tid in current: if len(self.rqdata.runtaskentries[p].depends) and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): continue - procdep = [] - for dep in self.rqdata.runtaskentries[tid].depends: - procdep.append(dep) orighash = self.rqdata.runtaskentries[tid].hash - newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)]) + newhash = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches[mc_from_tid(tid)]) origuni = self.rqdata.runtaskentries[tid].unihash newuni = bb.parse.siggen.get_unihash(tid) # FIXME, need to check it can come from sstate at all for determinism?
Signature · 1 line
2.17.1
PATCH 16/25 runqueue: Optimise task filtering
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
We were seeing this running thousands of times with hashequiv, do the filtering where it makes more sense and make it persist.
--- lib/bb/runqueue.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
lib/bb/runqueue.py +2 -3
@@ -148,8 +148,9 @@ class RunQueueScheduler(object): """ Return the id of the first task we find that is buildable """ + # Once tasks are running we don't need to worry about them again + self.buildable.difference_update(self.rq.runq_running) buildable = set(self.buildable) - buildable.difference_update(self.rq.runq_running) buildable.difference_update(self.rq.holdoff_tasks) buildable.intersection_update(self.rq.tasks_covered | self.rq.tasks_notcovered) if not buildable:
@@ -207,8 +208,6 @@ class RunQueueScheduler(object): def newbuildable(self, task): self.buildable.add(task) - # Once tasks are running we don't need to worry about them again - self.buildable.difference_update(self.rq.runq_running) def removebuildable(self, task): self.buildable.remove(task)
Signature · 1 line
2.17.1
PATCH 17/25 runqueue: Only call into the migrations function if migrations active
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
This doesn't save much time but does make the profile counts for the function more accurate which is in itself useful.
--- lib/bb/runqueue.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
lib/bb/runqueue.py +2 -1
@@ -1959,7 +1959,8 @@ class RunQueueExecute: """ self.rq.read_workers() - self.process_possible_migrations() + if self.updated_taskhash_queue or self.pending_migrations: + self.process_possible_migrations() if not hasattr(self, "sorted_setscene_tids"): # Don't want to sort this set every execution
Signature · 1 line
2.17.1
PATCH 18/25 lib/bb: Optimise out debug messages from cooker
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
We have bb.debug(2, xxx) messages in cooker which are useful for debugging but have really bad effects on performance, 640,000 calls on recent profile graphs taking tens of seconds.
Rather than commenting out debug which can be useful for debugging, don't create events for debug log messages from cooker which would never be seen. We already stop the messages hitting the IPC but this avoids the overhead of creating the log messages too, which has been shown to be signficiant on the profiles. This allows the code to perform whilst allowing debug messages to be availble when wanted/enabled.
--- lib/bb/__init__.py | 5 +++++ 1 file changed, 5 insertions(+)
lib/bb/__init__.py +5 -0
@@ -43,6 +43,11 @@ class BBLogger(Logger): Logger.__init__(self, name) def bbdebug(self, level, msg, *args, **kwargs): + if not bb.event.worker_pid: + if self.name in bb.msg.loggerDefaultDomains and level > (bb.msg.loggerDefaultDomains[self.name]): + return + if level > (bb.msg.loggerDefaultDebugLevel): + return return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs) def plain(self, msg, *args, **kwargs):
Signature · 1 line
2.17.1
PATCH 19/25 lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
In reply to Armin KusterFrom: Aníbal Limón <[email protected]>
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory.
Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"
--- lib/bb/checksum.py | 5 +++-- lib/bb/fetch2/__init__.py | 4 ++-- lib/bb/siggen.py | 5 +++-- 3 files changed, 8 insertions(+), 6 deletions(-)
lib/bb/checksum.py +3 -2
@@ -74,7 +74,7 @@ class FileChecksumCache(MultiProcessCache): else: dest[0][h] = source[0][h] - def get_checksums(self, filelist, pn): + def get_checksums(self, filelist, pn, localdirsexclude): """Get checksums for a list of files""" def checksum_file(f):
@@ -90,7 +90,8 @@ class FileChecksumCache(MultiProcessCache): if pth == "/": bb.fatal("Refusing to checksum /") dirchecksums = [] - for root, dirs, files in os.walk(pth): + for root, dirs, files in os.walk(pth, topdown=True): + [dirs.remove(d) for d in list(dirs) if d in localdirsexclude] for name in files: fullpth = os.path.join(root, name) checksum = checksum_file(fullpth)
lib/bb/fetch2/__init__.py +2 -2
@@ -1197,14 +1197,14 @@ def get_checksum_file_list(d): return " ".join(filelist) -def get_file_checksums(filelist, pn): +def get_file_checksums(filelist, pn, localdirsexclude): """Get a list of the checksums for a list of local files Returns the checksums for a list of local files, caching the results as it proceeds """ - return _checksum_cache.get_checksums(filelist, pn) + return _checksum_cache.get_checksums(filelist, pn, localdirsexclude) class FetchData(object):
lib/bb/siggen.py +3 -2
@@ -126,6 +126,7 @@ class SignatureGeneratorBasic(SignatureGenerator): self.unihash_cache = bb.cache.SimpleCache("1") self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {}) + self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc .p4 .repo .svn").split() def init_rundepcheck(self, data): self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None
@@ -222,9 +223,9 @@ class SignatureGeneratorBasic(SignatureGenerator): if task in dataCache.file_checksums[fn]: if self.checksum_cache: - checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename) + checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude) else: - checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename) + checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude) for (f,cs) in checksums: self.file_checksum_values[tid].append((f,cs))
Signature · 1 line
2.17.1
PATCH 20/25 prserv/serv: Use with while reading pidfile
In reply to Armin KusterFrom: Ola x Nilsson <[email protected]>
--- lib/prserv/serv.py | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
lib/prserv/serv.py +5 -7
@@ -292,10 +292,9 @@ class PRServer(SimpleXMLRPCServer): logger.addHandler(streamhandler) # write pidfile - pid = str(os.getpid()) - pf = open(self.pidfile, 'w') - pf.write("%s\n" % pid) - pf.close() + pid = str(os.getpid()) + with open(self.pidfile, 'w') as pf: + pf.write("%s\n" % pid) self.work_forever() self.delpid()
@@ -353,9 +352,8 @@ def start_daemon(dbfile, host, port, logfile): ip = socket.gethostbyname(host) pidfile = PIDPREFIX % (ip, port) try: - pf = open(pidfile,'r') - pid = int(pf.readline().strip()) - pf.close() + with open(pidfile) as pf: + pid = int(pf.readline().strip()) except IOError: pid = None
Signature · 1 line
2.17.1
PATCH 21/25 runqueue: Fix equiv hash handling build failures
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
Regardless of whether we remapped the hash on the server or not, we need to have bitbake work as if we did as we need to match how the stamp files look.
This change resolves build failures where tasks were rerunning when they shouldn't.
--- lib/bb/runqueue.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
lib/bb/runqueue.py +2 -1
@@ -2302,8 +2302,9 @@ class RunQueueExecute: remapped = True elif tid in self.scenequeue_covered or tid in self.sq_live: # Already ran this setscene task or it running. Report the new taskhash - remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) + bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) + remapped = True if not remapped: #logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni))
Signature · 1 line
2.17.1
PATCH 22/25 runqueue: Ensure task dependencies are run correctly
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run.
This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption.
Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures.
There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified.
--- lib/bb/runqueue.py | 6 ++++++ 1 file changed, 6 insertions(+)
lib/bb/runqueue.py +6 -0
@@ -2353,6 +2353,12 @@ class RunQueueExecute: if tid in self.tasks_scenequeue_done: self.tasks_scenequeue_done.remove(tid) for dep in self.sqdata.sq_covered_tasks[tid]: + if dep in self.runq_complete: + bb.error("Task %s marked as completed but now needing to rerun? Aborting build." % dep) + self.failed_tids.append(tid) + self.rq.state = runQueueCleanUp + return + if dep not in self.runq_complete: if dep in self.tasks_scenequeue_done and dep not in self.sqdata.unskippable: self.tasks_scenequeue_done.remove(dep)
Signature · 1 line
2.17.1
PATCH 23/25 runqueue: Fix task dependency corner case in sanity test
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly.
--- lib/bb/runqueue.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
lib/bb/runqueue.py +4 -1
@@ -1708,6 +1708,7 @@ class RunQueueExecute: self.runq_buildable = set() self.runq_running = set() self.runq_complete = set() + self.runq_tasksrun = set() self.build_stamps = {} self.build_stamps2 = []
@@ -1893,6 +1894,7 @@ class RunQueueExecute: self.stats.taskCompleted() bb.event.fire(runQueueTaskCompleted(task, self.stats, self.rq), self.cfgData) self.task_completeoutright(task) + self.runq_tasksrun.add(task) def task_fail(self, task, exitcode): """
@@ -2092,6 +2094,7 @@ class RunQueueExecute: logger.debug(2, "Stamp current task %s", task) self.task_skip(task, "existing") + self.runq_tasksrun.add(task) return True taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
@@ -2353,7 +2356,7 @@ class RunQueueExecute: if tid in self.tasks_scenequeue_done: self.tasks_scenequeue_done.remove(tid) for dep in self.sqdata.sq_covered_tasks[tid]: - if dep in self.runq_complete: + if dep in self.runq_complete and dep not in self.runq_tasksrun: bb.error("Task %s marked as completed but now needing to rerun? Aborting build." % dep) self.failed_tids.append(tid) self.rq.state = runQueueCleanUp
Signature · 1 line
2.17.1
PATCH 24/25 siggen: Test extra cross/native hashserv method
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
Hack the hashserv to allow extra data to be injected into the hashserv method. This allows OE-Core to handle cases where there are multiple sstate objects for the same taskhash, e.g. native/cross objects based upon BUILD_ARCH or the host distro (when uninative isn't used).
This has been tested and proven to be very effective. We will likely rework the code to improve how this is handled but for now this improves automated builds until we can get to that refactoring and more invasive changes.
--- lib/bb/siggen.py | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)
lib/bb/siggen.py +21 -6
@@ -391,12 +391,16 @@ class SignatureGeneratorBasicHash(SignatureGeneratorBasic): bb.build.write_taint(task, d, fn) class SignatureGeneratorUniHashMixIn(object): + def __init__(self, data): + self.extramethod = {} + super().__init__(data) + def get_taskdata(self): - return (self.server, self.method) + super().get_taskdata() + return (self.server, self.method, self.extramethod) + super().get_taskdata() def set_taskdata(self, data): - self.server, self.method = data[:2] - super().set_taskdata(data[2:]) + self.server, self.method, self.extramethod = data[:3] + super().set_taskdata(data[3:]) def client(self): if getattr(self, '_client', None) is None:
@@ -453,7 +457,10 @@ class SignatureGeneratorUniHashMixIn(object): unihash = taskhash try: - data = self.client().get_unihash(self.method, self.taskhash[tid]) + method = self.method + if tid in self.extramethod: + method = method + self.extramethod[tid] + data = self.client().get_unihash(method, self.taskhash[tid]) if data: unihash = data # A unique hash equal to the taskhash is not very interesting,
@@ -522,7 +529,11 @@ class SignatureGeneratorUniHashMixIn(object): extra_data['task'] = task extra_data['outhash_siginfo'] = sigfile.read().decode('utf-8') - data = self.client().report_unihash(taskhash, self.method, outhash, unihash, extra_data) + method = self.method + if tid in self.extramethod: + method = method + self.extramethod[tid] + + data = self.client().report_unihash(taskhash, method, outhash, unihash, extra_data) new_unihash = data['unihash'] if new_unihash != unihash:
@@ -549,7 +560,11 @@ class SignatureGeneratorUniHashMixIn(object): def report_unihash_equiv(self, tid, taskhash, wanted_unihash, current_unihash, datacaches): try: extra_data = {} - data = self.client().report_unihash_equiv(taskhash, self.method, wanted_unihash, extra_data) + method = self.method + if tid in self.extramethod: + method = method + self.extramethod[tid] + + data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data) bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) if data is None:
Signature · 1 line
2.17.1
PATCH 25/25 cache: Lower debug level for wold build messages
In reply to Armin KusterFrom: Richard Purdie <[email protected]>
These messages spam the logs for no good reason, they were useful for debugging a particular problem long ago but are distracting noise now. Disable them.
--- lib/bb/cache.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
lib/bb/cache.py +3 -3
@@ -208,10 +208,10 @@ class CoreRecipeInfo(RecipeInfoCommon): # Collect files we may need for possible world-dep # calculations - if self.not_world: - logger.debug(1, "EXCLUDE FROM WORLD: %s", fn) - else: + if not self.not_world: cachedata.possible_world.append(fn) + #else: + # logger.debug(2, "EXCLUDE FROM WORLD: %s", fn) # create a collection of all targets for sanity checking # tasks, such as upstream versions, license, and tools for
Signature · 1 line
2.17.1