toaster-related patchset
cover toaster-related patchset
From: Alexandru DAMIAN <[email protected]>
This is a rebased patchset consisting of patches that have been previously submitted to the oe-core mail list separately, but not yet merged.
In order to make merging easier, I'm resubmitting as a new patchset.
The following changes since commit e8476a9d19dba7a1a6c5a558a89f2c9f89d39ae2:
gettext: upgrade from 0.18.3.1 to 0.18.3.2 (2014-03-17 14:01:43 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib adamian/submission-17032014
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/submission-17032014
Alexandru DAMIAN (1):
sstate: list missing files for toaster
Cristiana Voicu (1):
toaster.bbclass: read the data needed for license manifest path
Marius Avram (1):
buildstats: use TaskBase time for elapsed time
meta/classes/buildstats.bbclass | 23 +++++++++++++++-------- meta/classes/sstate.bbclass | 11 +++++++++++ meta/classes/toaster.bbclass | 12 ++++++++++++ 3 files changed, 38 insertions(+), 8 deletions(-)
Signature · 1 line
1.8.3.2
PATCH 1/3 toaster.bbclass: read the data needed for license manifest path
From: Cristiana Voicu <[email protected]>
The license.manifest file is located in DEPLOY_DIR_IMAGE/licenses/ IMAGE_NAME dir. The data needed is collected after rootfs task.
--- meta/classes/toaster.bbclass | 12 ++++++++++++ 1 file changed, 12 insertions(+)
meta/classes/toaster.bbclass +12 -0
@@ -296,6 +296,17 @@ python toaster_buildhistory_dump() { } +# dump information related to license manifest path + +python toaster_licensemanifest_dump() { + deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE', True); + image_name = d.getVar('IMAGE_NAME', True); + + data = { 'deploy_dir_image' : deploy_dir_image, 'image_name' : image_name } + + bb.event.fire(bb.event.MetadataEvent("LicenseManifestPath", data), d) +} + # set event handlers addhandler toaster_layerinfo_dumpdata toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -308,3 +319,4 @@ toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted" do_package[postfuncs] += "toaster_package_dumpdata " do_rootfs[postfuncs] += "toaster_image_dumpdata " +do_rootfs[postfuncs] += "toaster_licensemanifest_dump "
Signature · 1 line
1.8.3.2
PATCH 2/3 sstate: list missing files for toaster
From: Alexandru DAMIAN <[email protected]>
Toaster needs to record the attempts to restore setscene tasks that don't have a sstate file.
We build a list of tasks for which we can't find an sstate file, and if we're running under Toaster data collection, we send it off with a MetadataEvent.
--- meta/classes/sstate.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/classes/sstate.bbclass +11 -0
@@ -625,6 +625,7 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): ret = [] + missed = [] def getpathcomponents(task, d): # Magic data from BB_HASHFILENAME
@@ -646,11 +647,13 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): spec, extrapath, tname = getpathcomponents(task, d) sstatefile = d.expand("${SSTATE_DIR}/" + extrapath + generate_sstatefn(spec, sq_hash[task], d) + "_" + tname + ".tgz.siginfo") + if os.path.exists(sstatefile): bb.debug(2, "SState: Found valid sstate file %s" % sstatefile) ret.append(task) continue else: + missed.append(task) bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile) mirrors = d.getVar("SSTATE_MIRRORS", True)
@@ -688,9 +691,17 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): bb.debug(2, "SState: Successful fetch test for %s" % srcuri) ret.append(task) except: + missed.append(task) bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri) pass + inheritlist = d.getVar("INHERIT", True) + if "toaster" in inheritlist: + evdata = [] + for task in missed: + evdata.append( (sq_fn[task], sq_task[task], sq_hash[task], generate_sstatefn(spec, sq_hash[task],d) ) ) + bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d) + return ret BB_SETSCENE_DEPVALID = "setscene_depvalid"
Signature · 1 line
1.8.3.2
PATCH 3/3 buildstats: use TaskBase time for elapsed time
In reply to Alex DAMIANFrom: Marius Avram <[email protected]>
To avoid any further inconsistencies between buildstats and buildinfohelper from toaster, buildstats will measure task duration using the time field from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.
--- meta/classes/buildstats.bbclass | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
meta/classes/buildstats.bbclass +15 -8
@@ -109,14 +109,17 @@ def get_diskdata(var, dev, data): diskdata["End"+key] = str(int(newdiskdata[key])) return diskdata -def set_timedata(var, data): +def set_timedata(var, data, server_time=None): import time - time = time.time() + if server_time: + time = server_time + else: + time = time.time() cputime = get_cputime() proctime = get_process_cputime(os.getpid()) data.setVar(var, (time, cputime, proctime)) -def get_timedata(var, data): +def get_timedata(var, data, server_time=None): import time timedata = data.getVar(var, False) if timedata is None:
@@ -124,7 +127,11 @@ def get_timedata(var, data): oldtime, oldcpu, oldproc = timedata procdiff = get_process_cputime(os.getpid()) - oldproc cpudiff = get_cputime() - oldcpu - timediff = time.time() - oldtime + if server_time: + end_time = server_time + else: + end_time = time.time() + timediff = end_time - oldtime if cpudiff > 0: cpuperc = float(procdiff) * 100 / cpudiff else:
@@ -136,7 +143,7 @@ def write_task_data(status, logfile, dev, e): bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn) taskdir = os.path.join(bsdir, e.data.expand("${PF}")) file = open(os.path.join(logfile), "a") - timedata = get_timedata("__timedata_task", e.data) + timedata = get_timedata("__timedata_task", e.data, e.time) if timedata: elapsedtime, cpu = timedata file.write(bb.data.expand("${PF}: %s: Elapsed time: %0.2f seconds \n" %
@@ -160,7 +167,7 @@ def write_task_data(status, logfile, dev, e): file.write("Status: PASSED \n") else: file.write("Status: FAILED \n") - file.write("Ended: %0.2f \n" % time.time()) + file.write("Ended: %0.2f \n" % e.time) file.close() python run_buildstats () {
@@ -234,7 +241,7 @@ python run_buildstats () { taskdir = os.path.join(bsdir, e.data.expand("${PF}")) if device != "NoLogicalDevice": set_diskdata("__diskdata_task", device, e.data) - set_timedata("__timedata_task", e.data) + set_timedata("__timedata_task", e.data, e.time) try: bb.utils.mkdirhier(taskdir) except:
@@ -242,7 +249,7 @@ python run_buildstats () { # write into the task event file the name and start time file = open(os.path.join(taskdir, e.task), "a") file.write("Event: %s \n" % bb.event.getName(e)) - file.write("Started: %0.2f \n" % time.time()) + file.write("Started: %0.2f \n" % e.time) file.close() elif isinstance(e, bb.build.TaskSucceeded):
Signature · 1 line
1.8.3.2
Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time
Quoted 6 lines · Alex DAMIAN
From: Marius Avram <[email protected]>
To avoid any further inconsistencies between buildstats and buildinfohelper from toaster, buildstats will measure task duration using the time field from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.
This patch seems to be causing some problems:
Quoted 16 lines
ERROR: Execution of event handler '
run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 72, inrun_buildstats(e=<bb.build.TaskStarted object at 0x6e45590>)
AttributeError: 'TaskStarted' object has no attribute 'time'ERROR: Execution of event handler '
run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 88, inrun_buildstats(e=<bb.build.TaskSucceeded object at 0x6e40190>)
File "buildstats.bbclass", line 5, in write_task_data(status='passed', logfile='/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_clean', dev='sda5', e=<bb.build.TaskSucceeded object at0x6e40190>)
IOError: [Errno 2] No such file or directory: '/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_clean'ERROR: Execution of event handler '
run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 72, inrun_buildstats(e=<bb.build.TaskStarted object at 0x74f59d0>)
AttributeError: 'TaskStarted' object has no attribute 'time'
Quoted 4 lines
ERROR: Execution of event handler '
run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 88, inrun_buildstats(e=<bb.build.TaskSucceeded object at 0x6e48dd0>)
File "buildstats.bbclass", line 5, in write_task_data(status='passed', logfile='/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_cleansstate', dev='sda5', e=<bb.build.TaskSucceeded object at0x6e48dd0>)
Sau!
Quoted 80 lines
Signed-off-by Marius Avram <[email protected]>--- meta/classes/buildstats.bbclass | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)meta/classes/buildstats.bbclass +15
-8@@ -109,14 +109,17 @@ def get_diskdata(var, dev, data): diskdata["End"+key] = str(int(newdiskdata[key])) return diskdata -def set_timedata(var, data): +def set_timedata(var, data, server_time=None): import time - time = time.time() + if server_time: + time = server_time + else: + time = time.time() cputime = get_cputime() proctime = get_process_cputime(os.getpid()) data.setVar(var, (time, cputime, proctime)) -def get_timedata(var, data): +def get_timedata(var, data, server_time=None): import time timedata = data.getVar(var, False) if timedata is None:@@ -124,7 +127,11 @@ def get_timedata(var, data): oldtime, oldcpu, oldproc = timedata procdiff = get_process_cputime(os.getpid()) - oldproc cpudiff = get_cputime() - oldcpu - timediff = time.time() - oldtime + if server_time: + end_time = server_time + else: + end_time = time.time() + timediff = end_time - oldtime if cpudiff > 0: cpuperc = float(procdiff) * 100 / cpudiff else:@@ -136,7 +143,7 @@ def write_task_data(status, logfile, dev, e): bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn) taskdir = os.path.join(bsdir, e.data.expand("${PF}")) file = open(os.path.join(logfile), "a") - timedata = get_timedata("__timedata_task", e.data) + timedata = get_timedata("__timedata_task", e.data, e.time) if timedata: elapsedtime, cpu = timedata file.write(bb.data.expand("${PF}: %s: Elapsed time: %0.2f seconds \n" %@@ -160,7 +167,7 @@ def write_task_data(status, logfile, dev, e): file.write("Status: PASSED \n") else: file.write("Status: FAILED \n") - file.write("Ended: %0.2f \n" % time.time()) + file.write("Ended: %0.2f \n" % e.time) file.close() python run_buildstats () {@@ -234,7 +241,7 @@ python run_buildstats () { taskdir = os.path.join(bsdir, e.data.expand("${PF}")) if device != "NoLogicalDevice": set_diskdata("__diskdata_task", device, e.data) - set_timedata("__timedata_task", e.data) + set_timedata("__timedata_task", e.data, e.time) try: bb.utils.mkdirhier(taskdir) except:@@ -242,7 +249,7 @@ python run_buildstats () { # write into the task event file the name and start time file = open(os.path.join(taskdir, e.task), "a") file.write("Event: %s \n" % bb.event.getName(e)) - file.write("Started: %0.2f \n" % time.time()) + file.write("Started: %0.2f \n" % e.time) file.close() elif isinstance(e, bb.build.TaskSucceeded):
Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time
Quoted 9 lines · Saul Wold
Quoted 6 lines · Alex DAMIAN
From: Marius Avram <[email protected]>
To avoid any further inconsistencies between buildstats and buildinfohelper from toaster, buildstats will measure task duration using the time field from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.
This patch seems to be causing some problems:
I suspect this is dependent on a bitbake change...
Cheers,
Richard
Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time
Yep, there is a counterpart change submitted to bitbake.
Alex
Quoted 21 lines · Richard Purdie
Quoted 4 lines · Saul Wold
Quoted 3 lines · Alex DAMIAN
From: Marius Avram <[email protected]>
To avoid any further inconsistencies between buildstats and
buildinfohelper
Quoted 1 line
Quoted 1 line
from toaster, buildstats will measure task duration using the time
field
Quoted 1 line
Quoted 1 line
from within the TaskBase events: TaskStarted and
TaskSucceeded/TaskFailed.
Quoted 3 lines
Quoted 1 line
This patch seems to be causing some problems:
I suspect this is dependent on a bitbake change...
Cheers,
Richard
Signature · 3 lines
Alex Damian Yocto Project SSG / OTC