Skip to contentSkip to conversation

toaster-related patchset

PATCH In review Alex DAMIAN 3 participants 7 messages Mar 17, 2014 – Mar 19, 2014

lore
Share & tools
lore
#0 lore raw

cover toaster-related patchset

From: Alexandru DAMIAN <[email protected]>

This is a rebased patchset consisting of patches that have been previously submitted to the oe-core mail list separately, but not yet merged.

In order to make merging easier, I'm resubmitting as a new patchset.

The following changes since commit e8476a9d19dba7a1a6c5a558a89f2c9f89d39ae2:

gettext: upgrade from 0.18.3.1 to 0.18.3.2 (2014-03-17 14:01:43 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib adamian/submission-17032014
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/submission-17032014

Alexandru DAMIAN (1):
sstate: list missing files for toaster

Cristiana Voicu (1):
toaster.bbclass: read the data needed for license manifest path

Marius Avram (1):
buildstats: use TaskBase time for elapsed time

 meta/classes/buildstats.bbclass | 23 +++++++++++++++--------
 meta/classes/sstate.bbclass     | 11 +++++++++++
 meta/classes/toaster.bbclass    | 12 ++++++++++++
 3 files changed, 38 insertions(+), 8 deletions(-)
Signature · 1 line
1.8.3.2
#1 lore raw

PATCH 1/3 toaster.bbclass: read the data needed for license manifest path

From: Cristiana Voicu <[email protected]>

The license.manifest file is located in DEPLOY_DIR_IMAGE/licenses/ IMAGE_NAME dir. The data needed is collected after rootfs task.

[YOCTO #5649]
---
 meta/classes/toaster.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)
meta/classes/toaster.bbclass +12 -0
index ddfceb5..563b1f7 100644
@@ -296,6 +296,17 @@ python toaster_buildhistory_dump() {
 
 }
 
+# dump information related to license manifest path
+
+python toaster_licensemanifest_dump() {
+    deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE', True);
+    image_name = d.getVar('IMAGE_NAME', True);
+
+    data = { 'deploy_dir_image' : deploy_dir_image, 'image_name' : image_name }
+
+    bb.event.fire(bb.event.MetadataEvent("LicenseManifestPath", data), d)
+}
+
 # set event handlers
 addhandler toaster_layerinfo_dumpdata
 toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -308,3 +319,4 @@ toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 do_package[postfuncs] += "toaster_package_dumpdata "
 
 do_rootfs[postfuncs] += "toaster_image_dumpdata "
+do_rootfs[postfuncs] += "toaster_licensemanifest_dump "
Signature · 1 line
1.8.3.2
#2 lore raw

PATCH 2/3 sstate: list missing files for toaster

From: Alexandru DAMIAN <[email protected]>

Toaster needs to record the attempts to restore setscene tasks that don't have a sstate file.

We build a list of tasks for which we can't find an sstate file, and if we're running under Toaster data collection, we send it off with a MetadataEvent.

---
 meta/classes/sstate.bbclass | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/classes/sstate.bbclass +11 -0
index 03ee820..25b8d72 100644
@@ -625,6 +625,7 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     ret = []
+    missed = []
 
     def getpathcomponents(task, d):
         # Magic data from BB_HASHFILENAME
@@ -646,11 +647,13 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
         spec, extrapath, tname = getpathcomponents(task, d)
 
         sstatefile = d.expand("${SSTATE_DIR}/" + extrapath + generate_sstatefn(spec, sq_hash[task], d) + "_" + tname + ".tgz.siginfo")
+
         if os.path.exists(sstatefile):
             bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
             ret.append(task)
             continue
         else:
+            missed.append(task)
             bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile)
 
     mirrors = d.getVar("SSTATE_MIRRORS", True)
@@ -688,9 +691,17 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
                 bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
                 ret.append(task)
             except:
+                missed.append(task)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri)
                 pass     
 
+    inheritlist = d.getVar("INHERIT", True)
+    if "toaster" in inheritlist:
+        evdata = []
+        for task in missed:
+            evdata.append( (sq_fn[task], sq_task[task], sq_hash[task], generate_sstatefn(spec, sq_hash[task],d) ) )
+        bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"
Signature · 1 line
1.8.3.2
#3 lore raw

PATCH 3/3 buildstats: use TaskBase time for elapsed time

In reply to Alex DAMIAN

From: Marius Avram <[email protected]>

To avoid any further inconsistencies between buildstats and buildinfohelper from toaster, buildstats will measure task duration using the time field from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.

---
 meta/classes/buildstats.bbclass | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)
meta/classes/buildstats.bbclass +15 -8
index 72fff11..89ae72c 100644
@@ -109,14 +109,17 @@ def get_diskdata(var, dev, data):
         diskdata["End"+key] = str(int(newdiskdata[key]))    
     return diskdata
     
-def set_timedata(var, data):
+def set_timedata(var, data, server_time=None):
     import time
-    time = time.time()
+    if server_time:
+        time = server_time
+    else:
+        time = time.time()
     cputime = get_cputime()
     proctime = get_process_cputime(os.getpid())
     data.setVar(var, (time, cputime, proctime))
 
-def get_timedata(var, data):
+def get_timedata(var, data, server_time=None):
     import time
     timedata = data.getVar(var, False)
     if timedata is None:
@@ -124,7 +127,11 @@ def get_timedata(var, data):
     oldtime, oldcpu, oldproc = timedata
     procdiff = get_process_cputime(os.getpid()) - oldproc
     cpudiff = get_cputime() - oldcpu
-    timediff = time.time() - oldtime
+    if server_time:
+        end_time = server_time
+    else:
+        end_time = time.time()
+    timediff = end_time - oldtime
     if cpudiff > 0:
         cpuperc = float(procdiff) * 100 / cpudiff
     else:
@@ -136,7 +143,7 @@ def write_task_data(status, logfile, dev, e):
     bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn)
     taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
     file = open(os.path.join(logfile), "a")
-    timedata = get_timedata("__timedata_task", e.data)
+    timedata = get_timedata("__timedata_task", e.data, e.time)
     if timedata:
         elapsedtime, cpu = timedata
         file.write(bb.data.expand("${PF}: %s: Elapsed time: %0.2f seconds \n" %
@@ -160,7 +167,7 @@ def write_task_data(status, logfile, dev, e):
 	    file.write("Status: PASSED \n")
     else:
         file.write("Status: FAILED \n")
-    file.write("Ended: %0.2f \n" % time.time())
+    file.write("Ended: %0.2f \n" % e.time)
     file.close()
 
 python run_buildstats () {
@@ -234,7 +241,7 @@ python run_buildstats () {
         taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
         if device != "NoLogicalDevice":
             set_diskdata("__diskdata_task", device, e.data)
-        set_timedata("__timedata_task", e.data)
+        set_timedata("__timedata_task", e.data, e.time)
         try:
             bb.utils.mkdirhier(taskdir)
         except:
@@ -242,7 +249,7 @@ python run_buildstats () {
         # write into the task event file the name and start time
         file = open(os.path.join(taskdir, e.task), "a")
         file.write("Event: %s \n" % bb.event.getName(e))
-        file.write("Started: %0.2f \n" % time.time())
+        file.write("Started: %0.2f \n" % e.time)
         file.close()
 
     elif isinstance(e, bb.build.TaskSucceeded):
Signature · 1 line
1.8.3.2
#4 lore raw

Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time

Quoted 6 lines · Alex DAMIAN

From: Marius Avram <[email protected]>

To avoid any further inconsistencies between buildstats and buildinfohelper from toaster, buildstats will measure task duration using the time field from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.

This patch seems to be causing some problems:

Quoted 16 lines

ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 72, in run_buildstats(e=<bb.build.TaskStarted object at 0x6e45590>)
AttributeError: 'TaskStarted' object has no attribute 'time'

ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 88, in run_buildstats(e=<bb.build.TaskSucceeded object at 0x6e40190>)
File "buildstats.bbclass", line 5, in write_task_data(status='passed', logfile='/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_clean', dev='sda5', e=<bb.build.TaskSucceeded object at 0x6e40190>)
IOError: [Errno 2] No such file or directory: '/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_clean'

ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 72, in run_buildstats(e=<bb.build.TaskStarted object at 0x74f59d0>)
AttributeError: 'TaskStarted' object has no attribute 'time'

Quoted 4 lines

ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 88, in run_buildstats(e=<bb.build.TaskSucceeded object at 0x6e48dd0>)
File "buildstats.bbclass", line 5, in write_task_data(status='passed', logfile='/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_cleansstate', dev='sda5', e=<bb.build.TaskSucceeded object at 0x6e48dd0>)

Sau!

Quoted 80 lines
---
  meta/classes/buildstats.bbclass | 23 +++++++++++++++--------
  1 file changed, 15 insertions(+), 8 deletions(-)
meta/classes/buildstats.bbclass +15 -8
index 72fff11..89ae72c 100644
@@ -109,14 +109,17 @@ def get_diskdata(var, dev, data):
          diskdata["End"+key] = str(int(newdiskdata[key]))
      return diskdata
 
-def set_timedata(var, data):
+def set_timedata(var, data, server_time=None):
      import time
-    time = time.time()
+    if server_time:
+        time = server_time
+    else:
+        time = time.time()
      cputime = get_cputime()
      proctime = get_process_cputime(os.getpid())
      data.setVar(var, (time, cputime, proctime))
 
-def get_timedata(var, data):
+def get_timedata(var, data, server_time=None):
      import time
      timedata = data.getVar(var, False)
      if timedata is None:
@@ -124,7 +127,11 @@ def get_timedata(var, data):
      oldtime, oldcpu, oldproc = timedata
      procdiff = get_process_cputime(os.getpid()) - oldproc
      cpudiff = get_cputime() - oldcpu
-    timediff = time.time() - oldtime
+    if server_time:
+        end_time = server_time
+    else:
+        end_time = time.time()
+    timediff = end_time - oldtime
      if cpudiff > 0:
          cpuperc = float(procdiff) * 100 / cpudiff
      else:
@@ -136,7 +143,7 @@ def write_task_data(status, logfile, dev, e):
      bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn)
      taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
      file = open(os.path.join(logfile), "a")
-    timedata = get_timedata("__timedata_task", e.data)
+    timedata = get_timedata("__timedata_task", e.data, e.time)
      if timedata:
          elapsedtime, cpu = timedata
          file.write(bb.data.expand("${PF}: %s: Elapsed time: %0.2f seconds \n" %
@@ -160,7 +167,7 @@ def write_task_data(status, logfile, dev, e):
  	    file.write("Status: PASSED \n")
      else:
          file.write("Status: FAILED \n")
-    file.write("Ended: %0.2f \n" % time.time())
+    file.write("Ended: %0.2f \n" % e.time)
      file.close()
 
  python run_buildstats () {
@@ -234,7 +241,7 @@ python run_buildstats () {
          taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
          if device != "NoLogicalDevice":
              set_diskdata("__diskdata_task", device, e.data)
-        set_timedata("__timedata_task", e.data)
+        set_timedata("__timedata_task", e.data, e.time)
          try:
              bb.utils.mkdirhier(taskdir)
          except:
@@ -242,7 +249,7 @@ python run_buildstats () {
          # write into the task event file the name and start time
          file = open(os.path.join(taskdir, e.task), "a")
          file.write("Event: %s \n" % bb.event.getName(e))
-        file.write("Started: %0.2f \n" % time.time())
+        file.write("Started: %0.2f \n" % e.time)
          file.close()
 
      elif isinstance(e, bb.build.TaskSucceeded):
#5 lore raw

Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time

Quoted 9 lines · Saul Wold
Quoted 6 lines · Alex DAMIAN

From: Marius Avram <[email protected]>

To avoid any further inconsistencies between buildstats and buildinfohelper from toaster, buildstats will measure task duration using the time field from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.

This patch seems to be causing some problems:

I suspect this is dependent on a bitbake change...

Cheers,

Richard

#6 lore raw

Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time

Yep, there is a counterpart change submitted to bitbake.

Alex

Quoted 21 lines · Richard Purdie
Quoted 4 lines · Saul Wold
Quoted 3 lines · Alex DAMIAN

From: Marius Avram <[email protected]>

To avoid any further inconsistencies between buildstats and

buildinfohelper

Quoted 1 line
Quoted 1 line

from toaster, buildstats will measure task duration using the time

field

Quoted 1 line
Quoted 1 line

from within the TaskBase events: TaskStarted and

TaskSucceeded/TaskFailed.

Quoted 3 lines
Quoted 1 line

This patch seems to be causing some problems:

I suspect this is dependent on a bitbake change...

Cheers,

Richard

Signature · 3 lines
Alex Damian
Yocto Project
SSG / OTC