Skip to contentSkip to conversation

1/2] buildsteps/CreateAutoConf.py: Add support for specify MACHINE by build prop

PATCH Superseded Aníbal Limón 3 participants 5 messages Jan 16, 2017 – Jan 19, 2017

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 1/2] buildsteps/CreateAutoConf.py: Add support for specify MACHINE by build prop

Sometimes is a good idea to could specify a machine to build via a build property.

---
 .../site-packages/autobuilder/buildsteps/CreateAutoConf.py          | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py +5 -1
index 021f542..7915dd5 100644
@@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):
                     fout = fout + 'baselib = \\042${@d.getVar(\\047BASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
             if self.distro == "poky-rt":
                 fout = fout + 'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n'
-            machine=self.machine
+            machine=self.getProperty('custom_machine')
+            if machine:
+                self.machine = machine
+            else:
+                machine = self.machine
             if layerversion is not None and int(layerversion) > 1:
                 if self.machine == "atom-pc":
                     machine = self.machine.replace("atom-pc", "genericx86")
Signature · 1 line
2.1.4
#1 lore raw

PATCH 1/1 2/2] buildset-config.yocto-qa: Add nightly-oe-build-perf-test buildset

From: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>

The new nightly-oe-build-perf-test buildset executes the build-perf-test-wrapper.sh into a worker and publish the results to specified folder by AB config.

When the performance script finish it sends an notifcation email to yocto alias users, the worker machine needs to configure the email addresses for yocto alias.

[YOCTO #9377]

Signed-off-by Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
---
 .../nightly-oe-build-perf-test.conf                | 28 ++++++++
 config/autobuilder.conf.example                    |  8 +++
 .../autobuilder/buildsteps/RunOeBuildPerfTest.py   | 78 ++++++++++++++++++++++
 .../autobuilder/buildsteps/SendOePerfEmail.py      | 64 ++++++++++++++++++
 .../buildsteps/TarballOeBuildPerfTest.py           | 33 +++++++++
 lib/python2.7/site-packages/autobuilder/config.py  |  2 +-
 6 files changed, 212 insertions(+), 1 deletion(-)
 create mode 100644 buildset-config.yocto-qa/nightly-oe-build-perf-test.conf
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/RunOeBuildPerfTest.py
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/SendOePerfEmail.py
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/TarballOeBuildPerfTest.py
buildset-config.yocto-qa/nightly-oe-build-perf-test.conf new file +28 -0
new file mode 100644
index 0000000..999036d
@@ -0,0 +1,28 @@
+[nightly-oe-build-perf-test]
+builders: 'example-worker'
+repos: [{'poky':
+            {'repourl':'git://git.yoctoproject.org/poky',
+             'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
+             'branch':'master'}}]
+props: [{'machine':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['qemux86', 'qemuarm', 'qemumips', 'qemuppc', 'qemux86-64', 'qemuarm64', 'qemumips64'],
+                       'name': 'machine',
+                       'label':'<hr>Selects machine for performance measurement.'}},
+        {'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
+steps: [{'SetDest':{}},
+        {'CheckOutLayers': {}},
+        {'RunPreamble': {}},
+        {'GetDistroVersion' : {'distro': 'poky'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'i686',
+                            'distro': 'poky', 'buildhistory' : True,
+                            'atextappend' : 'CONNECTIVITY_CHECK_URIS = ""'}},
+        {'CreateBBLayersConf': {'buildprovider' : 'yocto'}},
+        {'SyncPersistDB' : {'distro' : 'poky'}},
+        {'GetBitbakeVersion': {}},
+        {'RunOeBuildPerfTest': {}},
+        {'TarballOeBuildPerfTest': {}},
+        {'SendOePerfEmail': {}},
+        ]
config/autobuilder.conf.example +8 -0
index 2e42013..2ee11e6 100644
@@ -87,3 +87,11 @@ QA_MAIL_SIG = "Multiline\nSig\nLine"
 
 [Buildlogger]
 BUILDLOG_TO_WIKI = False
+
+[Performance]
+PERFORMANCE_PUBLISH_DIR = "/tmp/yocto-autobuilder/performance"
+PERFORMANCE_SEND_EMAIL = "False"
+PERFORMANCE_MAIL_TO = "root@localhost otherperson@localhost"
+PERFORMANCE_MAIL_CC = "buildcc@localhost"
+PERFORMANCE_MAIL_BCC = "buildbcc@localhost"
+PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
lib/python2.7/site-packages/autobuilder/buildsteps/RunOeBuildPerfTest.py new file +78 -0

78 lines not on this page · raw message

lib/python2.7/site-packages/autobuilder/buildsteps/SendOePerfEmail.py new file +64 -0

64 lines not on this page · raw message

lib/python2.7/site-packages/autobuilder/buildsteps/TarballOeBuildPerfTest.py new file +33 -0
new file mode 100644
index 0000000..ca60c93
@@ -0,0 +1,33 @@
+import os, datetime, subprocess
+from buildbot.steps.shell import ShellCommand
+from autobuilder.config import PERFORMANCE_PUBLISH_DIR
+
+class TarballOeBuildPerfTest(ShellCommand):
+    flunkOnFailure = True
+    name = "Tarball oe-build-perf-test results"
+
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.tests = None
+        self.factory = factory
+        for k, v in argdict.iteritems():
+                setattr(self, k, v)
+        self.description = "Running tarball oe-build-perf-test results"
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+        ShellCommand.__init__(self, **kwargs)
+  
+    def start(self):
+        results_dir = self.getProperty("oe_perf_results_dir")
+
+        results_basename = os.path.basename(results_dir)
+        archive_dir = os.path.join(PERFORMANCE_PUBLISH_DIR, 'archives')
+        archive_file = os.path.join(archive_dir, results_basename + '.tar.gz')
+
+        self.setProperty("oe_perf_archive_dir", archive_dir, "TarballOeBuildPerfTest")
+        self.setProperty("oe_perf_archive_file", archive_dir, "TarballOeBuildPerfTest")
+
+        self.command = "mkdir -p %s; " % archive_dir
+        self.command += "tar -czf \"%s\" \"%s\"" % (archive_file, results_dir)
+                
+        ShellCommand.start(self)
+
lib/python2.7/site-packages/autobuilder/config.py +1 -1
index eced0c7..9d945b1 100644
@@ -22,4 +22,4 @@ X8664TC_PUBLISH_DIR = os.environ.get("X8664TC_PUBLISH_DIR")
 RPM_PUBLISH_DIR = os.environ.get("RPM_PUBLISH_DIR")
 IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
 DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
-
+PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
Signature · 1 line
2.1.4
#2 lore raw
In reply to Aníbal Limón
Quoted 3 lines · Aníbal Limón

Sometimes is a good idea to could specify a machine to build via a build
property.

Why's that? I can guess but the commit log should really tell me what the motivation for the change is.

Quoted 33 lines

---
 .../site- packages/autobuilder/buildsteps/CreateAutoConf.py          | 6 +++++-

 1 file changed, 5 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

packages/autobuilder/buildsteps/CreateAutoConf.py
b/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py
index 021f542..7915dd5 100644
--- a/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py
+++ b/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py

(quoted diff) +0 -0
@@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):

                     fout = fout + 'baselib = \\042${@d.getVar(\\047B ASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
             if self.distro == "poky-rt":
                 fout = fout + 'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n'
-            machine=self.machine
+            machine=self.getProperty('custom_machine')
+            if machine:
+                self.machine = machine
+            else:
+                machine = self.machine
             if layerversion is not None and int(layerversion) > 1:
                 if self.machine == "atom-pc":
                     machine = self.machine.replace("atom-pc", "genericx86")

#3 lore raw
Quoted 7 lines · Joshua Lock
Quoted 3 lines · Aníbal Limón

Sometimes is a good idea to could specify a machine to build via a build
property.

Why's that? I can guess but the commit log should really tell me what the motivation for the change is.

In the performance buildset patch [1] there is a build property to specify the machine via the web interface, without this change the machine is overwrite to predefined in the buildset.

Cheers,
alimon

[1]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/tree/buildset-config.yocto-qa/nightly-oe-build-perf-test.conf?h=contrib/alimon/devel&id=6c25a37cd801f2cdecc21c76499625a117e08ff7#n7

Quoted 34 lines
Quoted 34 lines

---
.../site-
packages/autobuilder/buildsteps/CreateAutoConf.py | 6 +++++-

 1 file changed, 5 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

packages/autobuilder/buildsteps/CreateAutoConf.py
b/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py
index 021f542..7915dd5 100644
--- a/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py
+++ b/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py

(quoted diff) +0 -0
@@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):
                     fout = fout + 'baselib = \\042${@d.getVar(\\047B

ASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
if self.distro == "poky-rt":
fout = fout +
'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n'

lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py +5 -1
-            machine=self.machine
+            machine=self.getProperty('custom_machine')
+            if machine:
+                self.machine = machine
+            else:
+                machine = self.machine
             if layerversion is not None and int(layerversion) > 1:
                 if self.machine == "atom-pc":
                     machine = self.machine.replace("atom-pc",

"genericx86")

#4 lore raw
In reply to Joshua Lock
Quoted 7 lines · Joshua Lock
Quoted 4 lines · Aníbal Limón

Sometimes is a good idea to could specify a machine to build via a build
property.

Why's that? I can guess but the commit log should really tell me what the motivation for the change is.

I agree. Please submit a v2 with a bit better of a commit log.

-b

Quoted 37 lines
Quoted 37 lines

---
 .../site- packages/autobuilder/buildsteps/CreateAutoConf.py          | 6
+++++-

 1 file changed, 5 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

packages/autobuilder/buildsteps/CreateAutoConf.py
b/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py
index 021f542..7915dd5 100644
--- a/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py
+++ b/lib/python2.7/site-
packages/autobuilder/buildsteps/CreateAutoConf.py

(quoted diff) +0 -0
@@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):

                     fout = fout + 'baselib = \\042${@d.getVar(\\04 7B
ASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
             if self.distro == "poky-rt":
                 fout = fout + 'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n' -            machine=self.machine
+            machine=self.getProperty('custom_machine') +            if machine:
+                self.machine = machine
+            else:
+                machine = self.machine
             if layerversion is not None and int(layerversion) > 1:
                 if self.machine == "atom-pc":
                     machine = self.machine.replace("atom-pc", "genericx86")