remove ssh build controller
cover remove ssh build controller
Hi,
sshbecontroller code and tests are broken for months. Its functionality(running builds on remote machine) can be achieved by simply running manage.py runbuilds on remote machine. There is not need to keep it in the codebase and confuse users from my point of view.
This change would also make refactoring of buildcontroller code [YOCTO #8806] easier.
The following changes since commit c113ef1e40b9a5007c6b6af53226389fea706094:
toaster: tests Remove symlinks from toasteruitest folder (2016-01-20 12:07:44 +0200)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/toaster/remove-ssh-controller
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/remove-ssh-controller
Ed Bartosh (3):
toaster: raise NotImplementedError
toaster: don't use sshbecontroller
toaster: remove sshbecontroller module
bitbake/lib/toaster/bldcontrol/bbcontroller.py | 16 +- .../migrations/0002_auto_20160120_1250.py | 19 +++ bitbake/lib/toaster/bldcontrol/models.py | 10 +- bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 --------------------- bitbake/lib/toaster/bldcontrol/tests.py | 30 +--- 5 files changed, 32 insertions(+), 212 deletions(-) create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py
--
Regards,
Ed
PATCH 1/3 toaster: raise NotImplementedError
Raised NotImplementedError instead of Exception to be able to catch it.
This is a preparation for removing sshbecontroller module. It has to be done as code in bldcontrol/tests.py imports custom NotImplementedException from sshbecontroller.
--- bitbake/lib/toaster/bldcontrol/bbcontroller.py | 10 +++++----- bitbake/lib/toaster/bldcontrol/models.py | 8 ++++++-- bitbake/lib/toaster/bldcontrol/tests.py | 6 ++---- 3 files changed, 13 insertions(+), 11 deletions(-)
bitbake/lib/toaster/bldcontrol/bbcontroller.py +5 -5
@@ -138,7 +138,7 @@ class BuildEnvironmentController(object): After this method executes, self.be bbaddress/bbport MUST point to a running and free server, and the bbstate MUST be updated to "started". """ - raise Exception("FIXME: Must override in order to actually start the BB server") + raise NotImplementedError("FIXME: Must override in order to actually start the BB server") def setLayers(self, bitbake, ls):
@@ -149,7 +149,7 @@ class BuildEnvironmentController(object): a word of attention: by convention, the first layer for any build will be poky! """ - raise Exception("FIXME: Must override setLayers") + raise NotImplementedError("FIXME: Must override setLayers") def getBBController(self):
@@ -176,16 +176,16 @@ class BuildEnvironmentController(object): up to the implementing BEC. The return MUST be a REST URL where a GET will actually return the content of the artifact, e.g. for use as a "download link" in a web UI. """ - raise Exception("Must return the REST URL of the artifact") + raise NotImplementedError("Must return the REST URL of the artifact") def release(self): """ This stops the server and releases any resources. After this point, all resources are un-available for further reference """ - raise Exception("Must override BE release") + raise NotImplementedError("Must override BE release") def triggerBuild(self, bitbake, layers, variables, targets): - raise Exception("Must override BE release") + raise NotImplementedError("Must override BE release") class ShellCmdException(Exception): pass
bitbake/lib/toaster/bldcontrol/models.py +6 -2
@@ -42,13 +42,17 @@ class BuildEnvironment(models.Model): def get_artifact(self, path): if self.betype == BuildEnvironment.TYPE_LOCAL: return open(path, "r") - raise Exception("FIXME: artifact download not implemented for build environment type %s" % self.get_betype_display()) + raise NotImplementedError("FIXME: artifact download not implemented "\ + "for build environment type %s" % \ + self.get_betype_display()) def has_artifact(self, path): import os if self.betype == BuildEnvironment.TYPE_LOCAL: return os.path.exists(path) - raise Exception("FIXME: has artifact not implemented for build environment type %s" % self.get_betype_display()) + raise NotImplementedError("FIXME: has artifact not implemented for "\ + "build environment type %s" % \ + self.get_betype_display()) # a BuildRequest is a request that the scheduler will build using a BuildEnvironment # the build request queue is the table itself, ordered by state
bitbake/lib/toaster/bldcontrol/tests.py +2 -4
@@ -48,13 +48,12 @@ class BEControllerTests(object): self.assertTrue(err == '', "bitbake server pid %s not stopped" % err) def test_serverStartAndStop(self): - from bldcontrol.sshbecontroller import NotImplementedException obe = self._getBuildEnvironment() bc = self._getBEController(obe) try: # setting layers, skip any layer info bc.setLayers(BITBAKE_LAYER, POKY_LAYERS) - except NotImplementedException, e: + except NotImplementedError: print "Test skipped due to command not implemented yet" return True # We are ok with the exception as we're handling the git already exists
@@ -74,14 +73,13 @@ class BEControllerTests(object): self._serverForceStop(bc) def test_getBBController(self): - from bldcontrol.sshbecontroller import NotImplementedException obe = self._getBuildEnvironment() bc = self._getBEController(obe) layerSet = False try: # setting layers, skip any layer info layerSet = bc.setLayers(BITBAKE_LAYER, POKY_LAYERS) - except NotImplementedException: + except NotImplementedError: print "Test skipped due to command not implemented yet" return True # We are ok with the exception as we're handling the git already exists
Signature · 1 line
2.1.4
PATCH 2/3 toaster: don't use sshbecontroller
In reply to Ed BartoshRemoved usage of sshbecontroller from bbcontroller, models, tests and database schema.
--- bitbake/lib/toaster/bldcontrol/bbcontroller.py | 6 ------ .../migrations/0002_auto_20160120_1250.py | 19 +++++++++++++++++ bitbake/lib/toaster/bldcontrol/models.py | 2 -- bitbake/lib/toaster/bldcontrol/tests.py | 24 ---------------------- 4 files changed, 19 insertions(+), 32 deletions(-) create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
bitbake/lib/toaster/bldcontrol/bbcontroller.py +0 -6
@@ -76,13 +76,10 @@ def getBuildEnvironmentController(**kwargs): """ from localhostbecontroller import LocalhostBEController - from sshbecontroller import SSHBEController be = BuildEnvironment.objects.filter(Q(**kwargs))[0] if be.betype == BuildEnvironment.TYPE_LOCAL: return LocalhostBEController(be) - elif be.betype == BuildEnvironment.TYPE_SSH: - return SSHBEController(be) else: raise Exception("FIXME: Implement BEC for type %s" % str(be.betype))
@@ -105,9 +102,6 @@ class BuildEnvironmentController(object): on the local machine, with the "build/" directory under the "poky/" source checkout directory. Bash is expected to be available. - * SSH controller will run the Toaster BE on a remote machine, where the current user - can connect without raise Exception("FIXME: implement")word (set up with either ssh-agent or raise Exception("FIXME: implement")phrase-less key authentication) - """ def __init__(self, be): """ Takes a BuildEnvironment object as parameter that points to the settings of the BE.
bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py new file +19 -0
@@ -0,0 +1,19 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('bldcontrol', '0001_initial'), + ] + + operations = [ + migrations.AlterField( + model_name='buildenvironment', + name='betype', + field=models.IntegerField(choices=[(0, b'local')]), + ), + ]
bitbake/lib/toaster/bldcontrol/models.py +0 -2
@@ -12,10 +12,8 @@ class BuildEnvironment(models.Model): ) TYPE_LOCAL = 0 - TYPE_SSH = 1 TYPE = ( (TYPE_LOCAL, "local"), - (TYPE_SSH, "ssh"), ) LOCK_FREE = 0
bitbake/lib/toaster/bldcontrol/tests.py +0 -24
@@ -9,7 +9,6 @@ from django.test import TestCase from bldcontrol.bbcontroller import BitbakeController, BuildSetupException from bldcontrol.localhostbecontroller import LocalhostBEController -from bldcontrol.sshbecontroller import SSHBEController from bldcontrol.models import BuildEnvironment, BuildRequest from bldcontrol.management.commands.runbuilds import Command
@@ -110,29 +109,6 @@ class LocalhostBEControllerTests(TestCase, BEControllerTests): def _getBEController(self, obe): return LocalhostBEController(obe) -class SSHBEControllerTests(TestCase, BEControllerTests): - def __init__(self, *args): - super(SSHBEControllerTests, self).__init__(*args) - - def _getBuildEnvironment(self): - return BuildEnvironment.objects.create( - lock = BuildEnvironment.LOCK_FREE, - betype = BuildEnvironment.TYPE_SSH, - address = test_address, - sourcedir = test_sourcedir, - builddir = test_builddir ) - - def _getBEController(self, obe): - return SSHBEController(obe) - - def test_pathExists(self): - obe = BuildEnvironment.objects.create(betype = BuildEnvironment.TYPE_SSH, address= test_address) - sbc = SSHBEController(obe) - self.assertTrue(sbc._pathexists("/")) - self.assertFalse(sbc._pathexists("/.deadbeef")) - self.assertTrue(sbc._pathexists(sbc._shellcmd("pwd"))) - - class RunBuildsCommandTests(TestCase): def test_bec_select(self): """
Signature · 1 line
2.1.4
PATCH 3/3 toaster: remove sshbecontroller module
In reply to Ed BartoshThe code of this module is broken for a long time. The functionality of it can be easily achieved by running 'manage.py runbuilds.py' on remote machine.
[YOCTO #8806]
--- bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 ---------------------- 1 file changed, 169 deletions(-) delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py
bitbake/lib/toaster/bldcontrol/sshbecontroller.py deleted +0 -169
@@ -1,169 +0,0 @@ -# -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- -# -# BitBake Toaster Implementation -# -# Copyright (C) 2014 Intel Corporation -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - -import sys -import re -from django.db import transaction -from django.db.models import Q -from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget, BRBitbake -import subprocess - -from toastermain import settings - -from bbcontroller import BuildEnvironmentController, ShellCmdException, BuildSetupException - -class NotImplementedException(Exception): - pass - -def DN(path): - return "/".join(path.split("/")[0:-1]) - -class SSHBEController(BuildEnvironmentController): - """ Implementation of the BuildEnvironmentController for the localhost; - this controller manages the default build directory, - the server setup and system start and stop for the localhost-type build environment - - """ - - def __init__(self, be): - super(SSHBEController, self).__init__(be) - self.dburl = settings.getDATABASE_URL() - self.pokydirname = None - self.islayerset = False - - def _shellcmd(self, command, cwd = None): - if cwd is None: - cwd = self.be.sourcedir - - p = subprocess.Popen("ssh %s 'cd %s && %s'" % (self.be.address, cwd, command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True) - (out,err) = p.communicate() - if p.returncode: - if len(err) == 0: - err = "command: %s \n%s" % (command, out) - else: - err = "command: %s \n%s" % (command, err) - raise ShellCmdException(err) - else: - return out.strip() - - def _pathexists(self, path): - try: - self._shellcmd("test -e \"%s\"" % path) - return True - except ShellCmdException as e: - return False - - def _pathcreate(self, path): - self._shellcmd("mkdir -p \"%s\"" % path) - - def _setupBE(self): - assert self.pokydirname and self._pathexists(self.pokydirname) - self._pathcreate(self.be.builddir) - self._shellcmd("bash -c \"source %s/oe-init-build-env %s\"" % (self.pokydirname, self.be.builddir)) - - def startBBServer(self, brbe): - assert self.pokydirname and self._pathexists(self.pokydirname) - assert self.islayerset - cmd = self._shellcmd("bash -c \"source %s/oe-init-build-env %s && DATABASE_URL=%s source toaster start noweb brbe=%s\"" % (self.pokydirname, self.be.builddir, self.dburl, brbe)) - - port = "-1" - for i in cmd.split("\n"): - if i.startswith("Bitbake server address"): - port = i.split(" ")[-1] - print "Found bitbake server port ", port - - - assert self.be.sourcedir and self._pathexists(self.be.builddir) - self.be.bbaddress = self.be.address.split("@")[-1] - self.be.bbport = port - self.be.bbstate = BuildEnvironment.SERVER_STARTED - self.be.save() - - def _copyFile(self, filepath1, filepath2): - p = subprocess.Popen("scp '%s' '%s'" % (filepath1, filepath2), stdout=subprocess.PIPE, stderr = subprocess.PIPE, shell=True) - (out, err) = p.communicate() - if p.returncode: - raise ShellCmdException(err) - - def pullFile(self, local_filename, remote_filename): - _copyFile(local_filename, "%s:%s" % (self.be.address, remote_filename)) - - def pushFile(self, local_filename, remote_filename): - _copyFile("%s:%s" % (self.be.address, remote_filename), local_filename) - - def setLayers(self, bitbakes, layers): - """ a word of attention: by convention, the first layer for any build will be poky! """ - - assert self.be.sourcedir is not None - assert len(bitbakes) == 1 - # set layers in the layersource - - - raise NotImplementedException("Not implemented: SSH setLayers") - # 3. configure the build environment, so we have a conf/bblayers.conf - assert self.pokydirname is not None - self._setupBE() - - # 4. update the bblayers.conf - bblayerconf = os.path.join(self.be.builddir, "conf/bblayers.conf") - if not self._pathexists(bblayerconf): - raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf) - - import uuid - local_bblayerconf = "/tmp/" + uuid.uuid4() + "-bblayer.conf" - - self.pullFile(bblayerconf, local_bblayerconf) - - BuildEnvironmentController._updateBBLayers(local_bblayerconf, layerlist) - self.pushFile(local_bblayerconf, bblayerconf) - - os.unlink(local_bblayerconf) - - self.islayerset = True - return True - - def release(self): - assert self.be.sourcedir and self._pathexists(self.be.builddir) - import shutil - shutil.rmtree(os.path.join(self.be.sourcedir, "build")) - assert not self._pathexists(self.be.builddir) - - def triggerBuild(self, bitbake, layers, variables, targets): - # set up the buid environment with the needed layers - self.setLayers(bitbake, layers) - self.writeConfFile("conf/toaster-pre.conf", ) - self.writeConfFile("conf/toaster.conf", raw = "INHERIT+=\"toaster buildhistory\"") - - # get the bb server running with the build req id and build env id - bbctrl = self.getBBController() - - # trigger the build command - task = reduce(lambda x, y: x if len(y)== 0 else y, map(lambda y: y.task, targets)) - if len(task) == 0: - task = None - - bbctrl.build(list(map(lambda x:x.target, targets)), task) - - logger.debug("localhostbecontroller: Build launched, exiting. Follow build logs at %s/toaster_ui.log" % self.be.builddir) - - # disconnect from the server - bbctrl.disconnect()
Signature · 1 line
2.1.4
Quoted 7 lines · Ed Bartosh
sshbecontroller code and tests are broken for months. Its functionality(running
builds on remote machine) can be achieved by simply runningmanage.pyrunbuilds
on remote machine. There is not need to keep it in the codebase and confuse users from my point of view.
I am all in favour of this, but my preference would be to leave this until after the image customisation work is complete.
In particular, because this patchset includes migrations, it's going to add some complexity to merging it with the image customisation branch.
Also, it's not vital work for 2.1, yet it creates new opportunities for other stuff to break :)
Elliot
Quoted 42 lines
This change would also make refactoring of buildcontroller code [YOCTO #8806] easier.
The following changes since commit
c113ef1e40b9a5007c6b6af53226389fea706094:toaster: tests Remove symlinks from toasteruitest folder (2016-01-20
12:07:44 +0200)are available in the git repository at:
git://git.yoctoproject.org/poky-contribed/toaster/remove-ssh-controllerhttp://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/remove-ssh-controller
Ed Bartosh (3):
toaster: raise NotImplementedError
toaster: don't use sshbecontroller
toaster: remove sshbecontroller modulebitbake/lib/toaster/bldcontrol/bbcontroller.py | 16 +- .../migrations/0002_auto_20160120_1250.py | 19 +++ bitbake/lib/toaster/bldcontrol/models.py | 10 +- bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 --------------------- bitbake/lib/toaster/bldcontrol/tests.py | 30 +--- 5 files changed, 32 insertions(+), 212 deletions(-) create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py--
Regards,
Ed--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
Signature · 3 lines
Elliot Smith Software Engineer Intel Open Source Technology Centre
Quoted 9 lines · Ed Bartosh
Hi,
sshbecontroller code and tests are broken for months. Its functionality(running
builds on remote machine) can be achieved by simply runningmanage.pyrunbuilds
on remote machine. There is not need to keep it in the codebase and confuse users from my point of view.
It did confuse me :)
Quoted 42 lines
This change would also make refactoring of buildcontroller code [YOCTO #8806] easier.
The following changes since commit
c113ef1e40b9a5007c6b6af53226389fea706094:toaster: tests Remove symlinks from toasteruitest folder (2016-01-20
12:07:44 +0200)are available in the git repository at:
git://git.yoctoproject.org/poky-contribed/toaster/remove-ssh-controllerhttp://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/remove-ssh-controller
Ed Bartosh (3):
toaster: raise NotImplementedError
toaster: don't use sshbecontroller
toaster: remove sshbecontroller modulebitbake/lib/toaster/bldcontrol/bbcontroller.py | 16 +- .../migrations/0002_auto_20160120_1250.py | 19 +++ bitbake/lib/toaster/bldcontrol/models.py | 10 +- bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 --------------------- bitbake/lib/toaster/bldcontrol/tests.py | 30 +--- 5 files changed, 32 insertions(+), 212 deletions(-) create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py--
Regards,
Ed--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
Signature · 5 lines
സുജിത് ഹരിദാസന് Bangalore <Project>Contributor to KDE project http://fci.wikia.com/wiki/Anti-DRM-Campaign <Blog> http://sujithh.info
Hi,
I looked at this while I was in Atlanta. I had no problems with it but I tend to agree with elliot that I'd prefer to not let it get in michael's way for ic, so we should probably pend it til ic get's in. -b
Quoted 77 lines · sujith h
Quoted 10 lines · Ed Bartosh
Hi,
sshbecontroller code and tests are broken for months. Its functionality(running
builds on remote machine) can be achieved by simply runningmanage.pyrunbuilds
on remote machine. There is not need to keep it in the codebase and confuse
users from my point of view.It did confuse me :)
Quoted 37 lines
This change would also make refactoring of buildcontroller code [YOCTO #8806] easier.
The following changes since commit
c113ef1e40b9a5007c6b6af53226389fea706094:toaster: tests Remove symlinks from toasteruitest folder (2016-01-20
12:07:44 +0200)are available in the git repository at:
git://git.yoctoproject.org/poky-contribed/toaster/remove-ssh-controllerhttp://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/remove-ssh-controller
Ed Bartosh (3):
toaster: raise NotImplementedError
toaster: don't use sshbecontroller
toaster: remove sshbecontroller modulebitbake/lib/toaster/bldcontrol/bbcontroller.py | 16 +- .../migrations/0002_auto_20160120_1250.py | 19 +++ bitbake/lib/toaster/bldcontrol/models.py | 10 +- bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 --------------------- bitbake/lib/toaster/bldcontrol/tests.py | 30 +--- 5 files changed, 32 insertions(+), 212 deletions(-) create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py--
Regards,
EdQuoted 5 lines
--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster--
സുജിത് ഹരിദാസന്
Bangalore
<Project>Contributor to KDE project
http://fci.wikia.com/wiki/Anti-DRM-Campaign
<Blog> http://sujithh.info--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster