toaster: cummulative 28072017 patch
cover toaster: cummulative 28072017 patch
From: David Reyna <[email protected]>
WARNING: there is one new file added with this patch:
create mode 100644 bitbake/lib/toaster/toastergui/templates/health.html
* IMPORTANT: This fixes the fatal regressions to Toaster from xmlrpc changes in bitbake. This also addresses a new race condition that Toaster suddenly started hitting where the 'bitbake.lock' existed but was not populated with the server info for period of time (~ second).
toaster: move to new bitbake xmlrpc default
* This provides a health ping URL, plus adds a JSON API for fetching Toaster status for multi-instance Toaster aggregators
toaster: enable remote HTTP API for status aggregation
The following changes since commit 3c1c4ceaa470957fe5633d65215f165a38d61be7:
yocto-project-qs, ref-manual: Upgraded tar to 1.27 from 1.24 (2017-07-27 23:00:23 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib dreyna/submit/dreyna/toaster/cummulative_280717_patch
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dreyna/submit/dreyna/toaster/cummulative_280717_patch
David Reyna (2):
toaster: enable remote HTTP API for status aggregation
toaster: move to new bitbake xmlrpc default
lib/toaster/bldcontrol/bbcontroller.py | 4 +- lib/toaster/bldcontrol/localhostbecontroller.py | 19 ++++-- lib/toaster/orm/models.py | 2 +- lib/toaster/toastergui/templates/health.html | 6 ++ lib/toaster/toastergui/urls.py | 7 +- lib/toaster/toastergui/views.py | 86 ++++++++++++++++++++++++- lib/toaster/toastermain/urls.py | 5 +- 7 files changed, 118 insertions(+), 11 deletions(-) create mode 100644 bitbake/lib/toaster/toastergui/templates/health.html
Signature · 1 line
1.9.1
PATCH 1/2 toaster: enable remote HTTP API for status aggregation
From: David Reyna <[email protected]>
Add support for Toaster aggregators with a set of api links that return JSON data for (a) builds in progress, (b) builds completed,
(c) specific build data, and (d) an is-alive health ping link.
[YOCTO #11794]
--- lib/toaster/toastergui/templates/health.html | 6 ++ lib/toaster/toastergui/urls.py | 7 ++- lib/toaster/toastergui/views.py | 86 +++++++++++++++++++++++++++- lib/toaster/toastermain/urls.py | 5 +- 4 files changed, 100 insertions(+), 4 deletions(-) create mode 100644 bitbake/lib/toaster/toastergui/templates/health.html
lib/toaster/toastergui/templates/health.html new file +6 -0
@@ -0,0 +1,6 @@ +<!DOCTYPE html> +<html lang="en"> + <head><title>Toaster Health</title></head> + <body>Ok</body> +</html> +
lib/toaster/toastergui/urls.py +6 -1
@@ -244,6 +244,11 @@ urlpatterns = [ url(r'^mostrecentbuilds$', widgets.MostRecentBuildsView.as_view(), name='most_recent_builds'), - # default redirection + # JSON data for aggregators + url(r'^api/builds$', views.json_builds, name='json_builds'), + url(r'^api/building$', views.json_building, name='json_building'), + url(r'^api/build/(?P<build_id>\d+)$', views.json_build, name='json_build'), + + # default redirection url(r'^$', RedirectView.as_view(url='landing', permanent=True)), ]
lib/toaster/toastergui/views.py +85 -1
101 lines not on this page · raw message
lib/toaster/toastermain/urls.py +3 -2
@@ -20,9 +20,8 @@ # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. from django.conf.urls import patterns, include, url -from django.views.generic import RedirectView +from django.views.generic import RedirectView, TemplateView from django.views.decorators.cache import never_cache - import bldcollector.views import logging
@@ -46,6 +45,8 @@ urlpatterns = [ # in the future. url(r'^orm/eventfile$', bldcollector.views.eventfile), + url(r'^health$', TemplateView.as_view(template_name="health.html"), name='Toaster Health'), + # if no application is selected, we have the magic toastergui app here url(r'^$', never_cache(RedirectView.as_view(url='/toastergui/', permanent=True))), ]
Signature · 1 line
1.9.1
PATCH 2/2 toaster: move to new bitbake xmlrpc default
In reply to David ReynaFrom: David Reyna <[email protected]>
The bitbake option "-t SERVERTYPE" was deprecated and can be removed since the desired XMLRPC listener now the default.
The bitbake server port cannot be "-1" anymore and must be explicitly passed.
There is a race condition for the bblock file to not only be created but to actually be populated, so a delay test loop is required (usually only one pass).
The 'xmlrpcclient' is now the class that allows Toaster to for example kill builds.
The events for populating the recipe parsing now either show only the final result or are skipped entiredly, so the progress calculator needs to be changed to not block on the parsing phase.
[YOCTO #11851]
--- lib/toaster/bldcontrol/bbcontroller.py | 4 ++-- lib/toaster/bldcontrol/localhostbecontroller.py | 19 +++++++++++++++---- lib/toaster/orm/models.py | 2 +- 3 files changed, 18 insertions(+), 7 deletions(-)
lib/toaster/bldcontrol/bbcontroller.py +2 -2
@@ -37,8 +37,8 @@ class BitbakeController(object): """ def __init__(self, be): - import bb.server.xmlrpc - self.connection = bb.server.xmlrpc._create_server(be.bbaddress, + import bb.server.xmlrpcclient + self.connection = bb.server.xmlrpcclient._create_server(be.bbaddress, int(be.bbport))[0] def _runCommand(self, command):
lib/toaster/bldcontrol/localhostbecontroller.py +15 -4
@@ -24,6 +24,7 @@ import os import sys import re import shutil +import time from django.db import transaction from django.db.models import Q from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget, BRBitbake
@@ -331,12 +332,22 @@ class LocalhostBEController(BuildEnvironmentController): bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake') toasterlayers = os.path.join(builddir,"conf/toaster-bblayers.conf") self._shellcmd('bash -c \"source %s %s; BITBAKE_UI="knotty" %s --read %s --read %s ' - '--server-only -t xmlrpc -B 0.0.0.0:0\"' % (oe_init, + '--server-only -B 0.0.0.0:0\"' % (oe_init, builddir, bitbake, confpath, toasterlayers), self.be.sourcedir) # read port number from bitbake.lock self.be.bbport = "" bblock = os.path.join(builddir, 'bitbake.lock') + # allow 10 seconds for bb lock file to appear but also be populated + for lock_check in range(10): + if not os.path.exists(bblock): + logger.debug("localhostbecontroller: waiting for bblock file to appear") + time.sleep(1) + continue + if 10 < os.stat(bblock).st_size: + break + logger.debug("localhostbecontroller: waiting for bblock content to appear") + time.sleep(1) with open(bblock) as fplock: for line in fplock: if ":" in line:
@@ -365,10 +376,10 @@ class LocalhostBEController(BuildEnvironmentController): log = os.path.join(builddir, 'toaster_ui.log') local_bitbake = os.path.join(os.path.dirname(os.getenv('BBBASEDIR')), 'bitbake') - self._shellcmd(['bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" ' + self._shellcmd(['bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:%s" ' '%s %s -u toasterui --token="" >>%s 2>&1;' - 'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:-1 %s -m)&\"' \ - % (brbe, local_bitbake, bbtargets, log, bitbake)], + 'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:%s %s -m)&\"' \ + % (brbe, self.be.bbport, local_bitbake, bbtargets, log, self.be.bbport, bitbake)], builddir, nowait=True) logger.debug('localhostbecontroller: Build launched, exiting. '
lib/toaster/orm/models.py +1 -1
@@ -451,7 +451,7 @@ class Build(models.Model): recipes_to_parse = models.IntegerField(default=1) # number of recipes parsed so far for this build - recipes_parsed = models.IntegerField(default=0) + recipes_parsed = models.IntegerField(default=1) # number of repos to clone for this build repos_to_clone = models.IntegerField(default=1)
Signature · 1 line
1.9.1