Fix for #8918: Change the way we handle queued builds
cover Fix for #8918: Change the way we handle queued builds
Hi,
This patchset introduced a bit lighter way of handling builds. Instead of polling database every second we process builds only when runbuilds receives SIGUSR1. The signal is sent when build is either scheduled, canceled or finished. A little bit of code cleanup is also included into this patchset.
Test instructions:
- Create multiple builds in UI. Check if they'll be built sequentially
- Create multiple builds in UI, cancel one. Check if the next one starts building.
- Repeat first two test cases when command line build is run. Only one UI build should be run in parallel with command line build.
- Test any other combinations of running and cancelling builds to make sure there are no regressions.
The following changes since commit 75927f8f6809b9b4696fc438b240bc7a3db561e2:
toaster: don't kill all runserver processes (2016-09-05 15:32:25 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/toaster/nopoll-queued-builds-8918
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/nopoll-queued-builds-8918
Ed Bartosh (7):
toaster: implement signal_runbuilds function
toaster: notify runbuilds when build status changes
runbuilds: process builds on SIGUSR1
runbuilds: process builds on start
runbuilds: code cleanup - whitespaces, long lines
runbuilds: code cleanup - remove unused imports
toaster: unlock BuildEnvirnoment when build is done
bitbake/lib/bb/ui/buildinfohelper.py | 12 ++++-- .../bldcontrol/management/commands/runbuilds.py | 49 ++++++++++------------ bitbake/lib/toaster/orm/models.py | 10 ++++- 3 files changed, 41 insertions(+), 30 deletions(-)
--
Regards,
Ed
PATCH 1/7 toaster: implement signal_runbuilds function
This function reads pid of runbuilds process from BUILDDIR/.runbuilds.pid and sends SIGUSR1 to it. signal_runbuilds function will be used in Toaster code to notify runbuilds when build is scheduled, finished or cancelled.
--- bitbake/lib/toaster/orm/models.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
bitbake/lib/toaster/orm/models.py +7 -1
@@ -33,9 +33,10 @@ from django.conf import settings import django.db.models.signals import sys -import os.path +import os import re import itertools +from signal import SIGUSR1 import logging logger = logging.getLogger("toaster")
@@ -1736,6 +1737,11 @@ def invalidate_cache(**kwargs): except Exception as e: logger.warning("Problem with cache backend: Failed to clear cache: %s" % e) +def signal_runbuilds(): + """Send SIGUSR1 to runbuilds process""" + with open(os.path.join(os.getenv('BUILDDIR'), '.runbuilds.pid')) as pidf: + os.kill(int(pidf.read()), SIGUSR1) + django.db.models.signals.post_save.connect(invalidate_cache) django.db.models.signals.post_delete.connect(invalidate_cache) django.db.models.signals.m2m_changed.connect(invalidate_cache)
Signature · 1 line
2.1.4
PATCH 2/7 toaster: notify runbuilds when build status changes
In reply to Ed BartoshCalled signal_runbuilds API when build is scheduled, cancelled or finished to notify runbuilds process about builds status change.
[YOCTO #8918]
--- bitbake/lib/bb/ui/buildinfohelper.py | 4 ++++ bitbake/lib/toaster/orm/models.py | 2 ++ 2 files changed, 6 insertions(+)
bitbake/lib/bb/ui/buildinfohelper.py +4 -0
@@ -43,6 +43,7 @@ from orm.models import Package, Package_File, Target_Installed_Package, Target_F from orm.models import Task_Dependency, Package_Dependency from orm.models import Recipe_Dependency, Provides from orm.models import Project, CustomImagePackage, CustomImageRecipe +from orm.models import signal_runbuilds from bldcontrol.models import BuildEnvironment, BuildRequest
@@ -234,6 +235,7 @@ class ORMWrapper(object): build.completed_on = timezone.now() build.outcome = outcome build.save() + signal_runbuilds() def update_target_set_license_manifest(self, target, license_manifest_path): target.license_manifest_path = license_manifest_path
@@ -1354,6 +1356,7 @@ class BuildInfoHelper(object): self._ensure_build() self.internal_state['build'].outcome = Build.CANCELLED self.internal_state['build'].save() + signal_runbuilds() def store_dependency_information(self, event): assert '_depgraph' in vars(event)
@@ -1540,6 +1543,7 @@ class BuildInfoHelper(object): else: br.state = BuildRequest.REQ_FAILED br.save() + signal_runbuilds() def store_log_error(self, text): mockevent = MockEvent()
bitbake/lib/toaster/orm/models.py +2 -0
@@ -377,6 +377,8 @@ class Project(models.Model): except ProjectVariable.DoesNotExist: pass br.save() + signal_runbuilds() + except Exception: # revert the build request creation since we're not done cleanly br.delete()
Signature · 1 line
2.1.4
PATCH 3/7 runbuilds: process builds on SIGUSR1
In reply to Ed BartoshRun main processing function 'runbuild' only if SIGUSR1 is received. This signal is sent by Toaster when build status is changed (either started, cancelled or finished).
This should stop continuous database polling as run_builds function will be called only when needed, i.e. after build status is changed.
[YOCTO #8918]
--- bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +4 -2
@@ -11,9 +11,9 @@ from orm.models import Build, ToasterSetting, LogMessage, Target import os import logging -import time import sys import traceback +import signal logger = logging.getLogger("toaster")
@@ -175,6 +175,8 @@ class Command(NoArgsCommand): logger.warn("runbuilds: schedule exception %s" % str(e)) def handle_noargs(self, **options): + signal.signal(signal.SIGUSR1, lambda sig, frame: None) + while True: + signal.pause() self.runbuild() - time.sleep(1)
Signature · 1 line
2.1.4
PATCH 4/7 runbuilds: process builds on start
In reply to Ed BartoshIf Toaster is stopped incorrectly there could be some
build requests and builds in incorrect state left from the previous run. Running main processing function on start should take care of those.
--- bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 2 ++ 1 file changed, 2 insertions(+)
bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +2 -0
@@ -175,6 +175,8 @@ class Command(NoArgsCommand): logger.warn("runbuilds: schedule exception %s" % str(e)) def handle_noargs(self, **options): + self.runbuild() + signal.signal(signal.SIGUSR1, lambda sig, frame: None) while True:
Signature · 1 line
2.1.4
PATCH 5/7 runbuilds: code cleanup - whitespaces, long lines
In reply to Ed BartoshFixed following pylint warnings:
C0330(bad-continuation): Wrong hanging indentation before block. C0326(bad-whitespace): No space allowed around keyword argument assignment C0326(bad-whitespace): Exactly one space required before assignment C0301(line-too-long): Line too long
--- .../bldcontrol/management/commands/runbuilds.py | 34 ++++++++++------------ 1 file changed, 15 insertions(+), 19 deletions(-)
bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +15 -19
@@ -18,14 +18,13 @@ import signal logger = logging.getLogger("toaster") class Command(NoArgsCommand): - args = "" - help = "Schedules and executes build requests as possible." - "Does not return (interrupt with Ctrl-C)" - + args = "" + help = "Schedules and executes build requests as possible. "\ + "Does not return (interrupt with Ctrl-C)" @transaction.atomic def _selectBuildEnvironment(self): - bec = getBuildEnvironmentController(lock = BuildEnvironment.LOCK_FREE) + bec = getBuildEnvironmentController(lock=BuildEnvironment.LOCK_FREE) bec.be.lock = BuildEnvironment.LOCK_LOCK bec.be.save() return bec
@@ -75,17 +74,15 @@ class Command(NoArgsCommand): else: errmsg = str(e) - BRError.objects.create(req = br, - errtype = str(type(e)), - errmsg = errmsg, - traceback = traceback.format_exc()) + BRError.objects.create(req=br, errtype=str(type(e)), errmsg=errmsg, + traceback=traceback.format_exc()) br.state = BuildRequest.REQ_FAILED br.save() bec.be.lock = BuildEnvironment.LOCK_FREE bec.be.save() def archive(self): - for br in BuildRequest.objects.filter(state = BuildRequest.REQ_ARCHIVE): + for br in BuildRequest.objects.filter(state=BuildRequest.REQ_ARCHIVE): if br.build == None: br.state = BuildRequest.REQ_FAILED else:
@@ -102,14 +99,13 @@ class Command(NoArgsCommand): BuildRequest.REQ_COMPLETED, BuildRequest.REQ_CANCELLING]) & Q(lock=BuildEnvironment.LOCK_LOCK) & - Q(updated__lt=timezone.now() - timedelta(seconds = 30)) + Q(updated__lt=timezone.now() - timedelta(seconds=30)) ).update(lock=BuildEnvironment.LOCK_FREE) # update all Builds that were in progress and failed to start - for br in BuildRequest.objects.filter( - state=BuildRequest.REQ_FAILED, - build__outcome=Build.IN_PROGRESS): + for br in BuildRequest.objects.filter(state=BuildRequest.REQ_FAILED, + build__outcome=Build.IN_PROGRESS): # transpose the launch errors in ToasterExceptions br.build.outcome = Build.FAILED for brerror in br.brerror_set.all():
@@ -126,7 +122,7 @@ class Command(NoArgsCommand): # update all BuildRequests without a build created - for br in BuildRequest.objects.filter(build = None): + for br in BuildRequest.objects.filter(build=None): br.build = Build.objects.create(project=br.project, completed_on=br.updated, started_on=br.created)
@@ -151,10 +147,10 @@ class Command(NoArgsCommand): # Make sure the LOCK is removed for builds which have been fully # cancelled - for br in BuildRequest.objects.filter( - Q(build__outcome=Build.CANCELLED) & - Q(state=BuildRequest.REQ_CANCELLING) & - ~Q(environment=None)): + for br in BuildRequest.objects.filter(\ + Q(build__outcome=Build.CANCELLED) & + Q(state=BuildRequest.REQ_CANCELLING) & + ~Q(environment=None)): br.environment.lock = BuildEnvironment.LOCK_FREE br.environment.save()
Signature · 1 line
2.1.4
PATCH 6/7 runbuilds: code cleanup - remove unused imports
In reply to Ed BartoshFixed pylint warning: W0611(unused-import): Unused import
--- bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +2 -5
@@ -1,17 +1,14 @@ -from django.core.management.base import NoArgsCommand, CommandError +from django.core.management.base import NoArgsCommand from django.db import transaction from django.db.models import Q from bldcontrol.bbcontroller import getBuildEnvironmentController -from bldcontrol.bbcontroller import ShellCmdException, BuildSetupException from bldcontrol.models import BuildRequest, BuildEnvironment from bldcontrol.models import BRError, BRVariable -from orm.models import Build, ToasterSetting, LogMessage, Target +from orm.models import Build, LogMessage, Target -import os import logging -import sys import traceback import signal
Signature · 1 line
2.1.4
PATCH 7/7 toaster: unlock BuildEnvirnoment when build is done
In reply to Ed BartoshThere is no need to lock build environment before changing build status as this operation is very fast. However, there is a need to unlock it after changing build status.
Explicitly unlocked BuildEnvironment after build reaches final status SUCCEEDED, FAILED or CANCELLED. This should allow runbuilds process to pickup next build faster.
--- bitbake/lib/bb/ui/buildinfohelper.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
bitbake/lib/bb/ui/buildinfohelper.py +5 -3
@@ -1524,9 +1524,7 @@ class BuildInfoHelper(object): return br_id, be_id = self.brbe.split(":") - be = BuildEnvironment.objects.get(pk = be_id) - be.lock = BuildEnvironment.LOCK_LOCK - be.save() + br = BuildRequest.objects.get(pk = br_id) # if we're 'done' because we got cancelled update the build outcome
@@ -1543,6 +1541,10 @@ class BuildInfoHelper(object): else: br.state = BuildRequest.REQ_FAILED br.save() + + be = BuildEnvironment.objects.get(pk = be_id) + be.lock = BuildEnvironment.LOCK_FREE + be.save() signal_runbuilds() def store_log_error(self, text):
Signature · 1 line
2.1.4
Re: [PATCH 7/7] toaster: unlock BuildEnvirnoment when build is done
Thanks - send to bitbake-devel and applied to toaster-next
Quoted 39 lines · Ed Bartosh
There is no need to lock build environment before changing build status as this operation is very fast. However, there is a need to unlock it after changing build status.
Explicitly unlocked BuildEnvironment after build reaches final status SUCCEEDED, FAILED or CANCELLED. This should allow runbuilds process to pickup next build faster.
Signed-off-by Ed Bartosh <[email protected]>--- bitbake/lib/bb/ui/buildinfohelper.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)bitbake/lib/bb/ui/buildinfohelper.py +5
-3@@ -1524,9 +1524,7 @@ class BuildInfoHelper(object): return br_id, be_id = self.brbe.split(":") - be = BuildEnvironment.objects.get(pk = be_id) - be.lock = BuildEnvironment.LOCK_LOCK - be.save() + br = BuildRequest.objects.get(pk = br_id) # if we're 'done' because we got cancelled update the build outcome@@ -1543,6 +1541,10 @@ class BuildInfoHelper(object): else: br.state = BuildRequest.REQ_FAILED br.save() + + be = BuildEnvironment.objects.get(pk = be_id) + be.lock = BuildEnvironment.LOCK_FREE + be.save() signal_runbuilds() def store_log_error(self, text):