Skip to contentSkip to conversation

RFC: toaster: 8279: Provide a single way of starting Toaster

PATCH New Ed Bartosh 1 participant 21 messages Oct 11, 2015

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

cover RFC: toaster: 8279: Provide a single way of starting Toaster

Hi,

This patchset merges two Toaster modes in one:

To run toaster you should source it in already prepared build environment:
cd poky
. oe-init-build-env
. ../bitbake/bin/toaster

After that you should be able to trigger builds both ways:
from command line and from toaster UI.

Please test it and report your findings.

The following changes since commit 86371789e6eb6e5753105c48e5b4a6effaff683f:

toaster: Modify "New build" button behaviour for cli builds project (2015-10-11 14:32:25 +0300)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/toaster/8279-v2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/8279-v2

Ed Bartosh (20):
toaster: don't allow to run toaster as a script
toaster: implement get-dburl command
toaster: set DATABASE_URL in toaster script
toaster: run bitbake the same way
toaster: remove unused variable
toaster: don't use exit in shell script
toaster: check for toaster configuration later
toaster: use BBASEDIR to find config
toaster: get rid of TOASTER_MANAGED variable
toaster: use parent of the build dir
toaster: make runbuilds to loop
toaster: start 'manage.py runbuilds' in the script
toaster: update brbe and project attributes
toaster: implement stop_bitbake function
toaster: implement start_bitbake function
toaster: implement 'toaster restart-bitbake'
toaster: remove _setupBE function
toaster: reimplemented startBBServer method
toaster: remove stopBBServer API
toaster: do not terminate bb server

 bin/toaster                                        | 182 +++++++--------------
 lib/bb/ui/buildinfohelper.py                       |   6 +
 lib/bb/ui/toasterui.py                             |  15 +-
 lib/toaster/bldcontrol/bbcontroller.py             |   6 -
 lib/toaster/bldcontrol/localhostbecontroller.py    | 110 +++----------
 .../bldcontrol/management/commands/runbuilds.py    |  13 +-
 lib/toaster/bldcontrol/sshbecontroller.py          |  10 --
 lib/toaster/bldcontrol/tests.py                    |   4 -
 .../toastermain/management/commands/get-dburl.py   |   9 +
 9 files changed, 112 insertions(+), 243 deletions(-)
 create mode 100644 bitbake/lib/toaster/toastermain/management/commands/get-dburl.py

--
Regards,
Ed

author
#1 lore raw

PATCH 1/20 toaster: don't allow to run toaster as a script

Removed support of starting toaster as a script.

Sourcing a toaster script becomes the only way to start it. It's consistent with the way oe build system is started by sourcing oe-init-build-env. It also returns user back to shell, so user can continue running builds without having to open new terminal window.

[YOCTO #8279]

---
 bin/toaster | 68 ++-----------------------------------------------------------
 1 file changed, 2 insertions(+), 66 deletions(-)
bin/toaster +2 -66
index bc439e6..abe539d 100755
@@ -214,72 +214,8 @@ done
 [ -n "${BASH_SOURCE}" ] && SRCFILE=${BASH_SOURCE} || SRCFILE=$_
 
 if [ `basename \"$0\"` = `basename \"${SRCFILE}\"` ]; then
-    # We are called as standalone. We refuse to run in a build environment - we need the interactive mode for that.
-    # Start just the web server, point the web browser to the interface, and start any Django services.
-
-    if ! verify_prereq; then
-        echo "Error: Could not verify that the needed dependencies are installed. Please use virtualenv and pip to install dependencies listed in toaster-requirements.txt" 1>&2
-        exit 1
-    fi
-
-    if [ -n "$BUILDDIR" ]; then
-        printf "Error: It looks like you sourced oe-init-build-env. Toaster cannot start in build mode from an oe-core build environment.\n You should be starting Toaster from a new terminal window.\n" 1>&2
-        exit 1
-    fi
-
-    # Define a fake builddir where only the pid files are actually created. No real builds will take place here.
-    BUILDDIR=/tmp/toaster_$$
-    if [ -d "$BUILDDIR" ]; then
-        echo "Previous toaster run directory $BUILDDIR found, cowardly refusing to start. Please remove the directory when that toaster instance is over" 2>&1
-        exit 1
-    fi
-
-    mkdir -p "$BUILDDIR"
-
-    RUNNING=1
-    trap_ctrlc() {
-        echo "** Stopping system"
-        webserverKillAll
-        RUNNING=0
-    }
-
-    do_cleanup() {
-        find "$BUILDDIR" -type f | xargs rm
-        rmdir "$BUILDDIR"
-    }
-    cleanup() {
-        if grep -ir error "$BUILDDIR" >/dev/null; then
-            if grep -irn "That port is already in use" "$BUILDDIR"; then
-                echo "You can use the \"webport=PORTNUMBER\" parameter to start Toaster on a different port (port $WEB_PORT is already in use)"
-                do_cleanup
-            else
-                printf "\nErrors found in the Toaster log files present in '$BUILDDIR'. Directory will not be cleaned.\n Please review the errors and notify [email protected] or submit a bug https://bugzilla.yoctoproject.org/enter_bug.cgi?product=Toaster"
-            fi
-        else
-            echo "No errors found, removing the run directory '$BUILDDIR'"
-            do_cleanup
-        fi
-    }
-    TOASTER_MANAGED=1
-    export TOASTER_MANAGED=1
-    if [ $WEBSERVER -gt 0 ] && ! webserverStartAll; then
-        echo "Failed to start the web server, stopping" 1>&2
-        cleanup
-        exit 1
-    fi
-    if [ $WEBSERVER -gt 0 ] && [ $NOBROWSER -eq 0 ]  ; then
-        echo "Starting browser..."
-        xdg-open http://127.0.0.1:$WEB_PORT/ >/dev/null 2>&1 &
-    fi
-    trap trap_ctrlc 2
-    echo "Toaster is now running. You can stop it with Ctrl-C"
-    while [ $RUNNING -gt 0 ]; do
-        python $BBBASEDIR/lib/toaster/manage.py runbuilds 2>&1 | tee -a "$BUILDDIR/toaster.log"
-        sleep 1
-    done
-    cleanup
-    echo "**** Exit"
-    exit 0
+    echo "Error: This script needs to be sourced. Please run as '. $SRCFILE'"
+    exit 1
 fi
 
 
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH 2/20 toaster: implement get-dburl command

In reply to Ed Bartosh

Implemented management command to call getDATABASE_URL API. It will be used to get database url from toaster shell script by running 'manage.py get-dburl'

---
 lib/toaster/toastermain/management/commands/get-dburl.py | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100644 bitbake/lib/toaster/toastermain/management/commands/get-dburl.py
lib/toaster/toastermain/management/commands/get-dburl.py new file +9 -0
new file mode 100644
index 0000000..22b3eb7
@@ -0,0 +1,9 @@
+from toastermain.settings import getDATABASE_URL
+from django.core.management.base import NoArgsCommand
+
+class Command(NoArgsCommand):
+    args    = ""
+    help    = "get database url"
+
+    def handle_noargs(self,**options):
+        print getDATABASE_URL()
Signature · 1 line
2.1.4
author
#3 lore raw

PATCH 3/20 toaster: set DATABASE_URL in toaster script

In reply to Ed Bartosh

Set environment variable DATABASE_URL in bitbake/bit/toaster Variable value is obtained by running 'manage.py get-dburl'

---
 bin/toaster | 1 +
 1 file changed, 1 insertion(+)
bin/toaster +1 -0
index abe539d..0991030 100755
@@ -286,6 +286,7 @@ case $CMD in
             echo "Bitbake server start failed"
         else
             export BBSERVER=0.0.0.0:-1
+            export DATABASE_URL=`$BBBASEDIR/lib/toaster/manage.py get-dburl`
             if [ $NOTOASTERUI -eq 0 ]; then        # we start the TOASTERUI only if not inhibited
                 bitbake --observe-only -u toasterui >>${BUILDDIR}/toaster_ui.log 2>&1 & echo $! >${BUILDDIR}/.toasterui.pid
             fi
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH 4/20 toaster: run bitbake the same way

In reply to Ed Bartosh

Run bitbake in toaster script the same way as it's run by localbuildcontroller.

---
 bin/toaster | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
bin/toaster +4 -6
index 0991030..9f88dfc 100755
@@ -271,16 +271,13 @@ case $CMD in
     start )
         start_success=1
         addtoConfiguration toaster.conf "INHERIT+=\"toaster buildhistory\"" $TOASTER_BRBE
+        echo > ${BUILDDIR}/conf/toaster-pre.conf
         if [ $WEBSERVER -gt 0 ] && ! webserverStartAll; then
             echo "Failed ${CMD}."
             return 4
         fi
         unset BBSERVER
-        PREREAD=""
-        if [ -e ${BUILDDIR}/conf/toaster-pre.conf ]; then
-            rm ${BUILDDIR}/conf/toaster-pre.conf
-        fi
-        bitbake $PREREAD --postread conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
+        bitbake --read conf/toaster-pre.conf --postread conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
         if [ $? -ne 0 ]; then
             start_success=0
             echo "Bitbake server start failed"
@@ -288,7 +285,8 @@ case $CMD in
             export BBSERVER=0.0.0.0:-1
             export DATABASE_URL=`$BBBASEDIR/lib/toaster/manage.py get-dburl`
             if [ $NOTOASTERUI -eq 0 ]; then        # we start the TOASTERUI only if not inhibited
-                bitbake --observe-only -u toasterui >>${BUILDDIR}/toaster_ui.log 2>&1 & echo $! >${BUILDDIR}/.toasterui.pid
+                bitbake --observe-only -u toasterui --remote-server=$BBSERVER -t xmlrpc >>${BUILDDIR}/toaster_ui.log 2>&1 \
+                    & echo $! >${BUILDDIR}/.toasterui.pid
             fi
         fi
         if [ $start_success -eq 1 ]; then
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH 5/20 toaster: remove unused variable

In reply to Ed Bartosh

Removed variable RUNNING as it's not used anymore
in the toaster script.

---
 bin/toaster | 3 ---
 1 file changed, 3 deletions(-)
bin/toaster +0 -3
index 9f88dfc..2fc4e750 100755
@@ -156,9 +156,6 @@ else
 fi
 
 BBBASEDIR=`dirname $TOASTER`/..
-
-RUNNING=0
-
 NOTOASTERUI=0
 WEBSERVER=1
 TOASTER_BRBE=""
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH 6/20 toaster: don't use exit in shell script

In reply to Ed Bartosh

Replaced exit -> return as using exit in sourced script causes exit of running shell and shutting down terminal window.

---
 bin/toaster | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bin/toaster +1 -1
index 2fc4e750..badb2c1 100755
@@ -175,7 +175,7 @@ if [ "$TOASTER_CONF" = "" ]; then
 fi
 if [ ! -f $TOASTER_CONF ]; then
     echo "$TOASTER_CONF configuration file not found, exiting..."
-    exit 1;
+    return 1;
 fi
 # this defines the dir toaster will use for
 # 1) clones of layers (in _toaster_clones )
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH 7/20 toaster: check for toaster configuration later

In reply to Ed Bartosh

Moved check for toasterconf.json after check of build environment. We'll need some variables from build environment to find toasterconf.json better way.

---
 bin/toaster | 48 +++++++++++++++++++++++-------------------------
 1 file changed, 23 insertions(+), 25 deletions(-)
bin/toaster +23 -25
index badb2c1..b2e208e 100755
@@ -162,30 +162,6 @@ TOASTER_BRBE=""
 if [ "$WEB_PORT" = "" ]; then
     WEB_PORT="8000"
 fi
-# this is the configuraton file we are using for toaster
-# note default is assuming yocto. Override this if you are
-# running in a pure OE environment and use the toasterconf.json
-# in meta/conf/toasterconf.json
-# note: for future there are a number of relative path assumptions
-# in the local layers that currently prevent using an arbitrary
-# toasterconf.json
-if [ "$TOASTER_CONF" = "" ]; then
-    BIN_DIR=$(dirname -- "$0")
-    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"
-fi
-if [ ! -f $TOASTER_CONF ]; then
-    echo "$TOASTER_CONF configuration file not found, exiting..."
-    return 1;
-fi
-# this defines the dir toaster will use for
-# 1) clones of layers (in _toaster_clones )
-# 2) the build dir (in build)
-# 3) the sqlite db if that is being used.
-# 4) pid's we need to clean up on exit/shutdown
-# note: for future. in order to make this an arbitrary directory, we need to
-# make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
-export TOASTER_DIR=`pwd`
-
 
 NOBROWSER=0
 
@@ -221,13 +197,35 @@ if ! verify_prereq; then
     return 1
 fi
 
-
 # We make sure we're running in the current shell and in a good environment
 if [ -z "$BUILDDIR" ] ||  ! which bitbake >/dev/null 2>&1 ; then
     echo "Error: Build environment is not setup or bitbake is not in path." 1>&2
     return 2
 fi
 
+# this is the configuraton file we are using for toaster
+# note default is assuming yocto. Override this if you are
+# running in a pure OE environment and use the toasterconf.json
+# in meta/conf/toasterconf.json
+# note: for future there are a number of relative path assumptions
+# in the local layers that currently prevent using an arbitrary
+# toasterconf.json
+if [ "$TOASTER_CONF" = "" ]; then
+    BIN_DIR=$(dirname -- "$0")
+    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"
+fi
+if [ ! -f $TOASTER_CONF ]; then
+    echo "$TOASTER_CONF configuration file not found, exiting..."
+    return 1;
+fi
+# this defines the dir toaster will use for
+# 1) clones of layers (in _toaster_clones )
+# 2) the build dir (in build)
+# 3) the sqlite db if that is being used.
+# 4) pid's we need to clean up on exit/shutdown
+# note: for future. in order to make this an arbitrary directory, we need to
+# make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
+export TOASTER_DIR=`pwd`
 
 # Determine the action. If specified by arguments, fine, if not, toggle it
 if [ "$1" = 'start' ] || [ "$1" = 'stop' ]; then
Signature · 1 line
2.1.4
author
#8 lore raw

PATCH 8/20 toaster: use BBASEDIR to find config

In reply to Ed Bartosh

Used bitbake variable BBASEDIR to find toasterconf.json

---
 bin/toaster | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
bin/toaster +1 -2
index b2e208e..7084e0ff 100755
@@ -211,8 +211,7 @@ fi
 # in the local layers that currently prevent using an arbitrary
 # toasterconf.json
 if [ "$TOASTER_CONF" = "" ]; then
-    BIN_DIR=$(dirname -- "$0")
-    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"
+    export TOASTER_CONF="$BBBASEDIR/../meta-yocto/conf/toasterconf.json"
 fi
 if [ ! -f $TOASTER_CONF ]; then
     echo "$TOASTER_CONF configuration file not found, exiting..."
Signature · 1 line
2.1.4
author
#9 lore raw

PATCH 9/20 toaster: get rid of TOASTER_MANAGED variable

In reply to Ed Bartosh

As toaster modes are merged we don't need to use
this variable anymore.

---
 bin/toaster | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
bin/toaster +2 -4
index 7084e0ff..064602b 100755
@@ -67,10 +67,8 @@ webserverStartAll()
         retval=0
         python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
     fi
-    if [ "$TOASTER_MANAGED" = '1' ]; then
-        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
-        python $BBBASEDIR/lib/toaster/manage.py checksettings  --traceback || retval=1
-    fi
+    python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
+    python $BBBASEDIR/lib/toaster/manage.py checksettings  --traceback || retval=1
     if [ $retval -eq 0 ]; then
         echo "Starting webserver..."
         python $BBBASEDIR/lib/toaster/manage.py runserver "0.0.0.0:$WEB_PORT" </dev/null >>${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
Signature · 1 line
2.1.4
author
#10 lore raw

PATCH 10/20 toaster: use parent of the build dir

In reply to Ed Bartosh

Assigned TOASTER_DIR to the parent of the build directory. This should fix local controller crash as it assumes that TOASTER_DIR is a root of local poky and tries to clone it.

---
 bin/toaster | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bin/toaster +1 -1
index 064602b..d7653c2 100755
@@ -222,7 +222,7 @@ fi
 # 4) pid's we need to clean up on exit/shutdown
 # note: for future. in order to make this an arbitrary directory, we need to
 # make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
-export TOASTER_DIR=`pwd`
+export TOASTER_DIR=`dirname $BUILDDIR`
 
 # Determine the action. If specified by arguments, fine, if not, toggle it
 if [ "$1" = 'start' ] || [ "$1" = 'stop' ]; then
Signature · 1 line
2.1.4
author
#11 lore raw

PATCH 11/20 toaster: make runbuilds to loop

In reply to Ed Bartosh

Made runbuilds to loop to avoid having a loop in shell code and initialize heavy Django init machinery every second.

Ignored exceptions to prevent exiting a loop.

---
 lib/toaster/bldcontrol/management/commands/runbuilds.py | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
lib/toaster/bldcontrol/management/commands/runbuilds.py +10 -3
index 5243a50..48dc618 100644
@@ -5,6 +5,7 @@ from bldcontrol.bbcontroller import getBuildEnvironmentController, ShellCmdExcep
 from bldcontrol.models import BuildRequest, BuildEnvironment, BRError, BRVariable
 import os
 import logging
+import time
 
 logger = logging.getLogger("ToasterScheduler")
 
@@ -128,6 +129,12 @@ class Command(NoArgsCommand):
 
 
     def handle_noargs(self, **options):
-        self.cleanup()
-        self.archive()
-        self.schedule()
+        while True:
+            try:
+                self.cleanup()
+                self.archive()
+                self.schedule()
+            except:
+                pass
+
+            time.sleep(1)
Signature · 1 line
2.1.4
author
#12 lore raw

PATCH 12/20 toaster: start 'manage.py runbuilds' in the script

In reply to Ed Bartosh

Running runbuilds machinery in background allows Toaster to start builds initiated by toaster UI.

Added runbuilds pid file to the list in webserverKillAll to kill runbuilds the same way as runserver process.

---
 bin/toaster | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
bin/toaster +2 -1
index d7653c2..62f6526 100755
@@ -31,7 +31,7 @@
 webserverKillAll()
 {
     local pidfile
-    for pidfile in ${BUILDDIR}/.toastermain.pid; do
+    for pidfile in ${BUILDDIR}/.toastermain.pid ${BUILDDIR}/.runbuilds.pid; do
         if [ -f ${pidfile} ]; then
             pid=`cat ${pidfile}`
             while kill -0 $pid 2>/dev/null; do
@@ -280,6 +280,7 @@ case $CMD in
                 bitbake --observe-only -u toasterui --remote-server=$BBSERVER -t xmlrpc >>${BUILDDIR}/toaster_ui.log 2>&1 \
                     & echo $! >${BUILDDIR}/.toasterui.pid
             fi
+            python $BBBASEDIR/lib/toaster/manage.py runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
         fi
         if [ $start_success -eq 1 ]; then
             # set fail safe stop system on terminal exit
Signature · 1 line
2.1.4
author
#13 lore raw

PATCH 13/20 toaster: update brbe and project attributes

In reply to Ed Bartosh

Updated attributes of buildinfohelper object as they can be changed for every builds. For example brbe is set by runbuilds for every build triggered by Toaster UI.

---
 lib/bb/ui/buildinfohelper.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/ui/buildinfohelper.py +6 -0
index 67c239e..01a515d 100644
@@ -779,6 +779,8 @@ class BuildInfoHelper(object):
         build_info['cooker_log_path'] = consolelogfile
         build_info['build_name'] = self.server.runCommand(["getVariable", "BUILDNAME"])[0]
         build_info['bitbake_version'] = self.server.runCommand(["getVariable", "BB_VERSION"])[0]
+        build_info['brbe'] = self.server.runCommand(["getVariable", "TOASTER_BRBE"])[0]
+        build_info['project'] = self.project = self.server.runCommand(["getVariable", "TOASTER_PROJECT"])[0]
 
         return build_info
 
@@ -921,6 +923,10 @@ class BuildInfoHelper(object):
         assert '_pkgs' in vars(event)
         build_information = self._get_build_information(consolelogfile)
 
+        # Update brbe and project as they can be changed for every build
+        self.brbe = build_information['brbe']
+        self.project = build_information['project']
+
         build_obj = self.orm_wrapper.create_build_object(build_information, self.brbe, self.project)
 
         self.internal_state['build'] = build_obj
Signature · 1 line
2.1.4
author
#14 lore raw

PATCH 14/20 toaster: implement stop_bitbake function

In reply to Ed Bartosh

Separated functionality of stopping bitbake server and observer processes.

This functionality will be used by build controllers to restart bitbake processes.

---
 bin/toaster | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
bin/toaster +8 -4
index 62f6526..1deabd8 100755
@@ -105,16 +105,20 @@ stop_system()
         kill `cat ${BUILDDIR}/.toasterui.pid` 2>/dev/null
         rm ${BUILDDIR}/.toasterui.pid
     fi
-    BBSERVER=0.0.0.0:-1 bitbake -m
-    unset BBSERVER
+    stop_bitbake
     webserverKillAll
-    # force stop any misbehaving bitbake server
-    lsof bitbake.lock | awk '{print $2}' | grep "[0-9]\+" | xargs -n1 -r kill
     trap - SIGHUP
     #trap - SIGCHLD
     INSTOPSYSTEM=0
 }
 
+stop_bitbake() {
+    BBSERVER=0.0.0.0:-1 bitbake -m
+    unset BBSERVER
+    # force stop any misbehaving bitbake server
+    lsof bitbake.lock | awk '{print $2}' | grep "[0-9]\+" | xargs -n1 -r kill
+}
+
 check_pidbyfile() {
     [ -e $1 ] && kill -0 `cat $1` 2>/dev/null
 }
Signature · 1 line
2.1.4
author
#15 lore raw

PATCH 15/20 toaster: implement start_bitbake function

In reply to Ed Bartosh

Separated functionality of starting bitbake server and observer processes.

This functionality will be used by build controllers to restart bitbake processes.

---
 bin/toaster | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)
bin/toaster +18 -15
index 1deabd8..fee277c 100755
@@ -112,6 +112,22 @@ stop_system()
     INSTOPSYSTEM=0
 }
 
+start_bitbake() {
+    unset BBSERVER
+    bitbake --read conf/toaster-pre.conf --postread conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
+    if [ $? -ne 0 ]; then
+        echo "Bitbake server start failed"
+        return 1
+    fi
+    export BBSERVER=0.0.0.0:-1
+    export DATABASE_URL=`$BBBASEDIR/lib/toaster/manage.py get-dburl`
+    if [ $NOTOASTERUI -eq 0 ]; then        # we start the TOASTERUI only if not inhibited
+        bitbake --observe-only -u toasterui --remote-server=$BBSERVER -t xmlrpc >>${BUILDDIR}/toaster_ui.log 2>&1 \
+            & echo $! >${BUILDDIR}/.toasterui.pid
+    fi
+    return 0
+}
+
 stop_bitbake() {
     BBSERVER=0.0.0.0:-1 bitbake -m
     unset BBSERVER
@@ -265,28 +281,15 @@ fi
 
 case $CMD in
     start )
-        start_success=1
         addtoConfiguration toaster.conf "INHERIT+=\"toaster buildhistory\"" $TOASTER_BRBE
         echo > ${BUILDDIR}/conf/toaster-pre.conf
         if [ $WEBSERVER -gt 0 ] && ! webserverStartAll; then
             echo "Failed ${CMD}."
             return 4
         fi
-        unset BBSERVER
-        bitbake --read conf/toaster-pre.conf --postread conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
-        if [ $? -ne 0 ]; then
-            start_success=0
-            echo "Bitbake server start failed"
-        else
-            export BBSERVER=0.0.0.0:-1
-            export DATABASE_URL=`$BBBASEDIR/lib/toaster/manage.py get-dburl`
-            if [ $NOTOASTERUI -eq 0 ]; then        # we start the TOASTERUI only if not inhibited
-                bitbake --observe-only -u toasterui --remote-server=$BBSERVER -t xmlrpc >>${BUILDDIR}/toaster_ui.log 2>&1 \
-                    & echo $! >${BUILDDIR}/.toasterui.pid
-            fi
+        start_bitbake
+        if [ $? -eq 0 ]; then
             python $BBBASEDIR/lib/toaster/manage.py runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
-        fi
-        if [ $start_success -eq 1 ]; then
             # set fail safe stop system on terminal exit
             trap stop_system SIGHUP
             echo "Successful ${CMD}."
Signature · 1 line
2.1.4
author
#16 lore raw

PATCH 16/20 toaster: implement 'toaster restart-bitbake'

In reply to Ed Bartosh

This implementation allows to have functionality of restarting bitbake in toaster script. It can be used by toaster script and build controllers.

[YOCTO #8279]

---
 bin/toaster | 7 +++++++
 1 file changed, 7 insertions(+)
bin/toaster +7 -0
index fee277c..e1eb655 100755
@@ -209,6 +209,13 @@ if [ `basename \"$0\"` = `basename \"${SRCFILE}\"` ]; then
     exit 1
 fi
 
+if [ "$1" = 'restart-bitbake' ] ; then
+    stop_bitbake
+    sleep 1
+    start_bitbake
+    sleep 1
+    return $?
+fi
 
 if ! verify_prereq; then
     echo "Error: Could not verify that the needed dependencies are installed. Please use virtualenv and pip to install dependencies listed in toaster-requirements.txt" 1>&2
Signature · 1 line
2.1.4
author
#17 lore raw

PATCH 17/20 toaster: remove _setupBE function

In reply to Ed Bartosh

This function is not needed as build environment is always created because of the new way to run Toaster. It can be only sourced after oe-init-build-env is sourced.

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 19 ++-----------------
 1 file changed, 2 insertions(+), 17 deletions(-)
lib/toaster/bldcontrol/localhostbecontroller.py +2 -17
index b5cf559..9a3c818 100644
@@ -71,17 +71,6 @@ class LocalhostBEController(BuildEnvironmentController):
             logger.debug("localhostbecontroller: shellcmd success")
             return out
 
-    def _setupBE(self):
-        assert self.pokydirname and os.path.exists(self.pokydirname)
-        path = self.be.builddir
-        if not path:
-            raise Exception("Invalid path creation specified.")
-        if not os.path.exists(path):
-            os.makedirs(path, 0755)
-        self._shellcmd("bash -c \"source %s/oe-init-build-env %s\"" % (self.pokydirname, path))
-        # delete the templateconf.cfg; it may come from an unsupported layer configuration
-        os.remove(os.path.join(path, "conf/templateconf.cfg"))
-
 
     def writeConfFile(self, file_name, variable_list = None, raw = None):
         filepath = os.path.join(self.be.builddir, file_name)
@@ -290,16 +279,12 @@ class LocalhostBEController(BuildEnvironmentController):
 
         logger.debug("localhostbecontroller: current layer list %s " % pformat(layerlist))
 
-        # 4. configure the build environment, so we have a conf/bblayers.conf
-        assert self.pokydirname is not None
-        self._setupBE()
-
-        # 5. update the bblayers.conf
+        # 4. update the bblayers.conf
         bblayerconf = os.path.join(self.be.builddir, "conf/bblayers.conf")
         if not os.path.exists(bblayerconf):
             raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
 
-        # 6. create custom layer and add custom recipes to it
+        # 5. create custom layer and add custom recipes to it
         layerpath = os.path.join(self.be.sourcedir, "_meta-toaster-custom")
         if os.path.isdir(layerpath):
             shutil.rmtree(layerpath) # remove leftovers from previous builds
Signature · 1 line
2.1.4
author
#18 lore raw

PATCH 18/20 toaster: reimplemented startBBServer method

In reply to Ed Bartosh

Rewritten LocalhostBEController.startBBServer to use 'toaster restart-bitbake' and read bitbake port number from bitbake.lock.

Removed complicated logic of running oe-init-memres and looking for bitbake port number in the logs.

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 82 ++++++-------------------
 1 file changed, 18 insertions(+), 64 deletions(-)
lib/toaster/bldcontrol/localhostbecontroller.py +18 -64
index 9a3c818..f321537 100644
@@ -89,74 +89,28 @@ class LocalhostBEController(BuildEnvironmentController):
         # find our own toasterui listener/bitbake
         from toaster.bldcontrol.management.commands.loadconf import _reduce_canon_path
 
-        own_bitbake = _reduce_canon_path(os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../../bin/bitbake"))
-
-        assert os.path.exists(own_bitbake) and os.path.isfile(own_bitbake)
-
-        logger.debug("localhostbecontroller: running the listener at %s" % own_bitbake)
-
-        toaster_ui_log_filepath = os.path.join(self.be.builddir, "toaster_ui.log")
-        # get the file length; we need to detect the _last_ start of the toaster UI, not the first
-        toaster_ui_log_filelength = 0
-        if os.path.exists(toaster_ui_log_filepath):
-            with open(toaster_ui_log_filepath, "w") as f:
-                f.seek(0, 2)    # jump to the end
-                toaster_ui_log_filelength = f.tell()
-
-        cmd = "bash -c \"source %s/oe-init-build-env %s 2>&1 >toaster_server.log && bitbake --read %s/conf/toaster-pre.conf --postread %s/conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0 2>&1 >>toaster_server.log \"" % (self.pokydirname, self.be.builddir, self.be.builddir, self.be.builddir)
-
-        port = "-1"
-        logger.debug("localhostbecontroller: starting builder \n%s\n" % cmd)
-
-        cmdoutput = self._shellcmd(cmd)
-        with open(self.be.builddir + "/toaster_server.log", "r") as f:
-            for i in f.readlines():
-                if i.startswith("Bitbake server address"):
-                    port = i.split(" ")[-1]
-                    logger.debug("localhostbecontroller: Found bitbake server port %s" % port)
-
-        cmd = "bash -c \"source %s/oe-init-build-env-memres -1 %s && DATABASE_URL=%s %s --observe-only -u toasterui --remote-server=0.0.0.0:-1 -t xmlrpc\"" % (self.pokydirname, self.be.builddir, self.dburl, own_bitbake)
-        with open(toaster_ui_log_filepath, "a+") as f:
-            p = subprocess.Popen(cmd, cwd = self.be.builddir, shell=True, stdout=f, stderr=f)
-
-        def _toaster_ui_started(filepath, filepos = 0):
-            if not os.path.exists(filepath):
-                return False
-            with open(filepath, "r") as f:
-                f.seek(filepos)
-                for line in f:
-                    if line.startswith("NOTE: ToasterUI waiting for events"):
-                        return True
-            return False
-
-        retries = 0
-        started = False
-        while not started and retries < 50:
-            started = _toaster_ui_started(toaster_ui_log_filepath, toaster_ui_log_filelength)
-            import time
-            logger.debug("localhostbecontroller: Waiting bitbake server to start")
-            time.sleep(0.5)
-            retries += 1
-
-        if not started:
-            toaster_ui_log = open(os.path.join(self.be.builddir, "toaster_ui.log"), "r").read()
-            toaster_server_log = open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
-            raise BuildSetupException("localhostbecontroller: Bitbake server did not start in 25 seconds, aborting (Error: '%s' '%s')" % (toaster_ui_log, toaster_server_log))
-
-        logger.debug("localhostbecontroller: Started bitbake server")
-
-        while port == "-1":
-            # the port specification is "autodetect"; read the bitbake.lock file
-            with open("%s/bitbake.lock" % self.be.builddir, "r") as f:
-                for line in f.readlines():
+        toaster = _reduce_canon_path(os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../../bin/toaster"))
+        assert os.path.exists(toaster) and os.path.isfile(toaster)
+
+        # restart bitbake server and toastergui observer
+        self._shellcmd("bash -c 'source %s restart-bitbake'" % toaster, self.be.builddir)
+        logger.debug("localhostbecontroller: restarted bitbake server")
+
+        # read port number from bitbake.lock
+        self.be.bbport = ""
+        bblock = os.path.join(self.be.builddir, 'bitbake.lock')
+        if os.path.exists(bblock):
+            with open(bblock) as fplock:
+                for line in fplock:
                     if ":" in line:
-                        port = line.split(":")[1].strip()
-                        logger.debug("localhostbecontroller: Autodetected bitbake port %s", port)
+                        self.be.bbport = line.split(":")[-1].strip()
+                        logger.debug("localhostbecontroller: bitbake port %s", self.be.bbport)
                         break
 
-        assert self.be.sourcedir and os.path.exists(self.be.builddir)
+        if not self.be.bbport:
+            raise BuildSetupException("localhostbecontroller: can't read bitbake port from %s" % bblock)
+
         self.be.bbaddress = "localhost"
-        self.be.bbport = port
         self.be.bbstate = BuildEnvironment.SERVER_STARTED
         self.be.save()
 
Signature · 1 line
2.1.4
author
#19 lore raw

PATCH 19/20 toaster: remove stopBBServer API

In reply to Ed Bartosh

Removed stopBBServer API from build controller as toaster doesn't stop bitbake server anymore. It's reused for both types of builds: triggered by UI and started manually.

---
 lib/toaster/bldcontrol/bbcontroller.py          |  6 ------
 lib/toaster/bldcontrol/localhostbecontroller.py |  9 ---------
 lib/toaster/bldcontrol/sshbecontroller.py       | 10 ----------
 lib/toaster/bldcontrol/tests.py                 |  4 ----
 4 files changed, 29 deletions(-)
lib/toaster/bldcontrol/bbcontroller.py +0 -6
index ad70ac8..3d98ad7 100644
@@ -140,12 +140,6 @@ class BuildEnvironmentController(object):
         """
         raise Exception("FIXME: Must override in order to actually start the BB server")
 
-    def stopBBServer(self):
-        """ Stops the currently running BB server.
-            The bbstate MUST be updated to "stopped".
-            self.connection must be none.
-        """
-        raise Exception("FIXME: Must override stoBBServer")
 
     def setLayers(self, bbs, ls):
         """ Checks-out bitbake executor and layers from git repositories.
lib/toaster/bldcontrol/localhostbecontroller.py +0 -9
index f321537..9ab741f 100644
@@ -114,15 +114,6 @@ class LocalhostBEController(BuildEnvironmentController):
         self.be.bbstate = BuildEnvironment.SERVER_STARTED
         self.be.save()
 
-    def stopBBServer(self):
-        assert self.pokydirname and os.path.exists(self.pokydirname)
-        assert self.islayerset
-        self._shellcmd("bash -c \"source %s/oe-init-build-env %s && %s source toaster stop\"" %
-            (self.pokydirname, self.be.builddir, (lambda: "" if self.be.bbtoken is None else "BBTOKEN=%s" % self.be.bbtoken)()))
-        self.be.bbstate = BuildEnvironment.SERVER_STOPPED
-        self.be.save()
-        logger.debug("localhostbecontroller: Stopped bitbake server")
-
     def getGitCloneDirectory(self, url, branch):
         """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
lib/toaster/bldcontrol/sshbecontroller.py +0 -10
index 8ef434b..17dd66c 100644
@@ -98,16 +98,6 @@ class SSHBEController(BuildEnvironmentController):
         self.be.bbstate = BuildEnvironment.SERVER_STARTED
         self.be.save()
 
-    def stopBBServer(self):
-        assert self.pokydirname and self._pathexists(self.pokydirname)
-        assert self.islayerset
-        print self._shellcmd("bash -c \"source %s/oe-init-build-env %s && %s source toaster stop\"" %
-            (self.pokydirname, self.be.builddir, (lambda: "" if self.be.bbtoken is None else "BBTOKEN=%s" % self.be.bbtoken)()))
-        self.be.bbstate = BuildEnvironment.SERVER_STOPPED
-        self.be.save()
-        print "Stopped server"
-
-
     def _copyFile(self, filepath1, filepath2):
         p = subprocess.Popen("scp '%s' '%s'" % (filepath1, filepath2), stdout=subprocess.PIPE, stderr = subprocess.PIPE, shell=True)
         (out, err) = p.communicate()
lib/toaster/bldcontrol/tests.py +0 -4
index 5dbc77f..f54cf7f 100644
@@ -71,9 +71,6 @@ class BEControllerTests(object):
 
         self.assertFalse(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex((hostname, int(bc.be.bbport))), "Server not answering")
 
-        bc.stopBBServer()
-        self.assertTrue(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex((hostname, int(bc.be.bbport))), "Server not stopped")
-
         self._serverForceStop(bc)
 
     def test_getBBController(self):
@@ -96,7 +93,6 @@ class BEControllerTests(object):
 
         bbc = bc.getBBController()
         self.assertTrue(isinstance(bbc, BitbakeController))
-        bc.stopBBServer()
 
         self._serverForceStop(bc)
 
Signature · 1 line
2.1.4
author
#20 lore raw

PATCH 20/20 toaster: do not terminate bb server

In reply to Ed Bartosh

Toaster needs bb server to be running all the time due to merged analysis and managed modes. Server gets restarted before every build triggered by UI, but it shouldn't be terminated as it will influence command line builds.

[YOCTO #8279]

---
 lib/bb/ui/toasterui.py | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)
lib/bb/ui/toasterui.py +5 -10
index e3d3b8f..7a18ae3 100644
@@ -282,16 +282,11 @@ def main(server, eventHandler, params ):
                 logger.info("ToasterUI build done 1, brbe: %s", buildinfohelper.brbe )
 
                 # we start a new build info
-                if buildinfohelper.brbe is not None:
-
-                    logger.debug("ToasterUI under BuildEnvironment management - exiting after the build")
-                    server.terminateServer()
-                else:
-                    logger.debug("ToasterUI prepared for new build")
-                    errors = 0
-                    warnings = 0
-                    taskfailures = []
-                    buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+                logger.debug("ToasterUI prepared for new build")
+                errors = 0
+                warnings = 0
+                taskfailures = []
+                buildinfohelper = BuildInfoHelper(server, build_history_enabled)
 
                 logger.info("ToasterUI build done 2")
                 continue
Signature · 1 line
2.1.4