Skip to contentSkip to conversation

Fix errors thrown when viewing command line builds while build is running

PATCH In review Elliot Smith 2 participants 7 messages Sep 30, 2015 – Oct 13, 2015

lore
Share & tools
lore
#0 lore raw

v3 cover Fix errors thrown when viewing command line builds while build is running

Fix the OperationalError thrown if you attempt to view builds for the "Command line builds" project while a build is in progress.

* v2: Rebased on poky/master; includes a fix for a test broken during rebase.
* v3: Rebased on poky-contrib/toaster-next; remove fix for broken test, as it will conflict with a patch submitted by Ed.

Changes since 4220ebce223fa7cd0865c793b7dd3ffe07b8e130 are in
git://git.yoctoproject.org/poky-contrib, elliot/toaster/cli_builds_error-8277
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/cli_builds_error-8277

Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8277

Elliot Smith (3):
toaster: Always run bldcontrol migrations
toaster: Check whether buildrequest exists before using it
toaster: Test that exception isn't thrown by project page

 bitbake/bin/toaster                     | 58 ++++++++++++++++++++++-----------
 bitbake/lib/toaster/orm/models.py       | 20 ++++++++++--
 bitbake/lib/toaster/toastergui/tests.py | 31 ++++++++++++++++++
 3 files changed, 88 insertions(+), 21 deletions(-)

--
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

#1 lore raw

PATCH v3 1/3 toaster: Always run bldcontrol migrations

The toaster startup script conditionally migrates the database tables depending on whether you are in managed mode or not. This means that if you are in analysis mode, some of the bldcontrol* database tables used by managed mode are not available.

As a consequence, some of the code in toaster which refers to those tables can break in analysis mode, as there's no clean isolation of the two modes.

To prevent this from happening, always run the migrations for managed mode and create the bldcontrol* tables, even if in analysis mode.

Also clean up the function which starts up toaster so the logic is easier to follow.

[YOCTO #8277]

---
 bitbake/bin/toaster | 58 +++++++++++++++++++++++++++++++++++------------------
 1 file changed, 39 insertions(+), 19 deletions(-)
bitbake/bin/toaster +39 -19
index ac27826..05dabc9 100755
@@ -54,35 +54,55 @@ webserverStartAll()
     fi
 
     retval=0
-    if [ "$TOASTER_MANAGED" '=' '1' ]; then
+    if [ "$TOASTER_MANAGED" = '1' ]; then
         python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1
     else
         python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1
     fi
-    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
+
     if [ $retval -eq 1 ]; then
-        echo "Failed db sync, stopping system start" 1>&2
-    elif [ $retval -eq 2 ]; then
-        printf "\nError on migration, trying to recover... \n"
+        echo "Failed db sync, aborting system start" 1>&2
+        return $retval
+    fi
+
+    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError on orm migration, rolling back...\n"
         python $BBBASEDIR/lib/toaster/manage.py migrate orm 0001_initial --fake
-        retval=0
-        python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
+        return $retval
     fi
+
+    python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError on bldcontrol migration, rolling back...\n"
+        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol 0001_initial --fake
+        return $retval
+    fi
+
     if [ "$TOASTER_MANAGED" = '1' ]; then
-        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
-        python $BBBASEDIR/lib/toaster/manage.py checksettings  --traceback || retval=1
+        python $BBBASEDIR/lib/toaster/manage.py checksettings --traceback || retval=1
     fi
-    if [ $retval -eq 0 ]; then
-        echo "Starting webserver..."
-        python $BBBASEDIR/lib/toaster/manage.py runserver "0.0.0.0:$WEB_PORT" </dev/null >>${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
-        sleep 1
-        if ! cat "${BUILDDIR}/.toastermain.pid" | xargs -I{} kill -0 {} ; then
-            retval=1
-            rm "${BUILDDIR}/.toastermain.pid"
-        else
-            echo "Webserver address:  http://0.0.0.0:$WEB_PORT/"
-        fi
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError while checking settings; aborting\n"
+        return $retval
+    fi
+
+    echo "Starting webserver..."
+
+    python $BBBASEDIR/lib/toaster/manage.py runserver "0.0.0.0:$WEB_PORT" </dev/null >>${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
+
+    sleep 1
+
+    if ! cat "${BUILDDIR}/.toastermain.pid" | xargs -I{} kill -0 {} ; then
+        retval=1
+        rm "${BUILDDIR}/.toastermain.pid"
+    else
+        echo "Webserver address:  http://0.0.0.0:$WEB_PORT/"
     fi
+
     return $retval
 }
 
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#2 lore raw

Re: [review-request][PATCH 1/3] toaster: Always run bldcontrol migrations

Quoted 92 lines · Elliot Smith

The toaster startup script conditionally migrates the database tables depending on whether you are in managed mode or not. This means that if you are in analysis mode, some of the bldcontrol* database tables used by managed mode are not available.

As a consequence, some of the code in toaster which refers to those tables can break in analysis mode, as there's no clean isolation of the two modes.

To prevent this from happening, always run the migrations for managed mode and create the bldcontrol* tables, even if in analysis mode.

Also clean up the function which starts up toaster so the logic is easier to follow.

[YOCTO #8277]

---
  bitbake/bin/toaster | 58 +++++++++++++++++++++++++++++++++++------------------
  1 file changed, 39 insertions(+), 19 deletions(-)
bitbake/bin/toaster +33 -19
index ac27826..05dabc9 100755
@@ -54,35 +54,55 @@ webserverStartAll()
      fi
 
      retval=0
-    if [ "$TOASTER_MANAGED" '=' '1' ]; then
+    if [ "$TOASTER_MANAGED" = '1' ]; then
          python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1
      else
          python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1
      fi
-    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
+
      if [ $retval -eq 1 ]; then
-        echo "Failed db sync, stopping system start" 1>&2
-    elif [ $retval -eq 2 ]; then
-        printf "\nError on migration, trying to recover... \n"
+        echo "Failed db sync, aborting system start" 1>&2
+        return $retval
+    fi
+
+    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError on orm migration, rolling back...\n"
          python $BBBASEDIR/lib/toaster/manage.py migrate orm 0001_initial --fake
-        retval=0
-        python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
+        return $retval
      fi
+
+    python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError on bldcontrol migration, rolling back...\n"
+        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol 0001_initial --fake
+        return $retval
+    fi
+
      if [ "$TOASTER_MANAGED" = '1' ]; then
-        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
-        python $BBBASEDIR/lib/toaster/manage.py checksettings  --traceback || retval=1
+        python $BBBASEDIR/lib/toaster/manage.py checksettings --traceback || retval=1
      fi
-    if [ $retval -eq 0 ]; then
-        echo "Starting webserver..."
-        python $BBBASEDIR/lib/toaster/manage.py runserver "0.0.0.0:$WEB_PORT" </dev/null >>${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
-        sleep 1
-        if ! cat "${BUILDDIR}/.toastermain.pid" | xargs -I{} kill -0 {} ; then
-            retval=1
-            rm "${BUILDDIR}/.toastermain.pid"
-        else
-            echo "Webserver address:  http://0.0.0.0:$WEB_PORT/"
-        fi
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError while checking settings; aborting\n"
+        return $retval
+    fi
+
+    echo "Starting webserver..."
+
+    python $BBBASEDIR/lib/toaster/manage.py runserver "0.0.0.0:$WEB_PORT" </dev/null >>${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
+
+    sleep 1
+

bitbake/bin/toaster webserverStartAll() · line 108

Quoted 10 lines
bitbake/bin/toaster +6 -0
+    if ! cat "${BUILDDIR}/.toastermain.pid" | xargs -I{} kill -0 {} ; then
+        retval=1
+        rm "${BUILDDIR}/.toastermain.pid"
+    else
+        echo "Webserver address:  http://0.0.0.0:$WEB_PORT/"
      fi
+
      return $retval
  }
 

If you want to run all migrations we can do just do ./manage migrate --all

#3 lore raw

Re: [review-request][PATCH 1/3] toaster: Always run bldcontrol migrations

Quoted 112 lines · Michael Wood
Quoted 110 lines · Elliot Smith

The toaster startup script conditionally migrates the database tables depending on whether you are in managed mode or not. This means that if you are in analysis mode, some of the bldcontrol* database tables used by managed mode are not available.

As a consequence, some of the code in toaster which refers to those tables can break in analysis mode, as there's no clean isolation of the two modes.

To prevent this from happening, always run the migrations for managed mode and create the bldcontrol* tables, even if in analysis mode.

Also clean up the function which starts up toaster so the logic is easier to follow.

[YOCTO #8277]

---
  bitbake/bin/toaster | 58
+++++++++++++++++++++++++++++++++++------------------
  1 file changed, 39 insertions(+), 19 deletions(-)
bitbake/bin/toaster +1 -1
index ac27826..05dabc9 100755
@@ -54,35 +54,55 @@ webserverStartAll()
      fi
        retval=0
-    if [ "$TOASTER_MANAGED" '=' '1' ]; then
+    if [ "$TOASTER_MANAGED" = '1' ]; then
          python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1
      else
          python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput ||

retval=1
fi

bitbake/bin/toaster +9 -4
-    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
+
      if [ $retval -eq 1 ]; then
-        echo "Failed db sync, stopping system start" 1>&2
-    elif [ $retval -eq 2 ]; then
-        printf "\nError on migration, trying to recover... \n"
+        echo "Failed db sync, aborting system start" 1>&2
+        return $retval
+    fi
+
+    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError on orm migration, rolling back...\n"
          python $BBBASEDIR/lib/toaster/manage.py migrate orm

0001_initial --fake

bitbake/bin/toaster +3 -2
-        retval=0
-        python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
+        return $retval
      fi
+
+    python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol ||

retval=1

bitbake/bin/toaster +4 -0
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError on bldcontrol migration, rolling back...\n"
+        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol

0001_initial --fake

bitbake/bin/toaster +3 -1
+        return $retval
+    fi
+
      if [ "$TOASTER_MANAGED" = '1' ]; then
-        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol ||

retval=1

bitbake/bin/toaster +1 -6
-        python $BBBASEDIR/lib/toaster/manage.py checksettings
--traceback || retval=1
+        python $BBBASEDIR/lib/toaster/manage.py checksettings
--traceback || retval=1
      fi
-    if [ $retval -eq 0 ]; then
-        echo "Starting webserver..."
-        python $BBBASEDIR/lib/toaster/manage.py runserver "0.0.0.0:$WEB_PORT"

</dev/null >>${BUILDDIR}/toaster_web.log 2>&1 & echo $!

Quoted 1 line

${BUILDDIR}/.toastermain.pid

bitbake/bin/toaster +0 -2
-        sleep 1
-        if ! cat "${BUILDDIR}/.toastermain.pid" | xargs -I{} kill -0 {}

; then

bitbake/bin/toaster +9 -5
-            retval=1
-            rm "${BUILDDIR}/.toastermain.pid"
-        else
-            echo "Webserver address:  http://0.0.0.0:$WEB_PORT/"
-        fi
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError while checking settings; aborting\n"
+        return $retval
+    fi
+
+    echo "Starting webserver..."
+
+    python $BBBASEDIR/lib/toaster/manage.py runserver "0.0.0.0:$WEB_PORT"

</dev/null >>${BUILDDIR}/toaster_web.log 2>&1 & echo $!

Quoted 1 line

${BUILDDIR}/.toastermain.pid

bitbake/bin/toaster +4 -0
+
+    sleep 1
+
+    if ! cat "${BUILDDIR}/.toastermain.pid" | xargs -I{} kill -0 {} ;

then

bitbake/bin/toaster +4 -0
+        retval=1
+        rm "${BUILDDIR}/.toastermain.pid"
+    else
+        echo "Webserver address:  http://0.0.0.0:$WEB_PORT/"
Quoted 9 lines
Quoted 6 lines

fi
+
return $retval
}

If you want to run all migrations we can do just do ./manage migrate --all

The reason for doing them individually is so that if one series of migrations fails, we can roll just that series back and exit the startup sequence.

Elliot

Quoted 14 lines

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Signature · 3 lines
Elliot Smith
Software Engineer
Intel Open Source Technology Centre
#4 lore raw

PATCH v3 2/3 toaster: Check whether buildrequest exists before using it

In reply to Elliot Smith

Builds initiated from the command line don't have a buildrequest associated with them. The build.buildrequest association is only added if a build is triggered from toaster.

Some of the code for displaying the status of a build refers to build.buildrequest without checking whether it has been set, which causes an error to be thrown.

Add a guard to check whether the buildrequest has been set.

[YOCTO #8277]

---
 bitbake/lib/toaster/orm/models.py | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
bitbake/lib/toaster/orm/models.py +18 -2
index 5aed158..3ea821d 100644
@@ -344,6 +344,9 @@ class Build(models.Model):
         tgts = Target.objects.filter(build_id = self.id).order_by( 'target' );
         return( tgts );
 
+    def get_outcome_text(self):
+        return Build.BUILD_OUTCOME[int(self.outcome)][1]
+
     @property
     def toaster_exceptions(self):
         return self.logmessage_set.filter(level=LogMessage.EXCEPTION)
@@ -361,10 +364,23 @@ class Build(models.Model):
         return (self.completed_on - self.started_on).total_seconds()
 
     def get_current_status(self):
+        """
+        get the status string from the build request if the build
+        has one, or the text for the build outcome if it doesn't
+        """
+
         from bldcontrol.models import BuildRequest
-        if self.outcome == Build.IN_PROGRESS and self.buildrequest.state != BuildRequest.REQ_INPROGRESS:
+
+        build_request = None
+        if hasattr(self, 'buildrequest'):
+            build_request = self.buildrequest
+
+        if (build_request
+                and build_request.state != BuildRequest.REQ_INPROGRESS
+                and self.outcome == Build.IN_PROGRESS):
             return self.buildrequest.get_state_display()
-        return self.get_outcome_display()
+        else:
+            return self.get_outcome_text()
 
     def __str__(self):
         return "%d %s %s" % (self.id, self.project, ",".join([t.target for t in self.target_set.all()]))
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#5 lore raw

PATCH v3 3/3 toaster: Test that exception isn't thrown by project page

In reply to Elliot Smith

Add a test which checks that an exception is no longer thrown for the /toastergui/project/X page for the default project.

Note that we still get a spinning dialogue box on this page because the default project has no configuration to display, but at least it doesn't fail altogether.

[YOCTO #8277]

---
 bitbake/lib/toaster/toastergui/tests.py | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
bitbake/lib/toaster/toastergui/tests.py +31 -0
index 5d15ac9..fc6bc8e 100644
@@ -574,3 +574,34 @@ class ProjectBuildsDisplayTest(TestCase):
         response = self.client.get(url, follow=True)
         result = re.findall('bash:clean', response.content, re.MULTILINE)
         self.assertEqual(len(result), 3)
+
+class ProjectPageTests(TestCase):
+    """ Test project data at /project/X/ is displayed correctly """
+
+    PROJECT_NAME = 'Command line builds'
+
+    def test_command_line_builds_in_progress(self):
+        """
+        In progress builds should not cause an error to be thrown
+        when navigating to "command line builds" project page;
+        see https://bugzilla.yoctoproject.org/show_bug.cgi?id=8277
+        """
+
+        # add the "command line builds" default project; this mirrors what
+        # we do in migration 0026_set_default_project.py
+        default_project = Project.objects.create_project(self.PROJECT_NAME, None)
+        default_project.is_default = True
+        default_project.save()
+
+        # add an "in progress" build for the default project
+        now = timezone.now()
+        build = Build.objects.create(project=default_project,
+                                     started_on=now,
+                                     completed_on=now,
+                                     outcome=Build.IN_PROGRESS)
+
+        # navigate to the project page for the default project
+        url = reverse("project", args=(default_project.id,))
+        response = self.client.get(url, follow=True)
+
+        self.assertEqual(response.status_code, 200)
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#6 lore raw
In reply to Elliot Smith

Please ignore this patchset for now: it needs to be rebased, and I'm unable to test as command-line builds are not being picked up properly by Toaster.

I will resubmit a v4 once I can test (hopefully Ed's work on merging the two modes will remedy the issues).

Elliot

Quoted 44 lines · Elliot Smith

Fix the OperationalError thrown if you attempt to view builds for the "Command line builds" project while a build is in progress.

* v2: Rebased on poky/master; includes a fix for a test broken during rebase.
* v3: Rebased on poky-contrib/toaster-next; remove fix for broken test, as it will conflict with a patch submitted by Ed.

Changes since 4220ebce223fa7cd0865c793b7dd3ffe07b8e130 are in
git://git.yoctoproject.org/poky-contrib,
elliot/toaster/cli_builds_error-8277

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/cli_builds_error-8277

Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8277

Elliot Smith (3):
toaster: Always run bldcontrol migrations
toaster: Check whether buildrequest exists before using it
toaster: Test that exception isn't thrown by project page

 bitbake/bin/toaster                     | 58
++++++++++++++++++++++-----------
 bitbake/lib/toaster/orm/models.py       | 20 ++++++++++--
 bitbake/lib/toaster/toastergui/tests.py | 31 ++++++++++++++++++
 3 files changed, 88 insertions(+), 21 deletions(-)

--
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Signature · 3 lines
Elliot Smith
Software Engineer
Intel Open Source Technology Centre