Skip to contentSkip to conversation

Hide irrelevant builds

PATCH In review brian avery 3 participants 10 messages Sep 10, 2015 – Sep 19, 2015

lore
Share & tools
lore
#0 lore raw

cover Hide irrelevant builds

This patch fixes the project builds view so it doesn't show "in progress" builds or builds for other projects.

The following changes since commit db92bc23880794be77373b0f2d8c26279dcecb84:

bitbake: toaster: replace ETA with % of tasks done (2015-09-08 23:54:15 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib bavery/submit/elliot/20150902_hide_irrelevant_builds_v2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/elliot/20150902_hide_irrelevant_builds_v2

Alexandru DAMIAN (1):
toaster: hide irrelevant builds in the project builds view

Elliot Smith (4):
toaster: Add tests for /project/X/builds page
toaster: Show correct builds count on project pages
toaster: Fix date range pickers on the project builds page
toaster: Simplify redirects when build page parameters are missing

 lib/toaster/toastergui/static/js/base.js           |   4 +-
 .../toastergui/templates/projectbuilds.html        |   4 +-
 lib/toaster/toastergui/tests.py                    |  91 ++++++++++++++++-
 lib/toaster/toastergui/views.py                    | 108 +++++++++++++++------
 toaster-requirements.txt                           |   1 +
 5 files changed, 169 insertions(+), 39 deletions(-)
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/5 toaster: hide irrelevant builds in the project builds view

From: Alexandru DAMIAN <[email protected]>

This patch fixes the project builds view so it doesn't show "in progress" builds or builds for other projects.

Note that this also modifies the "all builds" view to use the same queryset filtering as the project builds. This is to avoid excluding "in progress" builds more than once, which is what was happening before.

The patch also has a minor change to ensure that when displaying the project builds page, only builds for that project are in the results.

The queryset filtering is now split over several lines so you can see what's going on.

[YOCTO #8236]
[YOCTO #8187]

---
 lib/toaster/toastergui/views.py | 54 ++++++++++++++++++++++++++++++++---------
 1 file changed, 42 insertions(+), 12 deletions(-)
lib/toaster/toastergui/views.py +42 -12
index 4e8f69e..c9831b1 100755
@@ -1896,7 +1896,7 @@ if True:
         # be able to display something.  'count' and 'page' are mandatory for all views
         # that use paginators.
 
-        queryset = Build.objects.exclude(outcome = Build.IN_PROGRESS)
+        queryset = Build.objects.all()
 
         try:
             context, pagesize, orderby = _build_list_helper(request, queryset)
@@ -1913,7 +1913,6 @@ if True:
 
     # helper function, to be used on "all builds" and "project builds" pages
     def _build_list_helper(request, queryset_all):
-
         default_orderby = 'completed_on:-'
         (pagesize, orderby) = _get_parameters_values(request, 10, default_orderby)
         mandatory_parameters = { 'count': pagesize,  'page' : 1, 'orderby' : orderby }
@@ -1924,11 +1923,42 @@ if True:
         # boilerplate code that takes a request for an object type and returns a queryset
         # for that object type. copypasta for all needed table searches
         (filter_string, search_term, ordering_string) = _search_tuple(request, Build)
+
         # post-process any date range filters
-        filter_string,daterange_selected = _modify_date_range_filter(filter_string)
-        queryset_all = queryset_all.select_related("project").annotate(errors_no = Count('logmessage', only=Q(logmessage__level=LogMessage.ERROR)|Q(logmessage__level=LogMessage.EXCEPTION))).annotate(warnings_no = Count('logmessage', only=Q(logmessage__level=LogMessage.WARNING))).extra(select={'timespent':'completed_on - started_on'})
-        queryset_with_search = _get_queryset(Build, queryset_all, None, search_term, ordering_string, '-completed_on')
-        queryset = _get_queryset(Build, queryset_all, filter_string, search_term, ordering_string, '-completed_on')
+        filter_string, daterange_selected = _modify_date_range_filter(filter_string)
+
+        # don't show "in progress" builds in "all builds" or "project builds"
+        queryset_all = queryset_all.exclude(outcome = Build.IN_PROGRESS)
+
+        # append project info
+        queryset_all = queryset_all.select_related("project")
+
+        # annotate with number of ERROR and EXCEPTION log messages
+        queryset_all = queryset_all.annotate(
+            errors_no = Count(
+                'logmessage',
+                only=Q(logmessage__level=LogMessage.ERROR) |
+                     Q(logmessage__level=LogMessage.EXCEPTION)
+            )
+        )
+
+        # annotate with number of warnings
+        q_warnings = Q(logmessage__level=LogMessage.WARNING)
+        queryset_all = queryset_all.annotate(
+            warnings_no = Count('logmessage', only=q_warnings)
+        )
+
+        # add timespent field
+        timespent = 'completed_on - started_on'
+        queryset_all = queryset_all.extra(select={'timespent': timespent})
+
+        queryset_with_search = _get_queryset(Build, queryset_all,
+                                             None, search_term,
+                                             ordering_string, '-completed_on')
+
+        queryset = _get_queryset(Build, queryset_all,
+                                 filter_string, search_term,
+                                 ordering_string, '-completed_on')
 
         # retrieve the objects that will be displayed in the table; builds a paginator and gets a page range to display
         build_info = _build_page_range(Paginator(queryset, pagesize), request.GET.get('page', 1))
@@ -2651,7 +2681,7 @@ if True:
             if 'buildDelete' in request.POST:
                 for i in request.POST['buildDelete'].strip().split(" "):
                     try:
-                        br = BuildRequest.objects.select_for_update().get(project = prj, pk = i, state__lte = BuildRequest.REQ_DELETED).delete()
+                        BuildRequest.objects.select_for_update().get(project = prj, pk = i, state__lte = BuildRequest.REQ_DELETED).delete()
                     except BuildRequest.DoesNotExist:
                         pass
 
@@ -2664,12 +2694,12 @@ if True:
                     else:
                         target = t
                         task = ""
-                    ProjectTarget.objects.create(project = prj, target = target, task = task)
-
-                br = prj.schedule_build()
-
+                    ProjectTarget.objects.create(project = prj,
+                                                 target = target,
+                                                 task = task)
+                prj.schedule_build()
 
-        queryset = Build.objects.filter(outcome__lte = Build.IN_PROGRESS)
+        queryset = Build.objects.filter(project_id = pid)
 
         try:
             context, pagesize, orderby = _build_list_helper(request, queryset)
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/5 toaster: Add tests for /project/X/builds page

From: Elliot Smith <[email protected]>

Add tests to check whether "in progress" builds are filtered out correctly, and that only builds for the current project are shown.

Adds a dependency on BeautifulSoup 4, which is used to simplify writing tests which verify the HTML.

Also requires a fix to how a date was converted in the view which only manifested when I wrote the tests.

[YOCTO #8236]
[YOCTO #8187]

---
 lib/toaster/toastergui/tests.py | 91 +++++++++++++++++++++++++++++++++++++++--
 lib/toaster/toastergui/views.py |  5 +--
 toaster-requirements.txt        |  1 +
 3 files changed, 91 insertions(+), 6 deletions(-)
lib/toaster/toastergui/tests.py +88 -3
index 1a8b478..4d1549b 100644
@@ -24,10 +24,11 @@
 from django.test import TestCase
 from django.core.urlresolvers import reverse
 from django.utils import timezone
-from orm.models import Project, Release, BitbakeVersion, Build
-from orm.models import ReleaseLayerSourcePriority, LayerSource, Layer
+from orm.models import Project, Release, BitbakeVersion, ProjectTarget
+from orm.models import ReleaseLayerSourcePriority, LayerSource, Layer, Build
 from orm.models import Layer_Version, Recipe, Machine, ProjectLayer
 import json
+from bs4 import BeautifulSoup
 
 PROJECT_NAME = "test project"
 
@@ -41,7 +42,6 @@ class ViewTests(TestCase):
                                          bitbake_version=bbv)
         self.project = Project.objects.create_project(name=PROJECT_NAME,
                                                       release=release)
-
         layersrc = LayerSource.objects.create(sourcetype=LayerSource.TYPE_IMPORTED)
         self.priority = ReleaseLayerSourcePriority.objects.create(release=release,
                                                                   layer_source=layersrc)
@@ -292,3 +292,88 @@ class ProjectsPageTests(TestCase):
                         'should be a project row in the page')
         self.assertTrue(self.PROJECT_NAME in response.content,
                         'default project "cli builds" should be in page')
+
+class ProjectBuildsDisplayTest(TestCase):
+    """ Test data at /project/X/builds is displayed correctly """
+
+    def setUp(self):
+        bbv = BitbakeVersion.objects.create(name="bbv1", giturl="/tmp/",
+                                            branch="master", dirpath="")
+        release = Release.objects.create(name="release1",
+                                         bitbake_version=bbv)
+        self.project1 = Project.objects.create_project(name=PROJECT_NAME,
+                                                       release=release)
+        self.project2 = Project.objects.create_project(name=PROJECT_NAME,
+                                                       release=release)
+
+        # parameters for builds to associate with the projects
+        now = timezone.now()
+
+        self.project1_build_success = {
+            "project": self.project1,
+            "started_on": now,
+            "completed_on": now,
+            "outcome": Build.SUCCEEDED
+        }
+
+        self.project1_build_in_progress = {
+            "project": self.project1,
+            "started_on": now,
+            "completed_on": now,
+            "outcome": Build.IN_PROGRESS
+        }
+
+        self.project2_build_success = {
+            "project": self.project2,
+            "started_on": now,
+            "completed_on": now,
+            "outcome": Build.SUCCEEDED
+        }
+
+        self.project2_build_in_progress = {
+            "project": self.project2,
+            "started_on": now,
+            "completed_on": now,
+            "outcome": Build.IN_PROGRESS
+        }
+
+    def _get_rows_for_project(self, project_id):
+        url = reverse("projectbuilds", args=(project_id,))
+        response = self.client.get(url, follow=True)
+        soup = BeautifulSoup(response.content)
+        return soup.select('tr[class="data"]')
+
+    def test_show_builds_for_project(self):
+        """ Builds for a project should be displayed """
+        build1a = Build.objects.create(**self.project1_build_success)
+        build1b = Build.objects.create(**self.project1_build_success)
+        build_rows = self._get_rows_for_project(self.project1.id)
+        self.assertEqual(len(build_rows), 2)
+
+    def test_show_builds_for_project_only(self):
+        """ Builds for other projects should be excluded """
+        build1a = Build.objects.create(**self.project1_build_success)
+        build1b = Build.objects.create(**self.project1_build_success)
+        build1c = Build.objects.create(**self.project1_build_success)
+
+        # shouldn't see these two
+        build2a = Build.objects.create(**self.project2_build_success)
+        build2b = Build.objects.create(**self.project2_build_in_progress)
+
+        build_rows = self._get_rows_for_project(self.project1.id)
+        self.assertEqual(len(build_rows), 3)
+
+    def test_show_builds_exclude_in_progress(self):
+        """ "in progress" builds should not be shown """
+        build1a = Build.objects.create(**self.project1_build_success)
+        build1b = Build.objects.create(**self.project1_build_success)
+
+        # shouldn't see this one
+        build1c = Build.objects.create(**self.project1_build_in_progress)
+
+        # shouldn't see these two either, as they belong to a different project
+        build2a = Build.objects.create(**self.project2_build_success)
+        build2b = Build.objects.create(**self.project2_build_in_progress)
+
+        build_rows = self._get_rows_for_project(self.project1.id)
+        self.assertEqual(len(build_rows), 2)
\ No newline at end of file
lib/toaster/toastergui/views.py +2 -3
index c9831b1..6ff510d 100755
@@ -40,7 +40,7 @@ from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
 from django.http import HttpResponseBadRequest, HttpResponseNotFound
 from django.utils import timezone
 from django.utils.html import escape
-from datetime import timedelta, datetime, date
+from datetime import timedelta, datetime
 from django.utils import formats
 from toastergui.templatetags.projecttags import json as jsonfilter
 import json
@@ -435,8 +435,7 @@ def _modify_date_range_filter(filter_string):
 def _add_daterange_context(queryset_all, request, daterange_list):
     # calculate the exact begining of local today and yesterday
     today_begin = timezone.localtime(timezone.now())
-    today_begin = date(today_begin.year,today_begin.month,today_begin.day)
-    yesterday_begin = today_begin-timedelta(days=1)
+    yesterday_begin = today_begin - timedelta(days=1)
     # add daterange persistent
     context_date = {}
     context_date['last_date_from'] = request.GET.get('last_date_from',timezone.localtime(timezone.now()).strftime("%d/%m/%Y"))
toaster-requirements.txt +1 -0
index 19b5293..1d7d21b 100644
@@ -2,3 +2,4 @@ Django==1.6
 South==0.8.4
 argparse==1.2.1
 wsgiref==0.1.2
+beautifulsoup4>=4.4.0
Signature · 1 line
1.9.1
#6 lore raw

PATCH 3/5 toaster: Show correct builds count on project pages

In reply to brian avery

From: Elliot Smith <[email protected]>

The counter for completed builds on the project pages includes builds in progress.

Instead use the completedbuilds queryset to count the number of completed builds for display on project pages.

Modify how the completedbuilds queryset is constructed so it only excludes builds with status "in progress".

---
 lib/toaster/toastergui/static/js/base.js | 4 ++--
 lib/toaster/toastergui/views.py          | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
lib/toaster/toastergui/static/js/base.js +2 -2
index e0df463..895e61b 100644
@@ -57,8 +57,8 @@ function basePageInit(ctx) {
 
   if ($(".total-builds").length !== 0){
     libtoaster.getProjectInfo(libtoaster.ctx.projectPageUrl, function(prjInfo){
-      if (prjInfo.builds)
-        $(".total-builds").text(prjInfo.builds.length);
+      if (prjInfo.completedbuilds)
+        $(".total-builds").text(prjInfo.completedbuilds.length);
     });
   }
 
lib/toaster/toastergui/views.py +1 -1
index 6ff510d..631d619 100755
@@ -2255,7 +2255,7 @@ if True:
         context = {
             "project" : prj,
             "lvs_nos" : Layer_Version.objects.all().count(),
-            "completedbuilds": Build.objects.filter(project_id = pid).filter(outcome__lte = Build.IN_PROGRESS),
+            "completedbuilds": Build.objects.exclude(outcome = Build.IN_PROGRESS).filter(project_id = pid),
             "prj" : {"name": prj.name, },
             "buildrequests" : prj.build_set.filter(outcome=Build.IN_PROGRESS),
             "builds" : _project_recent_build_list(prj),
Signature · 1 line
1.9.1
#7 lore raw

PATCH 4/5 toaster: Fix date range pickers on the project builds page

In reply to brian avery

From: Elliot Smith <[email protected]>

These were referring to the old HTML elements with ids "date_from_created" and "date_from_updated", but their ids have changed to "date_from_started_on" and "date_from_completed_on". This meant that they weren't functional.

This fixes the references.

Modified from an original patch by David Reyna <[email protected]>.

---
 lib/toaster/toastergui/templates/projectbuilds.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/toaster/toastergui/templates/projectbuilds.html +2 -2
index df809de..27cfcd7 100644
@@ -16,8 +16,8 @@
 <script>
     // initialize the date range controls
     $(document).ready(function () {
-        date_init('created','{{last_date_from}}','{{last_date_to}}','{{dateMin_started_on}}','{{dateMax_started_on}}','{{daterange_selected}}');
-        date_init('updated','{{last_date_from}}','{{last_date_to}}','{{dateMin_completed_on}}','{{dateMax_completed_on}}','{{daterange_selected}}');
+        date_init('started_on','{{last_date_from}}','{{last_date_to}}','{{dateMin_started_on}}','{{dateMax_started_on}}','{{daterange_selected}}');
+        date_init('completed_on','{{last_date_from}}','{{last_date_to}}','{{dateMin_completed_on}}','{{dateMax_completed_on}}','{{daterange_selected}}');
     });
 </script>
 
Signature · 1 line
1.9.1
#8 lore raw

PATCH 5/5 toaster: Simplify redirects when build page parameters are missing

In reply to brian avery

From: Elliot Smith <[email protected]>

A RedirectException is used to redirect the client to the correct page when their original request is missing the required parameters (page, orderby etc.). However, the code is difficult to follow.

Rather than catching RedirectExceptions and rethrowing them with different view URLs, ensure that the RedirectException has the correct URL in it when thrown by passing the original page name to the _build_list_helper() method.

Modified from an original patch by
David Reyna <[email protected]>.

---
 lib/toaster/toastergui/views.py | 49 +++++++++++++++++++++++++++--------------
 1 file changed, 32 insertions(+), 17 deletions(-)
lib/toaster/toastergui/views.py +32 -17
index 631d619..adfebab 100755
@@ -1889,6 +1889,10 @@ if True:
         pass
 
     # shows the "all builds" page for managed mode; it displays build requests (at least started!) instead of actual builds
+    # WARNING _build_list_helper() may raise a RedirectException, which
+    # will set the GET parameters and redirect back to the
+    # all-builds or projectbuilds page as appropriate;
+    # TODO don't use exceptions to control program flow
     @_template_renderer("builds.html")
     def builds(request):
         # define here what parameters the view needs in the GET portion in order to
@@ -1897,27 +1901,35 @@ if True:
 
         queryset = Build.objects.all()
 
-        try:
-            context, pagesize, orderby = _build_list_helper(request, queryset)
-            # all builds page as a Project column
-            context['tablecols'].append({'name': 'Project', 'clcalss': 'project_column', })
-        except RedirectException as re:
-            # rewrite the RedirectException
-            re.view = resolve(request.path_info).url_name
-            raise re
+        redirect_page = resolve(request.path_info).url_name
+
+        context, pagesize, orderby = _build_list_helper(request,
+                                                        queryset,
+                                                        redirect_page)
+        # all builds page as a Project column
+        context['tablecols'].append({
+            'name': 'Project',
+            'clclass': 'project_column'
+        })
 
         _set_parameters_values(pagesize, orderby, request)
         return context
 
 
     # helper function, to be used on "all builds" and "project builds" pages
-    def _build_list_helper(request, queryset_all):
+    def _build_list_helper(request, queryset_all, redirect_page, pid=None):
         default_orderby = 'completed_on:-'
         (pagesize, orderby) = _get_parameters_values(request, 10, default_orderby)
         mandatory_parameters = { 'count': pagesize,  'page' : 1, 'orderby' : orderby }
         retval = _verify_parameters( request.GET, mandatory_parameters )
         if retval:
-            raise RedirectException( None, request.GET, mandatory_parameters)
+            params = {}
+            if pid:
+                params = {'pid': pid}
+            raise RedirectException(redirect_page,
+                                    request.GET,
+                                    mandatory_parameters,
+                                    **params)
 
         # boilerplate code that takes a request for an object type and returns a queryset
         # for that object type. copypasta for all needed table searches
@@ -2661,6 +2673,10 @@ if True:
 
         return context
 
+    # WARNING _build_list_helper() may raise a RedirectException, which
+    # will set the GET parameters and redirect back to the
+    # all-builds or projectbuilds page as appropriate;
+    # TODO don't use exceptions to control program flow
     @_template_renderer('projectbuilds.html')
     def projectbuilds(request, pid):
         prj = Project.objects.get(id = pid)
@@ -2700,13 +2716,12 @@ if True:
 
         queryset = Build.objects.filter(project_id = pid)
 
-        try:
-            context, pagesize, orderby = _build_list_helper(request, queryset)
-        except RedirectException as re:
-            # rewrite the RedirectException with our current url information
-            re.view = resolve(request.path_info).url_name
-            re.okwargs = {"pid" : pid}
-            raise re
+        redirect_page = resolve(request.path_info).url_name
+
+        context, pagesize, orderby = _build_list_helper(request,
+                                                        queryset,
+                                                        redirect_page,
+                                                        pid)
 
         context['project'] = prj
         _set_parameters_values(pagesize, orderby, request)
Signature · 1 line
1.9.1
#9 lore raw
In reply to brian avery

submitted upstream.
-b

Quoted 31 lines · brian avery

This patch fixes the project builds view so it doesn't show "in progress" builds or builds for other projects.

The following changes since commit db92bc23880794be77373b0f2d8c26279dcecb84:

bitbake: toaster: replace ETA with % of tasks done (2015-09-08 23:54:15 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib bavery/submit/elliot/20150902_hide_irrelevant_builds_v2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/elliot/20150902_hide_irrelevant_builds_v2

Alexandru DAMIAN (1):
toaster: hide irrelevant builds in the project builds view

Elliot Smith (4):
toaster: Add tests for /project/X/builds page
toaster: Show correct builds count on project pages
toaster: Fix date range pickers on the project builds page
toaster: Simplify redirects when build page parameters are missing

 lib/toaster/toastergui/static/js/base.js           |   4 +-
 .../toastergui/templates/projectbuilds.html        |   4 +-
 lib/toaster/toastergui/tests.py                    |  91 ++++++++++++++++-
 lib/toaster/toastergui/views.py                    | 108 +++++++++++++++------
 toaster-requirements.txt                           |   1 +
 5 files changed, 169 insertions(+), 39 deletions(-)

--
1.9.1

#3 lore raw

Re: [PATCH 2/5] toaster: Add tests for /project/X/builds page

In reply to brian avery
Quoted 9 lines · brian avery

From: Elliot Smith <[email protected]>

Add tests to check whether "in progress" builds are filtered out correctly, and that only builds for the current project are shown.

Adds a dependency on BeautifulSoup 4, which is used to simplify writing tests which verify the HTML.

Quoted 8 lines

Also requires a fix to how a date was converted in the view which only manifested when I wrote the tests.

[YOCTO #8236]
[YOCTO #8187]

Could you rebase this patch and point me at a branch please. I can't seem to get it to apply cleanly. I did get the others to work with line offset warnings.

Cheers,

Richard

#4 lore raw

Re: [PATCH 2/5] toaster: Add tests for /project/X/builds page

Hello Richard,

The branch is:
elliot/toaster/hide_builds-8236

I just rebased it on poky master.

Thanks.
Elliot

Quoted 33 lines · Richard Purdie
Quoted 9 lines · brian avery

From: Elliot Smith <[email protected]>

Add tests to check whether "in progress" builds are filtered out correctly, and that only builds for the current project are shown.

Adds a dependency on BeautifulSoup 4, which is used to simplify writing tests which verify the HTML.

Quoted 8 lines

Also requires a fix to how a date was converted in the view which only manifested when I wrote the tests.

[YOCTO #8236]
[YOCTO #8187]

Could you rebase this patch and point me at a branch please. I can't seem to get it to apply cleanly. I did get the others to work with line offset warnings.

Cheers,

Richard

--
_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Signature · 3 lines
Elliot Smith
Software Engineer
Intel Open Source Technology Centre
#5 lore raw

Re: [PATCH 2/5] toaster: Add tests for /project/X/builds page

Quoted 4 lines · Elliot

The branch is:
elliot/toaster/hide_builds-8236

I just rebased it on poky master.

I'm guessing the problematic patch was in master-next at the time so the rebase didn't help, I still hit the merge issue.

Anyhow, it was a simple enough fix so I resolved and merged it, thanks.

Cheers,

Richard