Skip to contentSkip to conversation

add gitlab-ci jobs to the daily-mail

PATCH New Victor Huesca 3 participants 13 messages Aug 29, 2019 – Sep 5, 2019

lore
Share & tools
lore
#0 lore raw

cover add gitlab-ci jobs to the daily-mail

This patch series adds gitlab-ci job failures to the daily-mail, this includes both defconfig build failures and runtime tests failures.

Patches 1-3: gitlab-ci infrastructure for defconfigs and runtime-tests. Patches 4-5: add defconfigs and runtime-tests to per-developer
notifications.
Patches 6-8: add the formatting logic for defconfig and runtime-tests. Patches 9-10: adjust command-line arguments and enable gitlab-ci
integration.

Victor Huesca (10):
utils/daily-mail: new data: gitlab-ci jobs utils/daily-mail: new gitlab-ci data: defconfigs utils/daily-mail: new gitlab-ci data: runtime tests utils/daily-mail: add defconfig support in developers notifications utils/daily-mail: add runtime tests support in developers
notifications
utils/daily-mail: prepare the mail formatting to support gitlab-ci
results
utils/daily-mail: add formatting for defconfig results utils/daily-mail: add formatting for runtime-test results utils/daily-mail: add argument for gitlab-ci related data utils/daily-mail: enable gitlab-ci emails

 utils/daily-mail | 342 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 313 insertions(+), 29 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/10 utils/daily-mail: new data: gitlab-ci jobs

Buildroot have a mirror repository on gitlab.com that is used to run continious integration tasks. Among these tasks, a complete checkup of defconfigs and runtime-tests is regularly performed.

This patch allows to retrieve tests that have failed over a given period of time. It is based on the (very imperfect) gitlab-ci API.

Due to multiple limitations of the API (pagination of results, crucial data requiring a dedicated request, etc.), this implementation performs multiple optimisation to be able to retreive the tasks needed.

---
 utils/daily-mail | 157 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 156 insertions(+), 1 deletion(-)
utils/daily-mail +156 -1
index baf0e0b..2d4ed1c 100755
@@ -7,7 +7,7 @@ import textwrap
 import mysmtplib as smtplib
 from email.mime.text import MIMEText
 from email.utils import formatdate
-from datetime import date, timedelta
+from datetime import date, timedelta, datetime
 import localconfig
 import csv
 from collections import defaultdict
@@ -16,6 +16,10 @@ import argparse
 import re
 import json
 from packaging import version
+import certifi
+from urllib3 import HTTPSConnectionPool
+from urllib3.exceptions import HTTPError
+from multiprocessing import Pool as ProcessPool
 
 sys.path.append(os.path.join(localconfig.brbase, "utils"))
 import getdeveloperlib  # noqa: E402
@@ -27,6 +31,7 @@ baseurl = "autobuild.buildroot.net"
 http_baseurl = "http://" + baseurl
 
 developers = getdeveloperlib.parse_developers(localconfig.brbase)
+gitlab_ci_http_pool = None
 
 
 def get_branches():
@@ -469,6 +474,156 @@ def get_outdated_pkgs(path):
     return sorted(s, key=lambda pkg: pkg['name'])
 
 
+def request_paginated_url(pool, url, page_size=100, page_name='page', **request_kwargs):
+    '''
+    Simple helper to retreive data from call paginated url.
+    Request's result is assumed to be json list.
+    '''
+    request_kwargs.setdefault('fields', {})
+    request_kwargs['fields']['per_page'] = page_size
+    results = []
+    page = 1
+    while "len(res) == page_size":
+        request_kwargs['fields'][page_name] = page
+        try:
+            req = pool.request('GET', url, **request_kwargs)
+        except HTTPError:
+            print('error: there were an error with gitlab api. '
+                  'Please make sure your identification token is right and try again.')
+            break
+        if req.status != 200:
+            print('error: there were an error with gitlab api. '
+                  'Please make sure your identification token is right and try again.')
+            break
+        res = json.loads(req.data)
+        results += res
+        page += 1
+        if len(res) != page_size:
+            break
+    return results
+
+
+def get_pipeline_details(project_id, pipeline, **request_kwargs):
+    pool = gitlab_ci_http_pool
+    pipeline_url = '/api/v4/projects/{}/pipelines/{}'.format(project_id, pipeline)
+    try:
+        pipeline_req = pool.request('GET', pipeline_url, **request_kwargs)
+    except HTTPError:
+        print('error: there were an error while fetching the gitlab-ci pipeline\'s details. '
+              'Please make sure your identification token is right and try again.')
+        return {}
+    if pipeline_req.status != 200:
+        print('error: there were an error while fetching the gitlab-ci pipeline\'s details. '
+              'Please make sure your identification token is right and try again.')
+        return {}
+    infos = json.loads(pipeline_req.data)
+    return infos
+
+
+def get_pipeline_jobs(project_id, pipeline, kinds=None, **request_kwargs):
+    pool = gitlab_ci_http_pool
+    jobs_url = '/api/v4/projects/{}/pipelines/{}/jobs'.format(project_id, pipeline)
+    jobs = request_paginated_url(pool, jobs_url, **request_kwargs)
+    for job in jobs:
+        # Remove the unnecessary 'pipeline' field
+        del job['pipeline']
+        # Backup the actual name and set the job's kind and name depending on the naming scheme
+        job['full_name'] = job['name']
+        job['kind'] = 'other'
+    # Filter failed jobs then sort them
+    jobs = (job for job in jobs if job['status'] == 'failed')
+    if kinds:
+        jobs = (job for job in jobs if job['kind'] in kinds)
+    return sorted(jobs, key=lambda j: (j['name'], j['kind']))
+
+
+def get_gitlab_ci_pipelines(http_pool, process_pool, project_id, date, date_range=None):
+    '''
+    Returns all gitlab-ci pipelines ran after the giver `min_date`
+    '''
+    date_to = date
+    date_from = date - date_range if date_range else date
+    per_page = process_pool._processes
+    if per_page > 100:
+        raise ValueError('gitlab-ci supports a max pagination of 100 results per page. '
+                         'The pool passed as argument must have 100 or less processes.')
+    fields = {'per_page': per_page}
+    url = '/api/v4/projects/{}/pipelines'.format(project_id)
+    page = 1
+    pipelines = []
+
+    # Fetch pipelines `nb_processes` by `nb_processes` until `from_date` in reached
+    while True:
+        # List pipelines
+        fields['page'] = page
+        page += 1
+        try:
+            cur_pipelines = http_pool.request('GET', url, fields=fields)
+        except HTTPError:
+            print('error: there were an error while fetching the gitlab-ci pipelines. '
+                  'Please make sure your identification token is right and try again.')
+            return pipelines
+        if cur_pipelines.status != 200:
+            print('error: there were an error while fetching the gitlab-ci pipeline\'s details. '
+                  'Please make sure your identification token is right and try again.')
+            return pipelines
+        cur_pipelines = json.loads(cur_pipelines.data)
+
+        # Handle cases where the requested date is older than all pipelines
+        if len(cur_pipelines) == 0:
+            return pipelines
+
+        # Request more infos about each pipeline (for instance the date)
+        pipelines_infos = [process_pool.apply_async(get_pipeline_details, (project_id, p['id']))
+                           for p in cur_pipelines]
+        pipelines_infos = [p.get() for p in pipelines_infos]
+
+        # Add pipeline in the right period, break once the from_date is found
+        for p in pipelines_infos:
+            cur_date = datetime.strptime(p['finished_at'], '%Y-%m-%dT%H:%M:%S.%fZ').date()
+            if cur_date < date_from:
+                return pipelines
+            if cur_date > date_to:
+                continue
+            pipelines.append(p)
+
+    return pipelines
+
+
+def get_gitlab_ci(branches, date, date_range=None, kinds=None):
+    '''
+    Retreive failed jobs from gitlab-ci grouped by pipelines
+    for the specified date.
+    If `kinds` list is provided, only jobs matching one of the
+    given kind will be returned.
+    '''
+    header = {'PRIVATE-TOKEN': localconfig.gitlab_user_token}
+    project_id = localconfig.gitlab_project_id
+    failed_filter = {'fields': {'status': 'failed'}}
+
+    # Build pools
+    global gitlab_ci_http_pool
+    gitlab_ci_http_pool = HTTPSConnectionPool('gitlab.com', port=443, cert_reqs='CERT_REQUIRED',
+                                              ca_certs=certifi.where(), timeout=30, headers=header)
+    process_pool = ProcessPool(processes=64)
+
+    # Fetch pipelines 64 by 64 until `date_from` in reached
+    pipelines = get_gitlab_ci_pipelines(gitlab_ci_http_pool, process_pool, project_id, date, date_range)
+
+    # Drop unnecessary branches
+    pipelines = [p for p in pipelines if p['ref'] in branches]
+
+    # Get jobs ran by those pipelines
+    pipelines_jobs = [process_pool.apply_async(get_pipeline_jobs, (project_id, p['id'], kinds),
+                                               failed_filter) for p in pipelines]
+    for pipeline, jobs in zip(pipelines, pipelines_jobs):
+        pipeline['jobs'] = jobs.get()
+
+    # Group pipelines by branch
+    pipelines = {branch: [p for p in pipelines if p['ref'] == branch] for branch in branches}
+    return pipelines
+
+
 def calculate_notifications(results, outdated_pkgs):
     '''
     Prepare the notifications{} dict for the notifications to individual
Signature · 1 line
2.21.0
#2 lore raw

PATCH 2/10 utils/daily-mail: new gitlab-ci data: defconfigs

In reply to Victor Huesca

This patch adds defconfigs to the gitlab-ci data introduced by the previous patch.

---
 utils/daily-mail | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
utils/daily-mail +6 -2
index 2d4ed1c..9dc9729 100755
@@ -528,8 +528,12 @@ def get_pipeline_jobs(project_id, pipeline, kinds=None, **request_kwargs):
         # Remove the unnecessary 'pipeline' field
         del job['pipeline']
         # Backup the actual name and set the job's kind and name depending on the naming scheme
-        job['full_name'] = job['name']
-        job['kind'] = 'other'
+        name = job['full_name'] = job['name']
+        if name.endswith('_defconfig'):
+            job['kind'] = 'defconfig'
+            job['name'] = name[:-10]
+        else:
+            job['kind'] = 'other'
     # Filter failed jobs then sort them
     jobs = (job for job in jobs if job['status'] == 'failed')
     if kinds:
Signature · 1 line
2.21.0
#3 lore raw

PATCH 3/10 utils/daily-mail: new gitlab-ci data: runtime tests

In reply to Victor Huesca

This patch adds runtime tests to the recognized gitlab-ci results.

---
 utils/daily-mail | 3 +++
 1 file changed, 3 insertions(+)
utils/daily-mail +3 -0
index 9dc9729..53500a6 100755
@@ -532,6 +532,9 @@ def get_pipeline_jobs(project_id, pipeline, kinds=None, **request_kwargs):
         if name.endswith('_defconfig'):
             job['kind'] = 'defconfig'
             job['name'] = name[:-10]
+        elif name.startswith('tests.'):
+            job['kind'] = 'runtime-test'
+            job['name'] = name.split('.')[-1]
         else:
             job['kind'] = 'other'
     # Filter failed jobs then sort them
Signature · 1 line
2.21.0
#4 lore raw

PATCH 4/10 utils/daily-mail: add defconfig support in developers notifications

In reply to Victor Huesca

This patch adds defconfig job failures in the 'Notification' structure. This will allow to inform developers about failures related to the defconfigs they maintains.

This patch relys on the gitlab-ci infrasture introduced earlier on in this series as well as the patch adding support for defconfig in the getdeveloperlib already applied to master.

---
 utils/daily-mail | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)
utils/daily-mail +26 -2
index 53500a6..5a44fad 100755
@@ -77,6 +77,7 @@ class Notification:
     def __init__(self):
         self.arch_notifications = defaultdict(list)
         self.package_notifications = defaultdict(list)
+        self.defconfig_notifications = defaultdict(list)
         self.package_version_notification = []
 
 
@@ -166,6 +167,23 @@ def add_package_notification(branch, notifications, build_result):
     build_result['orphan'] = orphan
 
 
+def add_defconfig_notification(branch, notifications, job):
+    '''
+    Add to the notifications{} dict notifications that are related to
+    defconfig "maintainers".
+    '''
+    orphan = True
+    for dev in developers:
+        if job['name'] in dev.defconfigs:
+            orphan = False
+            notif = notifications.setdefault(dev, Notification())
+            notif.defconfig_notifications[branch].append(job)
+    if orphan:
+        job['orphan'] = True
+        notif = notifications.setdefault(get_orphan_developer(), Notification())
+        notif.defconfig_notifications[branch].append(job)
+
+
 def add_outdated_pkg_notification(notifications, package):
     '''
     Add to the notifications{} dict notifications that are related to
@@ -631,7 +649,7 @@ def get_gitlab_ci(branches, date, date_range=None, kinds=None):
     return pipelines
 
 
-def calculate_notifications(results, outdated_pkgs):
+def calculate_notifications(results, outdated_pkgs, gitlab_results):
     '''
     Prepare the notifications{} dict for the notifications to individual
     developers, based on architecture developers, package developers,
@@ -649,6 +667,12 @@ def calculate_notifications(results, outdated_pkgs):
     for pkg in outdated_pkgs:
         add_outdated_pkg_notification(notifications, pkg)
 
+    for branch, pipelines in gitlab_results.iteritems():
+        for pipeline in pipelines:
+            for job in pipeline['jobs']:
+                if job['kind'] == 'defconfig':
+                    add_defconfig_notification(branch, notifications, job)
+
     return notifications
 
 
@@ -691,7 +715,7 @@ def __main__():
         results = {}
         results_by_reason = {}
     outdated_pkgs = get_outdated_pkgs(localconfig.pkg_stats) if 'outdated' in email_data else []
-    notifications = calculate_notifications(results, outdated_pkgs)
+    notifications = calculate_notifications(results, outdated_pkgs, {})
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
Signature · 1 line
2.21.0
#5 lore raw

PATCH 4/10 utils/daily-mail: add defconfig support in developers notifications

Hello Victor and all,

Quoted 59 lines · Victor Huesca

This patch adds defconfig job failures in the 'Notification' structure. This will allow to inform developers about failures related to the defconfigs they maintains.

This patch relys on the gitlab-ci infrasture introduced earlier on in this series as well as the patch adding support for defconfig in the getdeveloperlib already applied to master.

---
  utils/daily-mail | 28 ++++++++++++++++++++++++++--
  1 file changed, 26 insertions(+), 2 deletions(-)
utils/daily-mail +19 -1
index 53500a6..5a44fad 100755
@@ -77,6 +77,7 @@ class Notification:
      def __init__(self):
          self.arch_notifications = defaultdict(list)
          self.package_notifications = defaultdict(list)
+        self.defconfig_notifications = defaultdict(list)
          self.package_version_notification = []
 
 
@@ -166,6 +167,23 @@ def add_package_notification(branch, notifications, build_result):
      build_result['orphan'] = orphan
 
 
+def add_defconfig_notification(branch, notifications, job):
+    '''
+    Add to the notifications{} dict notifications that are related to
+    defconfig "maintainers".
+    '''
+    orphan = True
+    for dev in developers:
+        if job['name'] in dev.defconfigs:
+            orphan = False
+            notif = notifications.setdefault(dev, Notification())
+            notif.defconfig_notifications[branch].append(job)
+    if orphan:
+        job['orphan'] = True
+        notif = notifications.setdefault(get_orphan_developer(), Notification())
+        notif.defconfig_notifications[branch].append(job)
+
+
  def add_outdated_pkg_notification(notifications, package):
      '''
      Add to the notifications{} dict notifications that are related to
@@ -631,7 +649,7 @@ def get_gitlab_ci(branches, date, date_range=None, kinds=None):
      return pipelines
 
 
-def calculate_notifications(results, outdated_pkgs):
+def calculate_notifications(results, outdated_pkgs, gitlab_results):
      '''
      Prepare the notifications{} dict for the notifications to individual
      developers, based on architecture developers, package developers,

utils/daily-mail def calculate_notifications(results, outdated_pkgs): · line 670

Quoted 4 lines
utils/daily-mail +1 -0
@@ -649,6 +667,12 @@ def calculate_notifications(results, outdated_pkgs):
      for pkg in outdated_pkgs:
          add_outdated_pkg_notification(notifications, pkg)
 
+    for branch, pipelines in gitlab_results.iteritems():

Not sure if relevant, but dict.iteritems() does not exists anymore in Py3, and dict.items() now returns a "view"
(https://docs.python.org/3.0/whatsnew/3.0.html#views-and-iterators-instead-of-lists).

Quoted 10 lines
utils/daily-mail +5 -0
+        for pipeline in pipelines:
+            for job in pipeline['jobs']:
+                if job['kind'] == 'defconfig':
+                    add_defconfig_notification(branch, notifications, job)
+
      return notifications
 
 

utils/daily-mail def __main__(): · line 721

Quoted 8 lines
utils/daily-mail +1 -1
@@ -691,7 +715,7 @@ def __main__():
          results = {}
          results_by_reason = {}
      outdated_pkgs = get_outdated_pkgs(localconfig.pkg_stats) if 'outdated' in email_data else []
-    notifications = calculate_notifications(results, outdated_pkgs)
+    notifications = calculate_notifications(results, outdated_pkgs, {})
      smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
      smtp.starttls()
      smtp.login(localconfig.smtpuser, localconfig.smtppass)

Regards,

Titouan

#6 lore raw

PATCH 5/10 utils/daily-mail: add runtime tests support in developers notifications

In reply to Victor Huesca

This patch adds runtime-test job failures in the 'Notification' structure.
This will allow to inform developers about failures related to the runtime-tests they maintains.

This patch relys on the gitlab-ci infrasture introduced earlier on in this series as well as the patch adding support for runtime-tests in the getdeveloperlib already applied to master.

---
 utils/daily-mail | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
utils/daily-mail +20 -0
index 5a44fad..5e8799a 100755
@@ -78,6 +78,7 @@ class Notification:
         self.arch_notifications = defaultdict(list)
         self.package_notifications = defaultdict(list)
         self.defconfig_notifications = defaultdict(list)
+        self.runtime_test_notifications = defaultdict(list)
         self.package_version_notification = []
 
 
@@ -184,6 +185,23 @@ def add_defconfig_notification(branch, notifications, job):
         notif.defconfig_notifications[branch].append(job)
 
 
+def add_runtime_test_notification(branch, notifications, job):
+    '''
+    Add to the notifications{} dict notifications that are related to
+    runtime-tests "maintainers".
+    '''
+    orphan = True
+    for dev in developers:
+        if job['full_name'] in dev.runtime_tests:
+            orphan = False
+            notif = notifications.setdefault(dev, Notification())
+            notif.runtime_test_notifications[branch].append(job)
+    if orphan:
+        job['orphan'] = True
+        notif = notifications.setdefault(get_orphan_developer(), Notification())
+        notif.runtime_test_notifications[branch].append(job)
+
+
 def add_outdated_pkg_notification(notifications, package):
     '''
     Add to the notifications{} dict notifications that are related to
@@ -672,6 +690,8 @@ def calculate_notifications(results, outdated_pkgs, gitlab_results):
             for job in pipeline['jobs']:
                 if job['kind'] == 'defconfig':
                     add_defconfig_notification(branch, notifications, job)
+                elif job['kind'] == 'runtime-test':
+                    add_runtime_test_notification(branch, notifications, job)
 
     return notifications
 
Signature · 1 line
2.21.0
#7 lore raw

PATCH 6/10 utils/daily-mail: prepare the mail formatting to support gitlab-ci results

In reply to Victor Huesca

This patch allows to add gitlab-ci related results to mails.

Since every gitlab-ci result share the same information, a single function was preferred to factorize this part for now (it could be split once different gitlab-ci results have specific formatting).

---
 utils/daily-mail | 68 ++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 54 insertions(+), 14 deletions(-)
utils/daily-mail +54 -14
index 5e8799a..9f0673b 100755
@@ -246,6 +246,12 @@ def shrink_str(string, length, align='right', fill='...'):
         raise ValueError('align must be either `left`, `right` or `center`')
 
 
+def fmt_duration(**timedelta_args):
+    '''Small helper to format a duration'''
+    t = datetime(1990, 1, 1) + timedelta(**timedelta_args)
+    return t.strftime('%Hh%Mm%Ss').lstrip('0:hms')
+
+
 def show_results(results, show_status, show_orphan=False):
     contents = ""
     # Header
@@ -314,6 +320,32 @@ def show_outdated(packages, show_orphan=False):
     return contents
 
 
+def show_gitlab_ci_jobs(jobs, job_column_description='test-name', job_column_length=40, show_orphan=False):
+    # Header
+    contents = ''
+    if show_orphan:
+        contents += '{:>{}} | {:^61} | {:>8} | {:^5}\n'.format(job_column_description, job_column_length,
+                                                               'link to the job', 'duration', 'orph?')
+        contents += '{0:->{1}}-+-{0:-^61}-+-{0:->8} | {0:-^5}\n'.format('', job_column_length)
+    else:
+        contents += '{:>{}} | {:^61} | {:>8}\n'.format(job_column_description, job_column_length,
+                                                       'link to the job', 'duration')
+        contents += '{0:->{1}}-+-{0:-^61}-+-{0:->8}\n'.format('', job_column_length)
+
+    # Data
+    for job in sorted(jobs, key=lambda j: (j['kind'], j['name'])):
+        name = shrink_str(job['name'], job_column_length, align='left')
+        duration = fmt_duration(seconds=job['duration'])
+        orphan = 'ORPH' if job.get('orphan') else ''
+
+        contents += '{:>{}} | {:^61} | {:>8}'.format(name, job_column_length, job['web_url'], duration)
+        if show_orphan:
+            contents += ' | {:^5}'.format(orphan)
+        contents += '\n'
+
+    return contents
+
+
 def developers_email(smtp, branches, notifications, datestr, dry_run):
     '''
     Send the e-mails to the individual developers
@@ -387,10 +419,8 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
             print "To: {}".format(dev.name)
 
 
-def global_email_branch_result(results, results_by_reason, branch):
-    contents = "Results for branch '{}'\n".format(branch)
-    contents += "=====================" + "=" * len(branch) + "\n\n"
-    contents += "Classification of failures by reason\n"
+def global_email_branch_result(results, results_by_reason):
+    contents = "Classification of failures by reason\n"
     contents += "------------------------------------\n\n"
     for r in results_by_reason:
         reason = shrink_str(r['reason'], 30)
@@ -400,11 +430,14 @@ def global_email_branch_result(results, results_by_reason, branch):
     contents += "Detail of failures\n"
     contents += "------------------\n\n"
     contents += show_results(results, show_status=True, show_orphan=True)
-    contents += "\n"
     return contents
 
 
-def global_email(smtp, results, results_by_reason, datestr, overall, outdated_pkgs, dry_run):
+def global_email_branch_gitlab_ci(pipelines):
+    pass  # Implement this to support gitlab-ci related results
+
+
+def global_email(smtp, branches, results, results_by_reason, datestr, overall, outdated_pkgs, gitlab_ci, dry_run):
     '''
     Send the global e-mail to the mailing list
     '''
@@ -421,14 +454,21 @@ def global_email(smtp, results, results_by_reason, datestr, overall, outdated_pk
         if stats[3] == 0:
             continue
         contents += "  {:^10} | {:^3} | {:^3} | {:^3} | {:^3} |\n".format(branch, stats[0], stats[1], stats[2], stats[3])
-    if len(overall) != 0:
+    if any(map(len, (overall, gitlab_ci))):
         contents += "\n"
-    for branch in results.keys():
-        if len(results[branch]) == 0:
+    for branch in branches:
+        res = results.get(branch, [])
+        git = gitlab_ci.get(branch, [])
+        if not any(map(len, (res, git))):
             continue
-        contents += global_email_branch_result(results[branch], results_by_reason[branch], branch)
-    if len(overall) != 0:
-        contents += "\n"
+        contents += "Results for branch '{}'\n".format(branch)
+        contents += "=====================" + "=" * len(branch) + "\n\n"
+        if len(res) != 0:
+            contents += global_email_branch_result(res, results_by_reason[branch])
+            contents += "\n\n"
+        if len(git) != 0:
+            contents += global_email_branch_gitlab_ci(git)
+            contents += "\n\n"
     if outdated_pkgs:
         contents += "Packages having a newer version\n"
         contents += "===============================\n\n"
@@ -742,8 +782,8 @@ def __main__():
     if 'dev' in email_dest:
         developers_email(smtp, branches, notifications, date_str, args.dry_run)
     if 'global' in email_dest:
-        global_email(smtp, results, results_by_reason, date_str,
-                     overall_stats, outdated_pkgs, args.dry_run)
+        global_email(smtp, branches, results, results_by_reason, date_str,
+                     overall_stats, outdated_pkgs, {}, args.dry_run)
     smtp.quit()
 
 
Signature · 1 line
2.21.0
#8 lore raw

PATCH 7/10 utils/daily-mail: add formatting for defconfig results

In reply to Victor Huesca

This patch uses the gitlab-ci results generic function to format defconfigs in mails.

---
 utils/daily-mail | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
utils/daily-mail +24 -9
index 9f0673b..f8a4bfb 100755
@@ -355,14 +355,14 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
         email_from = localconfig.fromaddr
         subject = "[{}] Your daily results for {}".format(baseurl, datestr)
         contents = "Hello,\n\n"
-
-        if notif.arch_notifications or notif.package_notifications:
-            contents += "Autobuilder failures\n"
-            contents += "====================\n\n"
+        if notif.arch_notifications or notif.package_notifications or notif.defconfig_notifications:
+            contents += "Recent build failures\n"
+            contents += "=====================\n\n"
             contents += textwrap.fill("This is the list of Buildroot build failures that occurred on {}, "
-                                      "and for which you are a registered architecture developer or package "
-                                      "developer. Please help us improving the quality of Buildroot by "
-                                      "investigating those build failures and sending patches to fix them. "
+                                      "and for which you are a registered architecture developer, package "
+                                      "developer or defconfig developer. Please help us improving the "
+                                      "quality of Buildroot by investigating those build failures and "
+                                      "sending patches to fix them. "
                                       "Thanks!".format(datestr))
             contents += "\n\n"
         show_orphan = dev.name == ORPHAN_DEVELOPER
@@ -370,8 +370,9 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
         for branch in branches:
             archs = notif.arch_notifications.get(branch, [])
             packages = notif.package_notifications.get(branch, [])
+            defconfigs = notif.defconfig_notifications.get(branch, [])
 
-            if len(archs) == 0 and len(packages) == 0:
+            if not any(map(len, (archs, packages, defconfigs))):
                 continue
 
             contents += "Results for the '{}' branch\n".format(branch)
@@ -384,6 +385,11 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
             if len(packages) != 0:
                 contents += "Build failures related to your packages:\n\n"
                 contents += show_results(packages, show_status=False, show_orphan=show_orphan)
+                contents += "\n\n"
+            if len(defconfigs) != 0:
+                contents += "Build failures related to your defconfigs:\n\n"
+                contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
+                                                job_column_length=33, show_orphan=show_orphan)
 
             contents += "\n"
 
@@ -434,7 +440,16 @@ def global_email_branch_result(results, results_by_reason):
 
 
 def global_email_branch_gitlab_ci(pipelines):
-    pass  # Implement this to support gitlab-ci related results
+    defconfigs = [j for p in pipelines for j in p['jobs'] if j['kind'] == 'defconfig']
+
+    contents = ""
+    if len(defconfigs) != 0:
+        contents += 'Detail of defconfig failures\n'
+        contents += '----------------------------\n\n'
+        contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
+                                        job_column_length=33, show_orphan=True)
+        contents += "\n\n"
+    return contents
 
 
 def global_email(smtp, branches, results, results_by_reason, datestr, overall, outdated_pkgs, gitlab_ci, dry_run):
Signature · 1 line
2.21.0
#9 lore raw

PATCH 8/10 utils/daily-mail: add formatting for runtime-test results

In reply to Victor Huesca

This patch uses the gitlab-ci results generic function to format runtime-test in mails.

---
 utils/daily-mail | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)
utils/daily-mail +25 -7
index f8a4bfb..0510c07 100755
@@ -355,14 +355,19 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
         email_from = localconfig.fromaddr
         subject = "[{}] Your daily results for {}".format(baseurl, datestr)
         contents = "Hello,\n\n"
-        if notif.arch_notifications or notif.package_notifications or notif.defconfig_notifications:
-            contents += "Recent build failures\n"
-            contents += "=====================\n\n"
+
+        archs = notif.arch_notifications
+        packages = notif.package_notifications
+        defconfigs = notif.defconfig_notifications
+        runtime_tests = notif.runtime_test_notifications
+        if archs or packages or defconfigs or runtime_tests:
+            contents += "Recent build failures and runtime-tests failures\n"
+            contents += "================================================\n\n"
             contents += textwrap.fill("This is the list of Buildroot build failures that occurred on {}, "
                                       "and for which you are a registered architecture developer, package "
-                                      "developer or defconfig developer. Please help us improving the "
-                                      "quality of Buildroot by investigating those build failures and "
-                                      "sending patches to fix them. "
+                                      "developer or defconfig developer. This list also include runtime "
+                                      "tests failures. Please help us improving the quality of Buildroot by "
+                                      "investigating those build failures and sending patches to fix them. "
                                       "Thanks!".format(datestr))
             contents += "\n\n"
         show_orphan = dev.name == ORPHAN_DEVELOPER
@@ -371,8 +376,9 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
             archs = notif.arch_notifications.get(branch, [])
             packages = notif.package_notifications.get(branch, [])
             defconfigs = notif.defconfig_notifications.get(branch, [])
+            runtimes = notif.runtime_test_notifications.get(branch, [])
 
-            if not any(map(len, (archs, packages, defconfigs))):
+            if not any(map(len, (archs, packages, defconfigs, runtimes))):
                 continue
 
             contents += "Results for the '{}' branch\n".format(branch)
@@ -390,6 +396,11 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
                 contents += "Build failures related to your defconfigs:\n\n"
                 contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
                                                 job_column_length=33, show_orphan=show_orphan)
+                contents += "\n\n"
+            if len(runtimes) != 0:
+                contents += "Build failures related to your runtime tests:\n\n"
+                contents += show_gitlab_ci_jobs(runtimes, job_column_description='runtime-test',
+                                                job_column_length=25, show_orphan=show_orphan)
 
             contents += "\n"
 
@@ -441,6 +452,7 @@ def global_email_branch_result(results, results_by_reason):
 
 def global_email_branch_gitlab_ci(pipelines):
     defconfigs = [j for p in pipelines for j in p['jobs'] if j['kind'] == 'defconfig']
+    runtime_tests = [j for p in pipelines for j in p['jobs'] if j['kind'] == 'runtime-test']
 
     contents = ""
     if len(defconfigs) != 0:
@@ -449,6 +461,12 @@ def global_email_branch_gitlab_ci(pipelines):
         contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
                                         job_column_length=33, show_orphan=True)
         contents += "\n\n"
+    if len(runtime_tests) != 0:
+        contents += "Detail of runtime-test failures\n"
+        contents += "-------------------------------\n\n"
+        contents += show_gitlab_ci_jobs(runtime_tests, job_column_description='runtime-test',
+                                        job_column_length=25, show_orphan=True)
+        contents += "\n\n"
     return contents
 
 
Signature · 1 line
2.21.0
#10 lore raw

PATCH 9/10 utils/daily-mail: add argument for gitlab-ci related data

In reply to Victor Huesca

This patch provides two new data to the args parser to allow enabling defconfigs, runtime-tests and sanity checks in global/developer emails. These tree data are grouped together in a gitlab-ci data group as they are fetched at the same time.

This new list will be use by the next patch to enable gitlab-ci jobs in email.

---
 utils/daily-mail | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
utils/daily-mail +7 -6
index 0510c07..6b0a1b4 100755
@@ -776,7 +776,8 @@ def parse_args():
     parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
     parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
                         help='List of emails type to send (ie. global, dev)')
-    parser.add_argument('--data', action='store', nargs='+', choices=['autobuild', 'outdated'],
+    parser.add_argument('--data', action='store', nargs='+',
+                        choices=['autobuild', 'outdated', 'defconfig', 'runtime-test'],
                         help='List of information to add in emails (blank separated)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
@@ -787,13 +788,13 @@ def parse_args():
 def __main__():
     args = parse_args()
     if args.date:
-        date_str = args.date
+        date_struct = datetime.strptime(args.date, '%Y-%m-%d').date()
     else:
-        yesterday = date.today() - timedelta(1)
-        date_str = yesterday.strftime('%Y-%m-%d')
-    branches = args.branches if args.branches else get_branches()
+        date_struct = date.today() - timedelta(1)
+    date_str = date_struct.isoformat()
+    branches = args.branches or get_branches()
     email_dest = set(args.dest) if args.dest else {'dev', 'global'}
-    email_data = set(args.data) if args.data else {'autobuild', 'outdated'}
+    email_data = set(args.data) if args.data else {'autobuild', 'outdated', 'defconfig', 'runtime-test'}
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
Signature · 1 line
2.21.0
#11 lore raw

PATCH 10/10 utils/daily-mail: enable gitlab-ci emails

In reply to Victor Huesca

This patch enable all gitlab-ci related results added by previous patches from the series in both the global and per-developer emails.

Defconfigs and runtime-tests entries are added in the per-branch results for both global and dev-mails.

---
 utils/daily-mail | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
utils/daily-mail +6 -2
index 6b0a1b4..c249376 100755
@@ -809,7 +809,11 @@ def __main__():
         results = {}
         results_by_reason = {}
     outdated_pkgs = get_outdated_pkgs(localconfig.pkg_stats) if 'outdated' in email_data else []
-    notifications = calculate_notifications(results, outdated_pkgs, {})
+    if email_data & {'defconfig', 'runtime-test'}:
+        gitlab_ci = get_gitlab_ci(branches, date_struct, kinds=email_data)
+    else:
+        gitlab_ci = {}
+    notifications = calculate_notifications(results, outdated_pkgs, gitlab_ci)
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
@@ -817,7 +821,7 @@ def __main__():
         developers_email(smtp, branches, notifications, date_str, args.dry_run)
     if 'global' in email_dest:
         global_email(smtp, branches, results, results_by_reason, date_str,
-                     overall_stats, outdated_pkgs, {}, args.dry_run)
+                     overall_stats, outdated_pkgs, gitlab_ci, args.dry_run)
     smtp.quit()
 
 
Signature · 1 line
2.21.0
#12 lore raw

cover add gitlab-ci jobs to the daily-mail

In reply to Victor Huesca

Hello Victor,

Quoted 5 lines · On Thu, 29 Aug 2019 15:23:38 +0200 Victor Huesca <[email protected]>…

Victor Huesca (10):
utils/daily-mail: new data: gitlab-ci jobs utils/daily-mail: new gitlab-ci data: defconfigs utils/daily-mail: new gitlab-ci data: runtime tests utils/daily-mail: add defconfig support in developers notifications

Quoted 8 lines

utils/daily-mail: add runtime tests support in developers
notifications
utils/daily-mail: prepare the mail formatting to support gitlab-ci
results
utils/daily-mail: add formatting for defconfig results utils/daily-mail: add formatting for runtime-test results utils/daily-mail: add argument for gitlab-ci related data utils/daily-mail: enable gitlab-ci emails

Thanks for this patch series! I have made a number of changes, and applied it.

The main changes are:

- More filtering of pipelines: only failed pipelines are retrieved

- Earlier filtering of pipelines: we can filter pipelines by branch and status before getting the details for each pipeline

- Removal of the job duration in the e-mails, it is not a very useful information IMO

- Further splitting of some of the patches

So, hopefully this Saturday we should see e-mails about defconfig failures, since we have at least one still failing on master.

Thanks a lot for this work!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com