Skip to contentSkip to conversation

improve modularity by adding an

PATCH New Victor Huesca 2 participants 7 messages Aug 4, 2019

lore
Share & tools
lore
#0 lore raw

cover improve modularity by adding an

This patch series replace the manual argument parsing by a python argument parser. The goal is to improve the modularity to help the integration of new information to the mails.

Victor Huesca (5):
utils/daily-mail: introduce argparse to ease modularity of the script utils/daily-mail: allow to pass a date in arguments utils/daily-mail: allow to specify branches from command-line utils/daily-mail: allow to select only a subset of emails to send utils/daily-mail: allow to select only a subset of data to send in
emails

 utils/daily-mail | 64 +++++++++++++++++++++++++++++++++++-------------
 1 file changed, 47 insertions(+), 17 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/5 utils/daily-mail: introduce argparse to ease modularity of the script

Having an actual argument parser instead of manually checking for command-line arguments will allow future patches to add divers options to the script. This patch only introduce the parser with the 'dry-run' argument.

---
 utils/daily-mail | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
utils/daily-mail +10 -5
index 2d5e684..d4c69f6 100755
@@ -12,6 +12,7 @@ import localconfig
 import csv
 from collections import defaultdict
 import math
+import argparse
 
 sys.path.append(os.path.join(localconfig.brbase, "utils"))
 import getdeveloperlib  # noqa: E402
@@ -379,7 +380,14 @@ def calculate_notifications(results):
     return notifications
 
 
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
+    return parser.parse_args()
+
+
 def __main__():
+    args = parse_args()
     yesterday = date.today() - timedelta(1)
     yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = get_branches()
@@ -392,15 +400,12 @@ def __main__():
     results = get_build_results(db, yesterday_str, branches)
     results_by_reason = get_build_results_grouped_by_reason(db, yesterday_str, branches)
     notifications = calculate_notifications(results)
-    dry_run = False
-    if len(sys.argv) == 2 and sys.argv[1] == "--dry-run":
-        dry_run = True
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
-    developers_email(smtp, branches, notifications, yesterday_str, dry_run)
+    developers_email(smtp, branches, notifications, yesterday_str, args.dry_run)
     global_email(smtp, results, results_by_reason, yesterday_str,
-                 overall_stats, dry_run)
+                 overall_stats, args.dry_run)
     smtp.quit()
 
 
Signature · 1 line
2.21.0
#2 lore raw

PATCH 2/5 utils/daily-mail: allow to pass a date in arguments

In reply to Victor Huesca

This allows to use a custorm date instead of the hardcoded yesterday.

---
 utils/daily-mail | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
utils/daily-mail +13 -3
index d4c69f6..cb7402f 100755
@@ -13,10 +13,13 @@ import csv
 from collections import defaultdict
 import math
 import argparse
+import re
 
 sys.path.append(os.path.join(localconfig.brbase, "utils"))
 import getdeveloperlib  # noqa: E402
 
+RE_DATE = re.compile(r'^\d\d\d\d-\d\d-\d\d$')
+
 baseurl = "autobuild.buildroot.net"
 http_baseurl = "http://" + baseurl
 
@@ -383,13 +386,20 @@ def calculate_notifications(results):
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
-    return parser.parse_args()
+    parser.add_argument('--date', action='store', help='Use this date instead of yesterday')
+    args = parser.parse_args()
+    if args.date and not RE_DATE.match(args.date):
+        parser.error('date must be in format YYYY-MM-DD')
+    return args
 
 
 def __main__():
     args = parse_args()
-    yesterday = date.today() - timedelta(1)
-    yesterday_str = yesterday.strftime('%Y-%m-%d')
+    if args.date:
+        yesterday_str = args.date
+    else:
+        yesterday = date.today() - timedelta(1)
+        yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = get_branches()
 
     db = _mysql.connect(host=localconfig.host,
Signature · 1 line
2.21.0
#3 lore raw

PATCH 3/5 utils/daily-mail: allow to specify branches from command-line

In reply to Victor Huesca

This patch allows to specify branches to include in emails instead of reading the 'branches' file. If no branch is provided, the default 'branches' file is used.

---
 utils/daily-mail | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
utils/daily-mail +2 -1
index cb7402f..040ee15 100755
@@ -387,6 +387,7 @@ def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
     parser.add_argument('--date', action='store', help='Use this date instead of yesterday')
+    parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
         parser.error('date must be in format YYYY-MM-DD')
@@ -400,7 +401,7 @@ def __main__():
     else:
         yesterday = date.today() - timedelta(1)
         yesterday_str = yesterday.strftime('%Y-%m-%d')
-    branches = get_branches()
+    branches = args.branches if args.branches else get_branches()
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
Signature · 1 line
2.21.0
#4 lore raw

PATCH 4/5 utils/daily-mail: allow to select only a subset of emails to send

In reply to Victor Huesca

This patch allows to send the global email and the per-developer email separately by specifying the destination as command-line argument. The default behavior -- when no destination is provided -- is to select both the global and developers emails.

---
 utils/daily-mail | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
utils/daily-mail +8 -3
index 040ee15..4407438 100755
@@ -388,6 +388,8 @@ def parse_args():
     parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
     parser.add_argument('--date', action='store', help='Use this date instead of yesterday')
     parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
+    parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
+                        help='List of emails type to send (ie. global, dev)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
         parser.error('date must be in format YYYY-MM-DD')
@@ -402,6 +404,7 @@ def __main__():
         yesterday = date.today() - timedelta(1)
         yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = args.branches if args.branches else get_branches()
+    email_dest = set(args.dest) if args.dest else {'dev', 'global'}
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
@@ -414,9 +417,11 @@ def __main__():
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
-    developers_email(smtp, branches, notifications, yesterday_str, args.dry_run)
-    global_email(smtp, results, results_by_reason, yesterday_str,
-                 overall_stats, args.dry_run)
+    if 'dev' in email_dest:
+        developers_email(smtp, branches, notifications, yesterday_str, args.dry_run)
+    if 'global' in email_dest:
+        global_email(smtp, results, results_by_reason, yesterday_str,
+                     overall_stats, args.dry_run)
     smtp.quit()
 
 
Signature · 1 line
2.21.0
#5 lore raw

PATCH 5/5 utils/daily-mail: allow to select only a subset of data to send in emails

In reply to Victor Huesca

This patch adds a modular way to select a subset of the available information collected by this script. Currently the range of possibilities is limited to enable or disable the autobuild results, but it allow future patches to easily add new kind of data to this script.

---
 utils/daily-mail | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
utils/daily-mail +17 -8
index 4407438..20e5886 100755
@@ -231,7 +231,7 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
     for dev, notif in notifications.iteritems():
         to = dev.name
         email_from = localconfig.fromaddr
-        subject = "[{}] Your build results for {}".format(baseurl, datestr)
+        subject = "[{}] Your daily results for {}".format(baseurl, datestr)
         contents = "Hello,\n\n"
         contents += textwrap.fill("This is the list of Buildroot build failures that occurred on {}, "
                                   "and for which you are a registered architecture developer or package "
@@ -304,11 +304,12 @@ def global_email(smtp, results, results_by_reason, datestr, overall, dry_run):
     '''
     to = "buildroot at buildroot.org"
     email_from = localconfig.fromaddr
-    subject = "[{}] Build results for {}".format(baseurl, datestr)
+    subject = "[{}] Daily results for {}".format(baseurl, datestr)
     contents = "Hello,\n\n"
-    contents += "Build statistics for {}\n".format(datestr)
-    contents += "===============================\n\n"
-    contents += "      branch |  OK | NOK | TIM | TOT |\n"
+    if len(overall) != 0:
+        contents += "Autobuild statistics for {}\n".format(datestr)
+        contents += "===============================\n\n"
+        contents += "      branch |  OK | NOK | TIM | TOT |\n"
     for branch in sorted(overall.iterkeys()):
         stats = overall[branch]
         if stats[3] == 0:
@@ -390,6 +391,8 @@ def parse_args():
     parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
     parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
                         help='List of emails type to send (ie. global, dev)')
+    parser.add_argument('--data', action='store', nargs='+', choices=['autobuild'],
+                        help='List of information to add in emails (blank separated)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
         parser.error('date must be in format YYYY-MM-DD')
@@ -405,14 +408,20 @@ def __main__():
         yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = args.branches if args.branches else get_branches()
     email_dest = set(args.dest) if args.dest else {'dev', 'global'}
+    email_data = set(args.data) if args.data else {'autobuild'}
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
                         passwd=localconfig.passwd,
                         db=localconfig.db)
-    overall_stats = get_overall_stats(db, yesterday_str, branches)
-    results = get_build_results(db, yesterday_str, branches)
-    results_by_reason = get_build_results_grouped_by_reason(db, yesterday_str, branches)
+    if 'autobuild' in email_data:
+        overall_stats = get_overall_stats(db, yesterday_str, branches)
+        results = get_build_results(db, yesterday_str, branches)
+        results_by_reason = get_build_results_grouped_by_reason(db, yesterday_str, branches)
+    else:
+        overall_stats = {}
+        results = {}
+        results_by_reason = {}
     notifications = calculate_notifications(results)
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
Signature · 1 line
2.21.0
#6 lore raw

cover improve modularity by adding an

In reply to Victor Huesca
Quoted 7 lines · On Sun, 4 Aug 2019 17:25:31 +0200 Victor Huesca <[email protected]>…

Victor Huesca (5):
utils/daily-mail: introduce argparse to ease modularity of the script utils/daily-mail: allow to pass a date in arguments utils/daily-mail: allow to specify branches from command-line utils/daily-mail: allow to select only a subset of emails to send utils/daily-mail: allow to select only a subset of data to send in
emails

Applied, thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com