add support for outdated packages to daily-mail
v2 cover add support for outdated packages to daily-mail
This patch-series provides a new data for daily-mails in addition to autobuild results: packages with a higher upstream version detected. The goal is to provide a more information to developers by in the daily-mail. This will help developers to know if packages they maintains have a newer version and should help to have latest version of packages in buildroot. Of course, this notification should not be included in the actual daily-mails since new version are not release every-day but it could be enabled for example every week.
Victor Huesca (4):
utils/daily-mail: new data: outdated packages utils/daily-mail: add outdated packages to the developer notification utils/daily-mail: add outdated-packages email formating part utils/daily-mail: enable outdated packages emails
utils/daily-mail | 106 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 100 insertions(+), 6 deletions(-)
Signature · 1 line
2.21.0
PATCH v2 1/4 utils/daily-mail: new data: outdated packages
This patch uses the buildroot pkg-stats json output to list all outdated packages. This will allow to ease the track of new upstream version of buildroot packages.
This patch requires the non-standard python module 'packaging' in order to compare the version numbers. This module can be install via the distro package manager as 'python-packaging'.
--- utils/daily-mail | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
utils/daily-mail +27 -0
@@ -14,11 +14,14 @@ from collections import defaultdict import math import argparse import re +import json +from packaging import version sys.path.append(os.path.join(localconfig.brbase, "utils")) import getdeveloperlib # noqa: E402 RE_DATE = re.compile(r'^\d\d\d\d-\d\d-\d\d$') +RE_HASH_40 = re.compile(r'.*[a-fA-F0-9]{40}.*') baseurl = "autobuild.buildroot.net" http_baseurl = "http://" + baseurl
@@ -367,6 +370,30 @@ def get_build_results_grouped_by_reason(db, datestr, branches): return results_by_reason +def get_outdated_pkg(path): + with open(path, 'r') as f: + stats = json.load(f) + s = [] + for name, pkg in stats['packages'].items(): + status, latest_ver, p_id = pkg['latest_version'] + cur_ver = pkg['current_version'] + if status not in (2, 3): + continue # an upstream version exists + if not cur_ver or not latest_ver: + continue # both version and latest version are actual strings + if RE_HASH_40.match(cur_ver): + continue # the version is a hash + if version.parse(str(cur_ver)) >= version.parse(str(latest_ver)): + continue # up to date + + s.append({'name': str(name), + 'id': p_id, + 'version': str(cur_ver), + 'upstream': str(latest_ver), + 'from': 'DISTRO' if status == 2 else 'GUESS'}) + return sorted(s, key=lambda pkg: pkg['name']) + + def calculate_notifications(results): ''' Prepare the notifications{} dict for the notifications to individual
Signature · 1 line
2.21.0
PATCH v2 2/4 utils/daily-mail: add outdated packages to the developer notification
In reply to Victor HuescaThis patch provides the logic to add outdated packages to the notification structure.
This will allow future patches to add this new data to the per-developer emails.
--- utils/daily-mail | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)
utils/daily-mail +24 -2
@@ -72,6 +72,7 @@ class Notification: def __init__(self): self.arch_notifications = defaultdict(list) self.package_notifications = defaultdict(list) + self.package_version_notification = [] def get_mklist(basepath):
@@ -160,6 +161,23 @@ def add_package_notification(branch, notifications, build_result): build_result['orphan'] = orphan +def add_outdated_pkg_notification(notifications, package): + ''' + Add to the notifications{} dict notifications that are related to + package "maintainers". + ''' + orphan = True + for dev in developers: + if package['name'] in dev.packages: + orphan = False + notif = notifications.setdefault(dev, Notification()) + notif.package_version_notification.append(package) + if orphan: + package['orphan'] = True + notif = notifications.setdefault(get_orphan_developer(), Notification()) + notif.package_version_notification.append(package) + + def shrink_str(string, length, align='right', fill='...'): ''' Returns the `string` shrinked to fit `length` characters and with `fill`
@@ -394,7 +412,7 @@ def get_outdated_pkg(path): return sorted(s, key=lambda pkg: pkg['name']) -def calculate_notifications(results): +def calculate_notifications(results, outdated_pkg): ''' Prepare the notifications{} dict for the notifications to individual developers, based on architecture developers, package developers,
@@ -408,6 +426,10 @@ def calculate_notifications(results): continue add_arch_notification(branch, notifications, result) add_package_notification(branch, notifications, result) + + for pkg in outdated_pkg: + add_outdated_pkg_notification(notifications, pkg) + return notifications
@@ -449,7 +471,7 @@ def __main__(): overall_stats = {} results = {} results_by_reason = {} - notifications = calculate_notifications(results) + notifications = calculate_notifications(results, []) smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport) smtp.starttls() smtp.login(localconfig.smtpuser, localconfig.smtppass)
Signature · 1 line
2.21.0
PATCH v2 3/4 utils/daily-mail: add outdated-packages email formating part
In reply to Victor HuescaThis patch provides the formating part for the outdated packages. This adds a new table after the autobuild results to inform about packages having a newer version.
--- utils/daily-mail | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)
utils/daily-mail +46 -2
@@ -245,6 +245,34 @@ def show_results(results, show_status, show_orphan=False): return contents +def show_outdated(packages, show_orphan=False): + contents = '' + # Header + contents += '{:^30} | {:^8} | {:^44} | {:^12} | {:^12}'.format('name', 'found by', + 'link to release-monitoring.org', + 'version', 'upstream') + if show_orphan: + contents += ' | {:^5}'.format('orph?') + contents += '\n{0:-^30}-+-{0:-^8}-+-{0:-^44}-+-{0:-^12}-+-{0:-^12}-'.format('') + if show_orphan: + contents += '+-{:-^5}-'.format('') + contents += '\n' + + # Actual data + for pkg in packages: + name = shrink_str(pkg['name'], 30) + version = shrink_str(pkg['version'], 12) + upstream = shrink_str(pkg['upstream'], 12) + orphan_str = 'ORPH' if pkg.get('orphan') else '' + url = 'https://release-monitoring.org/project/{:0>5}'.format(pkg['id']) + + contents += '{:>30} | {:^8} | {:0>5} | {:12} | {:12}'.format(name, pkg['from'], url, version, upstream) + if show_orphan: + contents += ' | {:4}'.format(orphan_str) + contents += '\n' + return contents + + def developers_email(smtp, branches, notifications, datestr, dry_run): ''' Send the e-mails to the individual developers
@@ -282,6 +310,17 @@ def developers_email(smtp, branches, notifications, datestr, dry_run): contents += "\n" + outdated = notif.package_version_notification + if len(outdated) != 0: + contents += '\n' + contents += textwrap.fill("This is the list of packages for which a new version has been " + "detected and for which you are a registered developer. Please " + "help us improving the quality of Buildroot by bumping these " + "packages to their latest version. Thanks!") + contents += '\n\n' + contents += show_outdated(outdated, show_orphan=show_orphan) + contents += '\n' + contents += "-- \n" contents += http_baseurl if dry_run:
@@ -319,7 +358,7 @@ def global_email_branch_result(results, results_by_reason, branch): return contents -def global_email(smtp, results, results_by_reason, datestr, overall, dry_run): +def global_email(smtp, results, results_by_reason, datestr, overall, outdated_pkg, dry_run): ''' Send the global e-mail to the mailing list '''
@@ -341,6 +380,11 @@ def global_email(smtp, results, results_by_reason, datestr, overall, dry_run): if len(results[branch]) == 0: continue contents += global_email_branch_result(results[branch], results_by_reason[branch], branch) + if outdated_pkg: + contents += '\n\n' + contents += "Packages having a newer version\n" + contents += "===============================\n\n" + contents += show_outdated(outdated_pkg, show_orphan=True) contents += "\n" contents += "-- \n" contents += http_baseurl
@@ -479,7 +523,7 @@ def __main__(): developers_email(smtp, branches, notifications, date_str, args.dry_run) if 'global' in email_dest: global_email(smtp, results, results_by_reason, date_str, - overall_stats, args.dry_run) + overall_stats, [], args.dry_run) smtp.quit()
Signature · 1 line
2.21.0
PATCH v2 4/4 utils/daily-mail: enable outdated packages emails
In reply to Victor HuescaThis patch adds an 'outdated' entry to the program data. This entry adds information about the outdated packages in the global and developers mails.
A new entry in the 'localconfig' file named 'pkg-stats' that contains the location of the JSON output of the buildroot's 'pkg-stats' script.
--- utils/daily-mail | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
utils/daily-mail +5 -4
@@ -484,7 +484,7 @@ def parse_args(): parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)') parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'], help='List of emails type to send (ie. global, dev)') - parser.add_argument('--data', action='store', nargs='+', choices=['autobuild'], + parser.add_argument('--data', action='store', nargs='+', choices=['autobuild', 'outdated'], help='List of information to add in emails (blank separated)') args = parser.parse_args() if args.date and not RE_DATE.match(args.date):
@@ -501,7 +501,7 @@ def __main__(): date_str = yesterday.strftime('%Y-%m-%d') branches = args.branches if args.branches else get_branches() email_dest = set(args.dest) if args.dest else {'dev', 'global'} - email_data = set(args.data) if args.data else {'autobuild'} + email_data = set(args.data) if args.data else {'autobuild', 'outdated'} db = _mysql.connect(host=localconfig.host, user=localconfig.user,
@@ -515,7 +515,8 @@ def __main__(): overall_stats = {} results = {} results_by_reason = {} - notifications = calculate_notifications(results, []) + outdate_pkg = get_outdated_pkg(localconfig.pkg_stats) if 'outdated' in email_data else [] + notifications = calculate_notifications(results, outdate_pkg) smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport) smtp.starttls() smtp.login(localconfig.smtpuser, localconfig.smtppass)
@@ -523,7 +524,7 @@ def __main__(): developers_email(smtp, branches, notifications, date_str, args.dry_run) if 'global' in email_dest: global_email(smtp, results, results_by_reason, date_str, - overall_stats, [], args.dry_run) + overall_stats, outdate_pkg, args.dry_run) smtp.quit()
Signature · 1 line
2.21.0
v2 cover add support for outdated packages to daily-mail
In reply to Victor HuescaHello Victor,
Quoted 5 lines · On Mon, 5 Aug 2019 15:15:40 +0200 Victor Huesca <[email protected]>…
Victor Huesca (4):
utils/daily-mail: new data: outdated packages utils/daily-mail: add outdated packages to the developer notification utils/daily-mail: add outdated-packages email formating part utils/daily-mail: enable outdated packages emails
Thanks a lot for this work, I have applied all 4 patches, with some minor changes. I also did a few formatting improvements:
https://git.buildroot.org/buildroot-test/commit/?id=ba42c3bdce31581dc9c270165ebe59393e91c51f
https://git.buildroot.org/buildroot-test/commit/?id=2e8ae4662d473d67398bf97eaf5e40bcbbc000be
https://git.buildroot.org/buildroot-test/commit/?id=e7ed66b8a9383de60ab59e8604df4215b1753d30
The e-mail of today (both global and per developer) was sent with your patches applied, so those notifications about outdated packages have been sent to many developers.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com