generate regression reports against proper releases
v3 cover generate regression reports against proper releases
Hi, this is the 3nd version of work initiated to improve regression reports usefulness, started around issue YOCTO #1465
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065).
Changes since v2:
- add symlink to preserve compatibility with current autobuilder configuration
Changes since v1:
- minor rework to be able to import send_qa_email.py as a standard python module
- properly manage non-releases versions since qe_send_email.py can be called with such "release" versions ("-r" parameter)
- add unit tests on previous version computation
- do not fetch yocto-testresults full history: identify needed revisions with git ls-remote and retrieve them with git fetch
Alexis Lothoré (5):
scripts/send_qa_email.py: Rename send-qa-email to send_qa_email.py scripts/send_qa_email.py: Wrap send_qa_email.py content in function scripts/send-qa-email: Generate regression reports against most
relevant release
scripts/send_qa_email.py: add unit tests on previous version
computation
scripts/send-qa-email: add symlink to preserve compatibility with
autobuilder
scripts/send-qa-email | 165 +------------------------ scripts/send_qa_email.py | 223 ++++++++++++++++++++++++++++++++++ scripts/shared-repo-unpack | 2 +- scripts/test_send_qa_email.py | 57 +++++++++ scripts/utils.py | 47 +++++++ 5 files changed, 329 insertions(+), 165 deletions(-) mode change 100755 => 120000 scripts/send-qa-email create mode 100755 scripts/send_qa_email.py create mode 100755 scripts/test_send_qa_email.py
Signature · 1 line
2.39.0
PATCH v3 1/5 scripts/send_qa_email.py: Rename send-qa-email to send_qa_email.py
This rename fixes send-qa-email regarding python guidelines, which allows to import the script content in another script (for unit testing purpose for example)
---
scripts/{send-qa-email => send_qa_email.py} | 0
scripts/shared-repo-unpack | 2 +-
2 files changed, 1 insertion(+), 1 deletion(-)
rename scripts/{send-qa-email => send_qa_email.py} (100%)scripts/send_qa_email.py renamed from scripts/send-qa-email +0 -0
scripts/shared-repo-unpack +1 -1
@@ -61,7 +61,7 @@ for repo in sorted(repos.keys()): utils.printheader("Copying in repo %s" % repo) utils.mkdir(targetsubdir) if args.target in ["a-full", "a-quick"]: - # full/quick need all repo data due to send-qa-email + # full/quick need all repo data due to send_qa_email.py subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir]) else: subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir, "./" + repo])
Signature · 1 line
2.39.0
PATCH v3 2/5 scripts/send_qa_email.py: Wrap send_qa_email.py content in function
In reply to Alexis LothoréThis wrapping allows importing send_qa_email.py in another script without executing the QA emailing routine. This is useful for unit testing the script internal functions
--- scripts/send_qa_email.py | 287 ++++++++++++++++++++------------------- 1 file changed, 145 insertions(+), 142 deletions(-)
scripts/send_qa_email.py +145 -142
@@ -15,150 +15,153 @@ import tempfile import utils -parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.') - -parser.add_argument('send', - help="True to send email, otherwise the script will display a message and exit") -parser.add_argument('repojson', - help="The json file containing the repositories to use") -parser.add_argument('sharedrepodir', - help="The shared repos directory (to resolve the repo revision hashes)") -parser.add_argument('-p', '--publish-dir', - action='store', - help="Where the artefacts were published") -parser.add_argument('-R', '--results-dir', - action='store', - help="Where the test results were published") -parser.add_argument('-r', '--release', - action='store', - help="The build/release 'name' for release purposes (optional)") - -args = parser.parse_args() - -scriptsdir = os.path.dirname(os.path.realpath(__file__)) -ourconfig = utils.loadconfig() - -with open(args.repojson) as f: - repos = json.load(f) - -resulttool = os.path.dirname(args.repojson) + "/build/scripts/resulttool" - -buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools" -if os.path.exists(buildtoolsdir): - utils.enable_buildtools_tarball(buildtoolsdir) - -repodir = os.path.dirname(args.repojson) + "/build/repos" - -if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: - # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/poky" % (repodir) - revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() - branch = repos['poky']['branch'] - repo = repos['poky']['url'] - - extraopts = None - basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch) - if basebranch: - extraopts = " --branch %s --commit %s" % (branch, revision) - if comparebranch: - extraopts = extraopts + " --branch2 %s" % (comparebranch) - elif basebranch: - print("No comparision branch found, comparing to %s" % basebranch) - extraopts = extraopts + " --branch2 %s" % basebranch - - report = subprocess.check_output([resulttool, "report", args.results_dir]) - with open(args.results_dir + "/testresult-report.txt", "wb") as f: - f.write(report) - - tempdir = tempfile.mkdtemp(prefix='sendqaemail.') - try: - cloneopts = [] +def send_qa_email(): + parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.') + + parser.add_argument('send', + help="True to send email, otherwise the script will display a message and exit") + parser.add_argument('repojson', + help="The json file containing the repositories to use") + parser.add_argument('sharedrepodir', + help="The shared repos directory (to resolve the repo revision hashes)") + parser.add_argument('-p', '--publish-dir', + action='store', + help="Where the artefacts were published") + parser.add_argument('-R', '--results-dir', + action='store', + help="Where the test results were published") + parser.add_argument('-r', '--release', + action='store', + help="The build/release 'name' for release purposes (optional)") + + args = parser.parse_args() + + scriptsdir = os.path.dirname(os.path.realpath(__file__)) + ourconfig = utils.loadconfig() + + with open(args.repojson) as f: + repos = json.load(f) + + resulttool = os.path.dirname(args.repojson) + "/build/scripts/resulttool" + + buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools" + if os.path.exists(buildtoolsdir): + utils.enable_buildtools_tarball(buildtoolsdir) + + repodir = os.path.dirname(args.repojson) + "/build/repos" + + if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: + # Need the finalised revisions (not 'HEAD') + targetrepodir = "%s/poky" % (repodir) + revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() + branch = repos['poky']['branch'] + repo = repos['poky']['url'] + + extraopts = None + basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch) + if basebranch: + extraopts = " --branch %s --commit %s" % (branch, revision) if comparebranch: - cloneopts = ["--branch", comparebranch] + extraopts = extraopts + " --branch2 %s" % (comparebranch) elif basebranch: - cloneopts = ["--branch", basebranch] - try: - subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"] + cloneopts) - except subprocess.CalledProcessError: - print("No comparision branch found, falling back to master") - subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"]) + print("No comparision branch found, comparing to %s" % basebranch) + extraopts = extraopts + " --branch2 %s" % basebranch - # If the base comparision branch isn't present regression comparision won't work - # at least until we can tell the tool to ignore internal branch information - if basebranch: + report = subprocess.check_output([resulttool, "report", args.results_dir]) + with open(args.results_dir + "/testresult-report.txt", "wb") as f: + f.write(report) + + tempdir = tempfile.mkdtemp(prefix='sendqaemail.') + try: + cloneopts = [] + if comparebranch: + cloneopts = ["--branch", comparebranch] + elif basebranch: + cloneopts = ["--branch", basebranch] try: - subprocess.check_call(["git", "rev-parse", "--verify", basebranch], cwd=tempdir) + subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"] + cloneopts) except subprocess.CalledProcessError: - # Doesn't exist so base it off master - # some older hosts don't have git branch old new - subprocess.check_call(["git", "checkout", "master"], cwd=tempdir) - subprocess.check_call(["git", "branch", basebranch], cwd=tempdir) - subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir) - extraopts = None - - subprocess.check_call([resulttool, "store", args.results_dir, tempdir]) - if comparebranch: - subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir) - subprocess.check_call(["git", "push", "--tags", "--force"], cwd=tempdir) - elif basebranch: - subprocess.check_call(["git", "push", "--all"], cwd=tempdir) - subprocess.check_call(["git", "push", "--tags"], cwd=tempdir) - - if extraopts: - regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split()) - with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f: - f.write(regreport) - - finally: - subprocess.check_call(["rm", "-rf", tempdir]) - pass - -if args.send.lower() != 'true' or not args.publish_dir or not args.release: - utils.printheader("Not sending QA email") - sys.exit(0) - -buildhashes = "" -for repo in sorted(repos.keys()): - # gplv2 is no longer built/tested in master - if repo == "meta-gplv2": - continue - # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/%s" % (repodir, repo) - revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() - buildhashes += "%s: %s\n" % (repo, revision) - -web_root = utils.getconfig('WEBPUBLISH_DIR', ourconfig) -web_url = utils.getconfig('WEBPUBLISH_URL', ourconfig) - -email = "" -mailto = utils.getconfig("QAMAIL_TO", ourconfig) -if mailto: - email += "To: " + mailto + "\n" -mailcc = utils.getconfig("QAMAIL_CC", ourconfig) -if mailcc: - email += "Cc: " + mailcc + "\n" -mailbcc = utils.getconfig("QAMAIL_BCC", ourconfig) -if mailbcc: - email += "Bcc: " + mailbcc + "\n" - -email += "Subject: " + "QA notification for completed autobuilder build (%s)\n" % args.release -email += '''\n -A build flagged for QA (%s) was completed on the autobuilder and is available at:\n\n - %s\n\n -Build hash information: \n -%s - -\nThis is an automated message from the Yocto Project Autobuilder\nGit: git://git.yoctoproject.org/yocto-autobuilder2\nEmail: [email protected]\n - -''' % (args.release, args.publish_dir.replace(web_root, web_url), buildhashes) - -# Store a copy of the email in case it doesn't reach the lists -with open(os.path.join(args.publish_dir, "qa-email"), "wb") as qa_email: - qa_email.write(email.encode('utf-8')) - -utils.printheader("Sending QA email") -env = os.environ.copy() -# Many distros have sendmail in */sbin -env["PATH"] = env["PATH"] + ":/usr/sbin:/sbin" -subprocess.check_call('echo "' + email +' " | sendmail -t', shell=True, env=env) - + print("No comparision branch found, falling back to master") + subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"]) + + # If the base comparision branch isn't present regression comparision won't work + # at least until we can tell the tool to ignore internal branch information + if basebranch: + try: + subprocess.check_call(["git", "rev-parse", "--verify", basebranch], cwd=tempdir) + except subprocess.CalledProcessError: + # Doesn't exist so base it off master + # some older hosts don't have git branch old new + subprocess.check_call(["git", "checkout", "master"], cwd=tempdir) + subprocess.check_call(["git", "branch", basebranch], cwd=tempdir) + subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir) + extraopts = None + + subprocess.check_call([resulttool, "store", args.results_dir, tempdir]) + if comparebranch: + subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir) + subprocess.check_call(["git", "push", "--tags", "--force"], cwd=tempdir) + elif basebranch: + subprocess.check_call(["git", "push", "--all"], cwd=tempdir) + subprocess.check_call(["git", "push", "--tags"], cwd=tempdir) + + if extraopts: + regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split()) + with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f: + f.write(regreport) + + finally: + subprocess.check_call(["rm", "-rf", tempdir]) + pass + + if args.send.lower() != 'true' or not args.publish_dir or not args.release: + utils.printheader("Not sending QA email") + sys.exit(0) + + buildhashes = "" + for repo in sorted(repos.keys()): + # gplv2 is no longer built/tested in master + if repo == "meta-gplv2": + continue + # Need the finalised revisions (not 'HEAD') + targetrepodir = "%s/%s" % (repodir, repo) + revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() + buildhashes += "%s: %s\n" % (repo, revision) + + web_root = utils.getconfig('WEBPUBLISH_DIR', ourconfig) + web_url = utils.getconfig('WEBPUBLISH_URL', ourconfig) + + email = "" + mailto = utils.getconfig("QAMAIL_TO", ourconfig) + if mailto: + email += "To: " + mailto + "\n" + mailcc = utils.getconfig("QAMAIL_CC", ourconfig) + if mailcc: + email += "Cc: " + mailcc + "\n" + mailbcc = utils.getconfig("QAMAIL_BCC", ourconfig) + if mailbcc: + email += "Bcc: " + mailbcc + "\n" + + email += "Subject: " + "QA notification for completed autobuilder build (%s)\n" % args.release + email += '''\n + A build flagged for QA (%s) was completed on the autobuilder and is available at:\n\n + %s\n\n + Build hash information: \n + %s + + \nThis is an automated message from the Yocto Project Autobuilder\nGit: git://git.yoctoproject.org/yocto-autobuilder2\nEmail: [email protected]\n + + ''' % (args.release, args.publish_dir.replace(web_root, web_url), buildhashes) + + # Store a copy of the email in case it doesn't reach the lists + with open(os.path.join(args.publish_dir, "qa-email"), "wb") as qa_email: + qa_email.write(email.encode('utf-8')) + + utils.printheader("Sending QA email") + env = os.environ.copy() + # Many distros have sendmail in */sbin + env["PATH"] = env["PATH"] + ":/usr/sbin:/sbin" + subprocess.check_call('echo "' + email +' " | sendmail -t', shell=True, env=env) + +if __name__ == "__main__": + send_qa_email()
Signature · 1 line
2.39.0
PATCH v3 3/5 scripts/send-qa-email: Generate regression reports against most relevant release
In reply to Alexis LothoréInstead of only generating regressions reports against HEAD of relevant branch, compute most relevant tag (ie : release) against which we can check for regressions. General rules introduced are the following :
- milestone release is checked against previous milestone if possible, otherwise against major release
- point release is checked against previous point release if possible, otherwise against major release
- major release is checked against previous major release
- a non release build is checked against base branch
Examples :
- 4.1.2.rc1 is checked against yocto-4.1.1
- 4.1.2 is checked against yocto-4.1.1
- 4.1.1.rc1 is checked against yocto-4.1
- 4.1.1 is checked against yocto-4.1
- 4.1 is checked against yocto-4.0
- 4.1.rc4 is checked against yocto-4.0
- 4.1_M2.rc1 is checked against 4.1_M1
- 4.1_M2 is checked against 4.1_M1
- 4.1_M1.rc1 is checked against yocto-4.0
- 4.1_M1 is checked against yocto-4.0
--- scripts/send_qa_email.py | 86 +++++++++++++++++++++++++++++++++------- scripts/utils.py | 47 ++++++++++++++++++++++ 2 files changed, 118 insertions(+), 15 deletions(-)
scripts/send_qa_email.py +71 -15
@@ -9,11 +9,79 @@ import json import os import sys import subprocess -import errno import tempfile +import re import utils +def is_non_release_version(version): + p = re.compile('\d{8}-\d+') + return p.match(version) is not None + +def get_previous_tag(targetrepodir, version): + previousversion = None + previousmilestone = None + if version: + if is_non_release_version(version): + return subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip() + compareversion, comparemilestone, _ = utils.get_version_from_string(version) + compareversionminor = compareversion[-1] + # After ignoring rc part, if we get a minor to 0 on point release (e.g 4.0.0), + # reject last digit since such versions do not exist + if len(compareversion) == 3 and compareversionminor == 0: + compareversion = compareversion[:-1] + + # Process milestone if not first in current release + if comparemilestone and comparemilestone > 1: + previousversion = compareversion + previousmilestone = comparemilestone-1 + # Process first milestone or release if not first in major release + elif compareversionminor > 0: + previousversion = compareversion[:-1] + [compareversion[-1] - 1] + # Otherwise : format it as tag (which must exist) and search previous tag + else: + comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone) + return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip() + + return utils.get_tag_from_version(previousversion, previousmilestone) + + # All other cases : merely check against latest tag reachable + defaultbaseversion, _, _ = utils.get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip()) + return utils.get_tag_from_version(defaultbaseversion, None) + +def get_sha1(targetrepodir, revision): + return subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=targetrepodir).decode('utf-8').strip() + +def fetch_testresults(resultdir, revision): + rawtags = subprocess.check_output(["git", "ls-remote", "--refs", "--tags", "origin", f"*{revision}*"], cwd=resultdir).decode('utf-8').strip() + if not rawtags: + raise Exception(f"No reference found for commit {revision} in {resultdir}") + for ref in [rawtag.split()[1] for rawtag in rawtags.splitlines()]: + print(f"Fetching matching revisions: {ref}") + subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{ref}:{ref}"], cwd=resultdir) + + +def generate_regression_report(resulttool, targetrepodir, basebranch, resultdir, outputdir, yoctoversion): + baseversion = get_previous_tag(targetrepodir, yoctoversion) + baserevision = get_sha1(targetrepodir, baseversion) + comparerevision = get_sha1(targetrepodir, basebranch) + print(f"Compare version : {basebranch} ({comparerevision})") + print(f"Base tag : {baseversion} ({baserevision})") + + try: + """ + Results directory is likely a shallow clone : + we need to fetch results corresponding to base revision before + running resulttool + """ + fetch_testresults(resultdir, baserevision) + regreport = subprocess.check_output([resulttool, "regression-git", "-B", basebranch, "--commit", baserevision, "--commit2", comparerevision, resultdir]) + with open(outputdir + "/testresult-regressions-report.txt", "wb") as f: + f.write(regreport) + except subprocess.CalledProcessError as e: + error = str(e) + print(f"Error while generating report between {baserevision} and {comparerevision} : {error}") + def send_qa_email(): parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.')
@@ -57,16 +125,7 @@ def send_qa_email(): branch = repos['poky']['branch'] repo = repos['poky']['url'] - extraopts = None basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch) - if basebranch: - extraopts = " --branch %s --commit %s" % (branch, revision) - if comparebranch: - extraopts = extraopts + " --branch2 %s" % (comparebranch) - elif basebranch: - print("No comparision branch found, comparing to %s" % basebranch) - extraopts = extraopts + " --branch2 %s" % basebranch - report = subprocess.check_output([resulttool, "report", args.results_dir]) with open(args.results_dir + "/testresult-report.txt", "wb") as f: f.write(report)
@@ -95,7 +154,6 @@ def send_qa_email(): subprocess.check_call(["git", "checkout", "master"], cwd=tempdir) subprocess.check_call(["git", "branch", basebranch], cwd=tempdir) subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir) - extraopts = None subprocess.check_call([resulttool, "store", args.results_dir, tempdir]) if comparebranch:
@@ -105,10 +163,8 @@ def send_qa_email(): subprocess.check_call(["git", "push", "--all"], cwd=tempdir) subprocess.check_call(["git", "push", "--tags"], cwd=tempdir) - if extraopts: - regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split()) - with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f: - f.write(regreport) + if basebranch: + generate_regression_report(resulttool, targetrepodir, basebranch, tempdir, args.results_dir, args.release) finally: subprocess.check_call(["rm", "-rf", tempdir])
scripts/utils.py +47 -0
51 lines not on this page · raw message
Signature · 1 line
2.39.0
Re: [yocto] [autobuilder][PATCH v3 3/5] scripts/send-qa-email: Generate regression reports against most relevant release
Quoted 120 lines · Alexis Lothoré via lists.yoctoproject.org
Instead of only generating regressions reports against HEAD of relevant branch, compute most relevant tag (ie : release) against which we can check for regressions. General rules introduced are the following :
- milestone release is checked against previous milestone if possible, otherwise against major release
- point release is checked against previous point release if possible, otherwise against major release
- major release is checked against previous major release
- a non release build is checked against base branch
Examples :
- 4.1.2.rc1 is checked against yocto-4.1.1
- 4.1.2 is checked against yocto-4.1.1
- 4.1.1.rc1 is checked against yocto-4.1
- 4.1.1 is checked against yocto-4.1
- 4.1 is checked against yocto-4.0
- 4.1.rc4 is checked against yocto-4.0
- 4.1_M2.rc1 is checked against 4.1_M1
- 4.1_M2 is checked against 4.1_M1
- 4.1_M1.rc1 is checked against yocto-4.0
- 4.1_M1 is checked against yocto-4.0Signed-off-by Alexis Lothoré <[email protected]>--- scripts/send_qa_email.py | 86 +++++++++++++++++++++++++++++++++------- scripts/utils.py | 47 ++++++++++++++++++++++ 2 files changed, 118 insertions(+), 15 deletions(-)scripts/send_qa_email.py +69
-3@@ -9,11 +9,79 @@ import json import os import sys import subprocess -import errno import tempfile +import re import utils +def is_non_release_version(version): + p = re.compile('\d{8}-\d+') + return p.match(version) is not None + +def get_previous_tag(targetrepodir, version): + previousversion = None + previousmilestone = None + if version: + if is_non_release_version(version): + return subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip() + compareversion, comparemilestone, _ = utils.get_version_from_string(version) + compareversionminor = compareversion[-1] + # After ignoring rc part, if we get a minor to 0 on point release (e.g 4.0.0), + # reject last digit since such versions do not exist + if len(compareversion) == 3 and compareversionminor == 0: + compareversion = compareversion[:-1] + + # Process milestone if not first in current release + if comparemilestone and comparemilestone > 1: + previousversion = compareversion + previousmilestone = comparemilestone-1 + # Process first milestone or release if not first in major release + elif compareversionminor > 0: + previousversion = compareversion[:-1] + [compareversion[-1] - 1] + # Otherwise : format it as tag (which must exist) and search previous tag + else: + comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone) + return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip() + + return utils.get_tag_from_version(previousversion, previousmilestone) + + # All other cases : merely check against latest tag reachable + defaultbaseversion, _, _ = utils.get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip()) + return utils.get_tag_from_version(defaultbaseversion, None) + +def get_sha1(targetrepodir, revision): + return subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=targetrepodir).decode('utf-8').strip() + +def fetch_testresults(resultdir, revision): + rawtags = subprocess.check_output(["git", "ls-remote", "--refs", "--tags", "origin", f"*{revision}*"], cwd=resultdir).decode('utf-8').strip() + if not rawtags: + raise Exception(f"No reference found for commit {revision} in {resultdir}") + for ref in [rawtag.split()[1] for rawtag in rawtags.splitlines()]: + print(f"Fetching matching revisions: {ref}") + subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{ref}:{ref}"], cwd=resultdir) + + +def generate_regression_report(resulttool, targetrepodir, basebranch, resultdir, outputdir, yoctoversion): + baseversion = get_previous_tag(targetrepodir, yoctoversion) + baserevision = get_sha1(targetrepodir, baseversion) + comparerevision = get_sha1(targetrepodir, basebranch) + print(f"Compare version : {basebranch} ({comparerevision})") + print(f"Base tag : {baseversion} ({baserevision})") + + try: + """ + Results directory is likely a shallow clone : + we need to fetch results corresponding to base revision before + running resulttool + """ + fetch_testresults(resultdir, baserevision) + regreport = subprocess.check_output([resulttool, "regression-git", "-B", basebranch, "--commit", baserevision, "--commit2", comparerevision, resultdir]) + with open(outputdir + "/testresult-regressions-report.txt", "wb") as f: + f.write(regreport) + except subprocess.CalledProcessError as e: + error = str(e) + print(f"Error while generating report between {baserevision} and {comparerevision} : {error}") + def send_qa_email(): parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.')@@ -57,16 +125,7 @@ def send_qa_email(): branch = repos['poky']['branch'] repo = repos['poky']['url'] - extraopts = None basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch) - if basebranch:
scripts/send_qa_email.py def send_qa_email(): · line 131
Quoted 10 lines
scripts/send_qa_email.py +0
-7- extraopts = " --branch %s --commit %s" % (branch, revision) - if comparebranch: - extraopts = extraopts + " --branch2 %s" % (comparebranch) - elif basebranch: - print("No comparision branch found, comparing to %s" % basebranch) - extraopts = extraopts + " --branch2 %s" % basebranch - report = subprocess.check_output([resulttool, "report", args.results_dir]) with open(args.results_dir + "/testresult-report.txt", "wb") as f: f.write(report)
There looks to be a potential issue here since you drop the use of comparebranch entirely. This comes from config.json:
"BUILD_HISTORY_FORKPUSH" : {"poky-contrib:ross/mut" : "poky:master", "poky:master-next" : "poky:master"},
and basically what it says is that master-next is a force push branch which should be compared against master (and ross/mut should be compared against master too).
We need this code so that when we build test branches, we run the regression comparison against the last known master test results. We should probably put Alexandre's test branch in there too.
Cheers,
Richard
Re: [yocto] [autobuilder][PATCH v3 3/5] scripts/send-qa-email: Generate regression reports against most relevant release
Hi Richard,
thanks for the feedback
Quoted 11 lines · Richard Purdie
Quoted 4 lines · Alexis Lothoré via lists.yoctoproject.org
report =
subprocess.check_output([resulttool, "report", args.results_dir])withopen(args.results_dir + "/testresult-report.txt", "wb")as f:
f.write(report)There looks to be a potential issue here since you drop the use of comparebranch entirely. This comes from config.json:
"
BUILD_HISTORY_FORKPUSH" : {"poky-contrib:ross/mut" : "poky:master", "poky:master-next" : "poky:master"},
Quoted 7 lines
and basically what it says is that master-next is a force push branch which should be compared against master (and ross/mut should be compared against master too).
We need this code so that when we build test branches, we run the regression comparison against the last known master test results. We should probably put Alexandre's test branch in there too.
I may have missed that. I need to dig a bit to understand exactly what is expected here in order to fix it. Noted, I will take a look at it.
Regards
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v3 4/5 scripts/send_qa_email.py: add unit tests on previous version computation
In reply to Alexis LothoréThe "previous version" computation bring many edge cases depending on the version under release. Add a basic test suite to validate currently implemented computation strategy and to prevent mistakes when ediiting it in the future
--- scripts/test_send_qa_email.py | 56 +++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100755 scripts/test_send_qa_email.py
scripts/test_send_qa_email.py new file +56 -0
@@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +""" +Test suite for send_qa_email.py + +The suite needs a valid poky clone to run since it will +fetch and return revisions from remote repository. To run the suite, +set POKY_PATH environment variable accordingly: +`POKY_PATH=~/src/poky ./scripts/test_send_qa_email.py` +""" +import os +import sys +import unittest +import send_qa_email + + +class TestVersion(unittest.TestCase): + test_data_get_version = [ + {"input": {"version": "4.1.2"}, "expected": "yocto-4.1.1"}, + {"input": {"version": "4.1"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1.1"}, "expected": "yocto-4.1"}, + {"input": {"version": "4.1_M2"}, "expected": "4.1_M1"}, + {"input": {"version": "4.1_M1"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1.1.rc1"}, "expected": "yocto-4.1"}, + {"input": {"version": "4.1.2.rc1"}, "expected": "yocto-4.1.1"}, + {"input": {"version": "4.1_M3.rc1"}, "expected": "4.1_M2"}, + {"input": {"version": "4.1_M3.rc4"}, "expected": "4.1_M2"}, + {"input": {"version": "4.1_M1.rc1"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1_M1.rc4"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1.rc4"}, "expected": "yocto-4.0"} + ] + + test_data_get_sha1 = [ + {"input": "yocto-4.0", "expected": "00cfdde791a0176c134f31e5a09eff725e75b905"}, + {"input": "4.1_M1", "expected": "95066dde6861ee08fdb505ab3e0422156cc24fae"}, + ] + + def test_versions(self): + for data in self.test_data_get_version: + test_name = data["input"]["version"] + with self.subTest(f"Test {test_name} previous tag"): + self.assertEqual(send_qa_email.get_previous_tag(os.environ.get( + "POKY_PATH"), data["input"]["version"]), data["expected"]) + + def test_get_sha1(self): + for data in self.test_data_get_sha1: + test_name = data["input"] + with self.subTest(f"Test SHA1 from {test_name}"): + self.assertEqual(send_qa_email.get_sha1(os.environ.get( + "POKY_PATH"), data["input"]), data["expected"]) + + +if __name__ == '__main__': + if os.environ.get("POKY_PATH") is None: + print("Please set POKY_PATH to proper poky clone location before running tests") + sys.exit(1) + unittest.main()
Signature · 1 line
2.39.0
PATCH v3 5/5 scripts/send-qa-email: add symlink to preserve compatibility with autobuilder
In reply to Alexis LothoréMerely renaming send-qa-email to send_qa_email.py would imply to update yocto-autobuilder2 scripts, which can be daunting. In order to preserve compatibility with current configuration, add a symlink pointing to the new script
--- scripts/send-qa-email | 1 + 1 file changed, 1 insertion(+) create mode 120000 scripts/send-qa-email
scripts/send-qa-email new file +1 -0
@@ -0,0 +1 @@ +send_qa_email.py
Signature · 1 line
2.39.0
Quoted 26 lines · Alexis Lothoré via lists.yoctoproject.org
Hi, this is the 3nd version of work initiated to improve regression reports usefulness, started around issue YOCTO #1465
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065).Changes since v2:
- add symlink to preserve compatibility with current autobuilder configurationChanges since v1:
- minor rework to be able to importsend_qa_email.pyas a standard python module
- properly manage non-releases versions sinceqe_send_email.pycan be called with such "release" versions ("-r" parameter)
- add unit tests on previous version computation
- do not fetch yocto-testresults full history: identify needed revisions with git ls-remote and retrieve them with git fetchAlexis Lothoré (5):
scripts/send_qa_email.py: Rename send-qa-email tosend_qa_email.pyscripts/send_qa_email.py: Wrapsend_qa_email.pycontent in functionscripts/send-qa-email: Generate regression reports against most
relevant release
scripts/send_qa_email.py: add unit tests on previous version
computation
scripts/send-qa-email: add symlink to preserve compatibility with
autobuilder
Quoted 9 lines
scripts/send-qa-email | 165 +------------------------ scripts/send_qa_email.py | 223 ++++++++++++++++++++++++++++++++++ scripts/shared-repo-unpack | 2 +- scripts/test_send_qa_email.py | 57 +++++++++ scripts/utils.py | 47 +++++++ 5 files changed, 329 insertions(+), 165 deletions(-) mode change 100755 => 120000 scripts/send-qa-email create mode 100755 scripts/send_qa_email.py create mode 100755 scripts/test_send_qa_email.py
Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.
I did notice one issue which we'll need to fix in some follow up patches, I'll reply to the patch about that.
Cheers,
Richard
Quoted 6 lines · Richard Purdie
Quoted 1 line · Alexis Lothoré via lists.yoctoproject.org
Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.
Thanks. Please feel free to reach back if you observe any unwanted side effect, since most of my knowledge is based on scripts reading and autobuilder builds history. I have tried to test it as much as possible on my machine, but as I was discussing with Alexandre Belloni, the many stubs and fake data used do not make the tests as accurate as the real world case.
Quoted 2 lines
I did notice one issue which we'll need to fix in some follow up patches, I'll reply to the patch about that.
ACK, I am working on it
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 8 lines · Alexis Lothoré
Quoted 6 lines · Richard Purdie
Quoted 1 line · Alexis Lothoré via lists.yoctoproject.org
Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.
Quoted 5 lines
Thanks. Please feel free to reach back if you observe any unwanted side effect, since most of my knowledge is based on scripts reading and autobuilder builds history. I have tried to test it as much as possible on my machine, but as I was discussing with Alexandre Belloni, the many stubs and fake data used do not make the tests as accurate as the real world case.
The result of the M2 build is here, specifically the regression report:
(warning, it is a 13MB file, which may hint at some problems)
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4844/steps/29/logs/stdio
was the log from running the script. It also suggests there may be a few issues.
I think we need some improvements to the report. Firstly it should state at the top which revisions are being compared and any tags they correspond to.
Next, we need to understand why the large chunk of tests "disappeared". it may be related to the "Failed to retrieved" messages?
I also happen to know the reproducibility tests failed in this build, yet the regression report has no mention of that.
So the patches are a start but there is much more to do.
Cheers,
Richard
Quoted 21 lines · Richard Purdie
Quoted 8 lines · Alexis Lothoré
Quoted 6 lines · Richard Purdie
Quoted 1 line · Alexis Lothoré via lists.yoctoproject.org
Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.
Quoted 5 lines
Thanks. Please feel free to reach back if you observe any unwanted side effect, since most of my knowledge is based on scripts reading and autobuilder builds history. I have tried to test it as much as possible on my machine, but as I was discussing with Alexandre Belloni, the many stubs and fake data used do not make the tests as accurate as the real world case.
The result of the M2 build is here, specifically the regression report:
(warning, it is a 13MB file, which may hint at some problems)
Quoted 8 lines
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4844/steps/29/logs/stdio
was the log from running the script. It also suggests there may be a few issues.
I think we need some improvements to the report. Firstly it should state at the top which revisions are being compared and any tags they correspond to.
Yes, I was thinking about this while tweaking send-qa-email. Another point I need to understand is how the regression report is managed if resulttool finds multiple tags in results dir concerning the same yocto revision. I feel that the result is not the same if it finds one tag or multiple tags for the same revision. Does it really make sense to check regression against multiple tags referencing the same yocto revision ? If so, indeed we should make the tag name appear. If not, how to distinguish which tag is appropriate ? This question is more linked to resulttool internals than to send-qa-email.
Quoted 6 lines
Next, we need to understand why the large chunk of tests "disappeared". it may be related to the "Failed to retrieved" messages?
I also happen to know the reproducibility tests failed in this build, yet the regression report has no mention of that.
ACK, I will take a look.
Quoted 2 lines
So the patches are a start but there is much more to do.
Regards,
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com