Skip to contentSkip to conversation

generate regression reports against proper releases

PATCH In review Alexis Lothoré 2 participants 12 messages Jan 24, 2023 – Jan 27, 2023

lore
Share & tools
lore
#0 lore raw

v3 cover generate regression reports against proper releases

Hi, this is the 3nd version of work initiated to improve regression reports usefulness, started around issue YOCTO #1465
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065).

Changes since v2:
- add symlink to preserve compatibility with current autobuilder configuration

Changes since v1:
- minor rework to be able to import send_qa_email.py as a standard python module
- properly manage non-releases versions since qe_send_email.py can be called with such "release" versions ("-r" parameter)
- add unit tests on previous version computation
- do not fetch yocto-testresults full history: identify needed revisions with git ls-remote and retrieve them with git fetch

Alexis Lothoré (5):
scripts/send_qa_email.py: Rename send-qa-email to send_qa_email.py scripts/send_qa_email.py: Wrap send_qa_email.py content in function scripts/send-qa-email: Generate regression reports against most
relevant release
scripts/send_qa_email.py: add unit tests on previous version
computation
scripts/send-qa-email: add symlink to preserve compatibility with
autobuilder

 scripts/send-qa-email         | 165 +------------------------
 scripts/send_qa_email.py      | 223 ++++++++++++++++++++++++++++++++++
 scripts/shared-repo-unpack    |   2 +-
 scripts/test_send_qa_email.py |  57 +++++++++
 scripts/utils.py              |  47 +++++++
 5 files changed, 329 insertions(+), 165 deletions(-)
 mode change 100755 => 120000 scripts/send-qa-email
 create mode 100755 scripts/send_qa_email.py
 create mode 100755 scripts/test_send_qa_email.py
Signature · 1 line
2.39.0
#1 lore raw

PATCH v3 1/5 scripts/send_qa_email.py: Rename send-qa-email to send_qa_email.py

This rename fixes send-qa-email regarding python guidelines, which allows to import the script content in another script (for unit testing purpose for example)

---
 scripts/{send-qa-email => send_qa_email.py} | 0
 scripts/shared-repo-unpack                  | 2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename scripts/{send-qa-email => send_qa_email.py} (100%)
scripts/send_qa_email.py renamed from scripts/send-qa-email +0 -0
similarity index 100%
rename from scripts/send-qa-email
rename to scripts/send_qa_email.py
scripts/shared-repo-unpack +1 -1
index b826c87..8e7ec95 100755
@@ -61,7 +61,7 @@ for repo in sorted(repos.keys()):
         utils.printheader("Copying in repo %s" % repo)
         utils.mkdir(targetsubdir)
         if args.target in ["a-full", "a-quick"]:
-            # full/quick need all repo data due to send-qa-email
+            # full/quick need all repo data due to send_qa_email.py
             subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir])
         else:
             subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir, "./" + repo])
Signature · 1 line
2.39.0
#2 lore raw

PATCH v3 2/5 scripts/send_qa_email.py: Wrap send_qa_email.py content in function

In reply to Alexis Lothoré

This wrapping allows importing send_qa_email.py in another script without executing the QA emailing routine. This is useful for unit testing the script internal functions

---
 scripts/send_qa_email.py | 287 ++++++++++++++++++++-------------------
 1 file changed, 145 insertions(+), 142 deletions(-)
scripts/send_qa_email.py +145 -142
index d2f63dc..4023918 100755
@@ -15,150 +15,153 @@ import tempfile
 import utils
 
 
-parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.')
-
-parser.add_argument('send',
-                    help="True to send email, otherwise the script will display a message and exit")
-parser.add_argument('repojson',
-                    help="The json file containing the repositories to use")
-parser.add_argument('sharedrepodir',
-                    help="The shared repos directory (to resolve the repo revision hashes)")
-parser.add_argument('-p', '--publish-dir',
-                    action='store',
-                    help="Where the artefacts were published")
-parser.add_argument('-R', '--results-dir',
-                    action='store',
-                    help="Where the test results were published")
-parser.add_argument('-r', '--release',
-                    action='store',
-                    help="The build/release 'name' for release purposes (optional)")
-
-args = parser.parse_args()
-
-scriptsdir = os.path.dirname(os.path.realpath(__file__))
-ourconfig = utils.loadconfig()
-
-with open(args.repojson) as f:
-    repos = json.load(f)
-
-resulttool = os.path.dirname(args.repojson) + "/build/scripts/resulttool"
-
-buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools"
-if os.path.exists(buildtoolsdir):
-    utils.enable_buildtools_tarball(buildtoolsdir)
-
-repodir = os.path.dirname(args.repojson) + "/build/repos"
-
-if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
-    # Need the finalised revisions (not 'HEAD')
-    targetrepodir = "%s/poky" % (repodir)
-    revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
-    branch = repos['poky']['branch']
-    repo = repos['poky']['url']
-
-    extraopts = None
-    basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch)
-    if basebranch:
-        extraopts = " --branch %s --commit %s" % (branch, revision)
-    if comparebranch:
-        extraopts = extraopts + " --branch2 %s" % (comparebranch)
-    elif basebranch:
-        print("No comparision branch found, comparing to %s" % basebranch)
-        extraopts = extraopts + " --branch2 %s" % basebranch
-
-    report = subprocess.check_output([resulttool, "report", args.results_dir])
-    with open(args.results_dir + "/testresult-report.txt", "wb") as f:
-        f.write(report)
-
-    tempdir = tempfile.mkdtemp(prefix='sendqaemail.')
-    try:
-        cloneopts = []
+def send_qa_email():
+    parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.')
+
+    parser.add_argument('send',
+                        help="True to send email, otherwise the script will display a message and exit")
+    parser.add_argument('repojson',
+                        help="The json file containing the repositories to use")
+    parser.add_argument('sharedrepodir',
+                        help="The shared repos directory (to resolve the repo revision hashes)")
+    parser.add_argument('-p', '--publish-dir',
+                        action='store',
+                        help="Where the artefacts were published")
+    parser.add_argument('-R', '--results-dir',
+                        action='store',
+                        help="Where the test results were published")
+    parser.add_argument('-r', '--release',
+                        action='store',
+                        help="The build/release 'name' for release purposes (optional)")
+
+    args = parser.parse_args()
+
+    scriptsdir = os.path.dirname(os.path.realpath(__file__))
+    ourconfig = utils.loadconfig()
+
+    with open(args.repojson) as f:
+        repos = json.load(f)
+
+    resulttool = os.path.dirname(args.repojson) + "/build/scripts/resulttool"
+
+    buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools"
+    if os.path.exists(buildtoolsdir):
+        utils.enable_buildtools_tarball(buildtoolsdir)
+
+    repodir = os.path.dirname(args.repojson) + "/build/repos"
+
+    if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
+        # Need the finalised revisions (not 'HEAD')
+        targetrepodir = "%s/poky" % (repodir)
+        revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
+        branch = repos['poky']['branch']
+        repo = repos['poky']['url']
+
+        extraopts = None
+        basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch)
+        if basebranch:
+            extraopts = " --branch %s --commit %s" % (branch, revision)
         if comparebranch:
-            cloneopts = ["--branch", comparebranch]
+            extraopts = extraopts + " --branch2 %s" % (comparebranch)
         elif basebranch:
-            cloneopts = ["--branch", basebranch]
-        try:
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
-        except subprocess.CalledProcessError:
-            print("No comparision branch found, falling back to master")
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"])
+            print("No comparision branch found, comparing to %s" % basebranch)
+            extraopts = extraopts + " --branch2 %s" % basebranch
 
-        # If the base comparision branch isn't present regression comparision won't work
-        # at least until we can tell the tool to ignore internal branch information
-        if basebranch:
+        report = subprocess.check_output([resulttool, "report", args.results_dir])
+        with open(args.results_dir + "/testresult-report.txt", "wb") as f:
+            f.write(report)
+
+        tempdir = tempfile.mkdtemp(prefix='sendqaemail.')
+        try:
+            cloneopts = []
+            if comparebranch:
+                cloneopts = ["--branch", comparebranch]
+            elif basebranch:
+                cloneopts = ["--branch", basebranch]
             try:
-                subprocess.check_call(["git", "rev-parse", "--verify", basebranch], cwd=tempdir)
+                subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
             except subprocess.CalledProcessError:
-                # Doesn't exist so base it off master
-                # some older hosts don't have git branch old new
-                subprocess.check_call(["git", "checkout", "master"], cwd=tempdir)
-                subprocess.check_call(["git", "branch", basebranch], cwd=tempdir)
-                subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir)
-                extraopts = None
-
-        subprocess.check_call([resulttool, "store", args.results_dir, tempdir])
-        if comparebranch:
-            subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir)
-            subprocess.check_call(["git", "push", "--tags", "--force"], cwd=tempdir)
-        elif basebranch:
-            subprocess.check_call(["git", "push", "--all"], cwd=tempdir)
-            subprocess.check_call(["git", "push", "--tags"], cwd=tempdir)
-
-        if extraopts:
-            regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split())
-            with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f:
-                f.write(regreport)
-
-    finally:
-        subprocess.check_call(["rm", "-rf",  tempdir])
-        pass
-
-if args.send.lower() != 'true' or not args.publish_dir or not args.release:
-    utils.printheader("Not sending QA email")
-    sys.exit(0)
-
-buildhashes = ""
-for repo in sorted(repos.keys()):
-    # gplv2 is no longer built/tested in master
-    if repo == "meta-gplv2":
-        continue
-    # Need the finalised revisions (not 'HEAD')
-    targetrepodir = "%s/%s" % (repodir, repo)
-    revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
-    buildhashes += "%s: %s\n" % (repo, revision)
-
-web_root = utils.getconfig('WEBPUBLISH_DIR', ourconfig)
-web_url = utils.getconfig('WEBPUBLISH_URL', ourconfig)
-
-email = ""
-mailto = utils.getconfig("QAMAIL_TO", ourconfig)
-if mailto:
-    email += "To: " + mailto + "\n"
-mailcc = utils.getconfig("QAMAIL_CC", ourconfig)
-if mailcc:
-    email += "Cc: " + mailcc + "\n"
-mailbcc = utils.getconfig("QAMAIL_BCC", ourconfig)
-if mailbcc:
-    email += "Bcc: " + mailbcc + "\n"
-
-email += "Subject: " + "QA notification for completed autobuilder build (%s)\n" % args.release
-email += '''\n
-A build flagged for QA (%s) was completed on the autobuilder and is available at:\n\n
-    %s\n\n
-Build hash information: \n
-%s
-
-\nThis is an automated message from the Yocto Project Autobuilder\nGit: git://git.yoctoproject.org/yocto-autobuilder2\nEmail: [email protected]\n
-
-''' % (args.release, args.publish_dir.replace(web_root, web_url), buildhashes)
-
-# Store a copy of the email in case it doesn't reach the lists
-with open(os.path.join(args.publish_dir, "qa-email"), "wb") as qa_email:
-    qa_email.write(email.encode('utf-8'))
-
-utils.printheader("Sending QA email")
-env = os.environ.copy()
-# Many distros have sendmail in */sbin
-env["PATH"] = env["PATH"] + ":/usr/sbin:/sbin"
-subprocess.check_call('echo "' + email +' " | sendmail -t', shell=True, env=env)
-
+                print("No comparision branch found, falling back to master")
+                subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"])
+
+            # If the base comparision branch isn't present regression comparision won't work
+            # at least until we can tell the tool to ignore internal branch information
+            if basebranch:
+                try:
+                    subprocess.check_call(["git", "rev-parse", "--verify", basebranch], cwd=tempdir)
+                except subprocess.CalledProcessError:
+                    # Doesn't exist so base it off master
+                    # some older hosts don't have git branch old new
+                    subprocess.check_call(["git", "checkout", "master"], cwd=tempdir)
+                    subprocess.check_call(["git", "branch", basebranch], cwd=tempdir)
+                    subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir)
+                    extraopts = None
+
+            subprocess.check_call([resulttool, "store", args.results_dir, tempdir])
+            if comparebranch:
+                subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir)
+                subprocess.check_call(["git", "push", "--tags", "--force"], cwd=tempdir)
+            elif basebranch:
+                subprocess.check_call(["git", "push", "--all"], cwd=tempdir)
+                subprocess.check_call(["git", "push", "--tags"], cwd=tempdir)
+
+            if extraopts:
+                regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split())
+                with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f:
+                    f.write(regreport)
+
+        finally:
+            subprocess.check_call(["rm", "-rf",  tempdir])
+            pass
+
+    if args.send.lower() != 'true' or not args.publish_dir or not args.release:
+        utils.printheader("Not sending QA email")
+        sys.exit(0)
+
+    buildhashes = ""
+    for repo in sorted(repos.keys()):
+        # gplv2 is no longer built/tested in master
+        if repo == "meta-gplv2":
+            continue
+        # Need the finalised revisions (not 'HEAD')
+        targetrepodir = "%s/%s" % (repodir, repo)
+        revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
+        buildhashes += "%s: %s\n" % (repo, revision)
+
+    web_root = utils.getconfig('WEBPUBLISH_DIR', ourconfig)
+    web_url = utils.getconfig('WEBPUBLISH_URL', ourconfig)
+
+    email = ""
+    mailto = utils.getconfig("QAMAIL_TO", ourconfig)
+    if mailto:
+        email += "To: " + mailto + "\n"
+    mailcc = utils.getconfig("QAMAIL_CC", ourconfig)
+    if mailcc:
+        email += "Cc: " + mailcc + "\n"
+    mailbcc = utils.getconfig("QAMAIL_BCC", ourconfig)
+    if mailbcc:
+        email += "Bcc: " + mailbcc + "\n"
+
+    email += "Subject: " + "QA notification for completed autobuilder build (%s)\n" % args.release
+    email += '''\n
+    A build flagged for QA (%s) was completed on the autobuilder and is available at:\n\n
+        %s\n\n
+    Build hash information: \n
+    %s
+
+    \nThis is an automated message from the Yocto Project Autobuilder\nGit: git://git.yoctoproject.org/yocto-autobuilder2\nEmail: [email protected]\n
+
+    ''' % (args.release, args.publish_dir.replace(web_root, web_url), buildhashes)
+
+    # Store a copy of the email in case it doesn't reach the lists
+    with open(os.path.join(args.publish_dir, "qa-email"), "wb") as qa_email:
+        qa_email.write(email.encode('utf-8'))
+
+    utils.printheader("Sending QA email")
+    env = os.environ.copy()
+    # Many distros have sendmail in */sbin
+    env["PATH"] = env["PATH"] + ":/usr/sbin:/sbin"
+    subprocess.check_call('echo "' + email +' " | sendmail -t', shell=True, env=env)
+
+if __name__ == "__main__":
+    send_qa_email()
Signature · 1 line
2.39.0
#3 lore raw

PATCH v3 3/5 scripts/send-qa-email: Generate regression reports against most relevant release

In reply to Alexis Lothoré

Instead of only generating regressions reports against HEAD of relevant branch, compute most relevant tag (ie : release) against which we can check for regressions. General rules introduced are the following :
- milestone release is checked against previous milestone if possible, otherwise against major release
- point release is checked against previous point release if possible, otherwise against major release
- major release is checked against previous major release
- a non release build is checked against base branch
Examples :
- 4.1.2.rc1 is checked against yocto-4.1.1
- 4.1.2 is checked against yocto-4.1.1
- 4.1.1.rc1 is checked against yocto-4.1
- 4.1.1 is checked against yocto-4.1
- 4.1 is checked against yocto-4.0
- 4.1.rc4 is checked against yocto-4.0
- 4.1_M2.rc1 is checked against 4.1_M1
- 4.1_M2 is checked against 4.1_M1
- 4.1_M1.rc1 is checked against yocto-4.0
- 4.1_M1 is checked against yocto-4.0

---
 scripts/send_qa_email.py | 86 +++++++++++++++++++++++++++++++++-------
 scripts/utils.py         | 47 ++++++++++++++++++++++
 2 files changed, 118 insertions(+), 15 deletions(-)
scripts/send_qa_email.py +71 -15
index 4023918..199fe4e 100755
@@ -9,11 +9,79 @@ import json
 import os
 import sys
 import subprocess
-import errno
 import tempfile
+import re
 
 import utils
 
+def is_non_release_version(version):
+    p = re.compile('\d{8}-\d+')
+    return p.match(version) is not None
+
+def get_previous_tag(targetrepodir, version):
+    previousversion = None
+    previousmilestone = None
+    if version:
+        if is_non_release_version(version):
+            return subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip()
+        compareversion, comparemilestone, _ = utils.get_version_from_string(version)
+        compareversionminor = compareversion[-1]
+        # After ignoring rc part, if we get a minor to 0 on point release (e.g 4.0.0),
+        # reject last digit since such versions do not exist
+        if len(compareversion) == 3 and compareversionminor == 0:
+            compareversion = compareversion[:-1]
+
+        # Process milestone if not first in current release
+        if comparemilestone and comparemilestone > 1:
+            previousversion = compareversion
+            previousmilestone = comparemilestone-1
+        # Process first milestone or release if not first in major release
+        elif compareversionminor > 0:
+            previousversion = compareversion[:-1] + [compareversion[-1] - 1]
+        # Otherwise : format it as tag (which must exist) and search previous tag
+        else:
+            comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone)
+            return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip()
+
+        return utils.get_tag_from_version(previousversion, previousmilestone)
+
+    # All other cases : merely check against latest tag reachable
+    defaultbaseversion, _, _ = utils.get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip())
+    return utils.get_tag_from_version(defaultbaseversion, None)
+
+def get_sha1(targetrepodir, revision):
+    return subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=targetrepodir).decode('utf-8').strip()
+
+def fetch_testresults(resultdir, revision):
+    rawtags = subprocess.check_output(["git", "ls-remote", "--refs", "--tags", "origin", f"*{revision}*"], cwd=resultdir).decode('utf-8').strip()
+    if not rawtags:
+        raise Exception(f"No reference found for commit {revision} in {resultdir}")
+    for ref in [rawtag.split()[1] for rawtag in rawtags.splitlines()]:
+        print(f"Fetching matching revisions: {ref}")
+        subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{ref}:{ref}"], cwd=resultdir)
+
+
+def generate_regression_report(resulttool, targetrepodir, basebranch, resultdir, outputdir, yoctoversion):
+    baseversion = get_previous_tag(targetrepodir, yoctoversion)
+    baserevision = get_sha1(targetrepodir, baseversion)
+    comparerevision = get_sha1(targetrepodir, basebranch)
+    print(f"Compare version : {basebranch} ({comparerevision})")
+    print(f"Base tag : {baseversion} ({baserevision})")
+
+    try:
+        """
+        Results directory is likely a shallow clone :
+        we need to fetch results corresponding to base revision before
+        running resulttool
+        """
+        fetch_testresults(resultdir, baserevision)
+        regreport = subprocess.check_output([resulttool, "regression-git", "-B", basebranch, "--commit", baserevision, "--commit2", comparerevision, resultdir])
+        with open(outputdir + "/testresult-regressions-report.txt", "wb") as f:
+           f.write(regreport)
+    except subprocess.CalledProcessError as e:
+        error = str(e)
+        print(f"Error while generating report between {baserevision} and {comparerevision} : {error}")
+
 
 def send_qa_email():
     parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.')
@@ -57,16 +125,7 @@ def send_qa_email():
         branch = repos['poky']['branch']
         repo = repos['poky']['url']
 
-        extraopts = None
         basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch)
-        if basebranch:
-            extraopts = " --branch %s --commit %s" % (branch, revision)
-        if comparebranch:
-            extraopts = extraopts + " --branch2 %s" % (comparebranch)
-        elif basebranch:
-            print("No comparision branch found, comparing to %s" % basebranch)
-            extraopts = extraopts + " --branch2 %s" % basebranch
-
         report = subprocess.check_output([resulttool, "report", args.results_dir])
         with open(args.results_dir + "/testresult-report.txt", "wb") as f:
             f.write(report)
@@ -95,7 +154,6 @@ def send_qa_email():
                     subprocess.check_call(["git", "checkout", "master"], cwd=tempdir)
                     subprocess.check_call(["git", "branch", basebranch], cwd=tempdir)
                     subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir)
-                    extraopts = None
 
             subprocess.check_call([resulttool, "store", args.results_dir, tempdir])
             if comparebranch:
@@ -105,10 +163,8 @@ def send_qa_email():
                 subprocess.check_call(["git", "push", "--all"], cwd=tempdir)
                 subprocess.check_call(["git", "push", "--tags"], cwd=tempdir)
 
-            if extraopts:
-                regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split())
-                with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f:
-                    f.write(regreport)
+            if basebranch:
+                generate_regression_report(resulttool, targetrepodir, basebranch, tempdir, args.results_dir, args.release)
 
         finally:
             subprocess.check_call(["rm", "-rf",  tempdir])
scripts/utils.py +47 -0

51 lines not on this page · raw message

Signature · 1 line
2.39.0
#4 lore raw

Re: [yocto] [autobuilder][PATCH v3 3/5] scripts/send-qa-email: Generate regression reports against most relevant release

Quoted 120 lines · Alexis Lothoré via lists.yoctoproject.org

Instead of only generating regressions reports against HEAD of relevant branch, compute most relevant tag (ie : release) against which we can check for regressions. General rules introduced are the following :
- milestone release is checked against previous milestone if possible, otherwise against major release
- point release is checked against previous point release if possible, otherwise against major release
- major release is checked against previous major release
- a non release build is checked against base branch
Examples :
- 4.1.2.rc1 is checked against yocto-4.1.1
- 4.1.2 is checked against yocto-4.1.1
- 4.1.1.rc1 is checked against yocto-4.1
- 4.1.1 is checked against yocto-4.1
- 4.1 is checked against yocto-4.0
- 4.1.rc4 is checked against yocto-4.0
- 4.1_M2.rc1 is checked against 4.1_M1
- 4.1_M2 is checked against 4.1_M1
- 4.1_M1.rc1 is checked against yocto-4.0
- 4.1_M1 is checked against yocto-4.0

---
 scripts/send_qa_email.py | 86 +++++++++++++++++++++++++++++++++-------
 scripts/utils.py         | 47 ++++++++++++++++++++++
 2 files changed, 118 insertions(+), 15 deletions(-)
scripts/send_qa_email.py +69 -3
index 4023918..199fe4e 100755
@@ -9,11 +9,79 @@ import json
 import os
 import sys
 import subprocess
-import errno
 import tempfile
+import re
 
 import utils
 
+def is_non_release_version(version):
+    p = re.compile('\d{8}-\d+')
+    return p.match(version) is not None
+
+def get_previous_tag(targetrepodir, version):
+    previousversion = None
+    previousmilestone = None
+    if version:
+        if is_non_release_version(version):
+            return subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip()
+        compareversion, comparemilestone, _ = utils.get_version_from_string(version)
+        compareversionminor = compareversion[-1]
+        # After ignoring rc part, if we get a minor to 0 on point release (e.g 4.0.0),
+        # reject last digit since such versions do not exist
+        if len(compareversion) == 3 and compareversionminor == 0:
+            compareversion = compareversion[:-1]
+
+        # Process milestone if not first in current release
+        if comparemilestone and comparemilestone > 1:
+            previousversion = compareversion
+            previousmilestone = comparemilestone-1
+        # Process first milestone or release if not first in major release
+        elif compareversionminor > 0:
+            previousversion = compareversion[:-1] + [compareversion[-1] - 1]
+        # Otherwise : format it as tag (which must exist) and search previous tag
+        else:
+            comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone)
+            return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip()
+
+        return utils.get_tag_from_version(previousversion, previousmilestone)
+
+    # All other cases : merely check against latest tag reachable
+    defaultbaseversion, _, _ = utils.get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip())
+    return utils.get_tag_from_version(defaultbaseversion, None)
+
+def get_sha1(targetrepodir, revision):
+    return subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=targetrepodir).decode('utf-8').strip()
+
+def fetch_testresults(resultdir, revision):
+    rawtags = subprocess.check_output(["git", "ls-remote", "--refs", "--tags", "origin", f"*{revision}*"], cwd=resultdir).decode('utf-8').strip()
+    if not rawtags:
+        raise Exception(f"No reference found for commit {revision} in {resultdir}")
+    for ref in [rawtag.split()[1] for rawtag in rawtags.splitlines()]:
+        print(f"Fetching matching revisions: {ref}")
+        subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{ref}:{ref}"], cwd=resultdir)
+
+
+def generate_regression_report(resulttool, targetrepodir, basebranch, resultdir, outputdir, yoctoversion):
+    baseversion = get_previous_tag(targetrepodir, yoctoversion)
+    baserevision = get_sha1(targetrepodir, baseversion)
+    comparerevision = get_sha1(targetrepodir, basebranch)
+    print(f"Compare version : {basebranch} ({comparerevision})")
+    print(f"Base tag : {baseversion} ({baserevision})")
+
+    try:
+        """
+        Results directory is likely a shallow clone :
+        we need to fetch results corresponding to base revision before
+        running resulttool
+        """
+        fetch_testresults(resultdir, baserevision)
+        regreport = subprocess.check_output([resulttool, "regression-git", "-B", basebranch, "--commit", baserevision, "--commit2", comparerevision, resultdir])
+        with open(outputdir + "/testresult-regressions-report.txt", "wb") as f:
+           f.write(regreport)
+    except subprocess.CalledProcessError as e:
+        error = str(e)
+        print(f"Error while generating report between {baserevision} and {comparerevision} : {error}")
+
 
 def send_qa_email():
     parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.')
@@ -57,16 +125,7 @@ def send_qa_email():
         branch = repos['poky']['branch']
         repo = repos['poky']['url']
 
-        extraopts = None
         basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch)
-        if basebranch:

scripts/send_qa_email.py def send_qa_email(): · line 131

Quoted 10 lines
scripts/send_qa_email.py +0 -7
-            extraopts = " --branch %s --commit %s" % (branch, revision)
-        if comparebranch:
-            extraopts = extraopts + " --branch2 %s" % (comparebranch)
-        elif basebranch:
-            print("No comparision branch found, comparing to %s" % basebranch)
-            extraopts = extraopts + " --branch2 %s" % basebranch
-
         report = subprocess.check_output([resulttool, "report", args.results_dir])
         with open(args.results_dir + "/testresult-report.txt", "wb") as f:
             f.write(report)

There looks to be a potential issue here since you drop the use of comparebranch entirely. This comes from config.json:

"BUILD_HISTORY_FORKPUSH" : {"poky-contrib:ross/mut" : "poky:master", "poky:master-next" : "poky:master"},

and basically what it says is that master-next is a force push branch which should be compared against master (and ross/mut should be compared against master too).

We need this code so that when we build test branches, we run the regression comparison against the last known master test results. We should probably put Alexandre's test branch in there too.

Cheers,

Richard

#5 lore raw

Re: [yocto] [autobuilder][PATCH v3 3/5] scripts/send-qa-email: Generate regression reports against most relevant release

Hi Richard,
thanks for the feedback

Quoted 11 lines · Richard Purdie
Quoted 4 lines · Alexis Lothoré via lists.yoctoproject.org

report = subprocess.check_output([resulttool, "report", args.results_dir]) with open(args.results_dir + "/testresult-report.txt", "wb") as f:
f.write(report)

There looks to be a potential issue here since you drop the use of comparebranch entirely. This comes from config.json:

"BUILD_HISTORY_FORKPUSH" : {"poky-contrib:ross/mut" : "poky:master", "poky:master-next" : "poky:master"},

Quoted 7 lines

and basically what it says is that master-next is a force push branch which should be compared against master (and ross/mut should be compared against master too).

We need this code so that when we build test branches, we run the regression comparison against the last known master test results. We should probably put Alexandre's test branch in there too.

I may have missed that. I need to dig a bit to understand exactly what is expected here in order to fix it. Noted, I will take a look at it.

Regards

Signature · 3 lines
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

PATCH v3 4/5 scripts/send_qa_email.py: add unit tests on previous version computation

In reply to Alexis Lothoré

The "previous version" computation bring many edge cases depending on the version under release. Add a basic test suite to validate currently implemented computation strategy and to prevent mistakes when ediiting it in the future

---
 scripts/test_send_qa_email.py | 56 +++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
 create mode 100755 scripts/test_send_qa_email.py
scripts/test_send_qa_email.py new file +56 -0
new file mode 100755
index 0000000..41e895e
@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+"""
+Test suite for send_qa_email.py
+
+The suite needs a valid poky clone to run since it will
+fetch and return revisions from remote repository. To run the suite,
+set POKY_PATH environment variable accordingly:
+`POKY_PATH=~/src/poky ./scripts/test_send_qa_email.py`
+"""
+import os
+import sys
+import unittest
+import send_qa_email
+
+
+class TestVersion(unittest.TestCase):
+    test_data_get_version = [
+        {"input": {"version": "4.1.2"}, "expected": "yocto-4.1.1"},
+        {"input": {"version": "4.1"}, "expected": "yocto-4.0"},
+        {"input": {"version": "4.1.1"}, "expected": "yocto-4.1"},
+        {"input": {"version": "4.1_M2"}, "expected": "4.1_M1"},
+        {"input": {"version": "4.1_M1"}, "expected": "yocto-4.0"},
+        {"input": {"version": "4.1.1.rc1"}, "expected": "yocto-4.1"},
+        {"input": {"version": "4.1.2.rc1"}, "expected": "yocto-4.1.1"},
+        {"input": {"version": "4.1_M3.rc1"}, "expected": "4.1_M2"},
+        {"input": {"version": "4.1_M3.rc4"}, "expected": "4.1_M2"},
+        {"input": {"version": "4.1_M1.rc1"}, "expected": "yocto-4.0"},
+        {"input": {"version": "4.1_M1.rc4"}, "expected": "yocto-4.0"},
+        {"input": {"version": "4.1.rc4"}, "expected": "yocto-4.0"}
+    ]
+
+    test_data_get_sha1 = [
+        {"input": "yocto-4.0", "expected": "00cfdde791a0176c134f31e5a09eff725e75b905"},
+        {"input": "4.1_M1", "expected": "95066dde6861ee08fdb505ab3e0422156cc24fae"},
+    ]
+
+    def test_versions(self):
+        for data in self.test_data_get_version:
+            test_name = data["input"]["version"]
+            with self.subTest(f"Test {test_name} previous tag"):
+                self.assertEqual(send_qa_email.get_previous_tag(os.environ.get(
+                    "POKY_PATH"), data["input"]["version"]), data["expected"])
+
+    def test_get_sha1(self):
+        for data in self.test_data_get_sha1:
+            test_name = data["input"]
+            with self.subTest(f"Test SHA1 from {test_name}"):
+                self.assertEqual(send_qa_email.get_sha1(os.environ.get(
+                    "POKY_PATH"), data["input"]), data["expected"])
+
+
+if __name__ == '__main__':
+    if os.environ.get("POKY_PATH") is None:
+        print("Please set POKY_PATH to proper poky clone location before running tests")
+        sys.exit(1)
+    unittest.main()
Signature · 1 line
2.39.0
#7 lore raw

PATCH v3 5/5 scripts/send-qa-email: add symlink to preserve compatibility with autobuilder

In reply to Alexis Lothoré

Merely renaming send-qa-email to send_qa_email.py would imply to update yocto-autobuilder2 scripts, which can be daunting. In order to preserve compatibility with current configuration, add a symlink pointing to the new script

---
 scripts/send-qa-email | 1 +
 1 file changed, 1 insertion(+)
 create mode 120000 scripts/send-qa-email
scripts/send-qa-email new file +1 -0
new file mode 120000
index 0000000..d82d07f
@@ -0,0 +1 @@
+send_qa_email.py
\ No newline at end of file
Signature · 1 line
2.39.0
#8 lore raw
In reply to Alexis Lothoré
Quoted 26 lines · Alexis Lothoré via lists.yoctoproject.org

Hi, this is the 3nd version of work initiated to improve regression reports usefulness, started around issue YOCTO #1465
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065).

Changes since v2:
- add symlink to preserve compatibility with current autobuilder configuration

Changes since v1:
- minor rework to be able to import send_qa_email.py as a standard python module
- properly manage non-releases versions since qe_send_email.py can be called with such "release" versions ("-r" parameter)
- add unit tests on previous version computation
- do not fetch yocto-testresults full history: identify needed revisions with git ls-remote and retrieve them with git fetch

Alexis Lothoré (5):
scripts/send_qa_email.py: Rename send-qa-email to send_qa_email.py scripts/send_qa_email.py: Wrap send_qa_email.py content in function scripts/send-qa-email: Generate regression reports against most
relevant release
scripts/send_qa_email.py: add unit tests on previous version
computation
scripts/send-qa-email: add symlink to preserve compatibility with
autobuilder

Quoted 9 lines
 scripts/send-qa-email         | 165 +------------------------
 scripts/send_qa_email.py      | 223 ++++++++++++++++++++++++++++++++++
 scripts/shared-repo-unpack    |   2 +-
 scripts/test_send_qa_email.py |  57 +++++++++
 scripts/utils.py              |  47 +++++++
 5 files changed, 329 insertions(+), 165 deletions(-)
 mode change 100755 => 120000 scripts/send-qa-email
 create mode 100755 scripts/send_qa_email.py
 create mode 100755 scripts/test_send_qa_email.py

Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.

I did notice one issue which we'll need to fix in some follow up patches, I'll reply to the patch about that.

Cheers,

Richard

#9 lore raw
Quoted 6 lines · Richard Purdie
Quoted 1 line · Alexis Lothoré via lists.yoctoproject.org

Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.

Thanks. Please feel free to reach back if you observe any unwanted side effect, since most of my knowledge is based on scripts reading and autobuilder builds history. I have tried to test it as much as possible on my machine, but as I was discussing with Alexandre Belloni, the many stubs and fake data used do not make the tests as accurate as the real world case.

Quoted 2 lines

I did notice one issue which we'll need to fix in some follow up patches, I'll reply to the patch about that.

ACK, I am working on it

Signature · 3 lines
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#10 lore raw
Quoted 8 lines · Alexis Lothoré
Quoted 6 lines · Richard Purdie
Quoted 1 line · Alexis Lothoré via lists.yoctoproject.org

Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.

Quoted 5 lines

Thanks. Please feel free to reach back if you observe any unwanted side effect, since most of my knowledge is based on scripts reading and autobuilder builds history. I have tried to test it as much as possible on my machine, but as I was discussing with Alexandre Belloni, the many stubs and fake data used do not make the tests as accurate as the real world case.

The result of the M2 build is here, specifically the regression report:

https://autobuilder.yocto.io/pub/releases/yocto-4.2_M2.rc2/testresults/testresult-regressions-report.txt

(warning, it is a 13MB file, which may hint at some problems)

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4844/steps/29/logs/stdio

was the log from running the script. It also suggests there may be a few issues.

I think we need some improvements to the report. Firstly it should state at the top which revisions are being compared and any tags they correspond to.

Next, we need to understand why the large chunk of tests "disappeared". it may be related to the "Failed to retrieved" messages?

I also happen to know the reproducibility tests failed in this build, yet the regression report has no mention of that.

So the patches are a start but there is much more to do.

Cheers,

Richard

#11 lore raw
Quoted 21 lines · Richard Purdie
Quoted 8 lines · Alexis Lothoré
Quoted 6 lines · Richard Purdie
Quoted 1 line · Alexis Lothoré via lists.yoctoproject.org

Thanks. Since I like living dangerously, I've put these on the master branch for the M2 release build. I wanted to see how this does in a real world test.

Quoted 5 lines

Thanks. Please feel free to reach back if you observe any unwanted side effect, since most of my knowledge is based on scripts reading and autobuilder builds history. I have tried to test it as much as possible on my machine, but as I was discussing with Alexandre Belloni, the many stubs and fake data used do not make the tests as accurate as the real world case.

The result of the M2 build is here, specifically the regression report:

https://autobuilder.yocto.io/pub/releases/yocto-4.2_M2.rc2/testresults/testresult-regressions-report.txt

(warning, it is a 13MB file, which may hint at some problems)

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4844/steps/29/logs/stdio

was the log from running the script. It also suggests there may be a few issues.

I think we need some improvements to the report. Firstly it should state at the top which revisions are being compared and any tags they correspond to.

Yes, I was thinking about this while tweaking send-qa-email. Another point I need to understand is how the regression report is managed if resulttool finds multiple tags in results dir concerning the same yocto revision. I feel that the result is not the same if it finds one tag or multiple tags for the same revision. Does it really make sense to check regression against multiple tags referencing the same yocto revision ? If so, indeed we should make the tag name appear. If not, how to distinguish which tag is appropriate ? This question is more linked to resulttool internals than to send-qa-email.

Quoted 6 lines

Next, we need to understand why the large chunk of tests "disappeared". it may be related to the "Failed to retrieved" messages?

I also happen to know the reproducibility tests failed in this build, yet the regression report has no mention of that.

ACK, I will take a look.

Quoted 2 lines

So the patches are a start but there is much more to do.

Regards,

Signature · 3 lines
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com