Skip to contentSkip to conversation

generate regression reports against proper releases

PATCH Superseded Alexis Lothoré 2 participants 7 messages Jan 13, 2023 – Jan 16, 2023 1 message not in the archive

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover generate regression reports against proper releases

This work has been initiated while discussing issue #14065
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065)
The topic initially covers ptest reports between releases, but after discussions with R. Purdie (https://lists.yoctoproject.org/g/automated-testing/message/1211),
a general regression report reliability topic emerged. Those first patches aims to make regression reports more useful during releases by computing against which revision it would be relevant to check for regressions.

A few notes about those patches :
- Since it affects release process, I struggled to properly test the new behavior, so testing has been made by stubbing send-qa-email side effects and picking some samples of build properties and layerinfo.json from autobuilder history to run manually send-qa-email on my machine. But I will gladly take into account any advice or procedure to test it more thoroughly
- I also have some uncommitted python unit tests on "previous version computation", but current scripts directory does not allow proper unit tests
integration. If appropriate, I could submit some minor reworks to make it viable for testing and submit those tests
- This new regression checking introduced the need to clone a wider testresults history, which is (very) slow on my machine, but I do not now how slow it is
on autobuilder. Could it be an issue for the delivery process ? Does it need to be smarter on this point ?
- If appropriate, new patch could follow to allow regression reports generation against multiple revisions : for example we could check for a milestone build against previous milestone AND previous major release

Alexis Lothoré (2):
scripts/send-qa-email : clone older history to allow comparing against
older releases
scripts/send-qa-email: Generate regression reports against most
relevant release

 scripts/send-qa-email | 69 ++++++++++++++++++++++++++++++++-----------
 scripts/utils.py      | 47 +++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+), 17 deletions(-)
Signature · 1 line
2.39.0
#1 lore raw

PATCH 1/2 scripts/send-qa-email : clone older history to allow comparing against older releases

In order to generate relevant regression reports, we need to use resulttool against taggued releases, which can be way older than the last five previous commits

---
 scripts/send-qa-email | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/send-qa-email +2 -2
index d2f63dc..c2c5a4b 100755
@@ -78,10 +78,10 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
         elif basebranch:
             cloneopts = ["--branch", basebranch]
         try:
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
+            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir] + cloneopts)
         except subprocess.CalledProcessError:
             print("No comparision branch found, falling back to master")
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"])
+            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir])
 
         # If the base comparision branch isn't present regression comparision won't work
         # at least until we can tell the tool to ignore internal branch information
Signature · 1 line
2.39.0
#2 lore raw

Re: [yocto] [autobuilder][PATCH 1/2] scripts/send-qa-email : clone older history to allow comparing against older releases

Quoted 16 lines · Alexis Lothoré via lists.yoctoproject.org

In order to generate relevant regression reports, we need to use resulttool against taggued releases, which can be way older than the last five previous commits

---
 scripts/send-qa-email | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/send-qa-email +0 -0
index d2f63dc..c2c5a4b 100755
@@ -78,10 +78,10 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
         elif basebranch:
             cloneopts = ["--branch", basebranch]

scripts/send-qa-email if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: · line 87

Quoted 10 lines
scripts/send-qa-email +2 -2
         try:
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
+            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir] + cloneopts)
         except subprocess.CalledProcessError:
             print("No comparision branch found, falling back to master")
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"])
+            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir])
 
         # If the base comparision branch isn't present regression comparision won't work
         # at least until we can tell the tool to ignore internal branch information

That is a somewhat "easy" option but it might not be practical.

The yocto-testresults repo is large. I tried a clone here and it is taking rather a while, it starts ok but then starts taking a long time with a lot of data transfer (over a GB so far) :(.

Can we check the tags with something like ls-remote and then only fetch what we need to compare against?

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

Re: [yocto] [autobuilder][PATCH 1/2] scripts/send-qa-email : clone older history to allow comparing against older releases

Quoted 31 lines · Richard Purdie via lists.yoctoproject.org
Quoted 16 lines · Alexis Lothoré via lists.yoctoproject.org

In order to generate relevant regression reports, we need to use resulttool against taggued releases, which can be way older than the last five previous commits

---
 scripts/send-qa-email | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/send-qa-email +0 -0
index d2f63dc..c2c5a4b 100755
@@ -78,10 +78,10 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
         elif basebranch:
             cloneopts = ["--branch", basebranch]

scripts/send-qa-email if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: · line 87

Quoted 10 lines
scripts/send-qa-email +2 -2
         try:
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
+            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir] + cloneopts)
         except subprocess.CalledProcessError:
             print("No comparision branch found, falling back to master")
-            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir, "--depth", "5"])
+            subprocess.check_call(["git", "clone", "[email protected]:yocto-testresults", tempdir])
 
         # If the base comparision branch isn't present regression comparision won't work
         # at least until we can tell the tool to ignore internal branch information

That is a somewhat "easy" option but it might not be practical.

Quoted 6 lines

The yocto-testresults repo is large. I tried a clone here and it is taking rather a while, it starts ok but then starts taking a long time with a lot of data transfer (over a GB so far) :(.

Can we check the tags with something like ls-remote and then only fetch what we need to compare against?

For reference the clone took 49 minutes and 12.45GiB so we will need to find a different approach.

Cheers,

Richard

#5 lore raw

PATCH 2/2 scripts/send-qa-email: Generate regression reports against most relevant release

In reply to Alexis Lothoré

Instead of only generating regressions reports against HEAD of relevant branch, compute most relevant tag (ie : release) against which we can check for regressions. General rules introduced are the following :
- milestone release is checked against previous milestone if possible, otherwise against major release
- point release is checked against previous point release if possible, otherwise against major release
- major release is checked against previous major release
- a non release build is checked against base branch
Examples :
- 4.1.2.rc1 is checked against yocto-4.1.1
- 4.1.2 is checked against yocto-4.1.1
- 4.1.1.rc1 is checked against yocto-4.1
- 4.1.1 is checked against yocto-4.1
- 4.1 is checked against yocto-4.0
- 4.1.rc4 is checked against yocto-4.0
- 4.1_M2.rc1 is checked against 4.1_M1
- 4.1_M2 is checked against 4.1_M1
- 4.1_M1.rc1 is checked against yocto-4.0
- 4.1_M1 is checked against yocto-4.0

---
 scripts/send-qa-email | 65 +++++++++++++++++++++++++++++++++----------
 scripts/utils.py      | 47 +++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 15 deletions(-)
scripts/send-qa-email +50 -15
index c2c5a4b..c2bdeb5 100755
@@ -49,6 +49,54 @@ if os.path.exists(buildtoolsdir):
 
 repodir = os.path.dirname(args.repojson) + "/build/repos"
 
+def get_previous_tag(targetrepodir, version):
+    previousversion = None
+    previousmilestone = None
+    if version:
+        compareversion, comparemilestone, _ = utils.get_version_from_string(version)
+        print(f"Searching before {compareversion}")
+        compareversionminor = compareversion[-1]
+        # After ignoring rc part, if we get a minor to 0 on point release (e.g 4.0.0),
+        # reject last digit since such versions do not exist
+        if len(compareversion) == 3 and compareversionminor == 0:
+            compareversion = compareversion[:-1]
+
+        # Process milestone if not first in current release
+        if comparemilestone and comparemilestone > 1:
+            previousversion = compareversion
+            previousmilestone = comparemilestone-1
+        # Process first milestone or release if not first in major release
+        elif compareversionminor > 0:
+            previousversion = compareversion[:-1] + [compareversion[-1] - 1]
+        # Otherwise : format it as tag (which must exist) and search previous tag
+        else:
+            comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone)
+            return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip()
+
+        return utils.get_tag_from_version(previousversion, previousmilestone)
+
+    # All other cases : merely check against latest tag reachable
+    defaultbaseversion, _, _ = utils.get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip())
+    return utils.get_tag_from_version(defaultbaseversion, None)
+
+def get_sha1(targetrepodir, revision):
+    return subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=targetrepodir).decode('utf-8').strip()
+
+def generate_regression_report(targetrepodir, basebranch, resultdir, yoctoversion):
+    baseversion = get_previous_tag(targetrepodir, yoctoversion)
+    baserevision = get_sha1(targetrepodir, baseversion)
+    comparerevision = get_sha1(targetrepodir, basebranch)
+    print(f"Compare version : {basebranch} ({comparerevision})")
+    print(f"Base tag : {baseversion} ({baserevision})")
+
+    try:
+        regreport = subprocess.check_output([resulttool, "regression-git", "-B", basebranch, "--commit", baserevision, "--commit2", comparerevision, resultdir])
+        with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f:
+           f.write(regreport)
+    except subprocess.CalledProcessError as e:
+        error = str(e)
+        print(f"Error while generating report between {baserevision} and {comparerevision} : {error}")
+
 if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
     # Need the finalised revisions (not 'HEAD')
     targetrepodir = "%s/poky" % (repodir)
@@ -56,16 +104,7 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
     branch = repos['poky']['branch']
     repo = repos['poky']['url']
 
-    extraopts = None
     basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch)
-    if basebranch:
-        extraopts = " --branch %s --commit %s" % (branch, revision)
-    if comparebranch:
-        extraopts = extraopts + " --branch2 %s" % (comparebranch)
-    elif basebranch:
-        print("No comparision branch found, comparing to %s" % basebranch)
-        extraopts = extraopts + " --branch2 %s" % basebranch
-
     report = subprocess.check_output([resulttool, "report", args.results_dir])
     with open(args.results_dir + "/testresult-report.txt", "wb") as f:
         f.write(report)
@@ -94,8 +133,6 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
                 subprocess.check_call(["git", "checkout", "master"], cwd=tempdir)
                 subprocess.check_call(["git", "branch", basebranch], cwd=tempdir)
                 subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir)
-                extraopts = None
-
         subprocess.check_call([resulttool, "store", args.results_dir, tempdir])
         if comparebranch:
             subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir)
@@ -104,10 +141,8 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
             subprocess.check_call(["git", "push", "--all"], cwd=tempdir)
             subprocess.check_call(["git", "push", "--tags"], cwd=tempdir)
 
-        if extraopts:
-            regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split())
-            with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f:
-                f.write(regreport)
+        if basebranch:
+            generate_regression_report(targetrepodir, basebranch, tempdir, args.release)
 
     finally:
         subprocess.check_call(["rm", "-rf",  tempdir])
scripts/utils.py +47 -0

51 lines not on this page · raw message

Signature · 1 line
2.39.0
#6 lore raw
In reply to Alexis Lothoré
Quoted 8 lines · Alexis Lothoré via lists.yoctoproject.org

This work has been initiated while discussing issue #14065
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065)
The topic initially covers ptest reports between releases, but after discussions with R. Purdie (https://lists.yoctoproject.org/g/automated-testing/message/1211),
a general regression report reliability topic emerged. Those first patches aims to make regression reports more useful during releases by computing against which revision it would be relevant to check for regressions.

Quoted 10 lines

A few notes about those patches :
- Since it affects release process, I struggled to properly test the new behavior, so testing has been made by stubbing send-qa-email side effects and picking some samples of build properties and layerinfo.json from autobuilder history to run manually send-qa-email on my machine. But I will gladly take into account any advice or procedure to test it more thoroughly
- I also have some uncommitted python unit tests on "previous version computation", but current scripts directory does not allow proper unit tests
integration. If appropriate, I could submit some minor reworks to make it viable for testing and submit those tests

I'm glad you mentioned this! Yes, I think it is important we add some tests for this. We can rework these scripts as needed as the project is their main user at this point.

Quoted 4 lines

- This new regression checking introduced the need to clone a wider testresults history, which is (very) slow on my machine, but I do not now how slow it is
on autobuilder. Could it be an issue for the delivery process ? Does it need to be smarter on this point ?

I think it will be slow on the autobuilder and we're going to have to do something different to make this work well. My clone still hasn't finished and is at 5.75GB of data now (15 mins to get the last 4.5GB) and still going.

Quoted 3 lines

- If appropriate, new patch could follow to allow regression reports generation against multiple revisions : for example we could check for a milestone build against previous milestone AND previous major release

I think that is the direction we need to go so yes please!

Cheers,

Richard

#7 lore raw

Hi Richard,

Quoted 11 lines · Richard Purdie
Quoted 5 lines · Alexis Lothoré via lists.yoctoproject.org

- I also have some uncommitted python unit tests on "previous version computation", but current scripts directory does not allow proper
unit tests
integration. If appropriate, I could submit some minor reworks to make it viable for testing and submit those tests

I'm glad you mentioned this! Yes, I think it is important we add some tests for this. We can rework these scripts as needed as the project is their main user at this point.

Good, I will then add some minor rework to allow introducing unit tests alongside the version computation in the next revision

Quoted 8 lines
Quoted 7 lines

- This new regression checking introduced the need to clone a wider testresults
history, which is (very) slow on my machine, but I do not now how
slow it is
on autobuilder. Could it be an issue for the delivery process ? Does it need to be smarter on this point ?

Quoted 4 lines

I think it will be slow on the autobuilder and we're going to have to do something different to make this work well. My clone still hasn't finished and is at 5.75GB of data now (15 mins to get the last 4.5GB) and still going.

OK. I observed a quite similar behaviour while pulling manually the whole repository, but I thought it might be linked to network issue on my side. I will add a smarter way based on your suggestion (git ls-remote)

Quoted 6 lines
Quoted 4 lines

- If appropriate, new patch could follow to allow regression reports generation
against multiple revisions : for example we could check for a milestone build against previous milestone AND previous major release

I think that is the direction we need to go so yes please!

Noted. If it is OK for you, I will keep this last point for another set of patches, once this initial work is approved and merged.

Regards,

Signature · 3 lines
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com