Skip to contentSkip to conversation

Add a display limit for regression report generation

PATCH New Alexis Lothoré 1 participant 5 messages Oct 22, 2023

lore
Share & tools
lore
#0 lore raw

v2 cover Add a display limit for regression report generation

It has been observed that useful information in regression report can be drowned in huge regression lists which are often false-positives (for example, a whole set of tests has been temporarily disabled). Additionally, the regression report usually starts with not-so-useful information when searching for regressions (basically, big lists of tests which have not regressed).

This series aims to improve regression display by rearranging info order as well as bringing a default limit to how many changes are displayed per base/target comparison. This new version follows up suggestions discussed in [1].

One consequence of this series is the new available option for yocto_testresults_query:
- yocto_testresults_query 4.3_M1 4.3_M2
-> will display at most 50 regressions per test type
- yocto_testresults_query -l 10 4.3_M1 4.3_M2
-> override the display limit and reduce it to 10 regressions per test
type
- yocto_testresults_query -l 0 4.3_M1 4.3_M2
-> disable the display limit, print all regressions

An example of regression report with those modifications can be found at
[2]

Note: this series does not take care yet of improving base/target pairs selection based on machine/distro (issue raised by Ross in [1]), because I still need to have a better look at this before tweaking anything there.

Changes in v2:
- use the display limit for each test type instead of using it as a global limit
- display regressions summaries before detailed list
- display regressions first, display matched/improved tests after, as "additional data"
- Reduce less useful info size
- fix testresult selftests

[1] https://lore.kernel.org/yocto/[email protected]/
[2] https://pastebin.com/JHFLF0sK

Alexis Lothoré (4):
scripts/resulttool: limit the number of changes displayed per test scripts/resulttool: rearrange regressions report order scripts/resulttool: make additional info more compact scripts/yocto_testresults_query: add option to change display limit

 scripts/lib/resulttool/regression.py | 60 +++++++++++++++++++++++-----
 scripts/yocto_testresults_query.py   | 13 ++++--
 2 files changed, 60 insertions(+), 13 deletions(-)
Signature · 1 line
2.42.0
#1 lore raw

PATCH v2 1/4 scripts/resulttool: limit the number of changes displayed per test

From: Alexis Lothoré <[email protected]>

Most of the changes list generated in regression reports fall in one of the two following categories:
- there is only a few (<10) changes listed and the info is valuable/relevant
- the list is huge (> 100 ? 1000 ?) and basically tells us that the whole tests category suffers the same status (test missing, test failing, test skipped, etc)

Prevent those huge, worthless lists by limiting the output for each test group:
- current default limit is arbitrarily set to 50
- limit can still be overriden with a new "-l"/"--limit" flag, either with custom value, or with 0 to print the whole lists of changes
- limit is applied per test family: currently it distinguishes only types of ptests, but it can be adapted to other kind of tests

---
Changes in v2:
- set the display limit per test category instead of one general limit
- add a "per group" count, additionally to the general regression count
- print total regressions count and "per group" regressions count on top of detailed report instead of bottom
- fix resulttool selftests

---
 scripts/lib/resulttool/regression.py | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)
scripts/lib/resulttool/regression.py +21 -4
index 3d64b8f4af7c..d98504f4ced8 100644
@@ -78,6 +78,8 @@ STATUS_STRINGS = {
     "None": "No matching test result"
 }
 
+REGRESSIONS_DISPLAY_LIMIT=50
+
 def test_has_at_least_one_matching_tag(test, tag_list):
     return "oetags" in test and any(oetag in tag_list for oetag in test["oetags"])
 
@@ -181,11 +183,15 @@ def get_status_str(raw_status):
     raw_status_lower = raw_status.lower() if raw_status else "None"
     return STATUS_STRINGS.get(raw_status_lower, raw_status)
 
-def compare_result(logger, base_name, target_name, base_result, target_result):
+def compare_result(logger, base_name, target_name, base_result, target_result, display_limit=None):
     base_result = base_result.get('result')
     target_result = target_result.get('result')
     result = {}
     new_tests = 0
+    regressions = {}
+    resultstring = ""
+
+    display_limit = int(display_limit) if display_limit else REGRESSIONS_DISPLAY_LIMIT
 
     if base_result and target_result:
         for k in base_result:
@@ -212,7 +218,17 @@ def compare_result(logger, base_name, target_name, base_result, target_result):
             resultstring = "Regression:  %s\n             %s\n" % (base_name, target_name)
             for k in sorted(result):
                 if not result[k]['target'] or not result[k]['target'].startswith("PASS"):
-                    resultstring += '    %s: %s -> %s\n' % (k, get_status_str(result[k]['base']), get_status_str(result[k]['target']))
+                    # Differentiate each ptest kind when listing regressions
+                    key = '.'.join(k.split('.')[:2]) if k.startswith('ptest') else k
+                    # Append new regression to corresponding test family
+                    regressions[key] = regressions.setdefault(key, []) + ['        %s: %s -> %s\n' % (k, get_status_str(result[k]['base']), get_status_str(result[k]['target']))]
+            resultstring += f"    Total: {sum([len(regressions[r]) for r in regressions])} new regression(s):\n"
+            for k in regressions:
+                resultstring += f"    {len(regressions[k])} regression(s) for {k}\n"
+                count_to_print=min([display_limit, len(regressions[k])]) if display_limit > 0 else len(regressions[k])
+                resultstring += ''.join(regressions[k][:count_to_print])
+                if count_to_print < len(regressions[k]):
+                    resultstring+='        [...]\n'
             if new_pass_count > 0:
                 resultstring += f'    Additionally, {new_pass_count} previously failing test(s) is/are now passing\n'
         else:
@@ -280,7 +296,7 @@ def regression_common(args, logger, base_results, target_results):
                 for b in target.copy():
                     if not can_be_compared(logger, base_results[a][c], target_results[a][b]):
                         continue
-                    res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b])
+                    res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b], args.limit)
                     if not res:
                         matches.append(resstr)
                         base.remove(c)
@@ -291,7 +307,7 @@ def regression_common(args, logger, base_results, target_results):
                 for b in target:
                     if not can_be_compared(logger, base_results[a][c], target_results[a][b]):
                         continue
-                    res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b])
+                    res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b], args.limit)
                     if res:
                         regressions.append(resstr)
         else:
@@ -403,4 +419,5 @@ def register_commands(subparsers):
     parser_build.add_argument('--commit-number', help="Revision number to search for, redundant if --commit is specified")
     parser_build.add_argument('--commit2', help="Revision to compare with")
     parser_build.add_argument('--commit-number2', help="Revision number to compare with, redundant if --commit2 is specified")
+    parser_build.add_argument('-l', '--limit', default=REGRESSIONS_DISPLAY_LIMIT, help="Maximum number of changes to display per test. Can be set to 0 to print all changes")
 
Signature · 1 line
2.42.0
#2 lore raw

PATCH v2 2/4 scripts/resulttool: rearrange regressions report order

In reply to Alexis Lothoré

From: Alexis Lothoré <[email protected]>

Regressions reports currently reports matching pairs and improved pairs first, then regressions.

Change order to print regressions first, which is the most valuable info in the report, and then print improvements and matches at the bottom.

---
 scripts/lib/resulttool/regression.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
scripts/lib/resulttool/regression.py +6 -2
index d98504f4ced8..560d102e172c 100644
@@ -80,6 +80,9 @@ STATUS_STRINGS = {
 
 REGRESSIONS_DISPLAY_LIMIT=50
 
+MISSING_TESTS_BANNER =   "-------------------------- Missing tests --------------------------"
+ADDITIONAL_DATA_BANNER = "--------------------- Matches and improvements --------------------"
+
 def test_has_at_least_one_matching_tag(test, tag_list):
     return "oetags" in test and any(oetag in tag_list for oetag in test["oetags"])
 
@@ -312,10 +315,11 @@ def regression_common(args, logger, base_results, target_results):
                         regressions.append(resstr)
         else:
             notfound.append("%s not found in target" % a)
-    print("\n".join(sorted(matches)))
-    print("\n")
     print("\n".join(sorted(regressions)))
+    print("\n" + MISSING_TESTS_BANNER + "\n")
     print("\n".join(sorted(notfound)))
+    print("\n" + ADDITIONAL_DATA_BANNER + "\n")
+    print("\n".join(sorted(matches)))
     return 0
 
 def regression_git(args, logger):
Signature · 1 line
2.42.0
#3 lore raw

PATCH v2 3/4 scripts/resulttool: make additional info more compact

In reply to Alexis Lothoré

From: Alexis Lothoré <[email protected]>

Since "matched" and "improved" tests are not as important as regressions, reduce the place they take in the regression report:

- merge "matched" and "improved" tests, while removing the label
- add a single line of additional info per pair

Those changes make the "Matches and improvements" look like the following sample:

oeselftest_almalinux-9.2_qemux86-64_20230910083156 oeselftest_almalinux-8.8_qemux86-64_20231018010951

    -> +7 test(s) present

oeselftest_almalinux-9.2_qemux86-64_20230911010538 oeselftest_debian-11_qemux86-64_20231017150459

oeselftest_debian-11_qemux86-64_20230910012927
oeselftest_debian-11_qemux86-64_20231017151319

    -> +7 test(s) present

[...]

---
 scripts/lib/resulttool/regression.py | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)
scripts/lib/resulttool/regression.py +23 -4
index 560d102e172c..8fbe5a54783a 100644
@@ -186,6 +186,18 @@ def get_status_str(raw_status):
     raw_status_lower = raw_status.lower() if raw_status else "None"
     return STATUS_STRINGS.get(raw_status_lower, raw_status)
 
+def get_additional_info_line(new_pass_count, new_tests):
+    result=[]
+    if new_tests:
+        result.append(f'+{new_tests} test(s) present')
+    if new_pass_count:
+        result.append(f'+{new_pass_count} test(s) now passing')
+
+    if not result:
+        return ""
+
+    return '    -> ' + ', '.join(result) + '\n'
+
 def compare_result(logger, base_name, target_name, base_result, target_result, display_limit=None):
     base_result = base_result.get('result')
     target_result = target_result.get('result')
@@ -193,6 +205,8 @@ def compare_result(logger, base_name, target_name, base_result, target_result, d
     new_tests = 0
     regressions = {}
     resultstring = ""
+    new_tests = 0
+    new_pass_count = 0
 
     display_limit = int(display_limit) if display_limit else REGRESSIONS_DISPLAY_LIMIT
 
@@ -234,14 +248,19 @@ def compare_result(logger, base_name, target_name, base_result, target_result, d
                     resultstring+='        [...]\n'
             if new_pass_count > 0:
                 resultstring += f'    Additionally, {new_pass_count} previously failing test(s) is/are now passing\n'
+            if new_tests > 0:
+                resultstring += f'    Additionally, {new_tests} new test(s) is/are present\n'
         else:
-            resultstring = "Improvement: %s\n             %s\n             (+%d test(s) passing)\n" % (base_name, target_name, new_pass_count)
+            resultstring = "%s\n%s\n" % (base_name, target_name)
             result = None
     else:
-        resultstring = "Match:       %s\n             %s\n" % (base_name, target_name)
+        resultstring = "%s\n%s\n" % (base_name, target_name)
+
+    if not result:
+        additional_info = get_additional_info_line(new_pass_count, new_tests)
+        if additional_info:
+            resultstring += additional_info
 
-    if new_tests > 0:
-        resultstring += f'    Additionally, {new_tests} new test(s) is/are present\n'
     return result, resultstring
 
 def get_results(logger, source):
Signature · 1 line
2.42.0
#4 lore raw

PATCH v2 4/4 scripts/yocto_testresults_query: add option to change display limit

In reply to Alexis Lothoré

From: Alexis Lothoré <[email protected]>

Add a "-l"/"--limit" option to allow changing the display limit in resulttool.
- If no value is passed, resulttool uses its default value.
- If 0 is passed, the display limit is removed and every regression will be displayed
- If a custom value is passed, this value overrides the vlaue configured in resulttool

---
 scripts/yocto_testresults_query.py | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
scripts/yocto_testresults_query.py +10 -3
index a5073736aab5..521ead8473ad 100755
@@ -56,9 +56,12 @@ def fetch_testresults(workdir, sha1):
         subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{rev}:{rev}"], cwd=workdir)
     return branch
 
-def compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision):
+def compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision, args):
     logger.info(f"Running resulttool regression between SHA1 {baserevision} and {targetrevision}")
-    report = subprocess.check_output([resulttool, "regression-git", "--branch", basebranch, "--commit", baserevision, "--branch2", targetbranch, "--commit2", targetrevision, workdir]).decode("utf-8")
+    command = [resulttool, "regression-git", "--branch", basebranch, "--commit", baserevision, "--branch2", targetbranch, "--commit2", targetrevision, workdir]
+    if args.limit:
+        command.extend(["-l", args.limit])
+    report = subprocess.check_output(command).decode("utf-8")
     return report
 
 def print_report_with_header(report, baseversion, baserevision, targetversion, targetrevision):
@@ -85,7 +88,7 @@ def regression(args):
             sys.exit(1)
         basebranch = fetch_testresults(workdir, baserevision)
         targetbranch = fetch_testresults(workdir, targetrevision)
-        report = compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision)
+        report = compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision, args)
         print_report_with_header(report, args.base, baserevision, args.target, targetrevision)
     finally:
         if not args.testresultsdir:
@@ -109,6 +112,10 @@ def main():
         '-t',
         '--testresultsdir',
         help=f"An existing test results directory. {sys.argv[0]} will automatically clone it and use default branch if not provided")
+    parser_regression_report.add_argument(
+        '-l',
+        '--limit',
+        help=f"Maximum number of changes to display per test. Can be set to 0 to print all changes")
     parser_regression_report.set_defaults(func=regression)
 
     args = parser.parse_args()
Signature · 1 line
2.42.0