Add a display limit for regression report generation
v2 cover Add a display limit for regression report generation
It has been observed that useful information in regression report can be drowned in huge regression lists which are often false-positives (for example, a whole set of tests has been temporarily disabled). Additionally, the regression report usually starts with not-so-useful information when searching for regressions (basically, big lists of tests which have not regressed).
This series aims to improve regression display by rearranging info order as well as bringing a default limit to how many changes are displayed per base/target comparison. This new version follows up suggestions discussed in [1].
One consequence of this series is the new available option for yocto_testresults_query:
- yocto_testresults_query 4.3_M1 4.3_M2
-> will display at most 50 regressions per test type
- yocto_testresults_query -l 10 4.3_M1 4.3_M2
-> override the display limit and reduce it to 10 regressions per test
type
- yocto_testresults_query -l 0 4.3_M1 4.3_M2
-> disable the display limit, print all regressions
An example of regression report with those modifications can be found at
[2]
Note: this series does not take care yet of improving base/target pairs selection based on machine/distro (issue raised by Ross in [1]), because I still need to have a better look at this before tweaking anything there.
Changes in v2:
- use the display limit for each test type instead of using it as a global limit
- display regressions summaries before detailed list
- display regressions first, display matched/improved tests after, as "additional data"
- Reduce less useful info size
- fix testresult selftests
[1] https://lore.kernel.org/yocto/[email protected]/
[2] https://pastebin.com/JHFLF0sK
Alexis Lothoré (4):
scripts/resulttool: limit the number of changes displayed per test scripts/resulttool: rearrange regressions report order scripts/resulttool: make additional info more compact scripts/yocto_testresults_query: add option to change display limit
scripts/lib/resulttool/regression.py | 60 +++++++++++++++++++++++----- scripts/yocto_testresults_query.py | 13 ++++-- 2 files changed, 60 insertions(+), 13 deletions(-)
Signature · 1 line
2.42.0
PATCH v2 1/4 scripts/resulttool: limit the number of changes displayed per test
From: Alexis Lothoré <[email protected]>
Most of the changes list generated in regression reports fall in one of the two following categories:
- there is only a few (<10) changes listed and the info is valuable/relevant
- the list is huge (> 100 ? 1000 ?) and basically tells us that the whole tests category suffers the same status (test missing, test failing, test skipped, etc)
Prevent those huge, worthless lists by limiting the output for each test group:
- current default limit is arbitrarily set to 50
- limit can still be overriden with a new "-l"/"--limit" flag, either with custom value, or with 0 to print the whole lists of changes
- limit is applied per test family: currently it distinguishes only types of ptests, but it can be adapted to other kind of tests
---
Changes in v2:
- set the display limit per test category instead of one general limit
- add a "per group" count, additionally to the general regression count
- print total regressions count and "per group" regressions count on top of detailed report instead of bottom
- fix resulttool selftests
--- scripts/lib/resulttool/regression.py | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)
scripts/lib/resulttool/regression.py +21 -4
@@ -78,6 +78,8 @@ STATUS_STRINGS = { "None": "No matching test result" } +REGRESSIONS_DISPLAY_LIMIT=50 + def test_has_at_least_one_matching_tag(test, tag_list): return "oetags" in test and any(oetag in tag_list for oetag in test["oetags"])
@@ -181,11 +183,15 @@ def get_status_str(raw_status): raw_status_lower = raw_status.lower() if raw_status else "None" return STATUS_STRINGS.get(raw_status_lower, raw_status) -def compare_result(logger, base_name, target_name, base_result, target_result): +def compare_result(logger, base_name, target_name, base_result, target_result, display_limit=None): base_result = base_result.get('result') target_result = target_result.get('result') result = {} new_tests = 0 + regressions = {} + resultstring = "" + + display_limit = int(display_limit) if display_limit else REGRESSIONS_DISPLAY_LIMIT if base_result and target_result: for k in base_result:
@@ -212,7 +218,17 @@ def compare_result(logger, base_name, target_name, base_result, target_result): resultstring = "Regression: %s\n %s\n" % (base_name, target_name) for k in sorted(result): if not result[k]['target'] or not result[k]['target'].startswith("PASS"): - resultstring += ' %s: %s -> %s\n' % (k, get_status_str(result[k]['base']), get_status_str(result[k]['target'])) + # Differentiate each ptest kind when listing regressions + key = '.'.join(k.split('.')[:2]) if k.startswith('ptest') else k + # Append new regression to corresponding test family + regressions[key] = regressions.setdefault(key, []) + [' %s: %s -> %s\n' % (k, get_status_str(result[k]['base']), get_status_str(result[k]['target']))] + resultstring += f" Total: {sum([len(regressions[r]) for r in regressions])} new regression(s):\n" + for k in regressions: + resultstring += f" {len(regressions[k])} regression(s) for {k}\n" + count_to_print=min([display_limit, len(regressions[k])]) if display_limit > 0 else len(regressions[k]) + resultstring += ''.join(regressions[k][:count_to_print]) + if count_to_print < len(regressions[k]): + resultstring+=' [...]\n' if new_pass_count > 0: resultstring += f' Additionally, {new_pass_count} previously failing test(s) is/are now passing\n' else:
@@ -280,7 +296,7 @@ def regression_common(args, logger, base_results, target_results): for b in target.copy(): if not can_be_compared(logger, base_results[a][c], target_results[a][b]): continue - res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b]) + res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b], args.limit) if not res: matches.append(resstr) base.remove(c)
@@ -291,7 +307,7 @@ def regression_common(args, logger, base_results, target_results): for b in target: if not can_be_compared(logger, base_results[a][c], target_results[a][b]): continue - res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b]) + res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b], args.limit) if res: regressions.append(resstr) else:
@@ -403,4 +419,5 @@ def register_commands(subparsers): parser_build.add_argument('--commit-number', help="Revision number to search for, redundant if --commit is specified") parser_build.add_argument('--commit2', help="Revision to compare with") parser_build.add_argument('--commit-number2', help="Revision number to compare with, redundant if --commit2 is specified") + parser_build.add_argument('-l', '--limit', default=REGRESSIONS_DISPLAY_LIMIT, help="Maximum number of changes to display per test. Can be set to 0 to print all changes")
Signature · 1 line
2.42.0
PATCH v2 2/4 scripts/resulttool: rearrange regressions report order
In reply to Alexis LothoréFrom: Alexis Lothoré <[email protected]>
Regressions reports currently reports matching pairs and improved pairs first, then regressions.
Change order to print regressions first, which is the most valuable info in the report, and then print improvements and matches at the bottom.
--- scripts/lib/resulttool/regression.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
scripts/lib/resulttool/regression.py +6 -2
@@ -80,6 +80,9 @@ STATUS_STRINGS = { REGRESSIONS_DISPLAY_LIMIT=50 +MISSING_TESTS_BANNER = "-------------------------- Missing tests --------------------------" +ADDITIONAL_DATA_BANNER = "--------------------- Matches and improvements --------------------" + def test_has_at_least_one_matching_tag(test, tag_list): return "oetags" in test and any(oetag in tag_list for oetag in test["oetags"])
@@ -312,10 +315,11 @@ def regression_common(args, logger, base_results, target_results): regressions.append(resstr) else: notfound.append("%s not found in target" % a) - print("\n".join(sorted(matches))) - print("\n") print("\n".join(sorted(regressions))) + print("\n" + MISSING_TESTS_BANNER + "\n") print("\n".join(sorted(notfound))) + print("\n" + ADDITIONAL_DATA_BANNER + "\n") + print("\n".join(sorted(matches))) return 0 def regression_git(args, logger):
Signature · 1 line
2.42.0
PATCH v2 3/4 scripts/resulttool: make additional info more compact
In reply to Alexis LothoréFrom: Alexis Lothoré <[email protected]>
Since "matched" and "improved" tests are not as important as regressions, reduce the place they take in the regression report:
- merge "matched" and "improved" tests, while removing the label
- add a single line of additional info per pair
Those changes make the "Matches and improvements" look like the following sample:
oeselftest_almalinux-9.2_qemux86-64_20230910083156 oeselftest_almalinux-8.8_qemux86-64_20231018010951
-> +7 test(s) present
oeselftest_almalinux-9.2_qemux86-64_20230911010538 oeselftest_debian-11_qemux86-64_20231017150459
oeselftest_debian-11_qemux86-64_20230910012927
oeselftest_debian-11_qemux86-64_20231017151319
-> +7 test(s) present
[...]
--- scripts/lib/resulttool/regression.py | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-)
scripts/lib/resulttool/regression.py +23 -4
@@ -186,6 +186,18 @@ def get_status_str(raw_status): raw_status_lower = raw_status.lower() if raw_status else "None" return STATUS_STRINGS.get(raw_status_lower, raw_status) +def get_additional_info_line(new_pass_count, new_tests): + result=[] + if new_tests: + result.append(f'+{new_tests} test(s) present') + if new_pass_count: + result.append(f'+{new_pass_count} test(s) now passing') + + if not result: + return "" + + return ' -> ' + ', '.join(result) + '\n' + def compare_result(logger, base_name, target_name, base_result, target_result, display_limit=None): base_result = base_result.get('result') target_result = target_result.get('result')
@@ -193,6 +205,8 @@ def compare_result(logger, base_name, target_name, base_result, target_result, d new_tests = 0 regressions = {} resultstring = "" + new_tests = 0 + new_pass_count = 0 display_limit = int(display_limit) if display_limit else REGRESSIONS_DISPLAY_LIMIT
@@ -234,14 +248,19 @@ def compare_result(logger, base_name, target_name, base_result, target_result, d resultstring+=' [...]\n' if new_pass_count > 0: resultstring += f' Additionally, {new_pass_count} previously failing test(s) is/are now passing\n' + if new_tests > 0: + resultstring += f' Additionally, {new_tests} new test(s) is/are present\n' else: - resultstring = "Improvement: %s\n %s\n (+%d test(s) passing)\n" % (base_name, target_name, new_pass_count) + resultstring = "%s\n%s\n" % (base_name, target_name) result = None else: - resultstring = "Match: %s\n %s\n" % (base_name, target_name) + resultstring = "%s\n%s\n" % (base_name, target_name) + + if not result: + additional_info = get_additional_info_line(new_pass_count, new_tests) + if additional_info: + resultstring += additional_info - if new_tests > 0: - resultstring += f' Additionally, {new_tests} new test(s) is/are present\n' return result, resultstring def get_results(logger, source):
Signature · 1 line
2.42.0
PATCH v2 4/4 scripts/yocto_testresults_query: add option to change display limit
In reply to Alexis LothoréFrom: Alexis Lothoré <[email protected]>
Add a "-l"/"--limit" option to allow changing the display limit in resulttool.
- If no value is passed, resulttool uses its default value.
- If 0 is passed, the display limit is removed and every regression will be displayed
- If a custom value is passed, this value overrides the vlaue configured in resulttool
--- scripts/yocto_testresults_query.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
scripts/yocto_testresults_query.py +10 -3
@@ -56,9 +56,12 @@ def fetch_testresults(workdir, sha1): subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{rev}:{rev}"], cwd=workdir) return branch -def compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision): +def compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision, args): logger.info(f"Running resulttool regression between SHA1 {baserevision} and {targetrevision}") - report = subprocess.check_output([resulttool, "regression-git", "--branch", basebranch, "--commit", baserevision, "--branch2", targetbranch, "--commit2", targetrevision, workdir]).decode("utf-8") + command = [resulttool, "regression-git", "--branch", basebranch, "--commit", baserevision, "--branch2", targetbranch, "--commit2", targetrevision, workdir] + if args.limit: + command.extend(["-l", args.limit]) + report = subprocess.check_output(command).decode("utf-8") return report def print_report_with_header(report, baseversion, baserevision, targetversion, targetrevision):
@@ -85,7 +88,7 @@ def regression(args): sys.exit(1) basebranch = fetch_testresults(workdir, baserevision) targetbranch = fetch_testresults(workdir, targetrevision) - report = compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision) + report = compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision, args) print_report_with_header(report, args.base, baserevision, args.target, targetrevision) finally: if not args.testresultsdir:
@@ -109,6 +112,10 @@ def main(): '-t', '--testresultsdir', help=f"An existing test results directory. {sys.argv[0]} will automatically clone it and use default branch if not provided") + parser_regression_report.add_argument( + '-l', + '--limit', + help=f"Maximum number of changes to display per test. Can be set to 0 to print all changes") parser_regression_report.set_defaults(func=regression) args = parser.parse_args()
Signature · 1 line
2.42.0