test-case-mgmt
v5 cover test-case-mgmt
v1:
Face key error from oe-git-archive
Undesirable behavior when storing to multiple git branch
v2:
Include fix for oe-git-archive
Include fix for store result to multiple git branch Improve git commit message
v3:
Enhance fix for oe-git-archive by using exception catch to improve code readability and easy to understand
v4:
Add new features, merge result files & regression analysis Add selftest to merge, store, report and regression functionalities Revise codebase for pythonic
v5:
Add required files for selftest testing store
Mazliana (1):
scripts/resultstool: enable manual execution and result creation
Yeoh Ee Peng (1):
resultstool: enable merge, store, report and regression analysis
meta/lib/oeqa/files/testresults/testresults.json | 40 ++++++ meta/lib/oeqa/selftest/cases/resultstooltests.py | 104 ++++++++++++++++ scripts/lib/resultstool/__init__.py | 0 scripts/lib/resultstool/manualexecution.py | 137 +++++++++++++++++++++ scripts/lib/resultstool/merge.py | 71 +++++++++++ scripts/lib/resultstool/regression.py | 134 ++++++++++++++++++++ scripts/lib/resultstool/report.py | 122 ++++++++++++++++++ scripts/lib/resultstool/resultsutils.py | 47 +++++++ scripts/lib/resultstool/store.py | 110 +++++++++++++++++ .../resultstool/template/test_report_full_text.txt | 35 ++++++ scripts/resultstool | 92 ++++++++++++++ 11 files changed, 892 insertions(+) create mode 100644 meta/lib/oeqa/files/testresults/testresults.json create mode 100644 meta/lib/oeqa/selftest/cases/resultstooltests.py create mode 100644 scripts/lib/resultstool/__init__.py create mode 100644 scripts/lib/resultstool/manualexecution.py create mode 100644 scripts/lib/resultstool/merge.py create mode 100644 scripts/lib/resultstool/regression.py create mode 100644 scripts/lib/resultstool/report.py create mode 100644 scripts/lib/resultstool/resultsutils.py create mode 100644 scripts/lib/resultstool/store.py create mode 100644 scripts/lib/resultstool/template/test_report_full_text.txt create mode 100755 scripts/resultstool
Signature · 1 line
2.7.4
PATCH v5 1/2 resultstool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file.
These scripts were developed as a test result tools to manage these testresults.json file.
Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json files to a target file.
Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository.
Using the "regression" operation, user can perform regression analysis on testresults.json files specified.
These resultstool operations expect the testresults.json file to use the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}To use these scripts, first source oe environment, then run the entry point script to look for help.
$ resultstool
To store test result from oeqa automated tests, execute the below
$ resultstool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resultstool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resultstool report <source_dir>
To perform regression analysis, execute the below
$ resultstool regression <base_result_file> <target_result_file>
[YOCTO# 13012]
[YOCTO# 12654]
--- meta/lib/oeqa/files/testresults/testresults.json | 40 ++++++ meta/lib/oeqa/selftest/cases/resultstooltests.py | 104 ++++++++++++++++ scripts/lib/resultstool/__init__.py | 0 scripts/lib/resultstool/merge.py | 71 +++++++++++ scripts/lib/resultstool/regression.py | 134 +++++++++++++++++++++ scripts/lib/resultstool/report.py | 122 +++++++++++++++++++ scripts/lib/resultstool/resultsutils.py | 47 ++++++++ scripts/lib/resultstool/store.py | 110 +++++++++++++++++ .../resultstool/template/test_report_full_text.txt | 35 ++++++ scripts/resultstool | 84 +++++++++++++ 10 files changed, 747 insertions(+) create mode 100644 meta/lib/oeqa/files/testresults/testresults.json create mode 100644 meta/lib/oeqa/selftest/cases/resultstooltests.py create mode 100644 scripts/lib/resultstool/__init__.py create mode 100644 scripts/lib/resultstool/merge.py create mode 100644 scripts/lib/resultstool/regression.py create mode 100644 scripts/lib/resultstool/report.py create mode 100644 scripts/lib/resultstool/resultsutils.py create mode 100644 scripts/lib/resultstool/store.py create mode 100644 scripts/lib/resultstool/template/test_report_full_text.txt create mode 100755 scripts/resultstool
meta/lib/oeqa/files/testresults/testresults.json new file +40 -0
@@ -0,0 +1,40 @@ +{ + "runtime_core-image-minimal_qemuarm_20181225195701": { + "configuration": { + "DISTRO": "poky", + "HOST_DISTRO": "ubuntu-16.04", + "IMAGE_BASENAME": "core-image-minimal", + "IMAGE_PKGTYPE": "rpm", + "LAYERS": { + "meta": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + }, + "meta-poky": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + }, + "meta-yocto-bsp": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + } + }, + "MACHINE": "qemuarm", + "STARTTIME": "20181225195701", + "TEST_TYPE": "runtime" + }, + "result": { + "apt.AptRepoTest.test_apt_install_from_repo": { + "log": "Test requires apt to be installed", + "status": "PASSED" + }, + "buildcpio.BuildCpioTest.test_cpio": { + "log": "Test requires autoconf to be installed", + "status": "ERROR" + } + } + } +}
meta/lib/oeqa/selftest/cases/resultstooltests.py new file +104 -0
104 lines not on this page · raw message
scripts/lib/resultstool/__init__.py new file +0 -0
scripts/lib/resultstool/merge.py new file +71 -0
71 lines not on this page · raw message
scripts/lib/resultstool/regression.py new file +134 -0
134 lines not on this page · raw message
scripts/lib/resultstool/report.py new file +122 -0
122 lines not on this page · raw message
scripts/lib/resultstool/resultsutils.py new file +47 -0
48 lines not on this page · raw message
scripts/lib/resultstool/store.py new file +110 -0
110 lines not on this page · raw message
scripts/lib/resultstool/template/test_report_full_text.txt new file +35 -0
@@ -0,0 +1,35 @@ +============================================================================================================== +Test Report (Count of passed, failed, skipped group by file_dir, result_id) +============================================================================================================== +-------------------------------------------------------------------------------------------------------------- +{{ 'file_dir'.ljust(max_len_dir) }} | {{ 'result_id'.ljust(max_len_result_id) }} | {{ 'passed'.ljust(10) }} | {{ 'failed'.ljust(10) }} | {{ 'skipped'.ljust(10) }} +-------------------------------------------------------------------------------------------------------------- +{% for report in test_count_reports |sort(attribute='test_file_dir_result_id') %} +{{ report.file_dir.ljust(max_len_dir) }} | {{ report.result_id.ljust(max_len_result_id) }} | {{ (report.passed|string).ljust(10) }} | {{ (report.failed|string).ljust(10) }} | {{ (report.skipped|string).ljust(10) }} +{% endfor %} +-------------------------------------------------------------------------------------------------------------- + +============================================================================================================== +Test Report (Percent of passed, failed, skipped group by file_dir, result_id) +============================================================================================================== +-------------------------------------------------------------------------------------------------------------- +{{ 'file_dir'.ljust(max_len_dir) }} | {{ 'result_id'.ljust(max_len_result_id) }} | {{ 'passed_%'.ljust(10) }} | {{ 'failed_%'.ljust(10) }} | {{ 'skipped_%'.ljust(10) }} +-------------------------------------------------------------------------------------------------------------- +{% for report in test_percent_reports |sort(attribute='test_file_dir_result_id') %} +{{ report.file_dir.ljust(max_len_dir) }} | {{ report.result_id.ljust(max_len_result_id) }} | {{ (report.passed|string).ljust(10) }} | {{ (report.failed|string).ljust(10) }} | {{ (report.skipped|string).ljust(10) }} +{% endfor %} +-------------------------------------------------------------------------------------------------------------- + +============================================================================================================== +Test Report (Failed test cases group by file_dir, result_id) +============================================================================================================== +-------------------------------------------------------------------------------------------------------------- +{% for report in test_count_reports |sort(attribute='test_file_dir_result_id') %} +{% if report.failed_testcases %} +file_dir | result_id : {{ report.file_dir }} | {{ report.result_id }} +{% for testcase in report.failed_testcases %} + {{ testcase }} +{% endfor %} +{% endif %} +{% endfor %} +--------------------------------------------------------------------------------------------------------------
scripts/resultstool new file +84 -0
84 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
RP,
The current patch allow files based regression, meaning if you have file#1 and file#2, the regression will select result instances for regression based on the configuration data available.
There are 2 more regression use cases that I have in mind:
Use case#1: directory based regression - Assumed that there are 2 directories storing list of result files. User shall provide these 2 directories for regression, regression scripts will first parse through all the available files inside each directories, then perform regression based on available configuration data to determine the regression pair (eg. select result_set_1 from directory#1 and result_set_x from directory#2 if they both have matching configurations).
Use case#2: git branch based regression - Assumed that result files are stored inside git repository with specific git branch storing result files for single commit. User shall provide the 2 specific git branches for regression, regression scripts will first parse through all the available files inside each git branch, then perform regression based on available configuration data to determine the regression pair (eg. select result_set_1 from git_branch_1 and result_set_x from git_branch_2 if they both have matching configurations).
Any idea which regression use cases above was needed? We shall develop the next regression functionalities based on your inputs.
Best regards,
Yeoh Ee Peng
-----Original Message-----
From: Yeoh, Ee Peng
Sent: Tuesday, January 22, 2019 5:42 PM
To: [email protected]
Cc: Yeoh, Ee Peng <[email protected]>
Subject: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file.
These scripts were developed as a test result tools to manage these testresults.json file.
Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json files to a target file.
Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository.
Using the "regression" operation, user can perform regression analysis on testresults.json files specified.
These resultstool operations expect the testresults.json file to use the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}To use these scripts, first source oe environment, then run the entry point script to look for help.
$ resultstool
To store test result from oeqa automated tests, execute the below
$ resultstool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resultstool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resultstool report <source_dir>
To perform regression analysis, execute the below
$ resultstool regression <base_result_file> <target_result_file>
[YOCTO# 13012]
[YOCTO# 12654]
--- meta/lib/oeqa/files/testresults/testresults.json | 40 ++++++ meta/lib/oeqa/selftest/cases/resultstooltests.py | 104 ++++++++++++++++ scripts/lib/resultstool/__init__.py | 0 scripts/lib/resultstool/merge.py | 71 +++++++++++ scripts/lib/resultstool/regression.py | 134 +++++++++++++++++++++ scripts/lib/resultstool/report.py | 122 +++++++++++++++++++ scripts/lib/resultstool/resultsutils.py | 47 ++++++++ scripts/lib/resultstool/store.py | 110 +++++++++++++++++ .../resultstool/template/test_report_full_text.txt | 35 ++++++ scripts/resultstool | 84 +++++++++++++ 10 files changed, 747 insertions(+) create mode 100644 meta/lib/oeqa/files/testresults/testresults.json create mode 100644 meta/lib/oeqa/selftest/cases/resultstooltests.py create mode 100644 scripts/lib/resultstool/__init__.py create mode 100644 scripts/lib/resultstool/merge.py create mode 100644 scripts/lib/resultstool/regression.py create mode 100644 scripts/lib/resultstool/report.py create mode 100644 scripts/lib/resultstool/resultsutils.py create mode 100644 scripts/lib/resultstool/store.py create mode 100644 scripts/lib/resultstool/template/test_report_full_text.txt create mode 100755 scripts/resultstool
meta/lib/oeqa/files/testresults/testresults.json new file +40 -0
@@ -0,0 +1,40 @@ +{ + "runtime_core-image-minimal_qemuarm_20181225195701": { + "configuration": { + "DISTRO": "poky", + "HOST_DISTRO": "ubuntu-16.04", + "IMAGE_BASENAME": "core-image-minimal", + "IMAGE_PKGTYPE": "rpm", + "LAYERS": { + "meta": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + }, + "meta-poky": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + }, + "meta-yocto-bsp": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + } + }, + "MACHINE": "qemuarm", + "STARTTIME": "20181225195701", + "TEST_TYPE": "runtime" + }, + "result": { + "apt.AptRepoTest.test_apt_install_from_repo": { + "log": "Test requires apt to be installed", + "status": "PASSED" + }, + "buildcpio.BuildCpioTest.test_cpio": { + "log": "Test requires autoconf to be installed", + "status": "ERROR" + } + } + } +}
meta/lib/oeqa/selftest/cases/resultstooltests.py new file +104 -0
104 lines not on this page · raw message
+ output_dir = store.store_to_existing(self.logger, source_dir, output_dir, git_branch)
+ self.assertTrue(checkout_git_dir(output_dir, git_branch), msg="store to existing git repository failed:%s" %
+ output_dir)
+ store._remove_temporary_workspace_dir(output_dir)
scripts/lib/resultstool/__init__.py new file +0 -0
scripts/lib/resultstool/merge.py new file +70 -0
70 lines not on this page · raw message
scripts/lib/resultstool/regression.py new file +134 -0
134 lines not on this page · raw message
+ help='(optional) default select regression based on configurations unless target result '
+ 'id was provided')
scripts/lib/resultstool/report.py new file +122 -0
122 lines not on this page · raw message
+ 'the provided git branch assuming
+source directory was a git repository')
scripts/lib/resultstool/resultsutils.py new file +47 -0
48 lines not on this page · raw message
scripts/lib/resultstool/store.py new file +110 -0
110 lines not on this page · raw message
+existing git repository as destination')
scripts/lib/resultstool/template/test_report_full_text.txt new file +35 -0
@@ -0,0 +1,35 @@ +======================================================================= +======================================= +Test Report (Count of passed, failed, skipped group by file_dir, +result_id) +======================================================================= +======================================= +----------------------------------------------------------------------- +--------------------------------------- +{{ 'file_dir'.ljust(max_len_dir) }} | {{ +'result_id'.ljust(max_len_result_id) }} | {{ 'passed'.ljust(10) }} | {{ +'failed'.ljust(10) }} | {{ 'skipped'.ljust(10) }} +----------------------------------------------------------------------- +--------------------------------------- +{% for report in test_count_reports +|sort(attribute='test_file_dir_result_id') %} {{ +report.file_dir.ljust(max_len_dir) }} | {{ +report.result_id.ljust(max_len_result_id) }} | {{ +(report.passed|string).ljust(10) }} | {{ +(report.failed|string).ljust(10) }} | {{ +(report.skipped|string).ljust(10) }} {% endfor %} +----------------------------------------------------------------------- +--------------------------------------- + +======================================================================= +======================================= +Test Report (Percent of passed, failed, skipped group by file_dir, +result_id) +======================================================================= +======================================= +----------------------------------------------------------------------- +--------------------------------------- +{{ 'file_dir'.ljust(max_len_dir) }} | {{ +'result_id'.ljust(max_len_result_id) }} | {{ 'passed_%'.ljust(10) }} | +{{ 'failed_%'.ljust(10) }} | {{ 'skipped_%'.ljust(10) }} +-----------------------------------------------------------------------
+---------------------------------------
+{% for report in test_percent_reports +|sort(attribute='test_file_dir_result_id') %} {{ +report.file_dir.ljust(max_len_dir) }} | {{ +report.result_id.ljust(max_len_result_id) }} | {{
+(report.passed|string).ljust(10) }} | {{
+(report.failed|string).ljust(10) }} | {{
+(report.skipped|string).ljust(10) }} {% endfor %}
+-----------------------------------------------------------------------
+---------------------------------------
+
+=======================================================================
+=======================================
+Test Report (Failed test cases group by file_dir, result_id)
+=======================================================================
+=======================================
+-----------------------------------------------------------------------
+---------------------------------------+{% for report in test_count_reports +|sort(attribute='test_file_dir_result_id') %} {% if +report.failed_testcases %} file_dir | result_id : {{ report.file_dir }} +| {{ report.result_id }} {% for testcase in report.failed_testcases %}
+ {{ testcase }}
+{% endfor %}
+{% endif %}
+{% endfor %}
+----------------------------------------------------------------------- +---------------------------------------
\ No newline at end of file
scripts/resultstool new file +83 -0
83 lines not on this page · raw message
--
2.7.4
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
In reply to Yeoh Ee PengQuoted 15 lines · Yeoh Ee Peng
OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file.
These scripts were developed as a test result tools to manage these testresults.json file.
Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json files to a target file.
Quoted 7 lines
Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository.
Using the "regression" operation, user can perform regression analysis
on testresults.json files specified.
Thanks Ee Peng, this version is much improved!
As an experiment I had a local test results file and I was able to run:
$ resultstool regression /tmp/repo/testresults.json /tmp/repo/testresults.json -b sdk_core-image-sato_x86_64_qemumips_20181219111311 -t sdk_core-image-sato_x86_64_qemumips_20181219200052 Successfully loaded base test results from: /tmp/repo/testresults.json Successfully loaded target test results from: /tmp/repo/testresults.json Getting base test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219111311 Getting target test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219200052 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== assimp.BuildAssimp.test_assimp : ERROR -> PASSED ==============================End Regression==============================
I was able to clearly see that my failing test case went from ERROR to PASSED which is good. The interface and the way the information is being presented and stored are now the things we need to work on.
What is odd about the current tool/behaviour is that it sometimes expects files that are stored in a repository and sometimes expects standalone files. It doesn't feel like the work flow and the way the user would interact with this is quite correct.
You can see from my test case above that the results file I wanted to compare were both in the same repo file which I'd already merged/stored. The tool can do it by passing the file twice and specifying the IDs but its awkward and not obvious.
I'm still a little unsure how we expect to use the tool and whether the layout of the files in the git repository is how we're going to ultimately want to do this.
For example, combining all results into a single json file means we can't really show useful comparison information directly from "git diff". Would we be better with a file per results ID for example with a layout allowing "git diff" to be useful? A directory structure like:
selftest/<HOST_DISTRO>/
runtime/<DISTRO>/<MACHINE>
sdk/<MACHINE>/<SDKMACHINE>
esdk/<MACHINE>/<SDKMACHINE>
build-perf/<HOST_DISTRO>/
for example? I don't have the "right" answer here as I've not experimented with this but its something we need to think about.
As another example, the regression command can't compare the file from two different git commits easily, you have to copy one file out the git repo and then checkout the other version to compare them. This isn't easy for the user.
In summary, I think the base underlying tools here are heading the right way but the layout of the git repo and the way people are expected to use and interact with it needs a little bit more work...
Cheers,
Richard
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP,
Thanks for providing the precious inputs.
Agreed with you that the current patch that enable files based regression was not enough for other use cases.
From the information that you had shared, there are 2 more regression use cases that I have in mind:
Use case#1: directory based regression
Given that Autobuilder stored result files inside /testresults directories, user shall be able to perform the directory based regression using output from Autobuilder directly, such as below Autobuilder directories.
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86/testresults.json
Assumed that there are 2 directories storing list of result files. User shall provide these 2 directories for regression, regression scripts will first parse through all the available files inside each directories, then perform regression based on available configuration data to determine the regression pair (eg. select result_set_1 from directory#1 and result_set_x from directory#2 if they both have matching configurations).
Use case#2: git branch based regression
Given that Autobuilder stored result files inside /testresults directories, user shall first store these directories and the result files in each git branch accordingly using the existing store plugin. After that, user can used the git branch based regression to analysis the information. Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git branch accordingly
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/
Assumed that result files are stored inside git repository with specific git branch storing result files for single commit. User shall provide the 2 specific git branches for regression, regression scripts will first parse through all the available files inside each git branch, then perform regression based on available configuration data to determine the regression pair (eg. select result_set_1 from git_branch_1 and result_set_x from git_branch_2 if they both have matching configurations).
The current codebase can be easily extended to enable both use cases above. Please let me know if both use cases above are important and please give us your inputs.
Thanks,
Ee Peng
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Friday, January 25, 2019 11:44 PM
To: Yeoh, Ee Peng <[email protected]>; [email protected]
Cc: Eggleton, Paul <[email protected]>; Burton, Ross <[email protected]>
Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Quoted 15 lines · Yeoh Ee Peng
OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file.
These scripts were developed as a test result tools to manage these testresults.json file.
Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json files to a target file.
Quoted 6 lines
Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository.
Using the "regression" operation, user can perform regression analysis on testresults.json files specified.
Thanks Ee Peng, this version is much improved!
As an experiment I had a local test results file and I was able to run:
$ resultstool regression /tmp/repo/testresults.json /tmp/repo/testresults.json -b sdk_core-image-sato_x86_64_qemumips_20181219111311 -t sdk_core-image-sato_x86_64_qemumips_20181219200052 Successfully loaded base test results from: /tmp/repo/testresults.json Successfully loaded target test results from: /tmp/repo/testresults.json Getting base test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219111311 Getting target test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219200052 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== assimp.BuildAssimp.test_assimp : ERROR -> PASSED ==============================End Regression==============================
I was able to clearly see that my failing test case went from ERROR to PASSED which is good. The interface and the way the information is being presented and stored are now the things we need to work on.
What is odd about the current tool/behaviour is that it sometimes expects files that are stored in a repository and sometimes expects standalone files. It doesn't feel like the work flow and the way the user would interact with this is quite correct.
You can see from my test case above that the results file I wanted to compare were both in the same repo file which I'd already merged/stored. The tool can do it by passing the file twice and specifying the IDs but its awkward and not obvious.
I'm still a little unsure how we expect to use the tool and whether the layout of the files in the git repository is how we're going to ultimately want to do this.
For example, combining all results into a single json file means we can't really show useful comparison information directly from "git diff". Would we be better with a file per results ID for example with a layout allowing "git diff" to be useful? A directory structure like:
selftest/<HOST_DISTRO>/
runtime/<DISTRO>/<MACHINE>
sdk/<MACHINE>/<SDKMACHINE>
esdk/<MACHINE>/<SDKMACHINE>
build-perf/<HOST_DISTRO>/
for example? I don't have the "right" answer here as I've not experimented with this but its something we need to think about.
As another example, the regression command can't compare the file from two different git commits easily, you have to copy one file out the git repo and then checkout the other version to compare them. This isn't easy for the user.
In summary, I think the base underlying tools here are heading the right way but the layout of the git repo and the way people are expected to use and interact with it needs a little bit more work...
Cheers,
Richard
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi Ee Peng,
Quoted 15 lines · Ee Peng
Thanks for providing the precious inputs.
Agreed with you that the current patch that enable files based regression was not enough for other use cases.From the information that you had shared, there are 2 more regression use cases that I have in mind:
Use case#1: directory based regression
Given that Autobuilder stored result files inside /testresults directories, user shall be able to perform the directory based regression using output from Autobuilder directly, such as below Autobuilder directories.
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86/testresults.json
Quoted 7 lines
Assumed that there are 2 directories storing list of result files. User shall provide these 2 directories for regression, regression scripts will first parse through all the available files inside each directories, then perform regression based on available configuration data to determine the regression pair (eg. select
result_set_1from directory#1 andresult_set_xfrom directory#2 if they both have matching configurations).
Yes, this would be very useful. I suspect you don't need to have matching layouts, just import from all the json files in a given directory for the comparison.
This way we can support arbitrary layouts.
Quoted 21 lines
Use case#2: git branch based regression
Given that Autobuilder stored result files inside /testresults directories, user shall first store these directories and the result files in each git branch accordingly using the existing store plugin. After that, user can used the git branch based regression to analysis the information.
Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git branch accordingly
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/Assumed that result files are stored inside git repository with specific git branch storing result files for single commit. User shall provide the 2 specific git branches for regression, regression scripts will first parse through all the available files inside each git branch, then perform regression based on available configuration data to determine the regression pair (eg. select
result_set_1fromgit_branch_1andresult_set_xfromgit_branch_2if they both have matching configurations).
Quoted 3 lines
The current codebase can be easily extended to enable both use cases above. Please let me know if both use cases above are important and please give us your inputs.
Yes, I think these are two key use cases we need to support.
I think there is a third thing we also need to look at:
It would be great if there was some way of allowing some kind of templating when storing into the git repository. This way a general local log file from tmp/log/oeqa could be stored into the git repo, being split according to the layout of the repo if needed.
Our default layout could match that from the autobuilder but the repository could define a layout?
As mentioned, we also need to think about ptest. Currently the runtime ptest code has some parsing and log generation code. I think pieces like:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101
and the log_as_files() code should be moved to the reporting tool and that the test code should just generate the json results file which can later be parsed/processed as needed. I did post on the list earlier about some of the other challenges we have with the ptest data.
buildperf is the other piece of this which we'll need to think about.
Cheers,
Richard
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP,
I had submitted the v6 patches with below changes. v6:
Add regression for directory and git repository Enable regression pairing base set to multiple target sets Revise selftest testing for regression
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278486.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278487.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278488.html
For regression directory and git, it can support arbitrary directory layout. The regression will select pair of result instances for comparison based on the unique configurations data inside the result instance itself.
I have some questions regarding below items:
Quoted 5 lines
I think there is a third thing we also need to look at:
It would be great if there was some way of allowing some kind of templating when storing into the git >repository. This way a general local log file from tmp/log/oeqa could be stored into the git repo, being >split according to the layout of the repo if needed.
Our default layout could match that from the autobuilder but the repository could define a layout?
Before developing custom template layout for store git repo, I would like to understand more so that I will make sure the output will fulfill the requirement. May I know what was the intention to store result into git repo with custom layout template? Can you share the use case?
For ptest and perform tests, let me look into them. Thank you for sharing the logparser.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101
Best regards,
Yeoh Ee Peng
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Tuesday, January 29, 2019 12:29 AM
To: Yeoh, Ee Peng <[email protected]>; [email protected]
Cc: Eggleton, Paul <[email protected]>; Burton, Ross <[email protected]>
Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi Ee Peng,
Quoted 17 lines · Ee Peng
Thanks for providing the precious inputs.
Agreed with you that the current patch that enable files based regression was not enough for other use cases.From the information that you had shared, there are 2 more regression use cases that I have in mind:
Use case#1: directory based regression Given that Autobuilder stored result files inside /testresults directories, user shall be able to perform the directory based regression using output from Autobuilder directly, such as below Autobuilder directories.
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults
/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults
/qemux86/testresults.json
Quoted 7 lines
Assumed that there are 2 directories storing list of result files. User shall provide these 2 directories for regression, regression scripts will first parse through all the available files inside each directories, then perform regression based on available configuration data to determine the regression pair (eg. select
result_set_1from directory#1 andresult_set_xfrom directory#2 if they both have matching configurations).
Yes, this would be very useful. I suspect you don't need to have matching layouts, just import from all the json files in a given directory for the comparison.
This way we can support arbitrary layouts.
Quoted 23 lines
Use case#2: git branch based regression Given that Autobuilder stored result files inside /testresults directories, user shall first store these directories and the result files in each git branch accordingly using the existing store plugin.
After that, user can used the git branch based regression to analysis the information.
Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git branch accordingly
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults
/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults
/Assumed that result files are stored inside git repository with specific git branch storing result files for single commit. User shall provide the 2 specific git branches for regression, regression scripts will first parse through all the available files inside each git branch, then perform regression based on available configuration data to determine the regression pair (eg. select
result_set_1fromgit_branch_1andresult_set_xfromgit_branch_2if they both have matching configurations).
Quoted 3 lines
The current codebase can be easily extended to enable both use cases above. Please let me know if both use cases above are important and please give us your inputs.
Yes, I think these are two key use cases we need to support.
I think there is a third thing we also need to look at:
It would be great if there was some way of allowing some kind of templating when storing into the git repository. This way a general local log file from tmp/log/oeqa could be stored into the git repo, being split according to the layout of the repo if needed.
Our default layout could match that from the autobuilder but the repository could define a layout?
As mentioned, we also need to think about ptest. Currently the runtime ptest code has some parsing and log generation code. I think pieces like:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101
and the log_as_files() code should be moved to the reporting tool and that the test code should just generate the json results file which can later be parsed/processed as needed. I did post on the list earlier about some of the other challenges we have with the ptest data.
buildperf is the other piece of this which we'll need to think about.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): E0805CCB83E6104E80E61FD34E5788AE55DD30DC@PGSMSX110.gar.corp.intel.com
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP,
I looked into ptest and regression. The existing "resultstool regression" can be used to perform regression on ptest, since the testresults.json capture ptest status. I had executed regression script for the below 2 ptest testresults.json. Attached was the regression report for ptest.
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86-64-ptest/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86-64-ptest/testresults.json
The only challenges now was since ptest result set was relatively large, it was taking some time for computing the regression. Also there was this "ptestresult.rawlogs" testcase that does not contain status but the large rawlog.
I did an experiment where I run the regression on testresults.json with and without the ptest rawlog. It shows the time taken for regression was significantly larger when it contain the rawlog. I will try to improve the regression time by throw away the rawlog at runtime when perform computing. testresults.json with rawlog
Regression start time: 20190131122805
Regression end time: 20190131124425
Time taken: 16 mins 20 sec
testresults.json without rawlog
Regression start time: 20190131124512
Regression end time: 20190131124529
Time taken: 17 sec
Thanks,
Ee Peng
-----Original Message-----
From: Yeoh, Ee Peng
Sent: Tuesday, January 29, 2019 5:15 PM
To: Richard Purdie <[email protected]>; [email protected]
Cc: Eggleton, Paul <[email protected]>; Burton, Ross <[email protected]>
Subject: RE: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP,
I had submitted the v6 patches with below changes.
v6:
Add regression for directory and git repository
Enable regression pairing base set to multiple target sets
Revise selftest testing for regression http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278486.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278487.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278488.html
For regression directory and git, it can support arbitrary directory layout. The regression will select pair of result instances for comparison based on the unique configurations data inside the result instance itself.
I have some questions regarding below items:
Quoted 5 lines
I think there is a third thing we also need to look at:
It would be great if there was some way of allowing some kind of templating when storing into the git >repository. This way a general local log file from tmp/log/oeqa could be stored into the git repo, being >split according to the layout of the repo if needed.
Our default layout could match that from the autobuilder but the repository could define a layout?
Before developing custom template layout for store git repo, I would like to understand more so that I will make sure the output will fulfill the requirement. May I know what was the intention to store result into git repo with custom layout template? Can you share the use case?
For ptest and perform tests, let me look into them. Thank you for sharing the logparser.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101
Best regards,
Yeoh Ee Peng
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Tuesday, January 29, 2019 12:29 AM
To: Yeoh, Ee Peng <[email protected]>; [email protected]
Cc: Eggleton, Paul <[email protected]>; Burton, Ross <[email protected]>
Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi Ee Peng,
Quoted 17 lines · Ee Peng
Thanks for providing the precious inputs.
Agreed with you that the current patch that enable files based regression was not enough for other use cases.From the information that you had shared, there are 2 more regression use cases that I have in mind:
Use case#1: directory based regression Given that Autobuilder stored result files inside /testresults directories, user shall be able to perform the directory based regression using output from Autobuilder directly, such as below Autobuilder directories.
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults
/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults
/qemux86/testresults.json
Quoted 7 lines
Assumed that there are 2 directories storing list of result files. User shall provide these 2 directories for regression, regression scripts will first parse through all the available files inside each directories, then perform regression based on available configuration data to determine the regression pair (eg. select
result_set_1from directory#1 andresult_set_xfrom directory#2 if they both have matching configurations).
Yes, this would be very useful. I suspect you don't need to have matching layouts, just import from all the json files in a given directory for the comparison.
This way we can support arbitrary layouts.
Quoted 23 lines
Use case#2: git branch based regression Given that Autobuilder stored result files inside /testresults directories, user shall first store these directories and the result files in each git branch accordingly using the existing store plugin.
After that, user can used the git branch based regression to analysis the information.
Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git branch accordingly
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults
/
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults
/Assumed that result files are stored inside git repository with specific git branch storing result files for single commit. User shall provide the 2 specific git branches for regression, regression scripts will first parse through all the available files inside each git branch, then perform regression based on available configuration data to determine the regression pair (eg. select
result_set_1fromgit_branch_1andresult_set_xfromgit_branch_2if they both have matching configurations).
Quoted 3 lines
The current codebase can be easily extended to enable both use cases above. Please let me know if both use cases above are important and please give us your inputs.
Yes, I think these are two key use cases we need to support.
I think there is a third thing we also need to look at:
It would be great if there was some way of allowing some kind of templating when storing into the git repository. This way a general local log file from tmp/log/oeqa could be stored into the git repo, being split according to the layout of the repo if needed.
Our default layout could match that from the autobuilder but the repository could define a layout?
As mentioned, we also need to think about ptest. Currently the runtime ptest code has some parsing and log generation code. I think pieces like:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101
and the log_as_files() code should be moved to the reporting tool and that the test code should just generate the json results file which can later be parsed/processed as needed. I did post on the list earlier about some of the other challenges we have with the ptest data.
buildperf is the other piece of this which we'll need to think about.
Cheers,
Richard
22,793 lines not shown · raw message
Part of this message is not shown here. View the raw message.
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Quoted 26 lines · Ee Peng
Hi RP,
I looked into ptest and regression. The existing "resultstool regression" can be used to perform regression on ptest, since the testresults.json capture ptest status. I had executed regression script for the below 2 ptest testresults.json. Attached was the regression report for ptest.
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86-64-ptest/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86-64-ptest/testresults.jsonThe only challenges now was since ptest result set was relatively large, it was taking some time for computing the regression. Also there was this "ptestresult.rawlogs" testcase that does not contain status but the large rawlog.
I did an experiment where I run the regression on testresults.json with and without the ptest rawlog. It shows the time taken for regression was significantly larger when it contain the rawlog. I will try to improve the regression time by throw away the rawlog at runtime when perform computing.
testresults.json with rawlog
Regression start time: 20190131122805
Regression end time: 20190131124425
Time taken: 16 mins 20 sec
Quoted 4 lines
testresults.json without rawlog
Regression start time: 20190131124512
Regression end time: 20190131124529
Time taken: 17 sec
Analysing the rawlog makes no sense so the tool needs to simply ignore that. 16 minutes is far too long!
I've just merged some changes which mean there are probably some other sections it will need to ignore now too since the logs are now being split out per ptest (section). I've left rawlogs in as its useful for debugging but once the section splits are working we could remove it.
This adds in timing data so we know how long each ptest took to run (in seconds), it also adds in exit code and timeout data. These all complicate the regression analysis but the fact that lttng has been timing out (for example) has been overlooked until now and shows we need to analyse these things.
I'm considering whether we should have a command in resulttool which takes json data and writes it out in a "filesystem" form.
The code in logparser.py already has a rudimentary version of this for ptest data. It could be extended to write out a X.log for each ptest based on the split out data and maybe duration and timeout information in some form too.
The idea behind flat filesystem representations of the data is that a user can more easily explore or compare them, they also show up well in git.
Its also worth thinking about how we'll end up using this. testresult will get called at the end of builds (particularly) release builds and we'll want it to generate a QA report for the automated test data. The autobuilder will likely put an http link in the "release build ready" email to an html like report stored alongside the testresults json files.
I'm still trying to figure out how to make this all fit together and allow automated comparisons but the build performance data would also fit into this (and already has html reports).
Cheers,
Richard
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP,
I had executed the runtime/ptest using the latest master with the latest changes to understand the new improvement.
For now, the resulttool regression will ignore both 'ptestresult.rawlogs' & 'ptestresult.sections' as current regression operation focuses on comparing the "status" differences and it does not need the log as well as the new section information. By ignoring both 'ptestresult.rawlogs' & 'ptestresult.sections' , the regression time was optimized to seconds instead of minutes for ptest.
For additional information inside 'ptestresult.sections', do we need similar regression? Any idea which data inside 'ptestresult.sections' will be useful for regression?
Currently, resulttool regression will only print text based report, if html report was needed, it can be extend by using jinja2 framework. Do we need html report for this regression? Any requirement for the html report?
http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278971.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278972.html
http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278973.html
Thanks,
Yeoh Ee Peng
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Friday, February 1, 2019 7:40 AM
To: Yeoh, Ee Peng <[email protected]>; '[email protected]' <[email protected]>
Cc: Eggleton, Paul <[email protected]>; Burton, Ross <[email protected]>
Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Quoted 28 lines · Ee Peng
Hi RP,
I looked into ptest and regression. The existing "resultstool regression" can be used to perform regression on ptest, since the testresults.json capture ptest status. I had executed regression script for the below 2 ptest testresults.json. Attached was the regression report for ptest.
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults
/qemux86-64-ptest/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults
/qemux86-64-ptest/testresults.jsonThe only challenges now was since ptest result set was relatively large, it was taking some time for computing the regression. Also there was this "ptestresult.rawlogs" testcase that does not contain status but the large rawlog.
I did an experiment where I run the regression on testresults.json with and without the ptest rawlog. It shows the time taken for regression was significantly larger when it contain the rawlog. I will try to improve the regression time by throw away the rawlog at runtime when perform computing.
testresults.json with rawlog
Regression start time: 20190131122805
Regression end time: 20190131124425
Time taken: 16 mins 20 sec
Quoted 4 lines
testresults.json without rawlog
Regression start time: 20190131124512
Regression end time: 20190131124529
Time taken: 17 sec
Analysing the rawlog makes no sense so the tool needs to simply ignore that. 16 minutes is far too long!
I've just merged some changes which mean there are probably some other sections it will need to ignore now too since the logs are now being split out per ptest (section). I've left rawlogs in as its useful for debugging but once the section splits are working we could remove it.
This adds in timing data so we know how long each ptest took to run (in seconds), it also adds in exit code and timeout data. These all complicate the regression analysis but the fact that lttng has been timing out (for example) has been overlooked until now and shows we need to analyse these things.
I'm considering whether we should have a command in resulttool which takes json data and writes it out in a "filesystem" form.
The code in logparser.py already has a rudimentary version of this for ptest data. It could be extended to write out a X.log for each ptest based on the split out data and maybe duration and timeout information in some form too.
The idea behind flat filesystem representations of the data is that a user can more easily explore or compare them, they also show up well in git.
Its also worth thinking about how we'll end up using this. testresult will get called at the end of builds (particularly) release builds and we'll want it to generate a QA report for the automated test data. The autobuilder will likely put an http link in the "release build ready" email to an html like report stored alongside the testresults json files.
I'm still trying to figure out how to make this all fit together and allow automated comparisons but the build performance data would also fit into this (and already has html reports).
Cheers,
Richard
Re: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
In reply to Yeoh Ee PengAs a further example of this I wanted to experiment with a real world example. We have:
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86/testresults.json
https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86/testresults.json
which are all results files for the same thing from 2.6.1, 2.7M1 and 2.7M2. I also have one for 2.6.0 locally but its the same as 2.6.1.
Running this gives:
resultstool regression qemux86-2.6.json qemux86-2.7-M2.json
Successfully loaded base test results from: qemux86-2.6.json
Successfully loaded target test results from: qemux86-2.7-M2.json
Getting regression for base=runtime_core-image-minimal_qemux86_20181029181546 target=runtime_core-image-minimal_qemux86_20190124185230 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
==============================End Regression==============================
Getting regression for base=runtime_core-image-sato-sdk_qemux86_20181029181546 target=runtime_core-image-sato-sdk_qemux86_20190124185230 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
==============================End Regression==============================
Getting regression for base=runtime_core-image-sato_qemux86_20181029181546 target=runtime_core-image-sato_qemux86_20190124185230 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
==============================End Regression==============================
Getting regression for base=sdk_core-image-sato_i686_qemux86_20181029190813 target=sdk_core-image-sato_i686_qemux86_20190124185230 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
python.PythonTest.test_python3 : UNKNOWN -> None
==============================End Regression==============================
Getting regression for base=sdk_core-image-sato_x86_64_qemux86_20181029192305 target=sdk_core-image-sato_x86_64_qemux86_20190124192316 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
python.PythonTest.test_python3 : UNKNOWN -> None
==============================End Regression==============================
Getting regression for base=sdkext_core-image-minimal_x86_64_qemux86_20181029192305 target=sdkext_core-image-minimal_x86_64_qemux86_20190124192316 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
python.PythonTest.test_python3 : PASSED -> None
==============================End Regression==============================
Getting regression for base=sdkext_core-image-sato_x86_64_qemux86_20181029192305 target=sdkext_core-image-sato_x86_64_qemux86_20190124192316 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
python.PythonTest.test_python3 : PASSED -> None
==============================End Regression==============================
which shows that:
a) In the case of no regressions it may be better simply print that rather than a header and empty body
b) It does highlight that the "python3" test somehow disappeared
c) Its hard to know that in fact another python3 test started passing.
Maybe we also need a compare command to show "new" results to make it easier to investigate these things?
For interest I ran the comparision backwards:
resultstool regression qemux86-2.7-M2.json qemux86-2.6.json
Successfully loaded base test results from: qemux86-2.7-M2.json
Successfully loaded target test results from: qemux86-2.6.json
Getting regression for base=runtime_core-image-minimal_qemux86_20190124185230 target=runtime_core-image-minimal_qemux86_20181029181546 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
apt.AptRepoTest.test_apt_install_from_repo : SKIPPED -> None
buildcpio.BuildCpioTest.test_cpio : SKIPPED -> None
buildgalculator.GalculatorTest.test_galculator : SKIPPED -> None
buildlzip.BuildLzipTest.test_lzip : SKIPPED -> None
connman.ConnmanTest.test_connmand_help : SKIPPED -> None
connman.ConnmanTest.test_connmand_running : SKIPPED -> None
date.DateTest.test_date : SKIPPED -> None
df.DfTest.test_df : SKIPPED -> None
dnf.DnfBasicTest.test_dnf_help : SKIPPED -> None
dnf.DnfBasicTest.test_dnf_history : SKIPPED -> None
dnf.DnfBasicTest.test_dnf_info : SKIPPED -> None
dnf.DnfBasicTest.test_dnf_search : SKIPPED -> None
dnf.DnfBasicTest.test_dnf_version : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_exclude : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_install : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_install_dependency : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_install_from_disk : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_install_from_http : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_installroot : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_makecache : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_reinstall : SKIPPED -> None
dnf.DnfRepoTest.test_dnf_repoinfo : SKIPPED -> None
gcc.GccCompileTest.test_gcc_compile : SKIPPED -> None
gcc.GccCompileTest.test_gpp2_compile : SKIPPED -> None
gcc.GccCompileTest.test_gpp_compile : SKIPPED -> None
gcc.GccCompileTest.test_make : SKIPPED -> None
gi.GObjectIntrospectionTest.test_python : SKIPPED -> None
kernelmodule.KernelModuleTest.test_kernel_module : SKIPPED -> None
ldd.LddTest.test_ldd : SKIPPED -> None
logrotate.LogrotateTest.test_1_logrotate_setup : SKIPPED -> None
logrotate.LogrotateTest.test_2_logrotate : SKIPPED -> None
oe_syslog.SyslogTest.test_syslog_running : SKIPPED -> None
oe_syslog.SyslogTestConfig.test_syslog_logger : SKIPPED -> None
oe_syslog.SyslogTestConfig.test_syslog_restart : SKIPPED -> None
oe_syslog.SyslogTestConfig.test_syslog_startup_config : SKIPPED -> None
opkg.OpkgRepoTest.test_opkg_install_from_repo : SKIPPED -> None
pam.PamBasicTest.test_pam : SKIPPED -> None
parselogs.ParseLogsTest.test_parselogs : SKIPPED -> None
perl.PerlTest.test_perl_works : SKIPPED -> None
ptest.PtestRunnerTest.test_ptestrunner : SKIPPED -> None
python.PythonTest.test_python3 : SKIPPED -> None
rpm.RpmBasicTest.test_rpm_help : SKIPPED -> None
rpm.RpmBasicTest.test_rpm_query : SKIPPED -> None
rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size : SKIPPED -> None rpm.RpmInstallRemoveTest.test_rpm_install : SKIPPED -> None
rpm.RpmInstallRemoveTest.test_rpm_query_nonroot : SKIPPED -> None
rpm.RpmInstallRemoveTest.test_rpm_remove : SKIPPED -> None
scp.ScpTest.test_scp_file : SKIPPED -> None
ssh.SSHTest.test_ssh : SKIPPED -> None
stap.StapTest.test_stap : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_basic : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_failed : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_list : SKIPPED -> None
systemd.SystemdJournalTests.test_systemd_boot_time : SKIPPED -> None
systemd.SystemdJournalTests.test_systemd_journal : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_disable_enable : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_status : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_stop_start : SKIPPED -> None
xorg.XorgTest.test_xorg_running : SKIPPED -> None
==============================End Regression==============================
Getting regression for base=runtime_core-image-sato-sdk_qemux86_20190124185230 target=runtime_core-image-sato-sdk_qemux86_20181029181546 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
apt.AptRepoTest.test_apt_install_from_repo : SKIPPED -> None
opkg.OpkgRepoTest.test_opkg_install_from_repo : SKIPPED -> None
pam.PamBasicTest.test_pam : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_basic : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_failed : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_list : SKIPPED -> None
systemd.SystemdJournalTests.test_systemd_boot_time : SKIPPED -> None
systemd.SystemdJournalTests.test_systemd_journal : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_disable_enable : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_status : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_stop_start : SKIPPED -> None
==============================End Regression==============================
Getting regression for base=runtime_core-image-sato_qemux86_20190124185230 target=runtime_core-image-sato_qemux86_20181029181546 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
apt.AptRepoTest.test_apt_install_from_repo : SKIPPED -> None
buildcpio.BuildCpioTest.test_cpio : SKIPPED -> None
buildgalculator.GalculatorTest.test_galculator : SKIPPED -> None
buildlzip.BuildLzipTest.test_lzip : SKIPPED -> None
gcc.GccCompileTest.test_gcc_compile : SKIPPED -> None
gcc.GccCompileTest.test_gpp2_compile : SKIPPED -> None
gcc.GccCompileTest.test_gpp_compile : SKIPPED -> None
gcc.GccCompileTest.test_make : SKIPPED -> None
kernelmodule.KernelModuleTest.test_kernel_module : SKIPPED -> None
ldd.LddTest.test_ldd : SKIPPED -> None
logrotate.LogrotateTest.test_1_logrotate_setup : SKIPPED -> None
logrotate.LogrotateTest.test_2_logrotate : SKIPPED -> None
opkg.OpkgRepoTest.test_opkg_install_from_repo : SKIPPED -> None
pam.PamBasicTest.test_pam : SKIPPED -> None
perl.PerlTest.test_perl_works : PASSED -> None
stap.StapTest.test_stap : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_basic : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_failed : SKIPPED -> None
systemd.SystemdBasicTests.test_systemd_list : SKIPPED -> None
systemd.SystemdJournalTests.test_systemd_boot_time : SKIPPED -> None
systemd.SystemdJournalTests.test_systemd_journal : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_disable_enable : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_status : SKIPPED -> None
systemd.SystemdServiceTests.test_systemd_stop_start : SKIPPED -> None
==============================End Regression==============================
Getting regression for base=sdk_core-image-sato_i686_qemux86_20190124185230 target=sdk_core-image-sato_i686_qemux86_20181029190813 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
buildepoxy.EpoxyTest.test_epoxy : PASSED -> None
python.Python2Test.test_python2 : SKIPPED -> None
python.Python3Test.test_python3 : PASSED -> None
==============================End Regression==============================
Getting regression for base=sdk_core-image-sato_x86_64_qemux86_20190124192316 target=sdk_core-image-sato_x86_64_qemux86_20181029192305 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
buildepoxy.EpoxyTest.test_epoxy : PASSED -> None
python.Python2Test.test_python2 : SKIPPED -> None
python.Python3Test.test_python3 : PASSED -> None
==============================End Regression==============================
Getting regression for base=sdkext_core-image-minimal_x86_64_qemux86_20190124192316 target=sdkext_core-image-minimal_x86_64_qemux86_20181029192305 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
buildepoxy.EpoxyTest.test_epoxy : SKIPPED -> None
python.Python2Test.test_python2 : SKIPPED -> None
python.Python3Test.test_python3 : SKIPPED -> None
==============================End Regression==============================
Getting regression for base=sdkext_core-image-sato_x86_64_qemux86_20190124192316 target=sdkext_core-image-sato_x86_64_qemux86_20181029192305 ============================Start Regression============================
Only print regression if base status not equal target
<test case> : <base status> -> <target status>
========================================================================
buildepoxy.EpoxyTest.test_epoxy : SKIPPED -> None
python.Python2Test.test_python2 : SKIPPED -> None
python.Python3Test.test_python3 : SKIPPED -> None
==============================End Regression==============================
and this shows there were a number of things which started showing up in the test results files, mostly more skips being logged which is understandable given the chanegs we made in recent versions of the QA code.
This does have a real world use in that we have been trying to figure out if the ADT test result changes in 2.6.1 are a regression. Given the data I've seen, I suspect the answer is now "no, its not a regression, we just have some test improvements in 2.7 which we may want to backport".
Cheers,
Richard
PATCH v5 2/2 scripts/resultstool: enable manual execution and result creation
In reply to Yeoh Ee PengFrom: Mazliana <[email protected]>
Integrated “manualexecution” operation to test-case-mgmt scripts. Manual execution script is a helper script to execute all manual test cases in baseline command, which consists of user guideline steps and the expected results. The last step will ask user to provide their input to execute result. The input options are passed/failed/blocked/skipped status. The result given will be written in testresults.json including log error from the user input and configuration if there is any.The output test result for json file is created by using OEQA library.
The configuration part is manually key-in by the user. The system allow user to specify how many configuration they want to add and they need to define the required configuration name and value pair. In QA perspective, "configuration" means the test environments and parameters used during QA setup before testing can be carry out. Example of configurations: image used for boot up, host machine distro used, poky configurations, etc.
The purpose of adding the configuration is to standardize the output test result format between automation and manual execution.
To use these scripts, first source oe environment, then run the entry point script to look for help.
$ resultstool
To execute manual test cases, execute the below
$ resultstool manualexecution <manualjsonfile>
By default testresults.json store in <build_dir>/tmp/log/manual/
[YOCTO #12651]
--- scripts/lib/resultstool/manualexecution.py | 137 +++++++++++++++++++++++++++++ scripts/resultstool | 8 ++ 2 files changed, 145 insertions(+) create mode 100644 scripts/lib/resultstool/manualexecution.py
scripts/lib/resultstool/manualexecution.py new file +137 -0
@@ -0,0 +1,137 @@ +# test case management tool - manual execution from testopia test cases +# +# Copyright (c) 2018, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +import argparse +import json +import os +import sys +import datetime +import re +from oeqa.core.runner import OETestResultJSONHelper +from resultstool.resultsutils import load_json_file + +class ManualTestRunner(object): + def __init__(self): + self.jdata = '' + self.test_module = '' + self.test_suite = '' + self.test_cases = '' + self.configuration = '' + self.starttime = '' + self.result_id = '' + self.write_dir = '' + + def _get_testcases(self, file): + self.jdata = load_json_file(file) + self.test_cases = [] + self.test_module = self.jdata[0]['test']['@alias'].split('.', 2)[0] + self.test_suite = self.jdata[0]['test']['@alias'].split('.', 2)[1] + for i in self.jdata: + self.test_cases.append(i['test']['@alias'].split('.', 2)[2]) + + def _get_input(self, config): + while True: + output = input('{} = '.format(config)) + if re.match('^[a-zA-Z0-9_]+$', output): + break + print('Only alphanumeric and underscore are allowed. Please try again') + return output + + def _create_config(self): + self.configuration = {} + while True: + try: + conf_total = int(input('\nPlease provide how many configuration you want to save \n')) + break + except ValueError: + print('Invalid input. Please provide input as a number not character.') + for i in range(conf_total): + print('---------------------------------------------') + print('This is configuration #%s ' % (i + 1) + '. Please provide configuration name and its value') + print('---------------------------------------------') + name_conf = self._get_input('Configuration Name') + value_conf = self._get_input('Configuration Value') + print('---------------------------------------------\n') + self.configuration[name_conf.upper()] = value_conf + current_datetime = datetime.datetime.now() + self.starttime = current_datetime.strftime('%Y%m%d%H%M%S') + self.configuration['STARTTIME'] = self.starttime + self.configuration['TEST_TYPE'] = self.test_module + + def _create_result_id(self): + self.result_id = 'manual_' + self.test_module + '_' + self.starttime + + def _execute_test_steps(self, test_id): + test_result = {} + testcase_id = self.test_module + '.' + self.test_suite + '.' + self.test_cases[test_id] + total_steps = len(self.jdata[test_id]['test']['execution'].keys()) + print('------------------------------------------------------------------------') + print('Executing test case:' + '' '' + self.test_cases[test_id]) + print('------------------------------------------------------------------------') + print('You have total ' + str(total_steps) + ' test steps to be executed.') + print('------------------------------------------------------------------------\n') + for step in sorted((self.jdata[test_id]['test']['execution']).keys()): + print('Step %s: ' % step + self.jdata[test_id]['test']['execution']['%s' % step]['action']) + print('Expected output: ' + self.jdata[test_id]['test']['execution']['%s' % step]['expected_results']) + done = input('\nPlease press ENTER when you are done to proceed to next step.\n') + while True: + done = input('\nPlease provide test results: (P)assed/(F)ailed/(B)locked/(S)kipped? \n') + done = done.lower() + result_types = {'p':'PASSED', + 'f':'FAILED', + 'b':'BLOCKED', + 's':'SKIPPED'} + if done in result_types: + for r in result_types: + if done == r: + res = result_types[r] + if res == 'FAILED': + log_input = input('\nPlease enter the error and the description of the log: (Ex:log:211 Error Bitbake)\n') + test_result.update({testcase_id: {'status': '%s' % res, 'log': '%s' % log_input}}) + else: + test_result.update({testcase_id: {'status': '%s' % res}}) + break + print('Invalid input!') + return test_result + + def _create_write_dir(self): + basepath = os.environ['BUILDDIR'] + self.write_dir = basepath + '/tmp/log/manual/' + + def run_test(self, file): + self._get_testcases(file) + self._create_config() + self._create_result_id() + self._create_write_dir() + test_results = {} + print('\nTotal number of test cases in this test suite: ' + '%s\n' % len(self.jdata)) + for i in range(0, len(self.jdata)): + test_result = self._execute_test_steps(i) + test_results.update(test_result) + return self.configuration, self.result_id, self.write_dir, test_results + +def manualexecution(args, logger): + testrunner = ManualTestRunner() + get_configuration, get_result_id, get_write_dir, get_test_results = testrunner.run_test(args.file) + resultjsonhelper = OETestResultJSONHelper() + resultjsonhelper.dump_testresult_file(get_write_dir, get_configuration, get_result_id, + get_test_results) + return 0 + +def register_commands(subparsers): + """Register subcommands from this plugin""" + parser_build = subparsers.add_parser('manualexecution', help='Helper script for results populating during manual test execution.', + description='Helper script for results populating during manual test execution. You can find manual test case JSON file in meta/lib/oeqa/manual/', + group='manualexecution') + parser_build.set_defaults(func=manualexecution) + parser_build.add_argument('file', help='Specify path to manual test case JSON file.Note: Please use \"\" to encapsulate the file path.')
scripts/resultstool +8 -0
@@ -17,6 +17,11 @@ # To perform regression analysis, execute the below # $ resultstool regression <base_result_file> <target_result_file> # +# To execute manual test cases, execute the below +# $ resultstool manualexecution <manualjsonfile> +# +# By default testresults.json for manualexecution store in <build>/tmp/log/manual/ +# # Copyright (c) 2019, Intel Corporation. # # This program is free software; you can redistribute it and/or modify it
@@ -42,6 +47,7 @@ import resultstool.merge import resultstool.store import resultstool.regression import resultstool.report +import resultstool.manualexecution logger = scriptutils.logger_create('resultstool') def _validate_user_input_arguments(args):
@@ -58,6 +64,8 @@ def main(): parser.add_argument('-q', '--quiet', help='print only errors', action='store_true') subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='<subcommand>') subparsers.required = True + subparsers.add_subparser_group('manualexecution', 'manual test cases', 300) + resultstool.manualexecution.register_commands(subparsers) subparsers.add_subparser_group('setup', 'setup', 200) resultstool.merge.register_commands(subparsers) resultstool.store.register_commands(subparsers)
Signature · 1 line
2.7.4