revive runtime/cases/_ptest.py
cover revive runtime/cases/_ptest.py
Hello,
These patches can make ptest test case work, RP has suggested we write a tool to do the regression check on ptest result, I think that the use case is like:
$ bitbake <image> -ctestiamge # Suppose we add ptest to default test cases in the future # Upgrade a recipe form V1.0 to V1.1 $ bitbake <image> -ctestiamge # Run the test again
Then the regression check tool can report what's different (passed, failed, skipped) between V1.0 and V1.1.
Currently, I'm not sure about where to save the ptest results, I saved it to ${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/testimage/ptest_log
But it will be removed after -cclean, then no regression check can be made, so I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no objections, and make -cclean not remove them. (Or only keep the latest 2 results).
And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression check among different build directories, and runtime/cases/_ptest.py can invoke it.
2) Add it to runtime/cases/_ptest.py directly
3) Add it to buildhistory
I prefer the first one, please feel free to give your comments.
// Robert
The following changes since commit ef68005a8c527e9b1d05b7769f0ec8ebe9ec3f91:
webkitgtk: Upgrade to 2.16.5 (2017-07-17 13:49:04 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/ptest
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ptest
Robert Yang (5):
oeqa/targetcontrol.py: simplify checking for qemu_use_kvm runtime/cases/_ptest.py: revive it oeqa/utils/logparser.py: add skip status runtime/cases/_ptest.py: add skip status runtime/cases/_ptest.py: rename it to ptest.py
meta/lib/oeqa/runtime/cases/_ptest.py | 103 ---------------------------------- meta/lib/oeqa/runtime/cases/ptest.py | 77 +++++++++++++++++++++++++ meta/lib/oeqa/targetcontrol.py | 5 +- meta/lib/oeqa/utils/logparser.py | 5 +- 4 files changed, 82 insertions(+), 108 deletions(-) delete mode 100644 meta/lib/oeqa/runtime/cases/_ptest.py create mode 100644 meta/lib/oeqa/runtime/cases/ptest.py
Signature · 1 line
2.11.0
RFC PATCH 1/5 oeqa/targetcontrol.py: simplify checking for qemu_use_kvm
The "if qemu_use_kvm" is not needed.
--- meta/lib/oeqa/targetcontrol.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
meta/lib/oeqa/targetcontrol.py +2 -3
@@ -132,9 +132,8 @@ class QemuTarget(BaseTarget): dump_host_cmds = d.getVar("testimage_dump_host") dump_dir = d.getVar("TESTIMAGE_DUMP_DIR") qemu_use_kvm = d.getVar("QEMU_USE_KVM") - if qemu_use_kvm and \ - (qemu_use_kvm == "True" and "x86" in d.getVar("MACHINE") or \ - d.getVar("MACHINE") in qemu_use_kvm.split()): + if qemu_use_kvm == "True" and "x86" in d.getVar("MACHINE") or \ + d.getVar("MACHINE") in qemu_use_kvm.split(): use_kvm = True else: use_kvm = False
Signature · 1 line
2.11.0
RFC PATCH 2/5 runtime/cases/_ptest.py: revive it
In reply to Robert Yang* Make it work with current oeqa
* Skip the test if ptest is not in DISTRO_FEATURES
* Skip the test if ptest-pkgs is not in IMAGE_FEATURES
* The logs are saved to:
testimage/ptest_log -> testimage/ptest_log.<datetime>
* Will write a tool to make regression check
[YOCTO #11547]
--- meta/lib/oeqa/runtime/cases/_ptest.py | 91 ++++++++++++----------------------- 1 file changed, 30 insertions(+), 61 deletions(-)
meta/lib/oeqa/runtime/cases/_ptest.py +30 -61
@@ -1,28 +1,10 @@ -import os -import shutil -import subprocess - from oeqa.runtime.case import OERuntimeTestCase from oeqa.core.decorator.depends import OETestDepends from oeqa.core.decorator.oeid import OETestID -from oeqa.core.decorator.data import skipIfNotDataVar, skipIfNotFeature -from oeqa.runtime.decorator.package import OEHasPackage - -from oeqa.runtime.cases.dnf import DnfTest -from oeqa.utils.logparser import * -from oeqa.utils.httpserver import HTTPService - -class PtestRunnerTest(DnfTest): +from oeqa.core.decorator.data import skipIfNotFeature +from oeqa.utils.logparser import Lparser, Result - @classmethod - def setUpClass(cls): - rpm_deploy = os.path.join(cls.tc.td['DEPLOY_DIR'], 'rpm') - cls.repo_server = HTTPService(rpm_deploy, cls.tc.target.server_ip) - cls.repo_server.start() - - @classmethod - def tearDownClass(cls): - cls.repo_server.stop() +class PtestRunnerTest(OERuntimeTestCase): # a ptest log parser def parse_ptest(self, logfile):
@@ -59,45 +41,32 @@ class PtestRunnerTest(DnfTest): result.sort_tests() return result - def _install_ptest_packages(self): - # Get ptest packages that can be installed in the image. - packages_dir = os.path.join(self.tc.td['DEPLOY_DIR'], 'rpm') - ptest_pkgs = [pkg[:pkg.find('-ptest')+6] - for _, _, filenames in os.walk(packages_dir) - for pkg in filenames - if 'ptest' in pkg - and pkg[:pkg.find('-ptest')] in self.tc.image_packages] - - repo_url = 'http://%s:%s' % (self.target.server_ip, - self.repo_server.port) - dnf_options = ('--repofrompath=oe-ptest-repo,%s ' - '--nogpgcheck ' - 'install -y' % repo_url) - self.dnf('%s %s ptest-runner' % (dnf_options, ' '.join(ptest_pkgs))) - - @skipIfNotFeature('package-management', - 'Test requires package-management to be in DISTRO_FEATURES') - @skipIfNotFeature('ptest', - 'Test requires package-management to be in DISTRO_FEATURES') - @skipIfNotDataVar('IMAGE_PKGTYPE', 'rpm', - 'RPM is not the primary package manager') - @OEHasPackage(['dnf']) + @OETestID(1600) + @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES') + @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES') @OETestDepends(['ssh.SSHTest.test_ssh']) def test_ptestrunner(self): - self.ptest_log = os.path.join(self.tc.td['TEST_LOG_DIR'], - 'ptest-%s.log' % self.tc.td['DATETIME']) - self._install_ptest_packages() - - (runnerstatus, result) = self.target.run('/usr/bin/ptest-runner > /tmp/ptest.log 2>&1', 0) - #exit code is !=0 even if ptest-runner executes because some ptest tests fail. - self.assertTrue(runnerstatus != 127, msg="Cannot execute ptest-runner!") - self.target.copyFrom('/tmp/ptest.log', self.ptest_log) - shutil.copyfile(self.ptest_log, "ptest.log") - - result = self.parse_ptest("ptest.log") - log_results_to_location = "./results" - if os.path.exists(log_results_to_location): - shutil.rmtree(log_results_to_location) - os.makedirs(log_results_to_location) - - result.log_as_files(log_results_to_location, test_status = ['pass','fail']) + import datetime + + test_log_dir = self.td['TEST_LOG_DIR'] + # Don't use self.td['DATETIME'], it's a cached value, not up-to-date, + # and may cause "File exists" when re-reun. + datetime = datetime.datetime.now().strftime('%Y%m%d%H%M%S') + ptest_log_dir_link = os.path.join(test_log_dir, 'ptest_log') + ptest_log_dir = '%s.%s' % (ptest_log_dir_link, datetime) + ptest_runner_log = os.path.join(ptest_log_dir, 'ptest-runner.log') + + status, output = self.target.run('ptest-runner', 0) + os.makedirs(ptest_log_dir) + with open(ptest_runner_log, 'w') as f: + f.write(output) + + # status !=0 is OK since some ptest tests may fail + self.assertTrue(status != 127, msg="Cannot execute ptest-runner!") + + # Parse and save results + parse_result = self.parse_ptest(ptest_runner_log) + parse_result.log_as_files(ptest_log_dir, test_status = ['pass','fail']) + if os.path.exists(ptest_log_dir_link): + os.remove(ptest_log_dir_link) + os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
Signature · 1 line
2.11.0
RFC PATCH 3/5 oeqa/utils/logparser.py: add skip status
In reply to Robert YangSome test cases maybe skipped, let's parse it.
[YOCTO #11547]
--- meta/lib/oeqa/utils/logparser.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
meta/lib/oeqa/utils/logparser.py +3 -2
@@ -9,7 +9,7 @@ from . import ftools # A parser that can be used to identify weather a line is a test result or a section statement. class Lparser(object): - def __init__(self, test_0_pass_regex, test_0_fail_regex, section_0_begin_regex=None, section_0_end_regex=None, **kwargs): + def __init__(self, test_0_pass_regex, test_0_fail_regex, test_0_skip_regex, section_0_begin_regex=None, section_0_end_regex=None, **kwargs): # Initialize the arguments dictionary if kwargs: self.args = kwargs
@@ -19,12 +19,13 @@ class Lparser(object): # Add the default args to the dictionary self.args['test_0_pass_regex'] = test_0_pass_regex self.args['test_0_fail_regex'] = test_0_fail_regex + self.args['test_0_skip_regex'] = test_0_skip_regex if section_0_begin_regex: self.args['section_0_begin_regex'] = section_0_begin_regex if section_0_end_regex: self.args['section_0_end_regex'] = section_0_end_regex - self.test_possible_status = ['pass', 'fail', 'error'] + self.test_possible_status = ['pass', 'fail', 'error', 'skip'] self.section_possible_status = ['begin', 'end'] self.initialized = False
Signature · 1 line
2.11.0
RFC PATCH 4/5 runtime/cases/_ptest.py: add skip status
In reply to Robert YangThe packages' test cases maybe skipped, check and save them.
[YOCTO #11547]
--- meta/lib/oeqa/runtime/cases/_ptest.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/_ptest.py +6 -1
@@ -10,6 +10,7 @@ class PtestRunnerTest(OERuntimeTestCase): def parse_ptest(self, logfile): parser = Lparser(test_0_pass_regex="^PASS:(.+)", test_0_fail_regex="^FAIL:(.+)", + test_0_skip_regex="^SKIP:(.+)", section_0_begin_regex="^BEGIN: .*/(.+)/ptest", section_0_end_regex="^END: .*/(.+)/ptest") parser.init()
@@ -38,6 +39,10 @@ class PtestRunnerTest(OERuntimeTestCase): result.store(current_section, name, status) continue + if line_type == 'test' and status == 'skip': + result.store(current_section, name, status) + continue + result.sort_tests() return result
@@ -66,7 +71,7 @@ class PtestRunnerTest(OERuntimeTestCase): # Parse and save results parse_result = self.parse_ptest(ptest_runner_log) - parse_result.log_as_files(ptest_log_dir, test_status = ['pass','fail']) + parse_result.log_as_files(ptest_log_dir, test_status = ['pass','fail', 'skip']) if os.path.exists(ptest_log_dir_link): os.remove(ptest_log_dir_link) os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
Signature · 1 line
2.11.0
RFC PATCH 5/5 runtime/cases/_ptest.py: rename it to ptest.py
In reply to Robert YangIt works now.
[YOCTO #11547]
---
meta/lib/oeqa/runtime/cases/{_ptest.py => ptest.py} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename meta/lib/oeqa/runtime/cases/{_ptest.py => ptest.py} (100%)meta/lib/oeqa/runtime/cases/ptest.py renamed from meta/lib/oeqa/runtime/cases/_ptest.py +0 -0
Signature · 1 line
2.11.0
Quoted 9 lines · Robert Yang
Hello,
These patches can make ptest test case work, RP has suggested we write a tool to do the regression check on ptest result, I think that the use case is like:
$ bitbake <image> -ctestiamge # Suppose we add ptest to default test cases in the future # Upgrade a recipe form V1.0 to V1.1 $ bitbake <image> -ctestiamge # Run the test again
Quoted 6 lines
Then the regression check tool can report what's different (passed, failed, skipped) between V1.0 and V1.1.
Currently, I'm not sure about where to save the ptest results, I saved it to ${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/testimage/ptest_log
I still have to think about more about where to put the test result.
Quoted 8 lines
But it will be removed after -cclean, then no regression check can be made, so I'd like to save the ptest results to
DEPLOY_DIR_IMAGEif no objections, and make -cclean not remove them. (Or only keep the latest 2 results).And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression check among different build directories, andruntime/cases/_ptest.pycan invoke it.
I'm leaning to add a script called ptest-regression-check which is more flexible, the cases/ptest.py or buildhistory (also the user) can run it when needed.
// Robert
Quoted 31 lines
2) Add it to
runtime/cases/_ptest.pydirectly
3) Add it to buildhistoryI prefer the first one, please feel free to give your comments.
// Robert
The following changes since commit
ef68005a8c527e9b1d05b7769f0ec8ebe9ec3f91:webkitgtk: Upgrade to 2.16.5 (2017-07-17 13:49:04 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contribrbt/ptest
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ptestRobert Yang (5):
oeqa/targetcontrol.py: simplify checking forqemu_use_kvmruntime/cases/_ptest.py: revive itoeqa/utils/logparser.py: add skip statusruntime/cases/_ptest.py: add skip statusruntime/cases/_ptest.py: rename it toptest.pymeta/lib/oeqa/runtime/cases/_ptest.py | 103 ---------------------------------- meta/lib/oeqa/runtime/cases/ptest.py | 77 +++++++++++++++++++++++++ meta/lib/oeqa/targetcontrol.py | 5 +- meta/lib/oeqa/utils/logparser.py | 5 +- 4 files changed, 82 insertions(+), 108 deletions(-) delete mode 100644 meta/lib/oeqa/runtime/cases/_ptest.py create mode 100644 meta/lib/oeqa/runtime/cases/ptest.py
Quoted 19 lines · Robert Yang
Quoted 17 lines · Robert Yang
Hello,
These patches can make ptest test case work, RP has suggested we write a tool to
do the regression check on ptest result, I think that the use case is like:$ bitbake <image> -ctestiamge # Suppose we add ptest to default test cases in the future # Upgrade a recipe form V1.0 to V1.1 $ bitbake <image> -ctestiamge # Run the test againThen the regression check tool can report what's different (passed, failed,
skipped) between V1.0 and V1.1.
Quoted 6 lines
Quoted 5 lines
Currently, I'm not sure about where to save the ptest results, I saved it to
${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0- r0/testimage/ptest_logI still have to think about more about where to put the test result.
Would buildhistory make sense for these since we already have a mechanism to store results there between builds?
Quoted 12 lines
Quoted 6 lines
But it will be removed after -cclean, then no regression check can be made, so
I'd like to save the ptest results toDEPLOY_DIR_IMAGEif no objections, and
make -cclean not remove them. (Or only keep the latest 2 results).Quoted 6 lines
And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression
check among different build directories, andruntime/cases/_ptest.pycan invoke it.
Quoted 3 lines
I'm leaning to add a script called ptest-regression-check which is more flexible, the
cases/ptest.pyor buildhistory (also the user) can run it when needed.
I'd either add a script, or perhaps an option to the buildhistory-diff? Paul might have some adivce on that. I would like to understand if there is a good reason we can't use buildhistory to at least help with some of this?
Cheers,
Richard
Quoted 25 lines · Richard Purdie
Quoted 24 lines · Robert Yang
Quoted 22 lines · Robert Yang
Hello,
These patches can make ptest test case work, RP has suggested we write a tool to
do the regression check on ptest result, I think that the use case is like:$ bitbake <image> -ctestiamge # Suppose we add ptest to default test cases in the future # Upgrade a recipe form V1.0 to V1.1 $ bitbake <image> -ctestiamge # Run the test againThen the regression check tool can report what's different (passed, failed,
skipped) between V1.0 and V1.1.Currently, I'm not sure about where to save the ptest results, I saved it to
${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0- r0/testimage/ptest_log
Quoted 4 lines
Quoted 1 line
I still have to think about more about where to put the test result.
Would buildhistory make sense for these since we already have a mechanism to store results there between builds?
Thanks, I will add it to buildhistory.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
But it will be removed after -cclean, then no regression check can be made, so
I'd like to save the ptest results toDEPLOY_DIR_IMAGEif no objections, and
make -cclean not remove them. (Or only keep the latest 2 results).And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression
check among different build directories, andruntime/cases/_ptest.pycan invoke it.
Quoted 8 lines
Quoted 3 lines
I'm leaning to add a script called ptest-regression-check which is more flexible, the
cases/ptest.pyor buildhistory (also the user) can run it when needed.I'd either add a script, or perhaps an option to the buildhistory-diff? Paul might have some adivce on that. I would like to understand if there is a good reason we can't use buildhistory to at least help with some of this?
Thanks, the script (I named it prc -- Ptest Result Compare or Ptest Regression Compile) already done, I will investigate buildhistory to see how to merge it into buildhistory.
// Robert
Quoted 5 lines
Cheers,
Richard