Skip to contentSkip to conversation

revive runtime/cases/_ptest.py

RFC PATCH In review Robert Yang 2 participants 9 messages Jul 19, 2017 – Jul 24, 2017

lore
Share & tools
lore
#0 lore raw

cover revive runtime/cases/_ptest.py

Hello,

These patches can make ptest test case work, RP has suggested we write a tool to do the regression check on ptest result, I think that the use case is like:

$ bitbake <image> -ctestiamge # Suppose we add ptest to default test cases in the future
# Upgrade a recipe form V1.0 to V1.1
$ bitbake <image> -ctestiamge # Run the test again

Then the regression check tool can report what's different (passed, failed, skipped) between V1.0 and V1.1.

Currently, I'm not sure about where to save the ptest results, I saved it to ${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/testimage/ptest_log

But it will be removed after -cclean, then no regression check can be made, so I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no objections, and make -cclean not remove them. (Or only keep the latest 2 results).

And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression check among different build directories, and runtime/cases/_ptest.py can invoke it.
2) Add it to runtime/cases/_ptest.py directly
3) Add it to buildhistory

I prefer the first one, please feel free to give your comments.

// Robert

The following changes since commit ef68005a8c527e9b1d05b7769f0ec8ebe9ec3f91:

webkitgtk: Upgrade to 2.16.5 (2017-07-17 13:49:04 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/ptest
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ptest

Robert Yang (5):
oeqa/targetcontrol.py: simplify checking for qemu_use_kvm runtime/cases/_ptest.py: revive it oeqa/utils/logparser.py: add skip status runtime/cases/_ptest.py: add skip status runtime/cases/_ptest.py: rename it to ptest.py

 meta/lib/oeqa/runtime/cases/_ptest.py | 103 ----------------------------------
 meta/lib/oeqa/runtime/cases/ptest.py  |  77 +++++++++++++++++++++++++
 meta/lib/oeqa/targetcontrol.py        |   5 +-
 meta/lib/oeqa/utils/logparser.py      |   5 +-
 4 files changed, 82 insertions(+), 108 deletions(-)
 delete mode 100644 meta/lib/oeqa/runtime/cases/_ptest.py
 create mode 100644 meta/lib/oeqa/runtime/cases/ptest.py
Signature · 1 line
2.11.0
#1 lore raw

RFC PATCH 1/5 oeqa/targetcontrol.py: simplify checking for qemu_use_kvm

The "if qemu_use_kvm" is not needed.

---
 meta/lib/oeqa/targetcontrol.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
meta/lib/oeqa/targetcontrol.py +2 -3
index 3255e3a5c63..11e6c820e85 100644
@@ -132,9 +132,8 @@ class QemuTarget(BaseTarget):
         dump_host_cmds = d.getVar("testimage_dump_host")
         dump_dir = d.getVar("TESTIMAGE_DUMP_DIR")
         qemu_use_kvm = d.getVar("QEMU_USE_KVM")
-        if qemu_use_kvm and \
-           (qemu_use_kvm == "True" and "x86" in d.getVar("MACHINE") or \
-            d.getVar("MACHINE") in qemu_use_kvm.split()):
+        if qemu_use_kvm == "True" and "x86" in d.getVar("MACHINE") or \
+            d.getVar("MACHINE") in qemu_use_kvm.split():
             use_kvm = True
         else:
             use_kvm = False
Signature · 1 line
2.11.0
#2 lore raw

RFC PATCH 2/5 runtime/cases/_ptest.py: revive it

In reply to Robert Yang

* Make it work with current oeqa
* Skip the test if ptest is not in DISTRO_FEATURES
* Skip the test if ptest-pkgs is not in IMAGE_FEATURES
* The logs are saved to:
testimage/ptest_log -> testimage/ptest_log.<datetime>
* Will write a tool to make regression check

[YOCTO #11547]

---
 meta/lib/oeqa/runtime/cases/_ptest.py | 91 ++++++++++++-----------------------
 1 file changed, 30 insertions(+), 61 deletions(-)
meta/lib/oeqa/runtime/cases/_ptest.py +30 -61
index aaed9a5352f..034bfe98959 100644
@@ -1,28 +1,10 @@
-import os
-import shutil
-import subprocess
-
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
-from oeqa.core.decorator.data import skipIfNotDataVar, skipIfNotFeature
-from oeqa.runtime.decorator.package import OEHasPackage
-
-from oeqa.runtime.cases.dnf import DnfTest
-from oeqa.utils.logparser import *
-from oeqa.utils.httpserver import HTTPService
-
-class PtestRunnerTest(DnfTest):
+from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.utils.logparser import Lparser, Result
 
-    @classmethod
-    def setUpClass(cls):
-        rpm_deploy = os.path.join(cls.tc.td['DEPLOY_DIR'], 'rpm')
-        cls.repo_server = HTTPService(rpm_deploy, cls.tc.target.server_ip)
-        cls.repo_server.start()
-
-    @classmethod
-    def tearDownClass(cls):
-        cls.repo_server.stop()
+class PtestRunnerTest(OERuntimeTestCase):
 
     # a ptest log parser
     def parse_ptest(self, logfile):
@@ -59,45 +41,32 @@ class PtestRunnerTest(DnfTest):
         result.sort_tests()
         return result
 
-    def _install_ptest_packages(self):
-        # Get ptest packages that can be installed in the image.
-        packages_dir = os.path.join(self.tc.td['DEPLOY_DIR'], 'rpm')
-        ptest_pkgs = [pkg[:pkg.find('-ptest')+6]
-                          for _, _, filenames in os.walk(packages_dir)
-                          for pkg in filenames
-                          if 'ptest' in pkg
-                          and pkg[:pkg.find('-ptest')] in self.tc.image_packages]
-
-        repo_url = 'http://%s:%s' % (self.target.server_ip,
-                                     self.repo_server.port)
-        dnf_options = ('--repofrompath=oe-ptest-repo,%s '
-                       '--nogpgcheck '
-                       'install -y' % repo_url)
-        self.dnf('%s %s ptest-runner' % (dnf_options, ' '.join(ptest_pkgs)))
-
-    @skipIfNotFeature('package-management',
-                      'Test requires package-management to be in DISTRO_FEATURES')
-    @skipIfNotFeature('ptest',
-                      'Test requires package-management to be in DISTRO_FEATURES')
-    @skipIfNotDataVar('IMAGE_PKGTYPE', 'rpm',
-                      'RPM is not the primary package manager')
-    @OEHasPackage(['dnf'])
+    @OETestID(1600)
+    @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
+    @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     def test_ptestrunner(self):
-        self.ptest_log = os.path.join(self.tc.td['TEST_LOG_DIR'],
-                                      'ptest-%s.log' % self.tc.td['DATETIME'])
-        self._install_ptest_packages()
-
-        (runnerstatus, result) = self.target.run('/usr/bin/ptest-runner > /tmp/ptest.log 2>&1', 0)
-        #exit code is !=0 even if ptest-runner executes because some ptest tests fail.
-        self.assertTrue(runnerstatus != 127, msg="Cannot execute ptest-runner!")
-        self.target.copyFrom('/tmp/ptest.log', self.ptest_log)
-        shutil.copyfile(self.ptest_log, "ptest.log")
-
-        result = self.parse_ptest("ptest.log")
-        log_results_to_location = "./results"
-        if os.path.exists(log_results_to_location):
-            shutil.rmtree(log_results_to_location)
-        os.makedirs(log_results_to_location)
-
-        result.log_as_files(log_results_to_location, test_status = ['pass','fail'])
+        import datetime
+
+        test_log_dir = self.td['TEST_LOG_DIR']
+        # Don't use self.td['DATETIME'], it's a cached value, not up-to-date,
+        # and may cause "File exists" when re-reun.
+        datetime = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
+        ptest_log_dir_link = os.path.join(test_log_dir, 'ptest_log')
+        ptest_log_dir = '%s.%s' % (ptest_log_dir_link, datetime)
+        ptest_runner_log = os.path.join(ptest_log_dir, 'ptest-runner.log')
+
+        status, output = self.target.run('ptest-runner', 0)
+        os.makedirs(ptest_log_dir)
+        with open(ptest_runner_log, 'w') as f:
+            f.write(output)
+
+        # status !=0 is OK since some ptest tests may fail
+        self.assertTrue(status != 127, msg="Cannot execute ptest-runner!")
+
+        # Parse and save results
+        parse_result = self.parse_ptest(ptest_runner_log)
+        parse_result.log_as_files(ptest_log_dir, test_status = ['pass','fail'])
+        if os.path.exists(ptest_log_dir_link):
+            os.remove(ptest_log_dir_link)
+        os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
Signature · 1 line
2.11.0
#3 lore raw

RFC PATCH 3/5 oeqa/utils/logparser.py: add skip status

In reply to Robert Yang

Some test cases maybe skipped, let's parse it.

[YOCTO #11547]

---
 meta/lib/oeqa/utils/logparser.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
meta/lib/oeqa/utils/logparser.py +3 -2
index b377dcd2716..4355ce0b767 100644
@@ -9,7 +9,7 @@ from . import ftools
 # A parser that can be used to identify weather a line is a test result or a section statement.
 class Lparser(object):
 
-    def __init__(self, test_0_pass_regex, test_0_fail_regex, section_0_begin_regex=None, section_0_end_regex=None, **kwargs):
+    def __init__(self, test_0_pass_regex, test_0_fail_regex, test_0_skip_regex, section_0_begin_regex=None, section_0_end_regex=None, **kwargs):
         # Initialize the arguments dictionary
         if kwargs:
             self.args = kwargs
@@ -19,12 +19,13 @@ class Lparser(object):
         # Add the default args to the dictionary
         self.args['test_0_pass_regex'] = test_0_pass_regex
         self.args['test_0_fail_regex'] = test_0_fail_regex
+        self.args['test_0_skip_regex'] = test_0_skip_regex
         if section_0_begin_regex:
             self.args['section_0_begin_regex'] = section_0_begin_regex
         if section_0_end_regex:
             self.args['section_0_end_regex'] = section_0_end_regex
 
-        self.test_possible_status = ['pass', 'fail', 'error']
+        self.test_possible_status = ['pass', 'fail', 'error', 'skip']
         self.section_possible_status = ['begin', 'end']
 
         self.initialized = False
Signature · 1 line
2.11.0
#4 lore raw

RFC PATCH 4/5 runtime/cases/_ptest.py: add skip status

In reply to Robert Yang

The packages' test cases maybe skipped, check and save them.

[YOCTO #11547]

---
 meta/lib/oeqa/runtime/cases/_ptest.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/_ptest.py +6 -1
index 034bfe98959..9810bd15178 100644
@@ -10,6 +10,7 @@ class PtestRunnerTest(OERuntimeTestCase):
     def parse_ptest(self, logfile):
         parser = Lparser(test_0_pass_regex="^PASS:(.+)",
                          test_0_fail_regex="^FAIL:(.+)",
+                         test_0_skip_regex="^SKIP:(.+)",
                          section_0_begin_regex="^BEGIN: .*/(.+)/ptest",
                          section_0_end_regex="^END: .*/(.+)/ptest")
         parser.init()
@@ -38,6 +39,10 @@ class PtestRunnerTest(OERuntimeTestCase):
                     result.store(current_section, name, status)
                     continue
 
+                if line_type == 'test' and status == 'skip':
+                    result.store(current_section, name, status)
+                    continue
+
         result.sort_tests()
         return result
 
@@ -66,7 +71,7 @@ class PtestRunnerTest(OERuntimeTestCase):
 
         # Parse and save results
         parse_result = self.parse_ptest(ptest_runner_log)
-        parse_result.log_as_files(ptest_log_dir, test_status = ['pass','fail'])
+        parse_result.log_as_files(ptest_log_dir, test_status = ['pass','fail', 'skip'])
         if os.path.exists(ptest_log_dir_link):
             os.remove(ptest_log_dir_link)
         os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
Signature · 1 line
2.11.0
#5 lore raw

RFC PATCH 5/5 runtime/cases/_ptest.py: rename it to ptest.py

In reply to Robert Yang

It works now.

[YOCTO #11547]

---
 meta/lib/oeqa/runtime/cases/{_ptest.py => ptest.py} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename meta/lib/oeqa/runtime/cases/{_ptest.py => ptest.py} (100%)
meta/lib/oeqa/runtime/cases/ptest.py renamed from meta/lib/oeqa/runtime/cases/_ptest.py +0 -0
similarity index 100%
rename from meta/lib/oeqa/runtime/cases/_ptest.py
rename to meta/lib/oeqa/runtime/cases/ptest.py
Signature · 1 line
2.11.0
#6 lore raw
In reply to Robert Yang
Quoted 9 lines · Robert Yang

Hello,

These patches can make ptest test case work, RP has suggested we write a tool to do the regression check on ptest result, I think that the use case is like:

$ bitbake <image> -ctestiamge # Suppose we add ptest to default test cases in the future
# Upgrade a recipe form V1.0 to V1.1
$ bitbake <image> -ctestiamge # Run the test again
Quoted 6 lines

Then the regression check tool can report what's different (passed, failed, skipped) between V1.0 and V1.1.

Currently, I'm not sure about where to save the ptest results, I saved it to ${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/testimage/ptest_log

I still have to think about more about where to put the test result.

Quoted 8 lines

But it will be removed after -cclean, then no regression check can be made, so I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no objections, and make -cclean not remove them. (Or only keep the latest 2 results).

And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression check among different build directories, and runtime/cases/_ptest.py can invoke it.

I'm leaning to add a script called ptest-regression-check which is more flexible, the cases/ptest.py or buildhistory (also the user) can run it when needed.

// Robert

Quoted 31 lines

2) Add it to runtime/cases/_ptest.py directly
3) Add it to buildhistory

I prefer the first one, please feel free to give your comments.

// Robert

The following changes since commit ef68005a8c527e9b1d05b7769f0ec8ebe9ec3f91:

webkitgtk: Upgrade to 2.16.5 (2017-07-17 13:49:04 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/ptest
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ptest

Robert Yang (5):
oeqa/targetcontrol.py: simplify checking for qemu_use_kvm runtime/cases/_ptest.py: revive it oeqa/utils/logparser.py: add skip status runtime/cases/_ptest.py: add skip status runtime/cases/_ptest.py: rename it to ptest.py

 meta/lib/oeqa/runtime/cases/_ptest.py | 103 ----------------------------------
 meta/lib/oeqa/runtime/cases/ptest.py  |  77 +++++++++++++++++++++++++
 meta/lib/oeqa/targetcontrol.py        |   5 +-
 meta/lib/oeqa/utils/logparser.py      |   5 +-
 4 files changed, 82 insertions(+), 108 deletions(-)
 delete mode 100644 meta/lib/oeqa/runtime/cases/_ptest.py
 create mode 100644 meta/lib/oeqa/runtime/cases/ptest.py
#7 lore raw
Quoted 19 lines · Robert Yang
Quoted 17 lines · Robert Yang

Hello,

These patches can make ptest test case work, RP has suggested we write a tool to
do the regression check on ptest result, I think that the use case is like:

$ bitbake <image> -ctestiamge # Suppose we add ptest to default
test cases in the future
# Upgrade a recipe form V1.0 to V1.1
$ bitbake <image> -ctestiamge # Run the test again

Then the regression check tool can report what's different (passed, failed,
skipped) between V1.0 and V1.1.

Quoted 6 lines
Quoted 5 lines

Currently, I'm not sure about where to save the ptest results, I saved it to
${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0- r0/testimage/ptest_log

I still have to think about more about where to put the test result.

Would buildhistory make sense for these since we already have a mechanism to store results there between builds?

Quoted 12 lines
Quoted 6 lines

But it will be removed after -cclean, then no regression check can be made, so
I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no objections, and
make -cclean not remove them. (Or only keep the latest 2 results).

Quoted 6 lines

And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression
   check among different build directories, and runtime/cases/_ptest.py can invoke it.

Quoted 3 lines

I'm leaning to add a script called ptest-regression-check which is more flexible, the cases/ptest.py or buildhistory (also the user) can run it when needed.

I'd either add a script, or perhaps an option to the buildhistory-diff? Paul might have some adivce on that. I would like to understand if there is a good reason we can't use buildhistory to at least help with some of this?

Cheers,

Richard

#8 lore raw
Quoted 25 lines · Richard Purdie
Quoted 24 lines · Robert Yang
Quoted 22 lines · Robert Yang

Hello,

These patches can make ptest test case work, RP has suggested we write a tool to
do the regression check on ptest result, I think that the use case is like:

$ bitbake <image> -ctestiamge # Suppose we add ptest to default
test cases in the future
# Upgrade a recipe form V1.0 to V1.1
$ bitbake <image> -ctestiamge # Run the test again

Then the regression check tool can report what's different (passed, failed,
skipped) between V1.0 and V1.1.

Currently, I'm not sure about where to save the ptest results, I saved it to
${WORKDIR}/testimage/ptest_log atm, e.g.:
tmp/work/qemux86-poky-linux/core-image-minimal/1.0- r0/testimage/ptest_log

Quoted 4 lines
Quoted 1 line

I still have to think about more about where to put the test result.

Would buildhistory make sense for these since we already have a mechanism to store results there between builds?

Thanks, I will add it to buildhistory.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

But it will be removed after -cclean, then no regression check can be made, so
I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no objections, and
make -cclean not remove them. (Or only keep the latest 2 results).

And I'm not sure where to add the regression check/tool, maybe one of:
1) Add a separate tool in oe-core/scripts, this can make it easy to do regression
check among different build directories, and
runtime/cases/_ptest.py can invoke it.

Quoted 8 lines
Quoted 3 lines

I'm leaning to add a script called ptest-regression-check which is more flexible, the cases/ptest.py or buildhistory (also the user) can run it when needed.

I'd either add a script, or perhaps an option to the buildhistory-diff? Paul might have some adivce on that. I would like to understand if there is a good reason we can't use buildhistory to at least help with some of this?

Thanks, the script (I named it prc -- Ptest Result Compare or Ptest Regression Compile) already done, I will investigate buildhistory to see how to merge it into buildhistory.

// Robert

Quoted 5 lines

Cheers,

Richard