Skip to contentSkip to conversation

ptest-runner

PATCH Superseded Aníbal Limón 2 participants 6 messages Dec 7, 2015 – Dec 14, 2015

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover ptest-runner

These two commits adds a new ptest-runner written in python the sh one is conserved to use in tiny systems.

What's new in ptest runner:

- Monitor/timeout stdout, stderr of the test suite to avoid block indefinetly.
- Add option for change ptest root directory.
- Add option for list available tests.
- Add option for only run certain tests.

The following changes since commit d9bc1f4b9a8aff859c59dbf9bbdd6b16e99381b2:

classes/sstate.bbclass: checkhashes modify to take into account siginfo. (2015-12-02 10:55:25 -0600)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib alimon/ptest-runner
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner

Aníbal Limón (2):
ptest-runner: Add version 2.0 re-implementation in python.
ptest-runner: Add a recipe for install ptest-runner 2.0.

 .../ptest-runner/files/ptest-runner_2.0.py         | 144 +++++++++++++++++++++
 .../ptest-runner/ptest-runner_2.0.bb               |  28 ++++
 2 files changed, 172 insertions(+)
 create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
Signature · 1 line
2.1.4
#1 lore raw

PATCH 1/2 ptest-runner: Add version 2.0 re-implementation in python.

The new ptest-runner supports timeout of upstream tests executed, it looks for stdout of process and if no information is available in certain time (defaults to 5m) the process is treaty as blocked and ptest-runner kills it, this handles problems of ptest-runner being blocked indefinitly for upstream test suites.

Also an options was added for list test available, specify directory of ptests and run only certain tests.

[YOCTO #8021]

---
 .../ptest-runner/files/ptest-runner_2.0.py         | 144 +++++++++++++++++++++
 1 file changed, 144 insertions(+)
 create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py new file +144 -0
new file mode 100755
index 0000000..fc0447c
@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+
+# Ptest-runner
+#
+# Copyright (C) 2014-2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import fcntl
+import time
+
+import argparse
+import logging 
+from datetime import datetime
+
+import subprocess
+
+DEFAULT_LIBDIR='/usr/lib'
+DEFAULT_TIMEOUT_SECS = 500
+
+logging.basicConfig(format="%(message)s")
+logger = logging.getLogger()
+logger.setLevel(logging.INFO)
+
+def get_available_ptests(libdir):
+    ptests = {}
+
+    if os.path.exists(libdir) and os.path.isdir(libdir):
+        for d in os.listdir(libdir):
+            full_d = os.path.join(libdir, d)
+            if os.path.isdir(full_d) and not os.path.islink(full_d):
+                run_ptest = os.path.join(full_d, 'ptest', 'run-ptest')
+                if os.path.exists(run_ptest):
+                    ptests[d] = run_ptest
+
+    return ptests
+
+def print_ptests(ptests):
+    if ptests.keys():
+        logger.info("Available ptests:\n")
+        for p in ptests.keys():
+            logger.info("%s\t%s" % (p, ptests[p]))
+        return 0
+    else:
+        logger.error("No ptests found.\n")
+        return 1
+
+def get_ptests_to_run(available_ptests, requested_ptests):
+    if requested_ptests:
+        for rp in requested_ptests:
+            if not rp in available_ptests.keys():
+                logger.error("%s ptest isn't available." % rp)
+                return 1
+
+        for ap in available_ptests.keys():
+            if not ap in requested_ptests:
+                del available_ptests[ap]
+    return 0
+
+def run_ptests(ptests, timeout):
+    def _set_nonblocking(fp):
+        fd = process.stdout.fileno()
+        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
+        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+
+    exit_code = 0
+
+    print "START: %s" % sys.argv[0]
+    for p in ptests.keys():
+        run_ptest = ptests[p]
+
+        print datetime.strftime(datetime.now(), "%Y-%m-%dT%H:%M")
+        print "BEGIN: %s" % run_ptest
+
+        os.chdir(os.path.dirname(run_ptest))
+
+        process = subprocess.Popen([run_ptest], stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE)
+        _set_nonblocking(process.stdout)
+        _set_nonblocking(process.stderr)
+
+        sentinel = datetime.now()
+        while process.poll() is None:
+            try:
+                data = process.stdout.read()
+                sys.stdout.write(data)
+                sentinel = datetime.now()
+            except:
+                if ((datetime.now() - sentinel).total_seconds() > timeout):
+                    print "TIMEOUT: %s" % run_ptest
+                    process.kill()
+                    break
+            time.sleep(0.5)
+
+        if process.returncode:
+            exit_code = process.returncode
+
+        print "END: %s" % run_ptest
+        print datetime.strftime(datetime.now(), "%Y-%m-%dT%H:%M")
+    print "STOP: %s" % sys.argv[0]
+
+    return exit_code
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description="ptest-runner runs ptests from upstream packages",
+            add_help=False,
+            epilog="Use %(prog)s [ptest1 ptest2 ...] --help to get help on a specific command")
+    parser.add_argument('ptests', metavar='ptests', help='ptests to run',
+            nargs='*')
+    parser.add_argument('-d', '--directory', help='Directory for search ptests',
+            action='store', default=DEFAULT_LIBDIR)
+    parser.add_argument('-l', '--list', help='List available ptests',
+            action='store_true')
+    parser.add_argument('-t', '--timeout',
+            help='Timeout (secs) for use when run ptests', action='store',
+            default=DEFAULT_TIMEOUT_SECS)
+    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
+                        help='show this help message and exit')
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    ptests = get_available_ptests(global_args.directory)
+    if global_args.list:
+        exit_code = print_ptests(ptests)
+        sys.exit(exit_code)
+
+    exit_code = get_ptests_to_run(ptests, global_args.ptests)
+    if exit_code:
+        sys.exit(exit_code)
+
+    sys.exit(run_ptests(ptests, global_args.timeout))
Signature · 1 line
2.1.4
#2 lore raw

Re: [PATCH 1/2] ptest-runner: Add version 2.0 re-implementation in python.

Quoted 5 lines · Aníbal Limón

The new ptest-runner supports timeout of upstream tests executed, it looks for stdout of process and if no information is available in certain time (defaults to 5m) the process is treaty as blocked and ptest-runner kills it, this handles problems of ptest-runner being blocked indefinitly for upstream test suites.

Having the option to specify a timeout is obviously useful, but please add the ability to wait indefinitely, and consider making this the default behavior. The appropriate value for a timeout, if any, will always be a function of the particular circumstances of the test run. An indefinite wait at least gives one the opportunity to investigate a misbehaving test without racing the test harness.

Quoted 1 line

+DEFAULT_TIMEOUT_SECS = 500

The change description says the default is five minutes, but this is not five minutes :-)

#3 lore raw

Re: [PATCH 1/2] ptest-runner: Add version 2.0 re-implementation in python.

Quoted 7 lines · Nathan Lynch
Quoted 5 lines · Aníbal Limón

The new ptest-runner supports timeout of upstream tests executed, it looks for stdout of process and if no information is available in certain time (defaults to 5m) the process is treaty as blocked and ptest-runner kills it, this handles problems of ptest-runner being blocked indefinitly for upstream test suites.

Quoted 6 lines

Having the option to specify a timeout is obviously useful, but please add the ability to wait indefinitely, and consider making this the default behavior. The appropriate value for a timeout, if any, will always be a function of the particular circumstances of the test run. An indefinite wait at least gives one the opportunity to investigate a misbehaving test without racing the test harness.

I'll add the ability to wait indefinitely adding a -1 to timeout option but for default i think is better to wait certain time by default because when ptest-runner is used in automatically manner (i.e.testimage) this cause the block indefinitely.

Quoted 5 lines
Quoted 2 lines

+DEFAULT_TIMEOUT_SECS = 500

The change description says the default is five minutes, but this is not five minutes :-)

Thanks for notice it, it was a mistake :).

Quoted 2 lines
#4 lore raw

PATCH 2/2 ptest-runner: Add a recipe for install ptest-runner 2.0.

In reply to Aníbal Limón

[YOCTO #8021]

---
 .../ptest-runner/ptest-runner_2.0.bb               | 28 ++++++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
meta/recipes-support/ptest-runner/ptest-runner_2.0.bb new file +28 -0
new file mode 100644
index 0000000..64fd0df
@@ -0,0 +1,28 @@
+SUMMARY = "A python script to run all installed ptests"
+
+DESCRIPTION = "The ptest-runner package installs a ptest-runner \
+python script which loops through all installed ptest test suites and \
+runs them in sequence."
+
+HOMEPAGE = "https://wiki.yoctoproject.org/wiki/Ptest"
+SRC_URI += "file://ptest-runner_2.0.py"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \
+                    file://${COREBASE}/meta/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+INHIBIT_DEFAULT_DEPS = "1"
+RDEPENDS_${PN} = "python python-fcntl python-argparse python-logging python-datetime \
+                  python-subprocess"
+
+S = "${WORKDIR}"
+
+do_install () {
+    mkdir -p ${D}${bindir}
+    install -m 0755 ${WORKDIR}/ptest-runner_2.0.py ${D}${bindir}/ptest-runner
+}
+
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_build[noexec] = "1"
Signature · 1 line
2.1.4
#5 lore raw
In reply to Aníbal Limón

ping

Quoted 29 lines · Aníbal Limón

These two commits adds a new ptest-runner written in python the sh one is conserved to use in tiny systems.

What's new in ptest runner:

- Monitor/timeout stdout, stderr of the test suite to avoid block indefinetly.
- Add option for change ptest root directory.
- Add option for list available tests.
- Add option for only run certain tests.

The following changes since commit d9bc1f4b9a8aff859c59dbf9bbdd6b16e99381b2:

classes/sstate.bbclass: checkhashes modify to take into account siginfo. (2015-12-02 10:55:25 -0600)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib alimon/ptest-runner
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner

Aníbal Limón (2):
ptest-runner: Add version 2.0 re-implementation in python.
ptest-runner: Add a recipe for install ptest-runner 2.0.

 .../ptest-runner/files/ptest-runner_2.0.py         | 144 +++++++++++++++++++++
 .../ptest-runner/ptest-runner_2.0.bb               |  28 ++++
 2 files changed, 172 insertions(+)
 create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb