Python/IPython unit test changes

PATCH New Andrey Smirnov 3 participants 12 messages Jul 12, 2017 – Jul 22, 2017

lore

Share & tools

lore

Andrey Smirnov

author

2017-07-12 02:40 UTC #0 lore raw

v2 cover Python/IPython unit test changes

Hi everyone,

This is a second version of the Python/IPython related patchset.

Changes since [v1]:

- Move timeout from login() to run()

- Split patch 2/3 in original submission into multiple discreet changes

Any feedback is much appreciated!

Thanks,
Andrey Smirnov

[v1]: http://lists.busybox.net/pipermail/buildroot/2017-July/196735.html

Andrey Smirnov (6):
testing/infra/emulator: allow to specify pexpect timeout testing/tests/package/test_python: refactor TestPythonBase testing/tests/package/test_python: add TestPython3 testing/tests/package/test_python: do not hardcode interpreter name testing/tests/package/test_python: allow to change timeout testing/tests/package: add basic unit test for IPython

 support/testing/infra/emulator.py             |  6 ++--
 support/testing/tests/package/test_ipython.py | 38 ++++++++++++++++++++
 support/testing/tests/package/test_python.py  | 52 +++++++++++++++++++++------
 3 files changed, 82 insertions(+), 14 deletions(-)
 create mode 100644 support/testing/tests/package/test_ipython.py

Signature · 1 line

2.9.4

Andrey Smirnov

author

2017-07-12 02:40 UTC #1 lore raw

PATCH v2 1/6 testing/infra/emulator: allow to specify pexpect timeout

Some commands take more than 5 seconds to complete under QEMU, so add provisions to allow individual unit-test to specify different duration to avoid false negative test failures.

Signed-off-by Andrey Smirnov <[email protected]>

---
 support/testing/infra/emulator.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

support/testing/infra/emulator.py +3 -3

index a39d59b..9b079cb 100644
@@ -89,11 +89,11 @@ class Emulator(object):
             raise SystemError("Cannot login")
         self.run("dmesg -n 1")
 
-    # Run the given 'cmd' on the target
+    # Run the given 'cmd' with a 'timeout' on the target
     # return a tuple (output, exit_code)
-    def run(self, cmd):
+    def run(self, cmd, timeout=-1):
         self.qemu.sendline(cmd)
-        self.qemu.expect("# ")
+        self.qemu.expect("# ", timeout=timeout)
         # Remove double carriage return from qemu stdout so str.splitlines()
         # works as expected.
         output = self.qemu.before.replace("\r\r", "\r").splitlines()[1:]

Signature · 1 line

2.9.4

Ricardo Martincoski

2017-07-15 04:21 UTC #2 lore raw

PATCH v2 1/6 testing/infra/emulator: allow to specify pexpect timeout

Hello,

Quoted 5 lines · Andrey Smirnov

Some commands take more than 5 seconds to complete under QEMU, so add provisions to allow individual unit-test to specify different duration to avoid false negative test failures.
Signed-off-by Andrey Smirnov <[email protected]>

Reviewed-by Ricardo Martincoski <[email protected]>

[I executed all tests in the current master with this patch]

Tested-by Ricardo Martincoski <[email protected]>

[snip]

support/testing/infra/emulator.py class Emulator(object): · line 94

Quoted 2 lines

support/testing/infra/emulator.py +1 -1
-    def run(self, cmd):
+    def run(self, cmd, timeout=-1):

Indeed.
From the docs: "When the keyword argument timeout is -1 (default), then TIMEOUT will raise after the default value specified by the class timeout attribute."

Regards,
Ricardo

Thomas Petazzoni

Applied

2017-07-22 20:39 UTC #3 lore raw

PATCH v2 1/6 testing/infra/emulator: allow to specify pexpect timeout

In reply to Andrey Smirnov

Hello,

Quoted 8 lines · Andrey Smirnov

Some commands take more than 5 seconds to complete under QEMU, so add provisions to allow individual unit-test to specify different duration to avoid false negative test failures.
Signed-off-by Andrey Smirnov <[email protected]>
---
 support/testing/infra/emulator.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Andrey Smirnov

author

2017-07-12 02:40 UTC #4 lore raw

PATCH v2 2/6 testing/tests/package/test_python: refactor TestPythonBase

In reply to Andrey Smirnov

Convert TestPythonBase to a true base class that only provides code implementing various tests without definig tests themselves in a "discoverable" form.

To retain correct testing functionality, add TestPython2 derived class that uses code from TestPythonBase to define actual runnable test.

Signed-off-by Andrey Smirnov <[email protected]>

---
 support/testing/tests/package/test_python.py | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

support/testing/tests/package/test_python.py +19 -4

index 5532fb5..250827e 100644
@@ -5,31 +5,46 @@ import infra.basetest
 class TestPythonBase(infra.basetest.BRTest):
     config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
 """
-BR2_PACKAGE_PYTHON=y
 BR2_TARGET_ROOTFS_CPIO=y
 # BR2_TARGET_ROOTFS_TAR is not set
 """
-
-    def test_run(self):
+    def login(self):
         cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
         self.emulator.boot(arch="armv5",
                            kernel="builtin",
                            options=["-initrd", cpio_file])
         self.emulator.login()
-        cmd = "python --version 2>&1 | grep '^Python 2'"
+
+    def version_test(self, version):
+        cmd = "python --version 2>&1 | grep '^{}'".format(version)
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
+    def math_floor_test(self):
         cmd = "python -c 'import math; math.floor(12.3)'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
+    def libc_time_test(self):
         cmd = "python -c 'import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
         cmd += "print libc.time(None)'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
+    def zlib_test(self):
         cmd = "python -c 'import zlib'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 1)
+
+class TestPython2(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON=y
+"""
+    def test_run(self):
+        self.login()
+        self.version_test("Python 2")
+        self.math_floor_test()
+        self.libc_time_test()
+        self.zlib_test()

Signature · 1 line

2.9.4

Ricardo Martincoski

2017-07-15 04:24 UTC #5 lore raw

PATCH v2 2/6 testing/tests/package/test_python: refactor TestPythonBase

Hello,

Quoted 2 lines · Andrey Smirnov

Convert TestPythonBase to a true base class that only provides code implementing various tests without definig tests themselves in a

s/definig/defining/

Quoted 6 lines

"discoverable" form.
To retain correct testing functionality, add TestPython2 derived class that uses code from TestPythonBase to define actual runnable test.
Signed-off-by Andrey Smirnov <[email protected]>

with the typo fixed

Reviewed-by Ricardo Martincoski <[email protected]>

[I executed the test and checked the -run.log is equivalent to the old one]

Tested-by Ricardo Martincoski <[email protected]>

Below a note to be read when applying...

[snip]

Quoted 1 line

+class TestPython2(TestPythonBase):

The runnable test got renamed, so the entry needs to be updated in .gitlab-ci.yml -tests.package.test_python.TestPythonBase: *runtime_test +tests.package.test_python.TestPython2: *runtime_test

Regards,
Ricardo

Thomas Petazzoni

Applied

2017-07-22 20:41 UTC #6 lore raw

PATCH v2 2/6 testing/tests/package/test_python: refactor TestPythonBase

In reply to Andrey Smirnov

Hello,

Quoted 11 lines · Andrey Smirnov

Convert TestPythonBase to a true base class that only provides code implementing various tests without definig tests themselves in a "discoverable" form.
To retain correct testing functionality, add TestPython2 derived class that uses code from TestPythonBase to define actual runnable test.
Signed-off-by Andrey Smirnov <[email protected]>
---
 support/testing/tests/package/test_python.py | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

Applied to master, after fixing the two issues pointed by Ricardo.

BTW Ricardo: you seem to be following the changes on the test infrastructure, and contributing to it. Would you be willing to be listed in DEVELOPERS for this support/testing/ folder ? Your review and testing of the testing infrastructure patches are definitely very useful.

Thanks!

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Andrey Smirnov

author

2017-07-12 02:40 UTC #7 lore raw

PATCH v2 3/6 testing/tests/package/test_python: add TestPython3

In reply to Andrey Smirnov

Add Python3 version of TestPython2 to make sure both versions of Python get unit-tested.

Modify the code of libc_time_test() to support that change (convert the code to use Python3-style "print").

Signed-off-by Andrey Smirnov <[email protected]>

---
 support/testing/tests/package/test_python.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

support/testing/tests/package/test_python.py +15 -2

index 250827e..218235b 100644
@@ -26,9 +26,10 @@ BR2_TARGET_ROOTFS_CPIO=y
         self.assertEqual(exit_code, 0)
 
     def libc_time_test(self):
-        cmd = "python -c 'import ctypes;"
+        cmd = "python -c 'from __future__ import print_function;"
+        cmd += "import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
-        cmd += "print libc.time(None)'"
+        cmd += "print(libc.time(None))'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)

@@ -48,3 +49,15 @@ BR2_PACKAGE_PYTHON=y
         self.math_floor_test()
         self.libc_time_test()
         self.zlib_test()
+
+class TestPython3(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON3=y
+"""
+    def test_run(self):
+        self.login()
+        self.version_test("Python 3")
+        self.math_floor_test()
+        self.libc_time_test()
+        self.zlib_test()

Signature · 1 line

2.9.4

Andrey Smirnov

author

2017-07-12 02:40 UTC #8 lore raw

PATCH v2 4/6 testing/tests/package/test_python: do not hardcode interpreter name

In reply to Andrey Smirnov

In order to be able to leverage the same test code for testing different python interpreters (or wrappers around CPython) allow classes derivate of TestPythonBase to override the name of the executable used to run tests.

Signed-off-by Andrey Smirnov <[email protected]>

---
 support/testing/tests/package/test_python.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

support/testing/tests/package/test_python.py +6 -4

index 218235b..a3ec31b 100644
@@ -8,6 +8,8 @@ class TestPythonBase(infra.basetest.BRTest):
 BR2_TARGET_ROOTFS_CPIO=y
 # BR2_TARGET_ROOTFS_TAR is not set
 """
+    interpreter = "python"
+
     def login(self):
         cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
         self.emulator.boot(arch="armv5",

@@ -16,17 +18,17 @@ BR2_TARGET_ROOTFS_CPIO=y
         self.emulator.login()
 
     def version_test(self, version):
-        cmd = "python --version 2>&1 | grep '^{}'".format(version)
+        cmd = self.interpreter + " --version 2>&1 | grep '^{}'".format(version)
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
     def math_floor_test(self):
-        cmd = "python -c 'import math; math.floor(12.3)'"
+        cmd = self.interpreter + " -c 'import math; math.floor(12.3)'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
     def libc_time_test(self):
-        cmd = "python -c 'from __future__ import print_function;"
+        cmd = self.interpreter + " -c 'from __future__ import print_function;"
         cmd += "import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
         cmd += "print(libc.time(None))'"

@@ -34,7 +36,7 @@ BR2_TARGET_ROOTFS_CPIO=y
         self.assertEqual(exit_code, 0)
 
     def zlib_test(self):
-        cmd = "python -c 'import zlib'"
+        cmd = self.interpreter + " -c 'import zlib'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 1)

Signature · 1 line

2.9.4

Andrey Smirnov

author

2017-07-12 02:40 UTC #9 lore raw

PATCH v2 5/6 testing/tests/package/test_python: allow to change timeout

In reply to Andrey Smirnov

Depending on Python implementation used for testing, time it takes to perform a given test can vary pretty significantly. To accout for that allow individual test functions to specify different timeout value.

Signed-off-by Andrey Smirnov <[email protected]>

---
 support/testing/tests/package/test_python.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

support/testing/tests/package/test_python.py +8 -8

index a3ec31b..bddf09d 100644
@@ -17,27 +17,27 @@ BR2_TARGET_ROOTFS_CPIO=y
                            options=["-initrd", cpio_file])
         self.emulator.login()
 
-    def version_test(self, version):
+    def version_test(self, version, timeout=-1):
         cmd = self.interpreter + " --version 2>&1 | grep '^{}'".format(version)
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 0)
 
-    def math_floor_test(self):
+    def math_floor_test(self, timeout=-1):
         cmd = self.interpreter + " -c 'import math; math.floor(12.3)'"
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 0)
 
-    def libc_time_test(self):
+    def libc_time_test(self, timeout=-1):
         cmd = self.interpreter + " -c 'from __future__ import print_function;"
         cmd += "import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
         cmd += "print(libc.time(None))'"
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 0)
 
-    def zlib_test(self):
+    def zlib_test(self, timeout=-1):
         cmd = self.interpreter + " -c 'import zlib'"
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 1)
 
 class TestPython2(TestPythonBase):

Signature · 1 line

2.9.4

Andrey Smirnov

author

2017-07-12 02:40 UTC #10 lore raw

PATCH v2 6/6 testing/tests/package: add basic unit test for IPython

In reply to Andrey Smirnov

Signed-off-by Andrey Smirnov <[email protected]>

---
 support/testing/tests/package/test_ipython.py | 38 +++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 support/testing/tests/package/test_ipython.py

support/testing/tests/package/test_ipython.py new file +38 -0

new file mode 100644
index 0000000..dd8bf50
@@ -0,0 +1,38 @@
+import os
+
+from tests.package.test_python import TestPythonBase
+#
+# The following pythong tests are not being used here:
+#
+# - version_test: IPython does not support --version option
+#
+# - zlib_test: IPython does not return a non-zero code the way CPython
+#              does, so this test ends up being a false-negative
+#
+class TestIPythonPy2(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON=y
+BR2_PACKAGE_PYTHON_IPYTHON=y
+"""
+    interpreter = "ipython"
+
+    def test_run(self):
+        self.login()
+        self.math_floor_test(40)
+        self.libc_time_test(40)
+
+class TestIPythonPy3(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON3=y
+BR2_PACKAGE_PYTHON_IPYTHON=y
+"""
+    interpreter = "ipython"
+
+    def test_run(self):
+        self.login()
+        self.math_floor_test(40)
+        self.libc_time_test(40)
+
+

Signature · 1 line

2.9.4

Thomas Petazzoni

2017-07-22 20:56 UTC #11 lore raw

v2 cover Python/IPython unit test changes

In reply to Andrey Smirnov

Hello,

Quoted 7 lines · Andrey Smirnov

Andrey Smirnov (6):
testing/infra/emulator: allow to specify pexpect timeout testing/tests/package/test_python: refactor TestPythonBase testing/tests/package/test_python: add TestPython3 testing/tests/package/test_python: do not hardcode interpreter name testing/tests/package/test_python: allow to change timeout testing/tests/package: add basic unit test for IPython

I've applied the remaining patches 3 to 6. One thing was missing:
updating the .gitlab-ci.yml file, so I've done that.

Thanks!

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com