test/py: Fixes for python 3.x
v2 cover test/py: Fixes for python 3.x
This series allows test/py to run on python 3.x in addition to the already supported python 2.x. With it applied I currently see only one failure (test_ut[ut_dm_usb_flash]) which doesn't appear to be python related when running with:
$ ./test/py/test.py --bd sandbox --build
Thanks,
Paul
Paul Burton (8):
test/py: Make print statements python 3.x safe test/py: Use range() rather than xrange()
test/py: Import 'configparser' lower case to be python 3.x safe test/py: Import StringIO from io module for python 3.x test/py: Encode/decode strings for stdio
test/py: fit: Open files as binary files
test/py: hush_if_test: Use open() in place of file() test/py: vboot: Remove stderr redirect from openssl command
test/py/conftest.py | 18 +++++++++++++----- test/py/multiplexed_log.py | 4 ++-- test/py/test.py | 6 ++++-- test/py/tests/test_fit.py | 14 ++++++++------ test/py/tests/test_hush_if_test.py | 2 +- test/py/tests/test_ut.py | 2 +- test/py/tests/test_vboot.py | 4 ++-- test/py/u_boot_console_sandbox.py | 4 ++-- test/py/u_boot_spawn.py | 12 ++++++------ test/py/u_boot_utils.py | 4 ++-- 10 files changed, 41 insertions(+), 29 deletions(-)
Signature · 1 line
2.14.1
PATCH v2 1/8 test/py: Make print statements python 3.x safe
In python 3.x print must be called as a function rather than used as a statement. Update uses of print to the function call syntax in order to be python 3.x safe.
---
Changes in v2: None
test/py/conftest.py | 2 +- test/py/test.py | 6 ++++-- test/py/tests/test_fit.py | 12 +++++++----- test/py/u_boot_console_sandbox.py | 2 +- test/py/u_boot_spawn.py | 2 +- 5 files changed, 14 insertions(+), 10 deletions(-)
test/py/conftest.py +1 -1
@@ -545,7 +545,7 @@ def pytest_runtest_protocol(item, nextitem): # is fixed, if this exception still exists, it will then be logged as # part of the test's stdout. import traceback - print 'Exception occurred while logging runtest status:' + print('Exception occurred while logging runtest status:') traceback.print_exc() # FIXME: Can we force a test failure here?
test/py/test.py +4 -2
@@ -8,6 +8,8 @@ # Wrapper script to invoke pytest with the directory name that contains the # U-Boot tests. +from __future__ import print_function + import os import os.path import sys
@@ -27,7 +29,7 @@ except: traceback.print_exc() # Hint to the user that they likely simply haven't installed the required # dependencies. - print >>sys.stderr, ''' + print(''' exec(py.test) failed; perhaps you are missing some dependencies? -See test/py/README.md for the list.''' +See test/py/README.md for the list.''', file=sys.stderr) sys.exit(1)
test/py/tests/test_fit.py +7 -5
@@ -4,6 +4,8 @@ # # Sanity check of the FIT handling in U-Boot +from __future__ import print_function + import os import pytest import struct
@@ -153,7 +155,7 @@ def test_fit(u_boot_console): src = make_fname('u-boot.dts') dtb = make_fname('u-boot.dtb') with open(src, 'w') as fd: - print >> fd, base_fdt + print(base_fdt, file=fd) util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb]) return dtb
@@ -167,7 +169,7 @@ def test_fit(u_boot_console): """ its = make_fname('test.its') with open(its, 'w') as fd: - print >> fd, base_its % params + print(base_its % params, file=fd) return its def make_fit(mkimage, params):
@@ -186,7 +188,7 @@ def test_fit(u_boot_console): its = make_its(params) util.run_and_log(cons, [mkimage, '-f', its, fit]) with open(make_fname('u-boot.dts'), 'w') as fd: - print >> fd, base_fdt + print(base_fdt, file=fd) return fit def make_kernel(filename, text):
@@ -202,7 +204,7 @@ def test_fit(u_boot_console): for i in range(100): data += 'this %s %d is unlikely to boot\n' % (text, i) with open(fname, 'w') as fd: - print >> fd, data + print(data, file=fd) return fname def make_ramdisk(filename, text):
@@ -216,7 +218,7 @@ def test_fit(u_boot_console): for i in range(100): data += '%s %d was seldom used in the middle ages\n' % (text, i) with open(fname, 'w') as fd: - print >> fd, data + print(data, file=fd) return fname def find_matching(text, match):
test/py/u_boot_console_sandbox.py +1 -1
@@ -42,7 +42,7 @@ class ConsoleSandbox(ConsoleBase): bcfg = self.config.buildconfig config_spl = bcfg.get('config_spl', 'n') == 'y' fname = '/spl/u-boot-spl' if config_spl else '/u-boot' - print fname + print(fname) cmd = [] if self.config.gdbserver: cmd += ['gdbserver', self.config.gdbserver]
test/py/u_boot_spawn.py +1 -1
@@ -59,7 +59,7 @@ class Spawn(object): os.chdir(cwd) os.execvp(args[0], args) except: - print 'CHILD EXECEPTION:' + print('CHILD EXECEPTION:') import traceback traceback.print_exc() finally:
Signature · 1 line
2.14.1
PATCH v2 1/8 test/py: Make print statements python 3.x safe
Quoted 8 lines · Paul Burton
In python 3.x print must be called as a function rather than used as a statement. Update uses of print to the function call syntax in order to be python 3.x safe.
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>---
Quoted 8 lines
Changes in v2: None
test/py/conftest.py | 2 +- test/py/test.py | 6 ++++-- test/py/tests/test_fit.py | 12 +++++++----- test/py/u_boot_console_sandbox.py | 2 +- test/py/u_boot_spawn.py | 2 +- 5 files changed, 14 insertions(+), 10 deletions(-)
Applied to u-boot-dm, thanks!
PATCH v2 2/8 test/py: Use range() rather than xrange()
In reply to Paul BurtonIn python 3.x the xrange() function has been removed, and range() returns an iterator much like Python 2.x's xrange(). Simply use range() in place of xrange() in order to work on both python 2.x & 3.x. This will mean a small cost on python 2.x since range() will return a list there rather than an iterator, but the cost should be negligible.
---
Changes in v2: None
test/py/u_boot_console_sandbox.py | 2 +- test/py/u_boot_spawn.py | 6 +++--- test/py/u_boot_utils.py | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
test/py/u_boot_console_sandbox.py +1 -1
@@ -82,7 +82,7 @@ class ConsoleSandbox(ConsoleBase): p = self.p self.p = None - for i in xrange(100): + for i in range(100): ret = not p.isalive() if ret: break
test/py/u_boot_spawn.py +3 -3
@@ -135,7 +135,7 @@ class Spawn(object): the expected time. """ - for pi in xrange(len(patterns)): + for pi in range(len(patterns)): if type(patterns[pi]) == type(''): patterns[pi] = re.compile(patterns[pi])
@@ -144,7 +144,7 @@ class Spawn(object): while True: earliest_m = None earliest_pi = None - for pi in xrange(len(patterns)): + for pi in range(len(patterns)): pattern = patterns[pi] m = pattern.search(self.buf) if not m:
@@ -199,7 +199,7 @@ class Spawn(object): """ os.close(self.fd) - for i in xrange(100): + for i in range(100): if not self.isalive(): break time.sleep(0.1)
test/py/u_boot_utils.py +2 -2
@@ -120,7 +120,7 @@ def wait_until_open_succeeds(fn): An open file handle to the file. """ - for i in xrange(100): + for i in range(100): fh = attempt_to_open_file(fn) if fh: return fh
@@ -143,7 +143,7 @@ def wait_until_file_open_fails(fn, ignore_errors): Nothing. """ - for i in xrange(100): + for i in range(100): fh = attempt_to_open_file(fn) if not fh: return
Signature · 1 line
2.14.1
PATCH v2 2/8 test/py: Use range() rather than xrange()
Quoted 8 lines · Paul Burton
In python 3.x the
xrange()function has been removed, andrange()returns an iterator much like Python 2.x'sxrange(). Simply userange()in place ofxrange()in order to work on both python 2.x & 3.x. This will mean a small cost on python 2.x sincerange()will return a list there rather than an iterator, but the cost should be negligible.Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>
Quoted 8 lines
---
Changes in v2: None
test/py/u_boot_console_sandbox.py | 2 +- test/py/u_boot_spawn.py | 6 +++--- test/py/u_boot_utils.py | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
Applied to u-boot-dm, thanks!
PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe
In reply to Paul BurtonIn python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.
---
Changes in v2:
- Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.
test/py/conftest.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
test/py/conftest.py +6 -2
@@ -19,11 +19,15 @@ import os import os.path import pytest from _pytest.runner import runtestprotocol -import ConfigParser import re import StringIO import sys +try: + import configparser +except: + import ConfigParser as configparser + # Globals: The HTML log file, and the connection to the U-Boot console. log = None console = None
@@ -167,7 +171,7 @@ def pytest_configure(config): with open(dot_config, 'rt') as f: ini_str = '[root]\n' + f.read() ini_sio = StringIO.StringIO(ini_str) - parser = ConfigParser.RawConfigParser() + parser = configparser.RawConfigParser() parser.readfp(ini_sio) ubconfig.buildconfig.update(parser.items('root'))
Signature · 1 line
2.14.1
PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe
Quoted 8 lines · Paul Burton
In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>
Quoted 7 lines
---
Changes in v2:
- Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.test/py/conftest.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
Applied to u-boot-dm, thanks!
PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe
In reply to Paul BurtonQuoted 39 lines · Paul Burton
In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>---
Changes in v2:
- Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.test/py/conftest.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)test/py/conftest.py +5
-1@@ -19,11 +19,15 @@ import os import os.path import pytest from _pytest.runner import runtestprotocol -import ConfigParser import re import StringIO import sys +try: + import configparser +except: + import ConfigParser as configparser + # Globals: The HTML log file, and the connection to the U-Boot console. log = None console = None
test/py/conftest.py def pytest_configure(config): · line 177
Quoted 8 lines
test/py/conftest.py +1
-1@@ -167,7 +171,7 @@ def pytest_configure(config): with open(dot_config, 'rt') as f: ini_str = '[root]\n' + f.read() ini_sio = StringIO.StringIO(ini_str) - parser = ConfigParser.RawConfigParser() + parser = configparser.RawConfigParser() parser.readfp(ini_sio) ubconfig.buildconfig.update(parser.items('root'))
I have bisect issue with test/py framework on my PC and I end up on this patch.
Below is the issue I am getting.
And here are my python version.
[u-boot](test)$ python3 --version
Python 3.4.3
[u-boot](test)$ python --version
Python 2.7.6
[u-boot](test)$ which python/usr/bin/python
Do you know what could be the issue?
Thanks,
Michal
=====================================================================================================
FAILURES
======================================================================================================
_____________________________________________________________________________________________________test_help
_____________________________________________________________________________________________________
u_boot_console = <u_boot_console_exec_attach.ConsoleExecAttach object at 0x7f1da3e6f290>
def test_help(u_boot_console):
"""Test that the "help" command can be executed."""
Quoted 1 line
u_boot_console.run_command('help')
test/py/tests/test_help.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/py/u_boot_console_base.py:181: in run_command
self.logstream.write(self.prompt, implicit=True)test/py/multiplexed_log.py:65: in write
self.logfile.write(self, data, implicit)test/py/multiplexed_log.py:688: in write
self.f.write(self._escape(data))test/py/multiplexed_log.py:335: in _escape
c for c in data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.0 = <iterator object@0x7f1da3d9bb90>
data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) orQuoted 1 line
c for c in data)
E UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
32: ordinal not in range(128)
test/py/multiplexed_log.py:335: UnicodeDecodeError
===============================================================================================
130 tests deselected
================================================================================================
=====================================================================================
1 failed, 130 deselected in 7.32 seconds
======================================================================================
PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe
Quoted 5 lines · Michal Simek
Quoted 4 lines · Paul Burton
In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.
...
Quoted 14 lines
I have bisect issue with test/py framework on my PC and I end up on this patch.
Below is the issue I am getting.
And here are my python version.
[u-boot](test)$ python3 --version
Python 3.4.3
[u-boot](test)$ python --version
Python 2.7.6
[u-boot](test)$ which python/usr/bin/python
Quoted 1 line
Do you know what could be the issue?
I honestly can't think why the configuration parsing could cause the kind of issue that was shown in the test log. It seems like there's a bug in the test/py code w.r.t. receiving characters from the target outside the ASCII range, which should be applicable in all cases, but I can't see how that would only happen based on how configparser was imported.
Quoted 57 lines
Thanks,
Michal=====================================================================================================
FAILURES
======================================================================================================
_____________________________________________________________________________________________________test_help
_____________________________________________________________________________________________________
u_boot_console= <u_boot_console_exec_attach.ConsoleExecAttach object at0x7f1da3e6f290>def
test_help(u_boot_console):
"""Test that the "help" command can be executed."""Quoted 1 line
u_boot_console.run_command('help')
test/py/tests/test_help.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _test/py/u_boot_console_base.py:181: inrun_command
self.logstream.write(self.prompt, implicit=True)test/py/multiplexed_log.py:65: in write
self.logfile.write(self, data, implicit)test/py/multiplexed_log.py:688: in write
self.f.write(self._escape(data))test/py/multiplexed_log.py:335: in_escape
c for c in data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.0 = <iterator object@0x7f1da3d9bb90>
data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) orQuoted 1 line
c for c in data)
E UnicodeDecodeError: 'ascii' codec can't decode byte
0x80in position
32: ordinal not in range(128)
test/py/multiplexed_log.py:335: UnicodeDecodeError
===============================================================================================
130 tests deselected
================================================================================================
=====================================================================================
1 failed, 130 deselected in 7.32 seconds
======================================================================================_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot
PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe
Quoted 21 lines · Stephen Warren
Quoted 5 lines · Michal Simek
Quoted 4 lines · Paul Burton
In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.
...
Quoted 14 lines
I have bisect issue with test/py framework on my PC and I end up on this patch.
Below is the issue I am getting.
And here are my python version.
[u-boot](test)$ python3 --version
Python 3.4.3
[u-boot](test)$ python --version
Python 2.7.6
[u-boot](test)$ which python/usr/bin/python
Quoted 8 lines
Quoted 1 line
Do you know what could be the issue?
I honestly can't think why the configuration parsing could cause the kind of issue that was shown in the test log. It seems like there's a bug in the test/py code w.r.t. receiving characters from the target outside the ASCII range, which should be applicable in all cases, but I can't see how that would only happen based on how configparser was imported.
hm. ok.
Thanks,
Michal
PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x
In reply to Paul BurtonIn python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.
---
Changes in v2:
- Try python 3.x style, fall back to python 2.x style rather than the reverse.
test/py/conftest.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
test/py/conftest.py +6 -2
@@ -20,7 +20,6 @@ import os.path import pytest from _pytest.runner import runtestprotocol import re -import StringIO import sys try:
@@ -28,6 +27,11 @@ try: except: import ConfigParser as configparser +try: + from io import StringIO +except ImportError: + from StringIO import StringIO + # Globals: The HTML log file, and the connection to the U-Boot console. log = None console = None
@@ -170,7 +174,7 @@ def pytest_configure(config): with open(dot_config, 'rt') as f: ini_str = '[root]\n' + f.read() - ini_sio = StringIO.StringIO(ini_str) + ini_sio = StringIO(ini_str) parser = configparser.RawConfigParser() parser.readfp(ini_sio) ubconfig.buildconfig.update(parser.items('root'))
Signature · 1 line
2.14.1
PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x
Quoted 7 lines · Paul Burton
In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>
This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But io.StringIO() needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/bda58c3e/attachment-0001.sig>
PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x
Hi Tom,
Quoted 6 lines · On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
Quoted 5 lines · Paul Burton
In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But
io.StringIO()needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!
Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO import first?)
https://patchwork.ozlabs.org/patch/813933/
Would you like me to submit a v3 or could you just take this one from v1?
Thanks,
Paul
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/ae89f6c5/attachment.sig>
PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x
Quoted 17 lines · Paul Burton
Hi Tom,
Quoted 6 lines · On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
Quoted 5 lines · Paul Burton
In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But
io.StringIO()needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!
Quoted 4 lines
Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO import first?)
https://patchwork.ozlabs.org/patch/813933/
If v1 works, that's fine by me too.
PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x
Quoted 18 lines · Stephen Warren
Quoted 17 lines · Paul Burton
Hi Tom,
Quoted 13 lines · On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
Quoted 7 lines · Paul Burton
In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But
io.StringIO()needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!
Quoted 6 lines
Quoted 4 lines
Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO import first?)
If v1 works, that's fine by me too.
OK, thanks. It does work and I'll grab just that.
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/c44f79b7/attachment.sig>
PATCH v2 5/8 test/py: Encode/decode strings for stdio
In reply to Paul BurtonWhen reading stdin python 3.x will give us byte arrays, and when writing stdout or stderr it will expect byte arrays. In order to insulate the rest of the code from this difference, call encode or decode at appropriate points when reading or writing stdio files. This works fine on python 2.x too.
---
Changes in v2: None
test/py/multiplexed_log.py | 4 ++-- test/py/tests/test_ut.py | 2 +- test/py/u_boot_spawn.py | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
test/py/multiplexed_log.py +2 -2
@@ -140,11 +140,11 @@ class RunAndLog(object): if stdout: if stderr: output += 'stdout:\n' - output += stdout + output += stdout.decode('utf8') if stderr: if stdout: output += 'stderr:\n' - output += stderr + output += stderr.decode('utf8') exit_status = p.returncode exception = None except subprocess.CalledProcessError as cpe:
test/py/tests/test_ut.py +1 -1
@@ -14,7 +14,7 @@ def test_ut_dm_init(u_boot_console): data = 'this is a test' data += '\x00' * ((4 * 1024 * 1024) - len(data)) with open(fn, 'wb') as fh: - fh.write(data) + fh.write(data.encode('utf-8')) fn = u_boot_console.config.source_dir + '/spi.bin' if not os.path.exists(fn):
test/py/u_boot_spawn.py +2 -2
@@ -114,7 +114,7 @@ class Spawn(object): Nothing. """ - os.write(self.fd, data) + os.write(self.fd, data.encode('utf8')) def expect(self, patterns): """Wait for the sub-process to emit specific data.
@@ -172,7 +172,7 @@ class Spawn(object): events = self.poll.poll(poll_maxwait) if not events: raise Timeout() - c = os.read(self.fd, 1024) + c = os.read(self.fd, 1024).decode('utf8') if not c: raise EOFError() if self.logfile_read:
Signature · 1 line
2.14.1
PATCH v2 5/8 test/py: Encode/decode strings for stdio
Quoted 8 lines · Paul Burton
When reading stdin python 3.x will give us byte arrays, and when writing stdout or stderr it will expect byte arrays. In order to insulate the rest of the code from this difference, call encode or decode at appropriate points when reading or writing stdio files. This works fine on python 2.x too.
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>
OK, this almost works. When I add in the following only the vboot test fails and I can't understand why:
test/py/multiplexed_log.py +1 -1
@@ -326,7 +326,7 @@ $(document).ready(function () { data = data.replace(chr(13), '') data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or - c for c in data) + c for c in data.encode('utf-8')) data = cgi.escape(data) return data
test/py/u_boot_spawn.py +1 -1
@@ -136,7 +136,7 @@ class Spawn(object): """ for pi in range(len(patterns)): - if type(patterns[pi]) == type(''): + if type(patterns[pi]) != type(re.compile('')): patterns[pi] = re.compile(patterns[pi]) tstart_s = time.time()
Please run 'make tests' on a host with python==python2, thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/55ca55dd/attachment.sig>
PATCH v2 6/8 test/py: fit: Open files as binary files
In reply to Paul BurtonThe read_file() function in test_fit is used with files that are not text files, as well as some that are. It is never used in a way that requires it to decode text files to characters, so open all files in binary mode such that read() doesn't attempt to decode characters for files which are not text files.
Without this test_fit fails on python 3.x when reading an FDT in run_fit_test() with:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position
0: invalid continuation byte
---
Changes in v2: None
test/py/tests/test_fit.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
test/py/tests/test_fit.py +1 -1
@@ -143,7 +143,7 @@ def test_fit(u_boot_console): Returns: Contents of file as a string """ - with open(fname, 'r') as fd: + with open(fname, 'rb') as fd: return fd.read() def make_dtb():
Signature · 1 line
2.14.1
PATCH v2 6/8 test/py: fit: Open files as binary files
Quoted 12 lines · Paul Burton
The
read_file()function intest_fitis used with files that are not text files, as well as some that are. It is never used in a way that requires it to decode text files to characters, so open all files in binary mode such thatread()doesn't attempt to decode characters for files which are not text files.Without this
test_fitfails on python 3.x when reading an FDT inrun_fit_test()with:UnicodeDecodeError: 'utf-8' codec can't decode byte
0xd0in position
0: invalid continuation byte
Quoted 8 lines
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>---
Changes in v2: None
test/py/tests/test_fit.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to u-boot-dm, thanks!
PATCH v2 7/8 test/py: hush_if_test: Use open() in place of file()
In reply to Paul BurtonIn python 3.x the file() function has been removed. Use open() instead, which works on both python 2.x & 3.x, and is described as the preferred method of opening a file by python 2.x documentation anyway.
---
Changes in v2: None
test/py/tests/test_hush_if_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
test/py/tests/test_hush_if_test.py +1 -1
@@ -149,7 +149,7 @@ def test_hush_if_test_host_file_exists(u_boot_console): exec_hush_if(u_boot_console, expr, False) try: - with file(test_file, 'wb'): + with open(test_file, 'wb'): pass assert os.path.exists(test_file)
Signature · 1 line
2.14.1
PATCH v2 7/8 test/py: hush_if_test: Use open() in place of file()
Quoted 4 lines · Paul Burton
In python 3.x the
file()function has been removed. Useopen()instead, which works on both python 2.x & 3.x, and is described as the preferred method of opening a file by python 2.x documentation anyway.
Quoted 8 lines
Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>---
Changes in v2: None
test/py/tests/test_hush_if_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to u-boot-dm, thanks!
PATCH v2 8/8 test/py: vboot: Remove stderr redirect from openssl command
In reply to Paul BurtonThe openssl command specified in test_with_algo() ultimately ends up being run by RunAndLog::run(), which uses it to construct a Popen object with the default shell=False. The stderr redirect in the command is therefore simply passed to openssl as an argument. With at least openssl 1.1.0f this causes openssl, and therefore test_vboot, to fail with:
genpkey: Use -help for summary.
Exit code: 1
Any stderr output ought to be captured & stored in the RunAndLog object's output field and returned from run() via run_and_log() to test_with_algo() which then ignores it anyway, so we can drop the shell-like redirection with no ill effects. With this fix test_vboot now passes for me.
---
Changes in v2:
- Place the % operator at the end of the last string line, not the start of the next.
test/py/tests/test_vboot.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
test/py/tests/test_vboot.py +2 -2
@@ -174,8 +174,8 @@ def test_vboot(u_boot_console): public_exponent = 65537 util.run_and_log(cons, 'openssl genpkey -algorithm RSA -out %sdev.key ' '-pkeyopt rsa_keygen_bits:2048 ' - '-pkeyopt rsa_keygen_pubexp:%d ' - '2>/dev/null' % (tmpdir, public_exponent)) + '-pkeyopt rsa_keygen_pubexp:%d' % + (tmpdir, public_exponent)) # Create a certificate containing the public key util.run_and_log(cons, 'openssl req -batch -new -x509 -key %sdev.key -out '
Signature · 1 line
2.14.1
PATCH v2 8/8 test/py: vboot: Remove stderr redirect from openssl command
Quoted 18 lines · Paul Burton
The openssl command specified in
test_with_algo()ultimately ends up being run byRunAndLog::run(), which uses it to construct a Popen object with the default shell=False. The stderr redirect in the command is therefore simply passed to openssl as an argument. With at least openssl 1.1.0f this causes openssl, and thereforetest_vboot, to fail with:genpkey: Use -help for summary.
Exit code: 1Any stderr output ought to be captured & stored in the RunAndLog object's output field and returned from
run()viarun_and_log()totest_with_algo()which then ignores it anyway, so we can drop the shell-like redirection with no ill effects. With this fixtest_vbootnow passes for me.Signed-off-by Paul Burton <[email protected]>Reviewed-by Stephen Warren <[email protected]>
Quoted 8 lines
---
Changes in v2:
- Place the % operator at the end of the last string line, not the start of the next.test/py/tests/test_vboot.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Applied to u-boot-dm, thanks!
v2 cover test/py: Fixes for python 3.x
In reply to Paul BurtonHi Paul,
Quoted 32 lines · Paul Burton
This series allows test/py to run on python 3.x in addition to the already supported python 2.x. With it applied I currently see only one failure (test_ut[ut_dm_usb_flash]) which doesn't appear to be python related when running with:
$ ./test/py/test.py --bd sandbox --buildThanks,
PaulPaul Burton (8):
test/py: Make print statements python 3.x safe test/py: Userange()rather thanxrange()
test/py: Import 'configparser' lower case to be python 3.x safe test/py: Import StringIO from io module for python 3.x test/py: Encode/decode strings for stdio
test/py: fit: Open files as binary files
test/py:hush_if_test: Useopen()in place offile()test/py: vboot: Remove stderr redirect from openssl commandtest/py/conftest.py | 18 +++++++++++++----- test/py/multiplexed_log.py | 4 ++-- test/py/test.py | 6 ++++-- test/py/tests/test_fit.py | 14 ++++++++------ test/py/tests/test_hush_if_test.py | 2 +- test/py/tests/test_ut.py | 2 +- test/py/tests/test_vboot.py | 4 ++-- test/py/u_boot_console_sandbox.py | 4 ++-- test/py/u_boot_spawn.py | 12 ++++++------ test/py/u_boot_utils.py | 4 ++-- 10 files changed, 41 insertions(+), 29 deletions(-)
Quoted 3 lines
--
2.14.1
I never did see a v3 of this but I would like to get these patches applied.
I'll apply the ones that seem cause no problems. If you have time, please respin these ones:
Quoted 2 lines
test/py: Import StringIO from io module for python 3.x test/py: Encode/decode strings for stdio
Regards,
Simon