Skip to contentSkip to conversation

test/py: Fixes for python 3.x

PATCH New Paul Burton 5 participants 24 messages Sep 14, 2017 – Aug 21, 2018

lore
Share & tools
lore
#0 lore raw

v2 cover test/py: Fixes for python 3.x

This series allows test/py to run on python 3.x in addition to the already supported python 2.x. With it applied I currently see only one failure (test_ut[ut_dm_usb_flash]) which doesn't appear to be python related when running with:

$ ./test/py/test.py --bd sandbox --build

Thanks,
Paul

Paul Burton (8):
test/py: Make print statements python 3.x safe test/py: Use range() rather than xrange()
test/py: Import 'configparser' lower case to be python 3.x safe test/py: Import StringIO from io module for python 3.x test/py: Encode/decode strings for stdio
test/py: fit: Open files as binary files
test/py: hush_if_test: Use open() in place of file() test/py: vboot: Remove stderr redirect from openssl command

 test/py/conftest.py                | 18 +++++++++++++-----
 test/py/multiplexed_log.py         |  4 ++--
 test/py/test.py                    |  6 ++++--
 test/py/tests/test_fit.py          | 14 ++++++++------
 test/py/tests/test_hush_if_test.py |  2 +-
 test/py/tests/test_ut.py           |  2 +-
 test/py/tests/test_vboot.py        |  4 ++--
 test/py/u_boot_console_sandbox.py  |  4 ++--
 test/py/u_boot_spawn.py            | 12 ++++++------
 test/py/u_boot_utils.py            |  4 ++--
 10 files changed, 41 insertions(+), 29 deletions(-)
Signature · 1 line
2.14.1
#1 lore raw

PATCH v2 1/8 test/py: Make print statements python 3.x safe

In python 3.x print must be called as a function rather than used as a statement. Update uses of print to the function call syntax in order to be python 3.x safe.

---

Changes in v2: None

 test/py/conftest.py               |  2 +-
 test/py/test.py                   |  6 ++++--
 test/py/tests/test_fit.py         | 12 +++++++-----
 test/py/u_boot_console_sandbox.py |  2 +-
 test/py/u_boot_spawn.py           |  2 +-
 5 files changed, 14 insertions(+), 10 deletions(-)
test/py/conftest.py +1 -1
index 65e1d75626..ced96f1006 100644
@@ -545,7 +545,7 @@ def pytest_runtest_protocol(item, nextitem):
         # is fixed, if this exception still exists, it will then be logged as
         # part of the test's stdout.
         import traceback
-        print 'Exception occurred while logging runtest status:'
+        print('Exception occurred while logging runtest status:')
         traceback.print_exc()
         # FIXME: Can we force a test failure here?
 
test/py/test.py +4 -2
index 74e560a4d3..215f2651a7 100755
@@ -8,6 +8,8 @@
 # Wrapper script to invoke pytest with the directory name that contains the
 # U-Boot tests.
 
+from __future__ import print_function
+
 import os
 import os.path
 import sys
@@ -27,7 +29,7 @@ except:
     traceback.print_exc()
     # Hint to the user that they likely simply haven't installed the required
     # dependencies.
-    print >>sys.stderr, '''
+    print('''
 exec(py.test) failed; perhaps you are missing some dependencies?
-See test/py/README.md for the list.'''
+See test/py/README.md for the list.''', file=sys.stderr)
     sys.exit(1)
test/py/tests/test_fit.py +7 -5
index 7e6b96dae4..29b2491f89 100755
@@ -4,6 +4,8 @@
 #
 # Sanity check of the FIT handling in U-Boot
 
+from __future__ import print_function
+
 import os
 import pytest
 import struct
@@ -153,7 +155,7 @@ def test_fit(u_boot_console):
         src = make_fname('u-boot.dts')
         dtb = make_fname('u-boot.dtb')
         with open(src, 'w') as fd:
-            print >> fd, base_fdt
+            print(base_fdt, file=fd)
         util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb])
         return dtb
 
@@ -167,7 +169,7 @@ def test_fit(u_boot_console):
         """
         its = make_fname('test.its')
         with open(its, 'w') as fd:
-            print >> fd, base_its % params
+            print(base_its % params, file=fd)
         return its
 
     def make_fit(mkimage, params):
@@ -186,7 +188,7 @@ def test_fit(u_boot_console):
         its = make_its(params)
         util.run_and_log(cons, [mkimage, '-f', its, fit])
         with open(make_fname('u-boot.dts'), 'w') as fd:
-            print >> fd, base_fdt
+            print(base_fdt, file=fd)
         return fit
 
     def make_kernel(filename, text):
@@ -202,7 +204,7 @@ def test_fit(u_boot_console):
         for i in range(100):
             data += 'this %s %d is unlikely to boot\n' % (text, i)
         with open(fname, 'w') as fd:
-            print >> fd, data
+            print(data, file=fd)
         return fname
 
     def make_ramdisk(filename, text):
@@ -216,7 +218,7 @@ def test_fit(u_boot_console):
         for i in range(100):
             data += '%s %d was seldom used in the middle ages\n' % (text, i)
         with open(fname, 'w') as fd:
-            print >> fd, data
+            print(data, file=fd)
         return fname
 
     def find_matching(text, match):
test/py/u_boot_console_sandbox.py +1 -1
index 647e1f879f..9a2fe9cb59 100644
@@ -42,7 +42,7 @@ class ConsoleSandbox(ConsoleBase):
         bcfg = self.config.buildconfig
         config_spl = bcfg.get('config_spl', 'n') == 'y'
         fname = '/spl/u-boot-spl' if config_spl else '/u-boot'
-        print fname
+        print(fname)
         cmd = []
         if self.config.gdbserver:
             cmd += ['gdbserver', self.config.gdbserver]
test/py/u_boot_spawn.py +1 -1
index 3a0fbfad90..15ca4ac838 100644
@@ -59,7 +59,7 @@ class Spawn(object):
                     os.chdir(cwd)
                 os.execvp(args[0], args)
             except:
-                print 'CHILD EXECEPTION:'
+                print('CHILD EXECEPTION:')
                 import traceback
                 traceback.print_exc()
             finally:
Signature · 1 line
2.14.1
#2 lore raw

PATCH v2 1/8 test/py: Make print statements python 3.x safe

Quoted 8 lines · Paul Burton

In python 3.x print must be called as a function rather than used as a statement. Update uses of print to the function call syntax in order to be python 3.x safe.

---

Quoted 8 lines

Changes in v2: None

 test/py/conftest.py               |  2 +-
 test/py/test.py                   |  6 ++++--
 test/py/tests/test_fit.py         | 12 +++++++-----
 test/py/u_boot_console_sandbox.py |  2 +-
 test/py/u_boot_spawn.py           |  2 +-
 5 files changed, 14 insertions(+), 10 deletions(-)

Applied to u-boot-dm, thanks!

#3 lore raw

PATCH v2 2/8 test/py: Use range() rather than xrange()

In reply to Paul Burton

In python 3.x the xrange() function has been removed, and range() returns an iterator much like Python 2.x's xrange(). Simply use range() in place of xrange() in order to work on both python 2.x & 3.x. This will mean a small cost on python 2.x since range() will return a list there rather than an iterator, but the cost should be negligible.

---

Changes in v2: None

 test/py/u_boot_console_sandbox.py | 2 +-
 test/py/u_boot_spawn.py           | 6 +++---
 test/py/u_boot_utils.py           | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
test/py/u_boot_console_sandbox.py +1 -1
index 9a2fe9cb59..5c91d926c5 100644
@@ -82,7 +82,7 @@ class ConsoleSandbox(ConsoleBase):
 
         p = self.p
         self.p = None
-        for i in xrange(100):
+        for i in range(100):
             ret = not p.isalive()
             if ret:
                 break
test/py/u_boot_spawn.py +3 -3
index 15ca4ac838..638c5dd31d 100644
@@ -135,7 +135,7 @@ class Spawn(object):
             the expected time.
         """
 
-        for pi in xrange(len(patterns)):
+        for pi in range(len(patterns)):
             if type(patterns[pi]) == type(''):
                 patterns[pi] = re.compile(patterns[pi])
 
@@ -144,7 +144,7 @@ class Spawn(object):
             while True:
                 earliest_m = None
                 earliest_pi = None
-                for pi in xrange(len(patterns)):
+                for pi in range(len(patterns)):
                     pattern = patterns[pi]
                     m = pattern.search(self.buf)
                     if not m:
@@ -199,7 +199,7 @@ class Spawn(object):
         """
 
         os.close(self.fd)
-        for i in xrange(100):
+        for i in range(100):
             if not self.isalive():
                 break
             time.sleep(0.1)
test/py/u_boot_utils.py +2 -2
index 2ba4baed07..264508da4e 100644
@@ -120,7 +120,7 @@ def wait_until_open_succeeds(fn):
         An open file handle to the file.
     """
 
-    for i in xrange(100):
+    for i in range(100):
         fh = attempt_to_open_file(fn)
         if fh:
             return fh
@@ -143,7 +143,7 @@ def wait_until_file_open_fails(fn, ignore_errors):
         Nothing.
     """
 
-    for i in xrange(100):
+    for i in range(100):
         fh = attempt_to_open_file(fn)
         if not fh:
             return
Signature · 1 line
2.14.1
#4 lore raw

PATCH v2 2/8 test/py: Use range() rather than xrange()

Quoted 8 lines · Paul Burton

In python 3.x the xrange() function has been removed, and range() returns an iterator much like Python 2.x's xrange(). Simply use range() in place of xrange() in order to work on both python 2.x & 3.x. This will mean a small cost on python 2.x since range() will return a list there rather than an iterator, but the cost should be negligible.

Quoted 8 lines

---

Changes in v2: None

 test/py/u_boot_console_sandbox.py | 2 +-
 test/py/u_boot_spawn.py           | 6 +++---
 test/py/u_boot_utils.py           | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

Applied to u-boot-dm, thanks!

#5 lore raw

PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe

In reply to Paul Burton

In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.

---

Changes in v2:
- Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.

 test/py/conftest.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
test/py/conftest.py +6 -2
index ced96f1006..bf55bd3d13 100644
@@ -19,11 +19,15 @@ import os
 import os.path
 import pytest
 from _pytest.runner import runtestprotocol
-import ConfigParser
 import re
 import StringIO
 import sys
 
+try:
+    import configparser
+except:
+    import ConfigParser as configparser
+
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
 console = None
@@ -167,7 +171,7 @@ def pytest_configure(config):
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
             ini_sio = StringIO.StringIO(ini_str)
-            parser = ConfigParser.RawConfigParser()
+            parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
 
Signature · 1 line
2.14.1
#6 lore raw

PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe

Quoted 8 lines · Paul Burton

In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.

Quoted 7 lines

---

Changes in v2:
- Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.

 test/py/conftest.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

#7 lore raw

PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe

In reply to Paul Burton
Quoted 39 lines · Paul Burton

In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.

---

Changes in v2:
- Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.

 test/py/conftest.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
test/py/conftest.py +5 -1
index ced96f1006..bf55bd3d13 100644
@@ -19,11 +19,15 @@ import os
 import os.path
 import pytest
 from _pytest.runner import runtestprotocol
-import ConfigParser
 import re
 import StringIO
 import sys
 
+try:
+    import configparser
+except:
+    import ConfigParser as configparser
+
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
 console = None

test/py/conftest.py def pytest_configure(config): · line 177

Quoted 8 lines
test/py/conftest.py +1 -1
@@ -167,7 +171,7 @@ def pytest_configure(config):
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
             ini_sio = StringIO.StringIO(ini_str)
-            parser = ConfigParser.RawConfigParser()
+            parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
 

I have bisect issue with test/py framework on my PC and I end up on this patch.

Below is the issue I am getting.

And here are my python version.
[u-boot](test)$ python3 --version
Python 3.4.3
[u-boot](test)$ python --version
Python 2.7.6
[u-boot](test)$ which python
/usr/bin/python

Do you know what could be the issue?

Thanks,
Michal

=====================================================================================================
FAILURES
======================================================================================================
_____________________________________________________________________________________________________
test_help
_____________________________________________________________________________________________________

u_boot_console = <u_boot_console_exec_attach.ConsoleExecAttach object at 0x7f1da3e6f290>

def test_help(u_boot_console):
"""Test that the "help" command can be executed."""

Quoted 1 line

u_boot_console.run_command('help')

test/py/tests/test_help.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/py/u_boot_console_base.py:181: in run_command
self.logstream.write(self.prompt, implicit=True)
test/py/multiplexed_log.py:65: in write
self.logfile.write(self, data, implicit)
test/py/multiplexed_log.py:688: in write
self.f.write(self._escape(data))
test/py/multiplexed_log.py:335: in _escape
c for c in data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <iterator object@0x7f1da3d9bb90>

    data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
Quoted 1 line

c for c in data)

E UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
32: ordinal not in range(128)

test/py/multiplexed_log.py:335: UnicodeDecodeError
===============================================================================================
130 tests deselected
================================================================================================
=====================================================================================
1 failed, 130 deselected in 7.32 seconds
======================================================================================

#8 lore raw

PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe

Quoted 5 lines · Michal Simek
Quoted 4 lines · Paul Burton

In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.

...

Quoted 14 lines

I have bisect issue with test/py framework on my PC and I end up on this patch.

Below is the issue I am getting.

And here are my python version.
[u-boot](test)$ python3 --version
Python 3.4.3
[u-boot](test)$ python --version
Python 2.7.6
[u-boot](test)$ which python
/usr/bin/python

Quoted 1 line

Do you know what could be the issue?

I honestly can't think why the configuration parsing could cause the kind of issue that was shown in the test log. It seems like there's a bug in the test/py code w.r.t. receiving characters from the target outside the ASCII range, which should be applicable in all cases, but I can't see how that would only happen based on how configparser was imported.

Quoted 57 lines

Thanks,
Michal

=====================================================================================================
FAILURES
======================================================================================================
_____________________________________________________________________________________________________
test_help
_____________________________________________________________________________________________________

u_boot_console = <u_boot_console_exec_attach.ConsoleExecAttach object at 0x7f1da3e6f290>

def test_help(u_boot_console):
"""Test that the "help" command can be executed."""

Quoted 1 line

u_boot_console.run_command('help')

test/py/tests/test_help.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/py/u_boot_console_base.py:181: in run_command
self.logstream.write(self.prompt, implicit=True)
test/py/multiplexed_log.py:65: in write
self.logfile.write(self, data, implicit)
test/py/multiplexed_log.py:688: in write
self.f.write(self._escape(data))
test/py/multiplexed_log.py:335: in _escape
c for c in data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <iterator object@0x7f1da3d9bb90>

     data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
Quoted 1 line

c for c in data)

E UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
32: ordinal not in range(128)

test/py/multiplexed_log.py:335: UnicodeDecodeError
===============================================================================================
130 tests deselected
================================================================================================
=====================================================================================
1 failed, 130 deselected in 7.32 seconds
======================================================================================

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot

#9 lore raw

PATCH v2 3/8 test/py: Import 'configparser' lower case to be python 3.x safe

Quoted 21 lines · Stephen Warren
Quoted 5 lines · Michal Simek
Quoted 4 lines · Paul Burton

In python 3.x the configparser module is named with all lower case. Import it as such in order to avoid errors when running on python 3.x, and fall back to the CamelCase version in order to keep working with python 2.x.

...

Quoted 14 lines

I have bisect issue with test/py framework on my PC and I end up on this patch.

Below is the issue I am getting.

And here are my python version.
[u-boot](test)$ python3 --version
Python 3.4.3
[u-boot](test)$ python --version
Python 2.7.6
[u-boot](test)$ which python
/usr/bin/python

Quoted 8 lines
Quoted 1 line

Do you know what could be the issue?

I honestly can't think why the configuration parsing could cause the kind of issue that was shown in the test log. It seems like there's a bug in the test/py code w.r.t. receiving characters from the target outside the ASCII range, which should be applicable in all cases, but I can't see how that would only happen based on how configparser was imported.

hm. ok.

Thanks,
Michal

#10 lore raw

PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x

In reply to Paul Burton

In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.

---

Changes in v2:
- Try python 3.x style, fall back to python 2.x style rather than the reverse.

 test/py/conftest.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
test/py/conftest.py +6 -2
index bf55bd3d13..0d564942d0 100644
@@ -20,7 +20,6 @@ import os.path
 import pytest
 from _pytest.runner import runtestprotocol
 import re
-import StringIO
 import sys
 
 try:
@@ -28,6 +27,11 @@ try:
 except:
     import ConfigParser as configparser
 
+try:
+    from io import StringIO
+except ImportError:
+    from StringIO import StringIO
+
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
 console = None
@@ -170,7 +174,7 @@ def pytest_configure(config):
 
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
-            ini_sio = StringIO.StringIO(ini_str)
+            ini_sio = StringIO(ini_str)
             parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
Signature · 1 line
2.14.1
#11 lore raw

PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x

Quoted 7 lines · Paul Burton

In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.

This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But io.StringIO() needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/bda58c3e/attachment-0001.sig>
#12 lore raw

PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x

Hi Tom,

Quoted 6 lines · On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
Quoted 5 lines · Paul Burton

In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.

Quoted 7 lines
Quoted 2 lines

This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But io.StringIO() needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!

Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO import first?)

https://patchwork.ozlabs.org/patch/813933/

Would you like me to submit a v3 or could you just take this one from v1?

Thanks,
Paul
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/ae89f6c5/attachment.sig>

#13 lore raw

PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x

Quoted 17 lines · Paul Burton

Hi Tom,

Quoted 6 lines · On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
Quoted 5 lines · Paul Burton

In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.

Quoted 7 lines
Quoted 2 lines

This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But io.StringIO() needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!

Quoted 4 lines

Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO import first?)

   https://patchwork.ozlabs.org/patch/813933/

If v1 works, that's fine by me too.

#14 lore raw

PATCH v2 4/8 test/py: Import StringIO from io module for python 3.x

Quoted 18 lines · Stephen Warren
Quoted 17 lines · Paul Burton

Hi Tom,

Quoted 13 lines · On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
Quoted 7 lines · Paul Burton

In python 3.x the StringIO module is gone, and instead StringIO can be imported from the io module. Do this in order to run on python 3.x, and fall back to importing StringIO as a module in order to continue working with python 2.x.

This breaks python2.7. There we have both StringIO and io.StringIO so we succeed in 'from io import StringIO'. But io.StringIO() needs unicode, and we don't have that by default in python2. My quick attempt here failed. Thanks!

Quoted 6 lines
Quoted 4 lines

Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO import first?)

https://patchwork.ozlabs.org/patch/813933/

If v1 works, that's fine by me too.

OK, thanks. It does work and I'll grab just that.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/c44f79b7/attachment.sig>
#15 lore raw

PATCH v2 5/8 test/py: Encode/decode strings for stdio

In reply to Paul Burton

When reading stdin python 3.x will give us byte arrays, and when writing stdout or stderr it will expect byte arrays. In order to insulate the rest of the code from this difference, call encode or decode at appropriate points when reading or writing stdio files. This works fine on python 2.x too.

---

Changes in v2: None

 test/py/multiplexed_log.py | 4 ++--
 test/py/tests/test_ut.py   | 2 +-
 test/py/u_boot_spawn.py    | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)
test/py/multiplexed_log.py +2 -2
index bf926c3e77..21bdcb7309 100644
@@ -140,11 +140,11 @@ class RunAndLog(object):
             if stdout:
                 if stderr:
                     output += 'stdout:\n'
-                output += stdout
+                output += stdout.decode('utf8')
             if stderr:
                 if stdout:
                     output += 'stderr:\n'
-                output += stderr
+                output += stderr.decode('utf8')
             exit_status = p.returncode
             exception = None
         except subprocess.CalledProcessError as cpe:
test/py/tests/test_ut.py +1 -1
index 5c25a2d465..1f40e2c2d0 100644
@@ -14,7 +14,7 @@ def test_ut_dm_init(u_boot_console):
         data = 'this is a test'
         data += '\x00' * ((4 * 1024 * 1024) - len(data))
         with open(fn, 'wb') as fh:
-            fh.write(data)
+            fh.write(data.encode('utf-8'))
 
     fn = u_boot_console.config.source_dir + '/spi.bin'
     if not os.path.exists(fn):
test/py/u_boot_spawn.py +2 -2
index 638c5dd31d..751302a529 100644
@@ -114,7 +114,7 @@ class Spawn(object):
             Nothing.
         """
 
-        os.write(self.fd, data)
+        os.write(self.fd, data.encode('utf8'))
 
     def expect(self, patterns):
         """Wait for the sub-process to emit specific data.
@@ -172,7 +172,7 @@ class Spawn(object):
                 events = self.poll.poll(poll_maxwait)
                 if not events:
                     raise Timeout()
-                c = os.read(self.fd, 1024)
+                c = os.read(self.fd, 1024).decode('utf8')
                 if not c:
                     raise EOFError()
                 if self.logfile_read:
Signature · 1 line
2.14.1
#16 lore raw

PATCH v2 5/8 test/py: Encode/decode strings for stdio

Quoted 8 lines · Paul Burton

When reading stdin python 3.x will give us byte arrays, and when writing stdout or stderr it will expect byte arrays. In order to insulate the rest of the code from this difference, call encode or decode at appropriate points when reading or writing stdio files. This works fine on python 2.x too.

OK, this almost works. When I add in the following only the vboot test fails and I can't understand why:

test/py/multiplexed_log.py +1 -1
index 21bdcb730917..d23bec0b9668 100644
@@ -326,7 +326,7 @@ $(document).ready(function () {
 
         data = data.replace(chr(13), '')
         data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
-                       c for c in data)
+                       c for c in data.encode('utf-8'))
         data = cgi.escape(data)
         return data
 
test/py/u_boot_spawn.py +1 -1
index 751302a529e8..0490230c83dc 100644
@@ -136,7 +136,7 @@ class Spawn(object):
         """
 
         for pi in range(len(patterns)):
-            if type(patterns[pi]) == type(''):
+            if type(patterns[pi]) != type(re.compile('')):
                 patterns[pi] = re.compile(patterns[pi])
 
         tstart_s = time.time()

Please run 'make tests' on a host with python==python2, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/55ca55dd/attachment.sig>
#17 lore raw

PATCH v2 6/8 test/py: fit: Open files as binary files

In reply to Paul Burton

The read_file() function in test_fit is used with files that are not text files, as well as some that are. It is never used in a way that requires it to decode text files to characters, so open all files in binary mode such that read() doesn't attempt to decode characters for files which are not text files.

Without this test_fit fails on python 3.x when reading an FDT in run_fit_test() with:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position
0: invalid continuation byte

---

Changes in v2: None

 test/py/tests/test_fit.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
test/py/tests/test_fit.py +1 -1
index 29b2491f89..01ecde27d5 100755
@@ -143,7 +143,7 @@ def test_fit(u_boot_console):
         Returns:
             Contents of file as a string
         """
-        with open(fname, 'r') as fd:
+        with open(fname, 'rb') as fd:
             return fd.read()
 
     def make_dtb():
Signature · 1 line
2.14.1
#18 lore raw

PATCH v2 6/8 test/py: fit: Open files as binary files

Quoted 12 lines · Paul Burton

The read_file() function in test_fit is used with files that are not text files, as well as some that are. It is never used in a way that requires it to decode text files to characters, so open all files in binary mode such that read() doesn't attempt to decode characters for files which are not text files.

Without this test_fit fails on python 3.x when reading an FDT in run_fit_test() with:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position
0: invalid continuation byte

Quoted 8 lines

---

Changes in v2: None

 test/py/tests/test_fit.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Applied to u-boot-dm, thanks!

#19 lore raw

PATCH v2 7/8 test/py: hush_if_test: Use open() in place of file()

In reply to Paul Burton

In python 3.x the file() function has been removed. Use open() instead, which works on both python 2.x & 3.x, and is described as the preferred method of opening a file by python 2.x documentation anyway.

---

Changes in v2: None

 test/py/tests/test_hush_if_test.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
test/py/tests/test_hush_if_test.py +1 -1
index c8f4208d31..cdad22b429 100644
@@ -149,7 +149,7 @@ def test_hush_if_test_host_file_exists(u_boot_console):
     exec_hush_if(u_boot_console, expr, False)
 
     try:
-        with file(test_file, 'wb'):
+        with open(test_file, 'wb'):
             pass
         assert os.path.exists(test_file)
 
Signature · 1 line
2.14.1
#20 lore raw

PATCH v2 7/8 test/py: hush_if_test: Use open() in place of file()

Quoted 4 lines · Paul Burton

In python 3.x the file() function has been removed. Use open() instead, which works on both python 2.x & 3.x, and is described as the preferred method of opening a file by python 2.x documentation anyway.

Quoted 8 lines

---

Changes in v2: None

 test/py/tests/test_hush_if_test.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Applied to u-boot-dm, thanks!

#21 lore raw

PATCH v2 8/8 test/py: vboot: Remove stderr redirect from openssl command

In reply to Paul Burton

The openssl command specified in test_with_algo() ultimately ends up being run by RunAndLog::run(), which uses it to construct a Popen object with the default shell=False. The stderr redirect in the command is therefore simply passed to openssl as an argument. With at least openssl 1.1.0f this causes openssl, and therefore test_vboot, to fail with:

genpkey: Use -help for summary.
Exit code: 1

Any stderr output ought to be captured & stored in the RunAndLog object's output field and returned from run() via run_and_log() to test_with_algo() which then ignores it anyway, so we can drop the shell-like redirection with no ill effects. With this fix test_vboot now passes for me.

---

Changes in v2:
- Place the % operator at the end of the last string line, not the start of the next.

 test/py/tests/test_vboot.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
test/py/tests/test_vboot.py +2 -2
index 6e62820743..b1170aa4b0 100644
@@ -174,8 +174,8 @@ def test_vboot(u_boot_console):
     public_exponent = 65537
     util.run_and_log(cons, 'openssl genpkey -algorithm RSA -out %sdev.key '
                      '-pkeyopt rsa_keygen_bits:2048 '
-                     '-pkeyopt rsa_keygen_pubexp:%d '
-                     '2>/dev/null'  % (tmpdir, public_exponent))
+                     '-pkeyopt rsa_keygen_pubexp:%d' %
+                     (tmpdir, public_exponent))
 
     # Create a certificate containing the public key
     util.run_and_log(cons, 'openssl req -batch -new -x509 -key %sdev.key -out '
Signature · 1 line
2.14.1
#22 lore raw

PATCH v2 8/8 test/py: vboot: Remove stderr redirect from openssl command

Quoted 18 lines · Paul Burton

The openssl command specified in test_with_algo() ultimately ends up being run by RunAndLog::run(), which uses it to construct a Popen object with the default shell=False. The stderr redirect in the command is therefore simply passed to openssl as an argument. With at least openssl 1.1.0f this causes openssl, and therefore test_vboot, to fail with:

genpkey: Use -help for summary.
Exit code: 1

Any stderr output ought to be captured & stored in the RunAndLog object's output field and returned from run() via run_and_log() to test_with_algo() which then ignores it anyway, so we can drop the shell-like redirection with no ill effects. With this fix test_vboot now passes for me.

Quoted 8 lines

---

Changes in v2:
- Place the % operator at the end of the last string line, not the start of the next.

 test/py/tests/test_vboot.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

#23 lore raw

v2 cover test/py: Fixes for python 3.x

In reply to Paul Burton

Hi Paul,

Quoted 32 lines · Paul Burton

This series allows test/py to run on python 3.x in addition to the already supported python 2.x. With it applied I currently see only one failure (test_ut[ut_dm_usb_flash]) which doesn't appear to be python related when running with:

$ ./test/py/test.py --bd sandbox --build

Thanks,
Paul

Paul Burton (8):
test/py: Make print statements python 3.x safe test/py: Use range() rather than xrange()
test/py: Import 'configparser' lower case to be python 3.x safe test/py: Import StringIO from io module for python 3.x test/py: Encode/decode strings for stdio
test/py: fit: Open files as binary files
test/py: hush_if_test: Use open() in place of file() test/py: vboot: Remove stderr redirect from openssl command

 test/py/conftest.py                | 18 +++++++++++++-----
 test/py/multiplexed_log.py         |  4 ++--
 test/py/test.py                    |  6 ++++--
 test/py/tests/test_fit.py          | 14 ++++++++------
 test/py/tests/test_hush_if_test.py |  2 +-
 test/py/tests/test_ut.py           |  2 +-
 test/py/tests/test_vboot.py        |  4 ++--
 test/py/u_boot_console_sandbox.py  |  4 ++--
 test/py/u_boot_spawn.py            | 12 ++++++------
 test/py/u_boot_utils.py            |  4 ++--
 10 files changed, 41 insertions(+), 29 deletions(-)
Quoted 3 lines

--
2.14.1

I never did see a v3 of this but I would like to get these patches applied.

I'll apply the ones that seem cause no problems. If you have time, please respin these ones:

Quoted 2 lines

test/py: Import StringIO from io module for python 3.x test/py: Encode/decode strings for stdio

Regards,
Simon