Skip to contentSkip to conversation

iotests: Require Python 3.6 or later

PATCH In review Kevin Wolf 6 participants 13 messages Sep 19, 2019 – Sep 20, 2019

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

v2 cover iotests: Require Python 3.6 or later

v2:

- Provide the right exit code from Python instead of having a potentially confusing negation in the shell script

- Raised the minimal version to 3.6. If we're going to use a different version than QEMU as a whole anyway, we can use a version that suits us best. 3.5 would only be for Debian Stretch, and we don't really care that much about running Python test cases on it.

- Added a patch to remove Python 2 compatibility code

Kevin Wolf (2):
iotests: Require Python 3.6 or later
iotests: Remove Python 2 compatibility code

 tests/qemu-iotests/044                   |  3 ---
 tests/qemu-iotests/163                   |  3 ---
 tests/qemu-iotests/check                 | 13 ++++++++++++-
 tests/qemu-iotests/iotests.py            | 13 +++----------
 tests/qemu-iotests/nbd-fault-injector.py |  7 +++----
 5 files changed, 18 insertions(+), 21 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v2 1/2 iotests: Require Python 3.6 or later

Running iotests is not required to build QEMU, so we can have stricter version requirements for Python here and can make use of new features and drop compatibility code earlier.

This makes qemu-iotests skip all Python tests if a Python version before 3.6 is used for the build.

---
 tests/qemu-iotests/check | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
tests/qemu-iotests/check +12 -1
index 875399d79f..588c453a94 100755
@@ -633,6 +633,12 @@ then
     export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
 fi
 
+python_usable=false
+if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
+then
+    python_usable=true
+fi
+
 default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
 default_alias_machine=$($QEMU_PROG -machine help | \
    sed -n "/(alias of $default_machine)/ { s/ .*//p; q; }")
@@ -809,7 +815,12 @@ do
         start=$(_wallclock)
 
         if [ "$(head -n 1 "$source_iotests/$seq")" == "#!/usr/bin/env python" ]; then
-            run_command="$PYTHON $seq"
+            if $python_usable; then
+                run_command="$PYTHON $seq"
+            else
+                run_command="false"
+                echo "Unsupported Python version" > $seq.notrun
+            fi
         else
             run_command="./$seq"
         fi
Signature · 1 line
2.20.1
Reviewed-by on v2 1/2
#2 lore raw
Quoted 9 lines · Kevin Wolf

Running iotests is not required to build QEMU, so we can have stricter version requirements for Python here and can make use of new features and drop compatibility code earlier.

This makes qemu-iotests skip all Python tests if a Python version before 3.6 is used for the build.

Signature · 1 line
Eduardo
Reviewed-by on v2 1/2
#3 lore raw
In reply to Kevin Wolf
Quoted 35 lines · Kevin Wolf

Running iotests is not required to build QEMU, so we can have stricter version requirements for Python here and can make use of new features and drop compatibility code earlier.

This makes qemu-iotests skip all Python tests if a Python version before 3.6 is used for the build.

---
 tests/qemu-iotests/check | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
tests/qemu-iotests/check +6 -0
index 875399d79f..588c453a94 100755
@@ -633,6 +633,12 @@ then
     export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
 fi
 
+python_usable=false
+if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
+then
+    python_usable=true
+fi
+
 default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
 default_alias_machine=$($QEMU_PROG -machine help | \
    sed -n "/(alias of $default_machine)/ { s/ .*//p; q; }")
@@ -809,7 +815,12 @@ do
         start=$(_wallclock)
 
         if [ "$(head -n 1 "$source_iotests/$seq")" == "#!/usr/bin/env python" ]; then

tests/qemu-iotests/check do · line 826

Quoted 10 lines
tests/qemu-iotests/check +6 -1
-            run_command="$PYTHON $seq"
+            if $python_usable; then
+                run_command="$PYTHON $seq"
+            else
+                run_command="false"
+                echo "Unsupported Python version" > $seq.notrun
+            fi
         else
             run_command="./$seq"
         fi
#4 lore raw
In reply to Kevin Wolf
Quoted 31 lines · Kevin Wolf

Running iotests is not required to build QEMU, so we can have stricter version requirements for Python here and can make use of new features and drop compatibility code earlier.

This makes qemu-iotests skip all Python tests if a Python version before 3.6 is used for the build.

---
  tests/qemu-iotests/check | 13 ++++++++++++-
  1 file changed, 12 insertions(+), 1 deletion(-)
tests/qemu-iotests/check +6 -0
index 875399d79f..588c453a94 100755
@@ -633,6 +633,12 @@ then
      export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
  fi
 
+python_usable=false
+if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
+then
+    python_usable=true
+fi
+
  default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
  default_alias_machine=$($QEMU_PROG -machine help | \
     sed -n "/(alias of $default_machine)/ { s/ .*//p; q; }")

tests/qemu-iotests/check do · line 818

Quoted 5 lines
tests/qemu-iotests/check +1 -1
@@ -809,7 +815,12 @@ do
          start=$(_wallclock)
 
          if [ "$(head -n 1 "$source_iotests/$seq")" == "#!/usr/bin/env python" ]; then
-            run_command="$PYTHON $seq"
+            if $python_usable; then

hmm I wanted to say that this should not work, as python_usable is a string. But I checked and see - it's work. Wow. Googled. And now I understand that here "false" or "true" command is called, to obtain it's return value.. I don't like bash and don't know its best practice, but I'd prefer python_usable to be just return value of your python command, like

above:

$PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)' python_usable=$?

and here:

if [ $python_usable -eq 0 ]; then

Quoted 9 lines
tests/qemu-iotests/check +5 -0
+                run_command="$PYTHON $seq"
+            else
+                run_command="false"
+                echo "Unsupported Python version" > $seq.notrun
+            fi
          else
              run_command="./$seq"
          fi
Signature · 2 lines
Best regards,
Vladimir
author
#5 lore raw

Am 20.09.2019 um 10:40 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 45 lines
Quoted 31 lines · Kevin Wolf

Running iotests is not required to build QEMU, so we can have stricter version requirements for Python here and can make use of new features and drop compatibility code earlier.

This makes qemu-iotests skip all Python tests if a Python version before 3.6 is used for the build.

---
  tests/qemu-iotests/check | 13 ++++++++++++-
  1 file changed, 12 insertions(+), 1 deletion(-)
tests/qemu-iotests/check +6 -0
index 875399d79f..588c453a94 100755
@@ -633,6 +633,12 @@ then
      export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
  fi
 
+python_usable=false
+if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
+then
+    python_usable=true
+fi
+
  default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
  default_alias_machine=$($QEMU_PROG -machine help | \
     sed -n "/(alias of $default_machine)/ { s/ .*//p; q; }")

tests/qemu-iotests/check do · line 818

Quoted 5 lines
tests/qemu-iotests/check +1 -1
@@ -809,7 +815,12 @@ do
          start=$(_wallclock)
 
          if [ "$(head -n 1 "$source_iotests/$seq")" == "#!/usr/bin/env python" ]; then
-            run_command="$PYTHON $seq"
+            if $python_usable; then

hmm I wanted to say that this should not work, as python_usable is a string. But I checked and see - it's work. Wow. Googled. And now I understand that here "false" or "true" command is called, to obtain it's return value.. I don't like bash and don't know its best practice, but I'd prefer python_usable to be just return value of your python command, like

Quoted 8 lines

above:

   $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
   python_usable=$?

and here:

if [ $python_usable -eq 0 ]; then

I'm just trying to be consistent with other variables used in the 'check' script. It has many boolean variables and they all end up calling the true/false commands.

And actually I think the version used is easier to read, even if maybe not as easy to understand why exactly it works.

Kevin

#6 lore raw
Quoted 23 lines · Kevin Wolf

Am 20.09.2019 um 10:40 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 22 lines
Quoted 21 lines · Kevin Wolf

Running iotests is not required to build QEMU, so we can have stricter version requirements for Python here and can make use of new features and drop compatibility code earlier.

This makes qemu-iotests skip all Python tests if a Python version before 3.6 is used for the build.

---
   tests/qemu-iotests/check | 13 ++++++++++++-
   1 file changed, 12 insertions(+), 1 deletion(-)
tests/qemu-iotests/check +0 -0
index 875399d79f..588c453a94 100755
@@ -633,6 +633,12 @@ then
       export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
   fi
 
Quoted 5 lines
Quoted 5 lines

tests/qemu-iotests/check then · line 640

Quoted 5 lines
tests/qemu-iotests/check +5 -0
+python_usable=false
+if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
+then
+    python_usable=true
+fi

don't we want
else
PYTHON="false"
fi

to prevent some occasional unchecked call below?

Quoted 23 lines
Quoted 23 lines

tests/qemu-iotests/check do · line 818

Quoted 10 lines
tests/qemu-iotests/check +2 -1
+
   default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
   default_alias_machine=$($QEMU_PROG -machine help | \
      sed -n "/(alias of $default_machine)/ { s/ .*//p; q; }")
@@ -809,7 +815,12 @@ do
           start=$(_wallclock)
 
           if [ "$(head -n 1 "$source_iotests/$seq")" == "#!/usr/bin/env python" ]; then
-            run_command="$PYTHON $seq"
+            if $python_usable; then

hmm I wanted to say that this should not work, as python_usable is a string. But I checked and see - it's work. Wow. Googled. And now I understand that here "false" or "true" command is called, to obtain it's return value.. I don't like bash and don't know its best practice, but I'd prefer python_usable to be just return value of your python command, like

above:

    $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
    python_usable=$?
Quoted 7 lines
Quoted 3 lines

and here:

if [ $python_usable -eq 0 ]; then

I'm just trying to be consistent with other variables used in the 'check' script. It has many boolean variables and they all end up calling the true/false commands.

Missed this, sorry. Then OK, of course..

still,
echo "Unsupported Python version (must be >= 3.6)" > $seq.notrun
may be a bit more useful, if someone see it.

With or without any of suggestions:

Signature · 2 lines
Best regards,
Vladimir
author
#7 lore raw

PATCH v2 2/2 iotests: Remove Python 2 compatibility code

In reply to Kevin Wolf

Some scripts check the Python version number and have two code paths to accomodate both Python 2 and 3. Remove the code specific to Python 2 and assert the minimum version of 3.6 instead (check skips Python tests in this case, so the assertion would only ever trigger if a Python script is executed manually).

---
 tests/qemu-iotests/044                   |  3 ---
 tests/qemu-iotests/163                   |  3 ---
 tests/qemu-iotests/iotests.py            | 13 +++----------
 tests/qemu-iotests/nbd-fault-injector.py |  7 +++----
 4 files changed, 6 insertions(+), 20 deletions(-)
tests/qemu-iotests/044 +0 -3
index 05ea1f49c5..8b2afa2a11 100755
@@ -28,9 +28,6 @@ import struct
 import subprocess
 import sys
 
-if sys.version_info.major == 2:
-    range = xrange
-
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
 class TestRefcountTableGrowth(iotests.QMPTestCase):
tests/qemu-iotests/163 +0 -3
index 081ccc8ac1..d94728e080 100755
@@ -21,9 +21,6 @@
 import os, random, iotests, struct, qcow2, sys
 from iotests import qemu_img, qemu_io, image_size
 
-if sys.version_info.major == 2:
-    range = xrange
-
 test_img = os.path.join(iotests.test_dir, 'test.img')
 check_img = os.path.join(iotests.test_dir, 'check.img')
 
tests/qemu-iotests/iotests.py +3 -10
index b26271187c..9fb5181c3d 100644
@@ -35,6 +35,7 @@ from collections import OrderedDict
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import qtest
 
+assert sys.version_info >= (3,6)
 
 # This will not work if arguments contain spaces but is necessary if we
 # want to support the override options that ./check supports.
@@ -250,10 +251,7 @@ def image_size(img):
     return json.loads(r)['virtual-size']
 
 def is_str(val):
-    if sys.version_info.major >= 3:
-        return isinstance(val, str)
-    else:
-        return isinstance(val, str) or isinstance(val, unicode)
+    return isinstance(val, str)
 
 test_dir_re = re.compile(r"%s" % test_dir)
 def filter_test_dir(msg):
@@ -935,12 +933,7 @@ def execute_test(test_function=None,
     else:
         # We need to filter out the time taken from the output so that
         # qemu-iotest can reliably diff the results against master output.
-        if sys.version_info.major >= 3:
-            output = io.StringIO()
-        else:
-            # io.StringIO is for unicode strings, which is not what
-            # 2.x's test runner emits.
-            output = io.BytesIO()
+        output = io.StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +3 -4
index 6b2d659dee..43f095ceef 100755
@@ -48,10 +48,9 @@ import sys
 import socket
 import struct
 import collections
-if sys.version_info.major >= 3:
-    import configparser
-else:
-    import ConfigParser as configparser
+import configparser
+
+assert sys.version_info >= (3,6)
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
Signature · 1 line
2.20.1
Reviewed-by on v2 2/2
#8 lore raw

Re: [PATCH v2 2/2] iotests: Remove Python 2 compatibility code

Quoted 7 lines · Kevin Wolf

Some scripts check the Python version number and have two code paths to accomodate both Python 2 and 3. Remove the code specific to Python 2 and assert the minimum version of 3.6 instead (check skips Python tests in this case, so the assertion would only ever trigger if a Python script is executed manually).

Signature · 1 line
Eduardo
Reviewed-by on v2 2/2
#9 lore raw

Re: [PATCH v2 2/2] iotests: Remove Python 2 compatibility code

In reply to Kevin Wolf
Quoted 88 lines · Kevin Wolf

Some scripts check the Python version number and have two code paths to accomodate both Python 2 and 3. Remove the code specific to Python 2 and assert the minimum version of 3.6 instead (check skips Python tests in this case, so the assertion would only ever trigger if a Python script is executed manually).

---
 tests/qemu-iotests/044                   |  3 ---
 tests/qemu-iotests/163                   |  3 ---
 tests/qemu-iotests/iotests.py            | 13 +++----------
 tests/qemu-iotests/nbd-fault-injector.py |  7 +++----
 4 files changed, 6 insertions(+), 20 deletions(-)
tests/qemu-iotests/044 +0 -3
index 05ea1f49c5..8b2afa2a11 100755
@@ -28,9 +28,6 @@ import struct
 import subprocess
 import sys
 
-if sys.version_info.major == 2:
-    range = xrange
-
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
 class TestRefcountTableGrowth(iotests.QMPTestCase):
tests/qemu-iotests/163 +0 -3
index 081ccc8ac1..d94728e080 100755
@@ -21,9 +21,6 @@
 import os, random, iotests, struct, qcow2, sys
 from iotests import qemu_img, qemu_io, image_size
 
-if sys.version_info.major == 2:
-    range = xrange
-
 test_img = os.path.join(iotests.test_dir, 'test.img')
 check_img = os.path.join(iotests.test_dir, 'check.img')
 
tests/qemu-iotests/iotests.py +3 -10
index b26271187c..9fb5181c3d 100644
@@ -35,6 +35,7 @@ from collections import OrderedDict
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import qtest
 
+assert sys.version_info >= (3,6)
 
 # This will not work if arguments contain spaces but is necessary if we
 # want to support the override options that ./check supports.
@@ -250,10 +251,7 @@ def image_size(img):
     return json.loads(r)['virtual-size']
 
 def is_str(val):
-    if sys.version_info.major >= 3:
-        return isinstance(val, str)
-    else:
-        return isinstance(val, str) or isinstance(val, unicode)
+    return isinstance(val, str)
 
 test_dir_re = re.compile(r"%s" % test_dir)
 def filter_test_dir(msg):
@@ -935,12 +933,7 @@ def execute_test(test_function=None,
     else:
         # We need to filter out the time taken from the output so that
         # qemu-iotest can reliably diff the results against master output.
-        if sys.version_info.major >= 3:
-            output = io.StringIO()
-        else:
-            # io.StringIO is for unicode strings, which is not what
-            # 2.x's test runner emits.
-            output = io.BytesIO()
+        output = io.StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +0 -0
index 6b2d659dee..43f095ceef 100755
@@ -48,10 +48,9 @@ import sys
 import socket
 import struct
 import collections

tests/qemu-iotests/nbd-fault-injector.py import sys · line 56

Quoted 10 lines
tests/qemu-iotests/nbd-fault-injector.py +3 -4
-if sys.version_info.major >= 3:
-    import configparser
-else:
-    import ConfigParser as configparser
+import configparser
+
+assert sys.version_info >= (3,6)
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
Reviewed-by on v2 2/2
#10 lore raw

Re: [PATCH v2 2/2] iotests: Remove Python 2 compatibility code

In reply to Kevin Wolf
Quoted 89 lines · Kevin Wolf

Some scripts check the Python version number and have two code paths to accomodate both Python 2 and 3. Remove the code specific to Python 2 and assert the minimum version of 3.6 instead (check skips Python tests in this case, so the assertion would only ever trigger if a Python script is executed manually).

---
 tests/qemu-iotests/044                   |  3 ---
 tests/qemu-iotests/163                   |  3 ---
 tests/qemu-iotests/iotests.py            | 13 +++----------
 tests/qemu-iotests/nbd-fault-injector.py |  7 +++----
 4 files changed, 6 insertions(+), 20 deletions(-)
tests/qemu-iotests/044 +0 -3
index 05ea1f49c5..8b2afa2a11 100755
@@ -28,9 +28,6 @@ import struct
 import subprocess
 import sys
 
-if sys.version_info.major == 2:
-    range = xrange
-
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
 class TestRefcountTableGrowth(iotests.QMPTestCase):
tests/qemu-iotests/163 +0 -3
index 081ccc8ac1..d94728e080 100755
@@ -21,9 +21,6 @@
 import os, random, iotests, struct, qcow2, sys
 from iotests import qemu_img, qemu_io, image_size
 
-if sys.version_info.major == 2:
-    range = xrange
-
 test_img = os.path.join(iotests.test_dir, 'test.img')
 check_img = os.path.join(iotests.test_dir, 'check.img')
 
tests/qemu-iotests/iotests.py +3 -10
index b26271187c..9fb5181c3d 100644
@@ -35,6 +35,7 @@ from collections import OrderedDict
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import qtest
 
+assert sys.version_info >= (3,6)
 
 # This will not work if arguments contain spaces but is necessary if we
 # want to support the override options that ./check supports.
@@ -250,10 +251,7 @@ def image_size(img):
     return json.loads(r)['virtual-size']
 
 def is_str(val):
-    if sys.version_info.major >= 3:
-        return isinstance(val, str)
-    else:
-        return isinstance(val, str) or isinstance(val, unicode)
+    return isinstance(val, str)
 
 test_dir_re = re.compile(r"%s" % test_dir)
 def filter_test_dir(msg):
@@ -935,12 +933,7 @@ def execute_test(test_function=None,
     else:
         # We need to filter out the time taken from the output so that
         # qemu-iotest can reliably diff the results against master output.
-        if sys.version_info.major >= 3:
-            output = io.StringIO()
-        else:
-            # io.StringIO is for unicode strings, which is not what
-            # 2.x's test runner emits.
-            output = io.BytesIO()
+        output = io.StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +0 -0
index 6b2d659dee..43f095ceef 100755
@@ -48,10 +48,9 @@ import sys
 import socket
 import struct
 import collections

tests/qemu-iotests/nbd-fault-injector.py import sys · line 56

Quoted 10 lines
tests/qemu-iotests/nbd-fault-injector.py +3 -4
-if sys.version_info.major >= 3:
-    import configparser
-else:
-    import ConfigParser as configparser
+import configparser
+
+assert sys.version_info >= (3,6)
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
Reviewed-by on v2 2/2
#11 lore raw

Re: [PATCH v2 2/2] iotests: Remove Python 2 compatibility code

In reply to Kevin Wolf
Quoted 7 lines · Kevin Wolf

Some scripts check the Python version number and have two code paths to accomodate both Python 2 and 3. Remove the code specific to Python 2 and assert the minimum version of 3.6 instead (check skips Python tests in this case, so the assertion would only ever trigger if a Python script is executed manually).

Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on the v2 series
#12 lore raw
In reply to Kevin Wolf
Quoted 16 lines · Kevin Wolf

v2:

- Provide the right exit code from Python instead of having a potentially confusing negation in the shell script

- Raised the minimal version to 3.6. If we're going to use a different version than QEMU as a whole anyway, we can use a version that suits us best. 3.5 would only be for Debian Stretch, and we don't really care that much about running Python test cases on it.

- Added a patch to remove Python 2 compatibility code

Kevin Wolf (2):
iotests: Require Python 3.6 or later
iotests: Remove Python 2 compatibility code

Quoted 6 lines
 tests/qemu-iotests/044                   |  3 ---
 tests/qemu-iotests/163                   |  3 ---
 tests/qemu-iotests/check                 | 13 ++++++++++++-
 tests/qemu-iotests/iotests.py            | 13 +++----------
 tests/qemu-iotests/nbd-fault-injector.py |  7 +++----
 5 files changed, 18 insertions(+), 21 deletions(-)