Switch to Python3

PATCH Applied Bart Van Assche 2 participants 7 messages Jun 7, 2020

lore

Share & tools

lore

Bart Van Assche

author

2020-06-07 19:10 UTC #0 lore raw

cover Switch to Python3

Hi Jens,

Since Python2 is in maintenance mode, this patch series converts the fio test scripts to Python3. Please consider this patch series for inclusion in the upstream fio repository. Thanks to Sitsofe Wheeler and Vincent Fu for the help with reviewing this series. See also
https://github.com/axboe/fio/pull/1002.

Thanks,

Bart.

Bart Van Assche (4):
.travis.yml: Move shell code out of .travis.yml ci/travis-install.sh: Install python3-six package Switch to Python3
t/run-fio-tests.py: Accept a wider range of IOPS values

Vincent Fu (1):
ci/travis-install.sh, MacOS: Install the Python 'six' package

 .travis.yml                         | 28 ++----------------
 ci/travis-build.sh                  | 16 +++++++++++
 ci/travis-install.sh                | 44 +++++++++++++++++++++++++++++
 t/run-fio-tests.py                  |  2 +-
 t/sgunmap-perf.py                   |  2 +-
 t/sgunmap-test.py                   |  2 +-
 tools/fio_jsonplus_clat2csv         |  2 +-
 tools/fiologparser.py               |  3 +-
 tools/hist/fio-histo-log-pctiles.py |  5 ++--
 tools/hist/fiologparser_hist.py     | 11 +++-----
 tools/hist/half-bins.py             |  2 +-
 tools/plot/fio2gnuplot              |  2 +-
 12 files changed, 78 insertions(+), 41 deletions(-)
 create mode 100755 ci/travis-build.sh
 create mode 100755 ci/travis-install.sh

Bart Van Assche

author

2020-06-07 19:11 UTC #1 lore raw

PATCH 1/5 .travis.yml: Move shell code out of .travis.yml

Shell code in a .travis.yml file is hard to test. Make it easy to run the shell code locally and to verify that code with shellcheck by moving the shell code into separate files.

Signed-off-by Bart Van Assche <[email protected]>

---
 .travis.yml          | 28 +++-------------------------
 ci/travis-build.sh   | 16 ++++++++++++++++
 ci/travis-install.sh | 31 +++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 25 deletions(-)
 create mode 100755 ci/travis-build.sh
 create mode 100755 ci/travis-install.sh

.travis.yml +3 ~~-25~~

index eba16baa2d03..b64f0a958bbc 100644
@@ -39,29 +39,7 @@ matrix:
       arch: arm64
 
 before_install:
-  - EXTRA_CFLAGS="-Werror"
-  - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
-        pkgs=(libaio-dev libcunit1 libcunit1-dev libgoogle-perftools4 libibverbs-dev libiscsi-dev libnuma-dev librbd-dev librdmacm-dev libz-dev);
-        if [[ "$BUILD_ARCH" == "x86" ]]; then
-            pkgs=("${pkgs[@]/%/:i386}");
-            pkgs+=(gcc-multilib python3-scipy);
-            EXTRA_CFLAGS="${EXTRA_CFLAGS} -m32";
-        else
-            pkgs+=(glusterfs-common python3-scipy);
-        fi;
-        sudo apt-get -qq update;
-        sudo apt-get install --no-install-recommends -qq -y "${pkgs[@]}";
-    fi;
-  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
-        brew update;
-        brew install cunit;
-        pip3 install scipy;
-    fi;
+  - ci/travis-install.sh
+
 script:
-  - ./configure --extra-cflags="${EXTRA_CFLAGS}" && make
-  - make test
-  - if [[ "$TRAVIS_CPU_ARCH" == "arm64" ]]; then
-        sudo python3 t/run-fio-tests.py --skip 6 1007 1008 --debug -p 1010:"--skip 15 16 17 18 19 20";
-    else
-        sudo python3 t/run-fio-tests.py --skip 6 1007 1008 --debug;
-    fi;
+  - ci/travis-build.sh

ci/travis-build.sh new file +16 -0

new file mode 100755
index 000000000000..fff9c088b35f
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+EXTRA_CFLAGS="-Werror"
+
+if [[ "$BUILD_ARCH" == "x86" ]]; then
+    EXTRA_CFLAGS="${EXTRA_CFLAGS} -m32"
+fi
+
+./configure --extra-cflags="${EXTRA_CFLAGS}" &&
+    make &&
+    make test &&
+    if [[ "$TRAVIS_CPU_ARCH" == "arm64" ]]; then
+	sudo python3 t/run-fio-tests.py --skip 6 1007 1008 --debug -p 1010:"--skip 15 16 17 18 19 20"
+    else
+	sudo python3 t/run-fio-tests.py --skip 6 1007 1008 --debug
+    fi

ci/travis-install.sh new file +31 -0

new file mode 100755
index 000000000000..096347ec662a
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+case "$TRAVIS_OS_NAME" in
+    "linux")
+	pkgs=(
+	    libaio-dev
+	    libcunit1
+	    libcunit1-dev
+	    libgoogle-perftools4
+	    libibverbs-dev
+	    libiscsi-dev
+	    libnuma-dev
+	    librbd-dev
+	    librdmacm-dev
+	    libz-dev
+	)
+	if [[ "$BUILD_ARCH" == "x86" ]]; then
+	    pkgs=("${pkgs[@]/%/:i386}")
+	    pkgs+=(gcc-multilib python3-scipy)
+	else
+	    pkgs+=(glusterfs-common python3-scipy)
+	fi
+	sudo apt-get -qq update
+	sudo apt-get install --no-install-recommends -qq -y "${pkgs[@]}"
+	;;
+    "osx")
+	brew update
+	brew install cunit
+	pip3 install scipy
+	;;
+esac

Bart Van Assche

author

2020-06-07 19:11 UTC #2 lore raw

PATCH 2/5 ci/travis-install.sh: Install python3-six package

In reply to Bart Van Assche

Additionally, report the Python3 path and version.

Signed-off-by Bart Van Assche <[email protected]>

---
 ci/travis-install.sh | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

ci/travis-install.sh +15 -3

index 096347ec662a..8a4f40e57a3a 100755
@@ -2,6 +2,7 @@
 
 case "$TRAVIS_OS_NAME" in
     "linux")
+	# Architecture-dependent packages.
 	pkgs=(
 	    libaio-dev
 	    libcunit1

@@ -16,16 +17,27 @@ case "$TRAVIS_OS_NAME" in
 	)
 	if [[ "$BUILD_ARCH" == "x86" ]]; then
 	    pkgs=("${pkgs[@]/%/:i386}")
-	    pkgs+=(gcc-multilib python3-scipy)
+	    pkgs+=(gcc-multilib)
 	else
-	    pkgs+=(glusterfs-common python3-scipy)
+	    pkgs+=(glusterfs-common)
 	fi
+	# Architecture-independent packages and packages for which we don't
+	# care about the architecture.
+	pkgs+=(
+	    python3
+	    python3-scipy
+	    python3-six
+	)
 	sudo apt-get -qq update
 	sudo apt-get install --no-install-recommends -qq -y "${pkgs[@]}"
 	;;
     "osx")
-	brew update
+	brew update >/dev/null 2>&1
 	brew install cunit
 	pip3 install scipy
 	;;
 esac
+
+echo "Python version: $(/usr/bin/python -V 2>&1)"
+echo "Python3 path: $(which python3 2>&1)"
+echo "Python3 version: $(python3 -V 2>&1)"

Bart Van Assche

author

2020-06-07 19:11 UTC #3 lore raw

PATCH 3/5 ci/travis-install.sh, MacOS: Install the Python 'six' package

In reply to Bart Van Assche

From: Vincent Fu <[email protected]>

Signed-off-by Vincent Fu <[email protected]>

[bvanassche: edited commit message / only retained "pip install six" command /
changed 'pip' into 'pip3']

Signed-off-by Bart Van Assche <[email protected]>

---
 ci/travis-install.sh | 1 +
 1 file changed, 1 insertion(+)

ci/travis-install.sh +1 -0

index 8a4f40e57a3a..232ab6b10c3c 100755
@@ -35,6 +35,7 @@ case "$TRAVIS_OS_NAME" in
 	brew update >/dev/null 2>&1
 	brew install cunit
 	pip3 install scipy
+	pip3 install six
 	;;
 esac

Bart Van Assche

author

2020-06-07 19:11 UTC #4 lore raw

PATCH 4/5 Switch to Python3

In reply to Bart Van Assche

Support for Python2.7 in Linux distributions is being phased out. As an example, some Linux distributions do not provide Python2.7 versions of some of the Python packages fio Python scripts relies on. Hence switch to Python3. Most of the Python code changes in this patch have been generated by running the Python scripts through 2to3, the Python2 to Python3 converter from python.org.

Notes:
- Python is only used by fio test scripts and not by fio itself.
- Older versions of RHEL / CentOS 6 do not support Python3. Python3 binaries for RHEL 6 / CentOS 6 are available e.g. in the EPEL repository (https://fedoraproject.org/wiki/EPEL).

Signed-off-by Bart Van Assche <[email protected]>

---
 t/sgunmap-perf.py                   |  2 +-
 t/sgunmap-test.py                   |  2 +-
 tools/fio_jsonplus_clat2csv         |  2 +-
 tools/fiologparser.py               |  3 ++-
 tools/hist/fio-histo-log-pctiles.py |  5 +++--
 tools/hist/fiologparser_hist.py     | 11 ++++-------
 tools/hist/half-bins.py             |  2 +-
 tools/plot/fio2gnuplot              |  2 +-
 8 files changed, 14 insertions(+), 15 deletions(-)

t/sgunmap-perf.py +1 -1

index fadbb85933e6..962d187da25b 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.7
+#!/usr/bin/env python3
 #
 # sgunmap-test.py
 #

t/sgunmap-test.py +1 -1

index f8f10ab38274..4960a040ea34 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.7
+#!/usr/bin/env python3
 # Note: this script is python2 and python 3 compatible.
 #
 # sgunmap-test.py

tools/fio_jsonplus_clat2csv +1 -1

index 9544ab747ce3..7f310fcc473c 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.7
+#!/usr/bin/env python3
 # Note: this script is python2 and python3 compatible.
 
 """

tools/fiologparser.py +2 -1

index cc29f1c7bcbf..054f1f607848 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.7
+#!/usr/bin/env python3
 # Note: this script is python2 and python 3 compatible.
 #
 # fiologparser.py

@@ -18,6 +18,7 @@ from __future__ import absolute_import
 from __future__ import print_function
 import argparse
 import math
+from functools import reduce
 
 def parse_args():
     parser = argparse.ArgumentParser()

tools/hist/fio-histo-log-pctiles.py +3 -2

index f9df2a3dada8..08e7722d04fe 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # module to parse fio histogram log files, not using pandas
 # runs in python v2 or v3

@@ -24,6 +24,7 @@
 import sys, os, math, copy, time
 from copy import deepcopy
 import argparse
+from functools import reduce
 
 unittest2_imported = True
 try:

@@ -82,7 +83,7 @@ def parse_hist_file(logfn, buckets_per_interval, log_hist_msec):
         except ValueError as e:
             raise FioHistoLogExc('non-integer value %s' % exception_suffix(k+1, logfn))
 
-        neg_ints = list(filter( lambda tk : tk < 0, int_tokens ))
+        neg_ints = list([tk for tk in int_tokens if tk < 0])
         if len(neg_ints) > 0:
             raise FioHistoLogExc('negative integer value %s' % exception_suffix(k+1, logfn))

tools/hist/fiologparser_hist.py +4 -7

index 8910d5fa9881..159454b15908 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.7
+#!/usr/bin/env python3
 """ 
     Utility for converting *_clat_hist* files generated by fio into latency statistics.

@@ -124,7 +124,7 @@ def gen_output_columns(ctx):
         columns = ["end-time", "dir", "samples", "min", "avg", "median"]
     else:
         columns = ["end-time", "samples", "min", "avg", "median"]
-    columns.extend(list(map(lambda x: x+'%', strpercs)))
+    columns.extend(list([x+'%' for x in strpercs]))
     columns.append("max")
 
 def fmt_float_list(ctx, num=1):

@@ -339,7 +339,7 @@ def guess_max_from_bins(ctx, hist_cols):
     else:
         bins = [1216,1280,1344,1408,1472,1536,1600,1664]
     coarses = range(max_coarse + 1)
-    fncn = lambda z: list(map(lambda x: z/2**x if z % 2**x == 0 else -10, coarses))
+    fncn = lambda z: list([z/2**x if z % 2**x == 0 else -10 for x in coarses])
     
     arr = np.transpose(list(map(fncn, bins)))
     idx = np.where(arr == hist_cols)

@@ -470,10 +470,7 @@ def output_interval_data(ctx,directions):
 def main(ctx):
 
     if ctx.job_file:
-        try:
-            from configparser import SafeConfigParser, NoOptionError
-        except ImportError:
-            from ConfigParser import SafeConfigParser, NoOptionError
+        from configparser import SafeConfigParser, NoOptionError
 
         cp = SafeConfigParser(allow_no_value=True)
         with open(ctx.job_file, 'r') as fp:

tools/hist/half-bins.py +1 -1

index 1bba8ff74802..42af954002c2 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.7
+#!/usr/bin/env python3
 """ Cut the number bins in half in fio histogram output. Example usage:
 
         $ half-bins.py -c 2 output_clat_hist.1.log > smaller_clat_hist.1.log

tools/plot/fio2gnuplot +1 -1

index 69aa791eb686..78ee82fb8025 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.7
+#!/usr/bin/env python3
 # Note: this script is python2 and python3 compatible.
 #
 #  Copyright (C) 2013 eNovance SAS <[email protected]>

Bart Van Assche

author

2020-06-07 19:11 UTC #5 lore raw

PATCH 5/5 t/run-fio-tests.py: Accept a wider range of IOPS values

In reply to Bart Van Assche

This patch prevents that test 11 sporadically fails as follows:

DEBUG:root:Test 11: return code: 0 DEBUG:root:Test 11: skipped 0 lines decoding JSON data DEBUG:root:Test 11: iops1: 997.670549 DEBUG:root:Test 11: ratio: 8.163684

Test 11 FAILED: iops value mismatch,

Signed-off-by Bart Van Assche <[email protected]>

---
 t/run-fio-tests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

t/run-fio-tests.py +1 -1

index e7063d3eb63a..c2352d80c2bf 100755
@@ -438,7 +438,7 @@ class FioJobTest_t0011(FioJobTest):
         logging.debug("Test %d: iops1: %f", self.testnum, iops1)
         logging.debug("Test %d: ratio: %f", self.testnum, ratio)
 
-        if iops1 < 998 or iops1 > 1002:
+        if iops1 < 997 or iops1 > 1003:
             self.failure_reason = "{0} iops value mismatch,".format(self.failure_reason)
             self.passed = False

Jens Axboe

Applied

2020-06-07 23:30 UTC #6 lore raw

In reply to Bart Van Assche

Quoted 7 lines · Bart Van Assche

Hi Jens,
Since Python2 is in maintenance mode, this patch series converts the fio test scripts to Python3. Please consider this patch series for inclusion in the upstream fio repository. Thanks to Sitsofe Wheeler and Vincent Fu for the help with reviewing this series. See also
https://github.com/axboe/fio/pull/1002.

Applied, thanks Bart!

Signature · 1 line

Jens Axboe