Skip to contentSkip to conversation

iotests: Make them work for both Python 2 and 3

PATCH Superseded Max Reitz 5 participants 48 messages Oct 15, 2018 – Oct 19, 2018

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

cover iotests: Make them work for both Python 2 and 3

This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO class from the 'io' or the 'StringIO' module), but most of the time, it just makes code work for both versions in general.

And when we make the switch to make Python 3 mandatory, we can simply drop the Python 2 branches.

Max Reitz (9):
iotests: Make nbd-fault-injector flush
iotests: Flush in iotests.py's QemuIoInteractive
iotests: Use Python byte strings where appropriate
iotests: Use // for Python integer division
iotests: Different iterator behavior in Python 3
iotests: Explicitly inherit FDs in Python
iotests: 'new' module replacement in 169
iotests: Modify imports for Python 3
iotests: Unify log outputs between Python 2 and 3

 scripts/qemu.py                          |  13 +-
 scripts/qmp/qmp.py                       |   7 +
 scripts/qtest.py                         |   2 +-
 tests/qemu-iotests/040                   |   4 +-
 tests/qemu-iotests/044                   |  20 +-
 tests/qemu-iotests/056                   |   2 +-
 tests/qemu-iotests/065                   |   4 +-
 tests/qemu-iotests/083.out               |   9 +
 tests/qemu-iotests/093                   |  18 +-
 tests/qemu-iotests/124                   |   4 +-
 tests/qemu-iotests/139                   |   2 +-
 tests/qemu-iotests/147                   |   7 +
 tests/qemu-iotests/149                   |  14 +-
 tests/qemu-iotests/151                   |  12 +-
 tests/qemu-iotests/163                   |   8 +-
 tests/qemu-iotests/169                   |   3 +-
 tests/qemu-iotests/194.out               |  22 +-
 tests/qemu-iotests/202.out               |  12 +-
 tests/qemu-iotests/203.out               |  14 +-
 tests/qemu-iotests/206.out               | 144 +++----
 tests/qemu-iotests/207                   |   4 +-
 tests/qemu-iotests/207.out               |  52 +--
 tests/qemu-iotests/208.out               |   8 +-
 tests/qemu-iotests/210.out               |  72 ++--
 tests/qemu-iotests/211.out               |  66 +--
 tests/qemu-iotests/212.out               | 102 ++---
 tests/qemu-iotests/213.out               | 124 +++---
 tests/qemu-iotests/216.out               |   4 +-
 tests/qemu-iotests/218.out               |  20 +-
 tests/qemu-iotests/219.out               | 526 +++++++++++------------
 tests/qemu-iotests/222.out               |  24 +-
 tests/qemu-iotests/iotests.py            |  64 ++-
 tests/qemu-iotests/nbd-fault-injector.py |  12 +-
 tests/qemu-iotests/qcow2.py              |  10 +-
 34 files changed, 745 insertions(+), 664 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/9 iotests: Make nbd-fault-injector flush

When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.

---
 tests/qemu-iotests/083.out               | 9 +++++++++
 tests/qemu-iotests/nbd-fault-injector.py | 1 +
 2 files changed, 10 insertions(+)
tests/qemu-iotests/083.out +9 -0
index be6079d27e..f9af8bb691 100644
@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
 === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
 === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/
 
 === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
 === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 === Check disconnect 8 neg2 ===
@@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
 === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 === Check disconnect after request ===
@@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock
 
 === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 *** done
tests/qemu-iotests/nbd-fault-injector.py +1 -0
index f9193c0fae..439a090eb6 100755
@@ -112,6 +112,7 @@ class FaultInjectionSocket(object):
             if rule.match(event, io):
                 if rule.when == 0 or bufsize is None:
                     print('Closing connection on rule match %s' % rule.name)
+                    self.sock.flush()
                     sys.exit(0)
                 if rule.when != -1:
                     return rule.when
Signature · 1 line
2.17.1
#4 lore raw

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

In reply to Max Reitz
Quoted 8 lines · Max Reitz

When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.

---
  tests/qemu-iotests/083.out               | 9 +++++++++
  tests/qemu-iotests/nbd-fault-injector.py | 1 +
  2 files changed, 10 insertions(+)

I already had a complaint that the error message in 083.out should NOT be printing a message (whether the server is python 2 and auto-flushes, or python 3 and needs an explicit flush, the message itself is pointless, and the test is racy as a result). We may need to revisit this patch when that thread is resolved.

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html

That said, I'm not opposed to this patch, if it gets iotests to be more useful in the meantime.

Quoted 69 lines
tests/qemu-iotests/083.out +9 -0
index be6079d27e..f9af8bb691 100644
@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
  === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
  Connection closed
  read failed: Input/output error
 
@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
  === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
  Connection closed
  read failed: Input/output error
 
@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/
 
  === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
  Connection closed
  read failed: Input/output error
 
@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
  === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
  read failed: Input/output error
 
  === Check disconnect 8 neg2 ===
@@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
  === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
  read failed: Input/output error
 
  === Check disconnect after request ===
@@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock
 
  === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
  read failed: Input/output error
 
  *** done
tests/qemu-iotests/nbd-fault-injector.py +1 -0
index f9193c0fae..439a090eb6 100755
@@ -112,6 +112,7 @@ class FaultInjectionSocket(object):
              if rule.match(event, io):
                  if rule.when == 0 or bufsize is None:
                      print('Closing connection on rule match %s' % rule.name)
+                    self.sock.flush()
                      sys.exit(0)
                  if rule.when != -1:
                      return rule.when
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#5 lore raw

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

Quoted 4 lines · Eric Blake
Quoted 3 lines · Max Reitz

When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.

Quoted 8 lines
Quoted 5 lines

---
  tests/qemu-iotests/083.out               | 9 +++++++++ tests/qemu-iotests/nbd-fault-injector.py | 1 +

  2 files changed, 10 insertions(+)

I already had a complaint that the error message in 083.out should NOT be printing a message

You mean the NBD server itself, right?

Quoted 6 lines

(whether the server is python 2 and auto-flushes,
or python 3 and needs an explicit flush, the message itself is pointless, and the test is racy as a result).  We may need to revisit this patch when that thread is resolved.

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html

Well, it's not like the flush hurts either way. :-)

Max

Quoted 60 lines

That said, I'm not opposed to this patch, if it gets iotests to be more useful in the meantime.

Quoted 56 lines
tests/qemu-iotests/083.out +0 -0
index be6079d27e..f9af8bb691 100644
@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo

    === Check disconnect after neg2 ===
  +Unable to read from socket: Connection reset by peer Connection closed
  read failed: Input/output error
  @@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect before request ===
  +Unable to read from socket: Connection reset by peer Connection closed
  read failed: Input/output error
  @@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/ === Check disconnect after neg-classic ===
  +Unable to read from socket: Connection reset by peer Connection closed
  read failed: Input/output error
  @@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
    === Check disconnect after neg2 ===
  +Unable to read from socket: Connection reset by peer +Connection closed
  read failed: Input/output error
    === Check disconnect 8 neg2 ===

tests/qemu-iotests/083.out +0 -0
@@ -173,6 +178,8 @@ can't open device

nbd+unix:///foo?socket=TEST_DIR/nbd.sock
    === Check disconnect before request === +Unable to read from socket: Connection reset by peer +Connection closed
  read failed: Input/output error
    === Check disconnect after request ===

tests/qemu-iotests/083.out +0 -0
@@ -234,6 +241,8 @@ can't open device

nbd+unix:///?socket=TEST_DIR/nbd.sock
    === Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer +Connection closed
  read failed: Input/output error
    *** done

tests/qemu-iotests/nbd-fault-injector.py +0 -0
index f9193c0fae..439a090eb6 100755
@@ -112,6 +112,7 @@ class FaultInjectionSocket(object):

              if rule.match(event, io):
                  if rule.when == 0 or bufsize is None:
                      print('Closing connection on rule match %s' % rule.name)
+                    self.sock.flush()
                      sys.exit(0)
                  if rule.when != -1:
                      return rule.when

Reviewed-by
#6 lore raw

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

Quoted 11 lines · Max Reitz
Quoted 10 lines · Eric Blake
Quoted 8 lines · Max Reitz

When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.

---
   tests/qemu-iotests/083.out               | 9 +++++++++
   tests/qemu-iotests/nbd-fault-injector.py | 1 +
   2 files changed, 10 insertions(+)
Quoted 4 lines
Quoted 2 lines

I already had a complaint that the error message in 083.out should NOT be printing a message

You mean the NBD server itself, right?

Yes, the NBD server should not be printing the redundant error messages that 083.out exposes.

Quoted 9 lines
Quoted 7 lines

(whether the server is python 2 and auto-flushes,
or python 3 and needs an explicit flush, the message itself is pointless, and the test is racy as a result).  We may need to revisit this patch when that thread is resolved.

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html

Well, it's not like the flush hurts either way. :-)

True. On that grounds:

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#7 lore raw

PATCH 2/9 iotests: Flush in iotests.py's QemuIoInteractive

In reply to Max Reitz

After issuing a command, flush the pipe. This does not change anything in Python 2, but it makes a difference in Python 3.

---
 tests/qemu-iotests/iotests.py | 1 +
 1 file changed, 1 insertion(+)
tests/qemu-iotests/iotests.py +1 -0
index 4e67fbbe96..10f2d17419 100644
@@ -178,6 +178,7 @@ class QemuIoInteractive:
         cmd = cmd.strip()
         assert cmd != 'q' and cmd != 'quit'
         self._p.stdin.write(cmd + '\n')
+        self._p.stdin.flush()
         return self._read_output()
 
 
Signature · 1 line
2.17.1
author
#10 lore raw

PATCH 3/9 iotests: Use Python byte strings where appropriate

In reply to Max Reitz

Since byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.

On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set univeral_newlines to True so that the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.

---
 scripts/qtest.py                         |  2 +-
 tests/qemu-iotests/044                   |  8 ++++----
 tests/qemu-iotests/149                   |  8 +++++---
 tests/qemu-iotests/207                   |  4 ++--
 tests/qemu-iotests/iotests.py            | 11 +++++++----
 tests/qemu-iotests/nbd-fault-injector.py |  4 ++--
 tests/qemu-iotests/qcow2.py              | 10 +++++-----
 7 files changed, 26 insertions(+), 21 deletions(-)
scripts/qtest.py +1 -1
index df0daf26ca..adf1fe3f26 100644
@@ -64,7 +64,7 @@ class QEMUQtestProtocol(object):
 
         @param qtest_cmd: qtest command text to be sent
         """
-        self._sock.sendall(qtest_cmd + "\n")
+        self._sock.sendall((qtest_cmd + "\n").encode('utf-8'))
 
     def close(self):
         self._sock.close()
tests/qemu-iotests/044 +4 -4
index 11ea0f4d35..69e736f687 100755
@@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             fd.seek(off_reftable)
 
             for i in xrange(0, h.refcount_table_clusters):
-                sector = ''.join(struct.pack('>Q',
+                sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
                     for j in xrange(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
-            sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256))
+            sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
             for block in xrange(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
-            table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
                 for j in xrange(0, h.l1_size))
             fd.write(table)
 
@@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
-            table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
                 for j in xrange(0, remaining / 512))
             fd.write(table)
 
tests/qemu-iotests/149 +5 -3
index 9e0cad76f9..1225334cb8 100755
@@ -79,7 +79,7 @@ class LUKSConfig(object):
 
     def first_password_base64(self):
         (pw, slot) = self.first_password()
-        return base64.b64encode(pw)
+        return base64.b64encode(pw.encode('ascii')).decode('ascii')
 
     def active_slots(self):
         slots = []
@@ -98,7 +98,8 @@ def verify_passwordless_sudo():
     proc = subprocess.Popen(args,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate()[0]
 
@@ -116,7 +117,8 @@ def cryptsetup(args, password=None):
     proc = subprocess.Popen(fullargs,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate(password)[0]
 
tests/qemu-iotests/207 +2 -2
index 444ae233ae..2d86a3da37 100755
@@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \
     md5_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
@@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \
     sha1_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
tests/qemu-iotests/iotests.py +7 -4
index 10f2d17419..7290c0b159 100644
@@ -104,7 +104,8 @@ def qemu_img_pipe(*args):
     '''Run qemu-img and return its output'''
     subp = subprocess.Popen(qemu_img_args + list(args),
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
@@ -128,7 +129,8 @@ def qemu_io(*args):
     '''Run qemu-io and return the stdout data'''
     args = qemu_io_args + list(args)
     subp = subprocess.Popen(args, stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))
@@ -149,7 +151,8 @@ class QemuIoInteractive:
         self.args = qemu_io_args + list(args)
         self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE,
                                    stdout=subprocess.PIPE,
-                                   stderr=subprocess.STDOUT)
+                                   stderr=subprocess.STDOUT,
+                                   universal_newlines=True)
         assert self._p.stdout.read(9) == 'qemu-io> '
 
     def close(self):
@@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
 
 def create_image(name, size):
     '''Create a fully-allocated raw image with sector markers'''
-    file = open(name, 'w')
+    file = open(name, 'wb')
     i = 0
     while i < size:
         sector = struct.pack('>l504xl', i / 512, i / 512)
tests/qemu-iotests/nbd-fault-injector.py +2 -2
index 439a090eb6..d45e2e0a6a 100755
@@ -86,7 +86,7 @@ def recvall(sock, bufsize):
             raise Exception('unexpected disconnect')
         chunks.append(chunk)
         received += len(chunk)
-    return ''.join(chunks)
+    return b''.join(chunks)
 
 class Rule(object):
     def __init__(self, name, event, io, when):
@@ -177,7 +177,7 @@ def handle_connection(conn, use_export):
         req = read_request(conn)
         if req.type == NBD_CMD_READ:
             write_reply(conn, 0, req.handle)
-            conn.send('\0' * req.len, event='data')
+            conn.send(b'\0' * req.len, event='data')
         elif req.type == NBD_CMD_WRITE:
             _ = conn.recv(req.len, event='data')
             write_reply(conn, 0, req.handle)
tests/qemu-iotests/qcow2.py +5 -5
index b95a837759..b392972d1b 100755
@@ -10,7 +10,7 @@ class QcowHeaderExtension:
     def __init__(self, magic, length, data):
         if length % 8 != 0:
             padding = 8 - (length % 8)
-            data += "\0" * padding
+            data += b"\0" * padding
 
         self.magic  = magic
         self.length = length
@@ -103,7 +103,7 @@ class QcowHeader:
 
         fd.seek(self.header_length)
         extensions = self.extensions
-        extensions.append(QcowHeaderExtension(0, 0, ""))
+        extensions.append(QcowHeaderExtension(0, 0, b""))
         for ex in extensions:
             buf = struct.pack('>II', ex.magic, ex.length)
             fd.write(buf)
@@ -137,8 +137,8 @@ class QcowHeader:
         for ex in self.extensions:
 
             data = ex.data[:ex.length]
-            if all(c in string.printable for c in data):
-                data = "'%s'" % data
+            if all(c in string.printable.encode('ascii') for c in data):
+                data = "'%s'" % data.decode('ascii')
             else:
                 data = "<binary>"
 
@@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data):
         sys.exit(1)
 
     h = QcowHeader(fd)
-    h.extensions.append(QcowHeaderExtension.create(magic, data))
+    h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii')))
     h.update(fd)
 
 def cmd_add_header_ext_stdio(fd, magic):
Signature · 1 line
2.17.1
Reviewed-by on v1 3/9
#11 lore raw

Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate

Quoted 15 lines · Max Reitz

Since byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.

On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set univeral_newlines to True so that the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.

[...]

tests/qemu-iotests/149 class LUKSConfig(object): · line 82

Quoted 10 lines
tests/qemu-iotests/149 +1 -1
index 9e0cad76f9..1225334cb8 100755
@@ -79,7 +79,7 @@ class LUKSConfig(object):
 
     def first_password_base64(self):
         (pw, slot) = self.first_password()
-        return base64.b64encode(pw)
+        return base64.b64encode(pw.encode('ascii')).decode('ascii')

Would we want to have a test case for non-ascii passwords in the future? In that case you would probably need to make self.passwords[] contain byte strings instead of text.

In either case, using 'ascii' as the encoding everywhere will ensure the code will not try to be too smart about string encodings if that happens. I like that.

Signature · 1 line
Eduardo
author
#12 lore raw

Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate

Quoted 23 lines · Eduardo Habkost
Quoted 15 lines · Max Reitz

Since byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.

On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set univeral_newlines to True so that the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.

[...]

Quoted 6 lines
tests/qemu-iotests/149 +0 -0
index 9e0cad76f9..1225334cb8 100755
@@ -79,7 +79,7 @@ class LUKSConfig(object):
 
Quoted 8 lines

tests/qemu-iotests/149 class LUKSConfig(object): · line 82

Quoted 4 lines
tests/qemu-iotests/149 +1 -1
     def first_password_base64(self):
         (pw, slot) = self.first_password()
-        return base64.b64encode(pw)
+        return base64.b64encode(pw.encode('ascii')).decode('ascii')

Would we want to have a test case for non-ascii passwords in the future? In that case you would probably need to make self.passwords[] contain byte strings instead of text.

I remember someone once providing a non-ASCII initial password to some system for me. I remember the system was running Windows, and I was running Linux, so the system expected ISO-8859-1, while I was sending UTF-8.

The moral of the story is that you probably don't want non-ASCII passwords. And if we do want to test them, well, we'll need to decide on an encoding then (or use byte strings, as you suggest).

Quoted 5 lines

In either case, using 'ascii' as the encoding everywhere will ensure the code will not try to be too smart about string encodings if that happens. I like that.

Thanks for reviewing!

Max

#13 lore raw

Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate

In reply to Max Reitz
Quoted 10 lines · Max Reitz

Since byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.

On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set univeral_newlines to True so that

'universal_newlines'

Quoted 229 lines

the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.

---
 scripts/qtest.py                         |  2 +-
 tests/qemu-iotests/044                   |  8 ++++----
 tests/qemu-iotests/149                   |  8 +++++---
 tests/qemu-iotests/207                   |  4 ++--
 tests/qemu-iotests/iotests.py            | 11 +++++++----
 tests/qemu-iotests/nbd-fault-injector.py |  4 ++--
 tests/qemu-iotests/qcow2.py              | 10 +++++-----
 7 files changed, 26 insertions(+), 21 deletions(-)
scripts/qtest.py +1 -1
index df0daf26ca..adf1fe3f26 100644
@@ -64,7 +64,7 @@ class QEMUQtestProtocol(object):
 
         @param qtest_cmd: qtest command text to be sent
         """
-        self._sock.sendall(qtest_cmd + "\n")
+        self._sock.sendall((qtest_cmd + "\n").encode('utf-8'))
 
     def close(self):
         self._sock.close()
tests/qemu-iotests/044 +4 -4
index 11ea0f4d35..69e736f687 100755
@@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             fd.seek(off_reftable)
 
             for i in xrange(0, h.refcount_table_clusters):
-                sector = ''.join(struct.pack('>Q',
+                sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
                     for j in xrange(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
-            sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256))
+            sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
             for block in xrange(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
-            table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
                 for j in xrange(0, h.l1_size))
             fd.write(table)
 
@@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
-            table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
                 for j in xrange(0, remaining / 512))
             fd.write(table)
 
tests/qemu-iotests/149 +5 -3
index 9e0cad76f9..1225334cb8 100755
@@ -79,7 +79,7 @@ class LUKSConfig(object):
 
     def first_password_base64(self):
         (pw, slot) = self.first_password()
-        return base64.b64encode(pw)
+        return base64.b64encode(pw.encode('ascii')).decode('ascii')
 
     def active_slots(self):
         slots = []
@@ -98,7 +98,8 @@ def verify_passwordless_sudo():
     proc = subprocess.Popen(args,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate()[0]
 
@@ -116,7 +117,8 @@ def cryptsetup(args, password=None):
     proc = subprocess.Popen(fullargs,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate(password)[0]
 
tests/qemu-iotests/207 +2 -2
index 444ae233ae..2d86a3da37 100755
@@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \
     md5_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
@@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \
     sha1_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
tests/qemu-iotests/iotests.py +7 -4
index 10f2d17419..7290c0b159 100644
@@ -104,7 +104,8 @@ def qemu_img_pipe(*args):
     '''Run qemu-img and return its output'''
     subp = subprocess.Popen(qemu_img_args + list(args),
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
@@ -128,7 +129,8 @@ def qemu_io(*args):
     '''Run qemu-io and return the stdout data'''
     args = qemu_io_args + list(args)
     subp = subprocess.Popen(args, stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))
@@ -149,7 +151,8 @@ class QemuIoInteractive:
         self.args = qemu_io_args + list(args)
         self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE,
                                    stdout=subprocess.PIPE,
-                                   stderr=subprocess.STDOUT)
+                                   stderr=subprocess.STDOUT,
+                                   universal_newlines=True)
         assert self._p.stdout.read(9) == 'qemu-io> '
 
     def close(self):
@@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
 
 def create_image(name, size):
     '''Create a fully-allocated raw image with sector markers'''
-    file = open(name, 'w')
+    file = open(name, 'wb')
     i = 0
     while i < size:
         sector = struct.pack('>l504xl', i / 512, i / 512)
tests/qemu-iotests/nbd-fault-injector.py +2 -2
index 439a090eb6..d45e2e0a6a 100755
@@ -86,7 +86,7 @@ def recvall(sock, bufsize):
             raise Exception('unexpected disconnect')
         chunks.append(chunk)
         received += len(chunk)
-    return ''.join(chunks)
+    return b''.join(chunks)
 
 class Rule(object):
     def __init__(self, name, event, io, when):
@@ -177,7 +177,7 @@ def handle_connection(conn, use_export):
         req = read_request(conn)
         if req.type == NBD_CMD_READ:
             write_reply(conn, 0, req.handle)
-            conn.send('\0' * req.len, event='data')
+            conn.send(b'\0' * req.len, event='data')
         elif req.type == NBD_CMD_WRITE:
             _ = conn.recv(req.len, event='data')
             write_reply(conn, 0, req.handle)
tests/qemu-iotests/qcow2.py +5 -5
index b95a837759..b392972d1b 100755
@@ -10,7 +10,7 @@ class QcowHeaderExtension:
     def __init__(self, magic, length, data):
         if length % 8 != 0:
             padding = 8 - (length % 8)
-            data += "\0" * padding
+            data += b"\0" * padding
 
         self.magic  = magic
         self.length = length
@@ -103,7 +103,7 @@ class QcowHeader:
 
         fd.seek(self.header_length)
         extensions = self.extensions
-        extensions.append(QcowHeaderExtension(0, 0, ""))
+        extensions.append(QcowHeaderExtension(0, 0, b""))
         for ex in extensions:
             buf = struct.pack('>II', ex.magic, ex.length)
             fd.write(buf)
@@ -137,8 +137,8 @@ class QcowHeader:
         for ex in self.extensions:
 
             data = ex.data[:ex.length]
-            if all(c in string.printable for c in data):
-                data = "'%s'" % data
+            if all(c in string.printable.encode('ascii') for c in data):
+                data = "'%s'" % data.decode('ascii')
             else:
                 data = "<binary>"
 
@@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data):
         sys.exit(1)
 
     h = QcowHeader(fd)
-    h.extensions.append(QcowHeaderExtension.create(magic, data))
+    h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii')))
     h.update(fd)
 
 def cmd_add_header_ext_stdio(fd, magic):
author
#14 lore raw

PATCH 4/9 iotests: Use // for Python integer division

In reply to Max Reitz

In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.

---
 tests/qemu-iotests/040        |  4 ++--
 tests/qemu-iotests/044        |  2 +-
 tests/qemu-iotests/093        | 18 +++++++++---------
 tests/qemu-iotests/149        |  6 +++---
 tests/qemu-iotests/151        | 12 ++++++------
 tests/qemu-iotests/163        |  2 +-
 tests/qemu-iotests/iotests.py |  2 +-
 7 files changed, 23 insertions(+), 23 deletions(-)
tests/qemu-iotests/040 +2 -2
index 1cb1ceeb33..b81133a474 100755
@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp('device_del', id='scsi0')
         self.assert_qmp(result, 'return', {})
@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
 
         result = self.vm.qmp('query-block')
tests/qemu-iotests/044 +1 -1
index 69e736f687..7ef5e46fe9 100755
@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining / 512))
+                for j in xrange(0, remaining // 512))
             fd.write(table)
 
 
tests/qemu-iotests/093 +9 -9
index 9d1971a56c..d88fbc182e 100755
@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase):
         # in. The throttled requests won't be executed until we
         # advance the virtual clock.
         rq_size = 512
-        rd_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_rd'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_rd'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_rd'])
         rd_nr *= seconds * 2
-        rd_nr /= ndrives
-        wr_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_wr'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr //= ndrives
+        wr_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_wr'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_wr'])
         wr_nr *= seconds * 2
-        wr_nr /= ndrives
+        wr_nr //= ndrives
 
         # Send I/O requests to all drives
         for i in range(rd_nr):
@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
             self.configure_throttle(ndrives, settings)
 
             # Wait for the bucket to empty so we can do bursts
-            wait_ns = nsec_per_sec * burst_length * burst_rate / rate
+            wait_ns = nsec_per_sec * burst_length * burst_rate // rate
             self.vm.qtest("clock_step %d" % wait_ns)
 
             # Test I/O at the max burst rate
tests/qemu-iotests/149 +3 -3
index 1225334cb8..4f363f295f 100755
@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False):
     image_size = 4 * oneTB
     if qemu_img:
         iotests.log("# Create image")
-        qemu_img_create(config, image_size / oneMB)
+        qemu_img_create(config, image_size // oneMB)
     else:
         iotests.log("# Create image")
-        create_image(config, image_size / oneMB)
+        create_image(config, image_size // oneMB)
 
     lowOffsetMB = 100
-    highOffsetMB = 3 * oneTB / oneMB
+    highOffsetMB = 3 * oneTB // oneMB
 
     try:
         if not qemu_img:
tests/qemu-iotests/151 +6 -6
index fe53b9f446..1bb74d67c4 100755
@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase):
                             'write -P 1 0 %i' % self.image_len);
 
         # Start some background requests
-        for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset)
-        for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Start the block job
@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         self.assert_qmp(result, 'return', {})
 
         # Start some more requests
-        for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Wait for the READY event
@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         # the source) should be settled using the active mechanism.
         # The mirror code itself asserts that the source BDS's dirty
         # bitmap will stay clean between READY and COMPLETED.
-        for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         if sync_source_and_target:
tests/qemu-iotests/163 +1 -1
index 403842354e..5fd424761b 100755
@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         entry_bits = 3
         entry_size = 1 << entry_bits
         l1_mask = 0x00fffffffffffe00
-        div_roundup = lambda n, d: (n + d - 1) / d
+        div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
             for x in xrange(0, len(data), n):
tests/qemu-iotests/iotests.py +1 -1
index 7290c0b159..7ca94e9278 100644
@@ -199,7 +199,7 @@ def create_image(name, size):
     file = open(name, 'wb')
     i = 0
     while i < size:
-        sector = struct.pack('>l504xl', i / 512, i / 512)
+        sector = struct.pack('>l504xl', i // 512, i // 512)
         file.write(sector)
         i = i + 512
     file.close()
Signature · 1 line
2.17.1
Reviewed-by on v1 4/9
#15 lore raw

Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division

Quoted 5 lines · Max Reitz

In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.

Signature · 1 line
Eduardo
#16 lore raw

Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division

In reply to Max Reitz
Quoted 170 lines · Max Reitz

In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.

---
 tests/qemu-iotests/040        |  4 ++--
 tests/qemu-iotests/044        |  2 +-
 tests/qemu-iotests/093        | 18 +++++++++---------
 tests/qemu-iotests/149        |  6 +++---
 tests/qemu-iotests/151        | 12 ++++++------
 tests/qemu-iotests/163        |  2 +-
 tests/qemu-iotests/iotests.py |  2 +-
 7 files changed, 23 insertions(+), 23 deletions(-)
tests/qemu-iotests/040 +2 -2
index 1cb1ceeb33..b81133a474 100755
@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp('device_del', id='scsi0')
         self.assert_qmp(result, 'return', {})
@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
 
         result = self.vm.qmp('query-block')
tests/qemu-iotests/044 +1 -1
index 69e736f687..7ef5e46fe9 100755
@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining / 512))
+                for j in xrange(0, remaining // 512))
             fd.write(table)
 
 
tests/qemu-iotests/093 +9 -9
index 9d1971a56c..d88fbc182e 100755
@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase):
         # in. The throttled requests won't be executed until we
         # advance the virtual clock.
         rq_size = 512
-        rd_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_rd'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_rd'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_rd'])
         rd_nr *= seconds * 2
-        rd_nr /= ndrives
-        wr_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_wr'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr //= ndrives
+        wr_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_wr'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_wr'])
         wr_nr *= seconds * 2
-        wr_nr /= ndrives
+        wr_nr //= ndrives
 
         # Send I/O requests to all drives
         for i in range(rd_nr):
@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
             self.configure_throttle(ndrives, settings)
 
             # Wait for the bucket to empty so we can do bursts
-            wait_ns = nsec_per_sec * burst_length * burst_rate / rate
+            wait_ns = nsec_per_sec * burst_length * burst_rate // rate
             self.vm.qtest("clock_step %d" % wait_ns)
 
             # Test I/O at the max burst rate
tests/qemu-iotests/149 +3 -3
index 1225334cb8..4f363f295f 100755
@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False):
     image_size = 4 * oneTB
     if qemu_img:
         iotests.log("# Create image")
-        qemu_img_create(config, image_size / oneMB)
+        qemu_img_create(config, image_size // oneMB)
     else:
         iotests.log("# Create image")
-        create_image(config, image_size / oneMB)
+        create_image(config, image_size // oneMB)
 
     lowOffsetMB = 100
-    highOffsetMB = 3 * oneTB / oneMB
+    highOffsetMB = 3 * oneTB // oneMB
 
     try:
         if not qemu_img:
tests/qemu-iotests/151 +6 -6
index fe53b9f446..1bb74d67c4 100755
@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase):
                             'write -P 1 0 %i' % self.image_len);
 
         # Start some background requests
-        for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset)
-        for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Start the block job
@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         self.assert_qmp(result, 'return', {})
 
         # Start some more requests
-        for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Wait for the READY event
@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         # the source) should be settled using the active mechanism.
         # The mirror code itself asserts that the source BDS's dirty
         # bitmap will stay clean between READY and COMPLETED.
-        for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         if sync_source_and_target:
tests/qemu-iotests/163 +1 -1
index 403842354e..5fd424761b 100755
@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         entry_bits = 3
         entry_size = 1 << entry_bits
         l1_mask = 0x00fffffffffffe00
-        div_roundup = lambda n, d: (n + d - 1) / d
+        div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
             for x in xrange(0, len(data), n):
tests/qemu-iotests/iotests.py +0 -0
index 7290c0b159..7ca94e9278 100644

tests/qemu-iotests/iotests.py def create_image(name, size): · line 205

Quoted 8 lines
tests/qemu-iotests/iotests.py +1 -1
@@ -199,7 +199,7 @@ def create_image(name, size):
     file = open(name, 'wb')
     i = 0
     while i < size:
-        sector = struct.pack('>l504xl', i / 512, i / 512)
+        sector = struct.pack('>l504xl', i // 512, i // 512)
         file.write(sector)
         i = i + 512
     file.close()

Found a few other occurrences:

030:''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))

- Cleber.

author
#17 lore raw

Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division

Quoted 184 lines · Cleber Rosa
Quoted 170 lines · Max Reitz

In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.

---
 tests/qemu-iotests/040        |  4 ++--
 tests/qemu-iotests/044        |  2 +-
 tests/qemu-iotests/093        | 18 +++++++++---------
 tests/qemu-iotests/149        |  6 +++---
 tests/qemu-iotests/151        | 12 ++++++------
 tests/qemu-iotests/163        |  2 +-
 tests/qemu-iotests/iotests.py |  2 +-
 7 files changed, 23 insertions(+), 23 deletions(-)
tests/qemu-iotests/040 +2 -2
index 1cb1ceeb33..b81133a474 100755
@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp('device_del', id='scsi0')
         self.assert_qmp(result, 'return', {})
@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
 
         result = self.vm.qmp('query-block')
tests/qemu-iotests/044 +1 -1
index 69e736f687..7ef5e46fe9 100755
@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining / 512))
+                for j in xrange(0, remaining // 512))
             fd.write(table)
 
 
tests/qemu-iotests/093 +9 -9
index 9d1971a56c..d88fbc182e 100755
@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase):
         # in. The throttled requests won't be executed until we
         # advance the virtual clock.
         rq_size = 512
-        rd_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_rd'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_rd'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_rd'])
         rd_nr *= seconds * 2
-        rd_nr /= ndrives
-        wr_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_wr'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr //= ndrives
+        wr_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_wr'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_wr'])
         wr_nr *= seconds * 2
-        wr_nr /= ndrives
+        wr_nr //= ndrives
 
         # Send I/O requests to all drives
         for i in range(rd_nr):
@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
             self.configure_throttle(ndrives, settings)
 
             # Wait for the bucket to empty so we can do bursts
-            wait_ns = nsec_per_sec * burst_length * burst_rate / rate
+            wait_ns = nsec_per_sec * burst_length * burst_rate // rate
             self.vm.qtest("clock_step %d" % wait_ns)
 
             # Test I/O at the max burst rate
tests/qemu-iotests/149 +3 -3
index 1225334cb8..4f363f295f 100755
@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False):
     image_size = 4 * oneTB
     if qemu_img:
         iotests.log("# Create image")
-        qemu_img_create(config, image_size / oneMB)
+        qemu_img_create(config, image_size // oneMB)
     else:
         iotests.log("# Create image")
-        create_image(config, image_size / oneMB)
+        create_image(config, image_size // oneMB)
 
     lowOffsetMB = 100
-    highOffsetMB = 3 * oneTB / oneMB
+    highOffsetMB = 3 * oneTB // oneMB
 
     try:
         if not qemu_img:
tests/qemu-iotests/151 +6 -6
index fe53b9f446..1bb74d67c4 100755
@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase):
                             'write -P 1 0 %i' % self.image_len);
 
         # Start some background requests
-        for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset)
-        for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Start the block job
@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         self.assert_qmp(result, 'return', {})
 
         # Start some more requests
-        for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Wait for the READY event
@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         # the source) should be settled using the active mechanism.
         # The mirror code itself asserts that the source BDS's dirty
         # bitmap will stay clean between READY and COMPLETED.
-        for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         if sync_source_and_target:
tests/qemu-iotests/163 +1 -1
index 403842354e..5fd424761b 100755
@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         entry_bits = 3
         entry_size = 1 << entry_bits
         l1_mask = 0x00fffffffffffe00
-        div_roundup = lambda n, d: (n + d - 1) / d
+        div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
             for x in xrange(0, len(data), n):
tests/qemu-iotests/iotests.py +0 -0
index 7290c0b159..7ca94e9278 100644

tests/qemu-iotests/iotests.py def create_image(name, size): · line 205

Quoted 8 lines
tests/qemu-iotests/iotests.py +1 -1
@@ -199,7 +199,7 @@ def create_image(name, size):
     file = open(name, 'wb')
     i = 0
     while i < size:
-        sector = struct.pack('>l504xl', i / 512, i / 512)
+        sector = struct.pack('>l504xl', i // 512, i // 512)
         file.write(sector)
         i = i + 512
     file.close()

Found a few other occurrences:

Quoted 3 lines

030:''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))

Oh, thanks. Right, those just worked because they were converted back to integers thanks to the %d in the format string.

Will fix. Thanks for reviewing!

Max

author
#18 lore raw

PATCH 5/9 iotests: Different iterator behavior in Python 3

In reply to Max Reitz

In Python 3, several functions now return iterators instead of lists. This includes range(), items(), map(), and filter(). This means that if we really want a list, we have to wrap those instances with list(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes used xrange() and iteritems() which no longer exist in Python 3. Just change these calls to be range() and items(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).

In one instance, we only wanted the first instance of the result of a filter() call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for with next() wrapped around, which works both in 2.7 and 3.

---
 tests/qemu-iotests/044 | 12 ++++++------
 tests/qemu-iotests/056 |  2 +-
 tests/qemu-iotests/065 |  4 ++--
 tests/qemu-iotests/124 |  4 ++--
 tests/qemu-iotests/139 |  2 +-
 tests/qemu-iotests/163 |  6 +++---
 6 files changed, 15 insertions(+), 15 deletions(-)
tests/qemu-iotests/044 +6 -6
index 7ef5e46fe9..e2d6c9b189 100755
@@ -52,23 +52,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             # Write a refcount table
             fd.seek(off_reftable)
 
-            for i in xrange(0, h.refcount_table_clusters):
+            for i in range(0, h.refcount_table_clusters):
                 sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
-                    for j in xrange(0, 64))
+                    for j in range(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
             sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
-            for block in xrange(0, h.refcount_table_clusters):
+            for block in range(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
             table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
-                for j in xrange(0, h.l1_size))
+                for j in range(0, h.l1_size))
             fd.write(table)
 
             # Write the L2 tables
@@ -79,14 +79,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             off = off_data
             while remaining > 1024 * 512:
                 pytable = list((1 << 63) | off + 512 * j
-                    for j in xrange(0, 1024))
+                    for j in range(0, 1024))
                 table = struct.pack('>1024Q', *pytable)
                 fd.write(table)
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining // 512))
+                for j in range(0, remaining // 512))
             fd.write(table)
 
 
tests/qemu-iotests/056 +1 -1
index 223292175a..3df323984d 100755
@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img')
 def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs):
     fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt))
     optargs = []
-    for k,v in kwargs.iteritems():
+    for k,v in kwargs.items():
         optargs = optargs + ['-o', '%s=%s' % (k,v)]
     args = ['create', '-f', fmt] + optargs + [fullname, size]
     iotests.qemu_img(*args)
tests/qemu-iotests/065 +2 -2
index 72aa9707c7..a339bf6069 100755
@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
                     :data.index('')]
         for field in data:
             self.assertTrue(re.match('^ {4}[^ ]', field) is not None)
-        data = map(lambda line: line.strip(), data)
+        data = list(map(lambda line: line.strip(), data))
         self.assertEqual(data, self.human_compare)
 
 class TestQMP(TestImageInfoSpecific):
@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
 
     def test_qmp(self):
         result = self.vm.qmp('query-block')['return']
-        drive = filter(lambda drive: drive['device'] == 'drive0', result)[0]
+        drive = next(drive for drive in result if drive['device'] == 'drive0')
         data = drive['inserted']['image']['format-specific']
         self.assertEqual(data['type'], iotests.imgfmt)
         self.assertEqual(data['data'], self.compare)
tests/qemu-iotests/124 +2 -2
index 3ea4ac53f5..9f189e3b54 100755
@@ -39,7 +39,7 @@ def try_remove(img):
 def transaction_action(action, **kwargs):
     return {
         'type': action,
-        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems())
+        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items())
     }
 
 
@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
     def img_create(self, img, fmt=iotests.imgfmt, size='64M',
                    parent=None, parentFormat=None, **kwargs):
         optargs = []
-        for k,v in kwargs.iteritems():
+        for k,v in kwargs.items():
             optargs = optargs + ['-o', '%s=%s' % (k,v)]
         args = ['create', '-f', fmt] + optargs + [img, size]
         if parent:
tests/qemu-iotests/139 +1 -1
index cc7fe337f3..e00f10b8c8 100755
@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
     # Check whether a BlockDriverState exists
     def checkBlockDriverState(self, node, must_exist = True):
         result = self.vm.qmp('query-named-block-nodes')
-        nodes = filter(lambda x: x['node-name'] == node, result['return'])
+        nodes = list(filter(lambda x: x['node-name'] == node, result['return']))
         self.assertLessEqual(len(nodes), 1)
         self.assertEqual(must_exist, len(nodes) == 1)
 
tests/qemu-iotests/163 +3 -3
index 5fd424761b..35c1a2bafc 100755
@@ -41,7 +41,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
-            for x in xrange(0, len(data), n):
+            for x in range(0, len(data), n):
                 yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask
 
         def check_l1_table(h, l1_data):
@@ -135,8 +135,8 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         self.image_verify()
 
     def test_random_write(self):
-        offs_list = range(0, size_to_int(self.image_len),
-                          size_to_int(self.chunk_size))
+        offs_list = list(range(0, size_to_int(self.image_len),
+                               size_to_int(self.chunk_size)))
         random.shuffle(offs_list)
         for offs in offs_list:
             qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),
Signature · 1 line
2.17.1
Reviewed-by on v1 5/9
#19 lore raw

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

Quoted 9 lines · Max Reitz

In Python 3, several functions now return iterators instead of lists. This includes range(), items(), map(), and filter(). This means that if we really want a list, we have to wrap those instances with list(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes used xrange() and iteritems() which no longer exist in Python 3. Just change these calls to be range() and items(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).

Quoted 7 lines

In one instance, we only wanted the first instance of the result of a filter() call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for with next() wrapped around, which works both in 2.7 and 3.

Minor comments below:

Quoted 76 lines
---
 tests/qemu-iotests/044 | 12 ++++++------
 tests/qemu-iotests/056 |  2 +-
 tests/qemu-iotests/065 |  4 ++--
 tests/qemu-iotests/124 |  4 ++--
 tests/qemu-iotests/139 |  2 +-
 tests/qemu-iotests/163 |  6 +++---
 6 files changed, 15 insertions(+), 15 deletions(-)
tests/qemu-iotests/044 +6 -6
index 7ef5e46fe9..e2d6c9b189 100755
@@ -52,23 +52,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             # Write a refcount table
             fd.seek(off_reftable)
 
-            for i in xrange(0, h.refcount_table_clusters):
+            for i in range(0, h.refcount_table_clusters):
                 sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
-                    for j in xrange(0, 64))
+                    for j in range(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
             sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
-            for block in xrange(0, h.refcount_table_clusters):
+            for block in range(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
             table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
-                for j in xrange(0, h.l1_size))
+                for j in range(0, h.l1_size))
             fd.write(table)
 
             # Write the L2 tables
@@ -79,14 +79,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             off = off_data
             while remaining > 1024 * 512:
                 pytable = list((1 << 63) | off + 512 * j
-                    for j in xrange(0, 1024))
+                    for j in range(0, 1024))
                 table = struct.pack('>1024Q', *pytable)
                 fd.write(table)
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining // 512))
+                for j in range(0, remaining // 512))
             fd.write(table)
 
 
tests/qemu-iotests/056 +1 -1
index 223292175a..3df323984d 100755
@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img')
 def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs):
     fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt))
     optargs = []
-    for k,v in kwargs.iteritems():
+    for k,v in kwargs.items():
         optargs = optargs + ['-o', '%s=%s' % (k,v)]
     args = ['create', '-f', fmt] + optargs + [fullname, size]
     iotests.qemu_img(*args)
tests/qemu-iotests/065 +0 -0
index 72aa9707c7..a339bf6069 100755

tests/qemu-iotests/065 class TestQemuImgInfo(TestImageInfoSpecific): · line 62

Quoted 5 lines
tests/qemu-iotests/065 +1 -1
@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
                     :data.index('')]
         for field in data:
             self.assertTrue(re.match('^ {4}[^ ]', field) is not None)
-        data = map(lambda line: line.strip(), data)
+        data = list(map(lambda line: line.strip(), data))

I would find this more readable:

data = [line.strip() for line in data]

Not a blocker, though.

tests/qemu-iotests/065 class TestQMP(TestImageInfoSpecific): · line 83

Quoted 9 lines
tests/qemu-iotests/065 +1 -1
         self.assertEqual(data, self.human_compare)
 
 class TestQMP(TestImageInfoSpecific):
@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
 
     def test_qmp(self):
         result = self.vm.qmp('query-block')['return']
-        drive = filter(lambda drive: drive['device'] == 'drive0', result)[0]
+        drive = next(drive for drive in result if drive['device'] == 'drive0')

I didn't understand what you meant by "single-line for" on the commit message, until I saw the generator expression here. :)

This will raise an exception if there's no drive0 in the list, but that was already true on the original code.

Quoted 30 lines
tests/qemu-iotests/065 +0 -0
         data = drive['inserted']['image']['format-specific']
         self.assertEqual(data['type'], iotests.imgfmt)
         self.assertEqual(data['data'], self.compare)
tests/qemu-iotests/124 +2 -2
index 3ea4ac53f5..9f189e3b54 100755
@@ -39,7 +39,7 @@ def try_remove(img):
 def transaction_action(action, **kwargs):
     return {
         'type': action,
-        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems())
+        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items())
     }
 
 
@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
     def img_create(self, img, fmt=iotests.imgfmt, size='64M',
                    parent=None, parentFormat=None, **kwargs):
         optargs = []
-        for k,v in kwargs.iteritems():
+        for k,v in kwargs.items():
             optargs = optargs + ['-o', '%s=%s' % (k,v)]
         args = ['create', '-f', fmt] + optargs + [img, size]
         if parent:
tests/qemu-iotests/139 +0 -0
index cc7fe337f3..e00f10b8c8 100755

tests/qemu-iotests/139 class TestBlockdevDel(iotests.QMPTestCase): · line 54

Quoted 5 lines
tests/qemu-iotests/139 +1 -1
@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
     # Check whether a BlockDriverState exists
     def checkBlockDriverState(self, node, must_exist = True):
         result = self.vm.qmp('query-named-block-nodes')
-        nodes = filter(lambda x: x['node-name'] == node, result['return'])
+        nodes = list(filter(lambda x: x['node-name'] == node, result['return']))

I'd prefer a list expression:

        nodes = [x for x in result['return'] if x['node-name'] == node]

Also not a blocker.

Quoted 30 lines
tests/qemu-iotests/139 +0 -0
         self.assertLessEqual(len(nodes), 1)
         self.assertEqual(must_exist, len(nodes) == 1)
 
tests/qemu-iotests/163 +3 -3
index 5fd424761b..35c1a2bafc 100755
@@ -41,7 +41,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
-            for x in xrange(0, len(data), n):
+            for x in range(0, len(data), n):
                 yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask
 
         def check_l1_table(h, l1_data):
@@ -135,8 +135,8 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         self.image_verify()
 
     def test_random_write(self):
-        offs_list = range(0, size_to_int(self.image_len),
-                          size_to_int(self.chunk_size))
+        offs_list = list(range(0, size_to_int(self.image_len),
+                               size_to_int(self.chunk_size)))
         random.shuffle(offs_list)
         for offs in offs_list:
             qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),
Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
author
#20 lore raw

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

Quoted 21 lines · Eduardo Habkost
Quoted 9 lines · Max Reitz

In Python 3, several functions now return iterators instead of lists. This includes range(), items(), map(), and filter(). This means that if we really want a list, we have to wrap those instances with list(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes used xrange() and iteritems() which no longer exist in Python 3. Just change these calls to be range() and items(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).

Quoted 7 lines

In one instance, we only wanted the first instance of the result of a filter() call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for with next() wrapped around, which works both in 2.7 and 3.

Minor comments below:

[...]

Quoted 13 lines

tests/qemu-iotests/065 class TestQemuImgInfo(TestImageInfoSpecific): · line 62

Quoted 10 lines
tests/qemu-iotests/065 +1 -1
index 72aa9707c7..a339bf6069 100755
@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
                     :data.index('')]
         for field in data:
             self.assertTrue(re.match('^ {4}[^ ]', field) is not None)
-        data = map(lambda line: line.strip(), data)
+        data = list(map(lambda line: line.strip(), data))

I would find this more readable:

Quoted 3 lines

data = [line.strip() for line in data]

Not a blocker, though.

Yes, I agree, that looks better.

Quoted 5 lines
Quoted 5 lines
tests/qemu-iotests/065 +0 -0
         self.assertEqual(data, self.human_compare)
 
 class TestQMP(TestImageInfoSpecific):
@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
 
Quoted 7 lines

tests/qemu-iotests/065 class TestQMP(TestImageInfoSpecific): · line 83

Quoted 4 lines
tests/qemu-iotests/065 +1 -1
     def test_qmp(self):
         result = self.vm.qmp('query-block')['return']
-        drive = filter(lambda drive: drive['device'] == 'drive0', result)[0]
+        drive = next(drive for drive in result if drive['device'] == 'drive0')

I didn't understand what you meant by "single-line for" on the commit message, until I saw the generator expression here. :)

Maybe I should at least put quotes around the "for", because the combination of "for with" really makes it difficult to read...

Quoted 42 lines

This will raise an exception if there's no drive0 in the list, but that was already true on the original code.

Quoted 30 lines
tests/qemu-iotests/065 +0 -0
         data = drive['inserted']['image']['format-specific']
         self.assertEqual(data['type'], iotests.imgfmt)
         self.assertEqual(data['data'], self.compare)
tests/qemu-iotests/124 +2 -2
index 3ea4ac53f5..9f189e3b54 100755
@@ -39,7 +39,7 @@ def try_remove(img):
 def transaction_action(action, **kwargs):
     return {
         'type': action,
-        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems())
+        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items())
     }
 
 
@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
     def img_create(self, img, fmt=iotests.imgfmt, size='64M',
                    parent=None, parentFormat=None, **kwargs):
         optargs = []
-        for k,v in kwargs.iteritems():
+        for k,v in kwargs.items():
             optargs = optargs + ['-o', '%s=%s' % (k,v)]
         args = ['create', '-f', fmt] + optargs + [img, size]
         if parent:
tests/qemu-iotests/139 +0 -0
index cc7fe337f3..e00f10b8c8 100755

tests/qemu-iotests/139 class TestBlockdevDel(iotests.QMPTestCase): · line 54

Quoted 5 lines
tests/qemu-iotests/139 +1 -1
@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
     # Check whether a BlockDriverState exists
     def checkBlockDriverState(self, node, must_exist = True):
         result = self.vm.qmp('query-named-block-nodes')
-        nodes = filter(lambda x: x['node-name'] == node, result['return'])
+        nodes = list(filter(lambda x: x['node-name'] == node, result['return']))

I'd prefer a list expression:

Quoted 3 lines
        nodes = [x for x in result['return'] if x['node-name'] == node]

Also not a blocker.

I don't mind either way, so why not.

Max

Reviewed-by on v1 5/9
#21 lore raw

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

In reply to Max Reitz
Quoted 16 lines · Max Reitz

In Python 3, several functions now return iterators instead of lists. This includes range(), items(), map(), and filter(). This means that if we really want a list, we have to wrap those instances with list(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes used xrange() and iteritems() which no longer exist in Python 3. Just change these calls to be range() and items(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).

In one instance, we only wanted the first instance of the result of a filter() call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for with next() wrapped around, which works both in 2.7 and 3.

Quoted 9 lines
---
 tests/qemu-iotests/044 | 12 ++++++------
 tests/qemu-iotests/056 |  2 +-
 tests/qemu-iotests/065 |  4 ++--
 tests/qemu-iotests/124 |  4 ++--
 tests/qemu-iotests/139 |  2 +-
 tests/qemu-iotests/163 |  6 +++---
 6 files changed, 15 insertions(+), 15 deletions(-)

You have 2 files here which use xrange (which is a manageable size, and whose occurrences involve a moderate size of items) to also consider:

if sys.version_info.major == 2:
range = xrange

Defaulting to the Python 3 names, but behaving the same across Python 2 and 3.

To do the same for dict.iteritems() => dict.items() requires a lot more code, so I'd stay away from it for now. Also, it looks like most of those dicts are small in size (**kwargs and the like).

Other than that suggestion,

author
#22 lore raw

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

Quoted 29 lines · Cleber Rosa
Quoted 16 lines · Max Reitz

In Python 3, several functions now return iterators instead of lists. This includes range(), items(), map(), and filter(). This means that if we really want a list, we have to wrap those instances with list(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes used xrange() and iteritems() which no longer exist in Python 3. Just change these calls to be range() and items(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).

In one instance, we only wanted the first instance of the result of a filter() call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for with next() wrapped around, which works both in 2.7 and 3.

Quoted 9 lines
---
 tests/qemu-iotests/044 | 12 ++++++------
 tests/qemu-iotests/056 |  2 +-
 tests/qemu-iotests/065 |  4 ++--
 tests/qemu-iotests/124 |  4 ++--
 tests/qemu-iotests/139 |  2 +-
 tests/qemu-iotests/163 |  6 +++---
 6 files changed, 15 insertions(+), 15 deletions(-)
Quoted 5 lines

You have 2 files here which use xrange (which is a manageable size, and whose occurrences involve a moderate size of items) to also consider:

if sys.version_info.major == 2:
range = xrange

I don't think it's necessary, but it's so easy to grep for "sys.version_info" once we want to completely switch to Python 3 that I can't find good arguments against it. O:-)

Max

Quoted 11 lines

Defaulting to the Python 3 names, but behaving the same across Python 2 and 3.

To do the same for dict.iteritems() => dict.items() requires a lot more code, so I'd stay away from it for now. Also, it looks like most of those dicts are small in size (**kwargs and the like).

Other than that suggestion,

author
#23 lore raw

PATCH 6/9 iotests: Explicitly inherit FDs in Python

In reply to Max Reitz

Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if close_fds is explicitly set to False.

Adhere to this by setting close_fds to False when working with subprocesses that may want to inherit FDs, and by trying to set_inheritable() on FDs that we do want to bequeath to them.

---
 scripts/qemu.py        | 13 +++++++++++--
 scripts/qmp/qmp.py     |  7 +++++++
 tests/qemu-iotests/147 |  7 +++++++
 3 files changed, 25 insertions(+), 2 deletions(-)
scripts/qemu.py +11 -2
index f099ce7278..28366c4a67 100644
@@ -142,10 +142,18 @@ class QEMUMachine(object):
         if opts:
             options.append(opts)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(fd, True)
+        except AttributeError:
+            pass
+
         self._args.append('-add-fd')
         self._args.append(','.join(options))
         return self
 
+    # The caller needs to make sure the FD is inheritable
     def send_fd_scm(self, fd_file_path):
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
@@ -159,7 +167,7 @@ class QEMUMachine(object):
                     "%s" % fd_file_path]
         devnull = open(os.path.devnull, 'rb')
         proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT, close_fds=False)
         output = proc.communicate()[0]
         if output:
             LOG.debug(output)
@@ -280,7 +288,8 @@ class QEMUMachine(object):
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
                                        stderr=subprocess.STDOUT,
-                                       shell=False)
+                                       shell=False,
+                                       close_fds=False)
         self._post_launch()
 
     def wait(self):
scripts/qmp/qmp.py +7 -0
index 5c8cf6a056..009be8345b 100644
@@ -10,6 +10,7 @@
 
 import json
 import errno
+import os
 import socket
 import logging
 
@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
         return self.__sock.fileno()
 
     def is_scm_available(self):
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(self.get_sock_fd(), True)
+        except AttributeError:
+            pass
         return self.__sock.family == socket.AF_UNIX
tests/qemu-iotests/147 +7 -0
index d2081df84b..b58455645b 100755
@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
         sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sockfd.connect(unix_socket)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(sockfd.fileno(), True)
+        except AttributeError:
+            pass
+
         result = self.vm.send_fd_scm(str(sockfd.fileno()))
         self.assertEqual(result, 0, 'Failed to send socket FD')
 
Signature · 1 line
2.17.1
#24 lore raw

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

Quoted 21 lines · Max Reitz

Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if close_fds is explicitly set to False.

Adhere to this by setting close_fds to False when working with subprocesses that may want to inherit FDs, and by trying to set_inheritable() on FDs that we do want to bequeath to them.

---
 scripts/qemu.py        | 13 +++++++++++--
 scripts/qmp/qmp.py     |  7 +++++++
 tests/qemu-iotests/147 |  7 +++++++
 3 files changed, 25 insertions(+), 2 deletions(-)
scripts/qemu.py +0 -0
index f099ce7278..28366c4a67 100644

scripts/qemu.py class QEMUMachine(object): · line 151

Quoted 10 lines
scripts/qemu.py +7 -0
@@ -142,10 +142,18 @@ class QEMUMachine(object):
         if opts:
             options.append(opts)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(fd, True)
+        except AttributeError:
+            pass
+

This is add_fd(), so calling set_inheritable() automatically here makes sense.

Quoted 40 lines
scripts/qemu.py +4 -2
         self._args.append('-add-fd')
         self._args.append(','.join(options))
         return self
 
+    # The caller needs to make sure the FD is inheritable
     def send_fd_scm(self, fd_file_path):
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
@@ -159,7 +167,7 @@ class QEMUMachine(object):
                     "%s" % fd_file_path]
         devnull = open(os.path.devnull, 'rb')
         proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT, close_fds=False)
         output = proc.communicate()[0]
         if output:
             LOG.debug(output)
@@ -280,7 +288,8 @@ class QEMUMachine(object):
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
                                        stderr=subprocess.STDOUT,
-                                       shell=False)
+                                       shell=False,
+                                       close_fds=False)
         self._post_launch()
 
     def wait(self):
scripts/qmp/qmp.py +1 -0
index 5c8cf6a056..009be8345b 100644
@@ -10,6 +10,7 @@
 
 import json
 import errno
+import os
 import socket
 import logging
 

scripts/qmp/qmp.py class QEMUMonitorProtocol(object): · line 262

Quoted 9 lines
scripts/qmp/qmp.py +6 -0
@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
         return self.__sock.fileno()
 
     def is_scm_available(self):
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(self.get_sock_fd(), True)
+        except AttributeError:
+            pass

Why did you decide to place this code inside is_scm_available()?

For reference, this is the only caller of is_scm_available():

def send_fd_scm(self, fd_file_path):
# In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()
...

In addition to making a method called is_*() have an unexpected side-effect, the method won't be called at all if running with debugging disabled.

I suggest simply placing the os.set_inheritable() call inside send_fd_scm(), as close as possible to the subprocess.Popen() call.

Quoted 6 lines
scripts/qmp/qmp.py +0 -0
         return self.__sock.family == socket.AF_UNIX
tests/qemu-iotests/147 +0 -0
index d2081df84b..b58455645b 100755

tests/qemu-iotests/147 class BuiltinNBD(NBDBlockdevAddBase): · line 238

Quoted 10 lines
tests/qemu-iotests/147 +7 -0
@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
         sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sockfd.connect(unix_socket)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(sockfd.fileno(), True)
+        except AttributeError:
+            pass
+

Why not make send_fd_scm() responsible for calling os.set_inheritable(), making this hunk unnecessary?

Quoted 6 lines
tests/qemu-iotests/147 +0 -0
         result = self.vm.send_fd_scm(str(sockfd.fileno()))
         self.assertEqual(result, 0, 'Failed to send socket FD')
 
Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
author
#25 lore raw

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

Quoted 90 lines · Eduardo Habkost
Quoted 21 lines · Max Reitz

Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if close_fds is explicitly set to False.

Adhere to this by setting close_fds to False when working with subprocesses that may want to inherit FDs, and by trying to set_inheritable() on FDs that we do want to bequeath to them.

---
 scripts/qemu.py        | 13 +++++++++++--
 scripts/qmp/qmp.py     |  7 +++++++
 tests/qemu-iotests/147 |  7 +++++++
 3 files changed, 25 insertions(+), 2 deletions(-)
scripts/qemu.py +0 -0
index f099ce7278..28366c4a67 100644

scripts/qemu.py class QEMUMachine(object): · line 151

Quoted 10 lines
scripts/qemu.py +7 -0
@@ -142,10 +142,18 @@ class QEMUMachine(object):
         if opts:
             options.append(opts)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(fd, True)
+        except AttributeError:
+            pass
+

This is add_fd(), so calling set_inheritable() automatically here makes sense.

Quoted 40 lines
scripts/qemu.py +4 -2
         self._args.append('-add-fd')
         self._args.append(','.join(options))
         return self
 
+    # The caller needs to make sure the FD is inheritable
     def send_fd_scm(self, fd_file_path):
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
@@ -159,7 +167,7 @@ class QEMUMachine(object):
                     "%s" % fd_file_path]
         devnull = open(os.path.devnull, 'rb')
         proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT, close_fds=False)
         output = proc.communicate()[0]
         if output:
             LOG.debug(output)
@@ -280,7 +288,8 @@ class QEMUMachine(object):
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
                                        stderr=subprocess.STDOUT,
-                                       shell=False)
+                                       shell=False,
+                                       close_fds=False)
         self._post_launch()
 
     def wait(self):
scripts/qmp/qmp.py +1 -0
index 5c8cf6a056..009be8345b 100644
@@ -10,6 +10,7 @@
 
 import json
 import errno
+import os
 import socket
 import logging
 

scripts/qmp/qmp.py class QEMUMonitorProtocol(object): · line 262

Quoted 9 lines
scripts/qmp/qmp.py +6 -0
@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
         return self.__sock.fileno()
 
     def is_scm_available(self):
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(self.get_sock_fd(), True)
+        except AttributeError:
+            pass

Why did you decide to place this code inside is_scm_available()?

For reference, this is the only caller of is_scm_available():

Quoted 7 lines

def send_fd_scm(self, fd_file_path):
# In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()
...

In addition to making a method called is_*() have an unexpected side-effect,

True. My idea was that a function that asks for SCM to be available might as well make it available.

Quoted 2 lines

the method won't be called at all if running with
debugging disabled.

Well, I sure hope we don't disable debugging in the iotests. We use assert quite a number of times there.

On the other hand, someone might want to use this outside of the iotests but I don't even know whether that works, considering the SCM helper program is part of the iotests.

Quoted 3 lines

I suggest simply placing the os.set_inheritable() call inside send_fd_scm(), as close as possible to the subprocess.Popen() call.

Yes, I think you're right. I didn't want to put it into send_fd_scm(), because that method is only there to send some FD over QMP/SCM; it isn't really supposed to send the QMP socket FD somewhere. But sending something over QMP/SCM is different from bequeathing something to a child process (the socket_scm_helper). And since send_fd_scm() needs to bequeath the QMP socket FD to that helper, it should be responsible for making it inheritable.

And maybe even more importantly, whether the socket allows for SCM really has nothing to do with whether it's inheritable. So it's actually just wrong to put it here.

Quoted 9 lines
Quoted 9 lines
scripts/qmp/qmp.py +0 -0
         return self.__sock.family == socket.AF_UNIX
tests/qemu-iotests/147 +0 -0
index d2081df84b..b58455645b 100755
@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
         sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sockfd.connect(unix_socket)
 
Quoted 10 lines

tests/qemu-iotests/147 class BuiltinNBD(NBDBlockdevAddBase): · line 238

Quoted 7 lines
tests/qemu-iotests/147 +7 -0
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(sockfd.fileno(), True)
+        except AttributeError:
+            pass
+

Why not make send_fd_scm() responsible for calling os.set_inheritable(), making this hunk unnecessary?

My idea was: Because send_fd_scm() takes a string and not an integer. The socket_scm_helper takes either an FD or a path, so send_fd_scm() accepts either.

But now I realize that if we pass a path, we don't need to make the FD inheritable. So send_fd_scm() can check whether it's supposed to pass an FD, and if so, make it inheritable, yes.

Max

Quoted 7 lines
Quoted 6 lines
tests/qemu-iotests/147 +0 -0
         result = self.vm.send_fd_scm(str(sockfd.fileno()))
         self.assertEqual(result, 0, 'Failed to send socket FD')
 
Signature · 1 line
2.17.1
#26 lore raw

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

In reply to Max Reitz
Quoted 5 lines · Max Reitz

Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if close_fds is explicitly set to False.

It's actually a change that was introduced in 3.4:

https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors

Quoted 16 lines

Adhere to this by setting close_fds to False when working with subprocesses that may want to inherit FDs, and by trying to set_inheritable() on FDs that we do want to bequeath to them.

---
 scripts/qemu.py        | 13 +++++++++++--
 scripts/qmp/qmp.py     |  7 +++++++
 tests/qemu-iotests/147 |  7 +++++++
 3 files changed, 25 insertions(+), 2 deletions(-)
scripts/qemu.py +0 -0
index f099ce7278..28366c4a67 100644

scripts/qemu.py class QEMUMachine(object): · line 147

Quoted 6 lines
scripts/qemu.py +3 -0
@@ -142,10 +142,18 @@ class QEMUMachine(object):
         if opts:
             options.append(opts)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:

Version should be 3.4 here as well.

scripts/qemu.py class QEMUMachine(object): · line 151

Quoted 4 lines
scripts/qemu.py +4 -0
+            os.set_inheritable(fd, True)
+        except AttributeError:
+            pass
+

Doing hasattr(os, "set_inheritable") is cheaper.

- Cleber.

Quoted 69 lines
scripts/qemu.py +4 -2
         self._args.append('-add-fd')
         self._args.append(','.join(options))
         return self
 
+    # The caller needs to make sure the FD is inheritable
     def send_fd_scm(self, fd_file_path):
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
@@ -159,7 +167,7 @@ class QEMUMachine(object):
                     "%s" % fd_file_path]
         devnull = open(os.path.devnull, 'rb')
         proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT, close_fds=False)
         output = proc.communicate()[0]
         if output:
             LOG.debug(output)
@@ -280,7 +288,8 @@ class QEMUMachine(object):
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
                                        stderr=subprocess.STDOUT,
-                                       shell=False)
+                                       shell=False,
+                                       close_fds=False)
         self._post_launch()
 
     def wait(self):
scripts/qmp/qmp.py +7 -0
index 5c8cf6a056..009be8345b 100644
@@ -10,6 +10,7 @@
 
 import json
 import errno
+import os
 import socket
 import logging
 
@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
         return self.__sock.fileno()
 
     def is_scm_available(self):
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(self.get_sock_fd(), True)
+        except AttributeError:
+            pass
         return self.__sock.family == socket.AF_UNIX
tests/qemu-iotests/147 +7 -0
index d2081df84b..b58455645b 100755
@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
         sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sockfd.connect(unix_socket)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(sockfd.fileno(), True)
+        except AttributeError:
+            pass
+
         result = self.vm.send_fd_scm(str(sockfd.fileno()))
         self.assertEqual(result, 0, 'Failed to send socket FD')
 
author
#27 lore raw

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

Quoted 9 lines · Cleber Rosa
Quoted 5 lines · Max Reitz

Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if close_fds is explicitly set to False.

Quoted 3 lines

It's actually a change that was introduced in 3.4:

https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors

Oops, don't know how I got that wrong.

Quoted 19 lines
Quoted 19 lines

Adhere to this by setting close_fds to False when working with subprocesses that may want to inherit FDs, and by trying to set_inheritable() on FDs that we do want to bequeath to them.

---
 scripts/qemu.py        | 13 +++++++++++--
 scripts/qmp/qmp.py     |  7 +++++++
 tests/qemu-iotests/147 |  7 +++++++
 3 files changed, 25 insertions(+), 2 deletions(-)
scripts/qemu.py +0 -0
index f099ce7278..28366c4a67 100644
@@ -142,10 +142,18 @@ class QEMUMachine(object):
         if opts:
             options.append(opts)
 
Quoted 5 lines

scripts/qemu.py class QEMUMachine(object): · line 147

Quoted 3 lines
scripts/qemu.py +3 -0
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:

Version should be 3.4 here as well.

Yes, will fix everywhere.

Quoted 8 lines

scripts/qemu.py class QEMUMachine(object): · line 151

Quoted 4 lines
scripts/qemu.py +4 -0
+            os.set_inheritable(fd, True)
+        except AttributeError:
+            pass
+

Doing hasattr(os, "set_inheritable") is cheaper.

- Cleber.

Nice, I'll use that then.

Max

Quoted 70 lines
Quoted 69 lines
scripts/qemu.py +4 -2
         self._args.append('-add-fd')
         self._args.append(','.join(options))
         return self
 
+    # The caller needs to make sure the FD is inheritable
     def send_fd_scm(self, fd_file_path):
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
@@ -159,7 +167,7 @@ class QEMUMachine(object):
                     "%s" % fd_file_path]
         devnull = open(os.path.devnull, 'rb')
         proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT, close_fds=False)
         output = proc.communicate()[0]
         if output:
             LOG.debug(output)
@@ -280,7 +288,8 @@ class QEMUMachine(object):
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
                                        stderr=subprocess.STDOUT,
-                                       shell=False)
+                                       shell=False,
+                                       close_fds=False)
         self._post_launch()
 
     def wait(self):
scripts/qmp/qmp.py +7 -0
index 5c8cf6a056..009be8345b 100644
@@ -10,6 +10,7 @@
 
 import json
 import errno
+import os
 import socket
 import logging
 
@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
         return self.__sock.fileno()
 
     def is_scm_available(self):
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(self.get_sock_fd(), True)
+        except AttributeError:
+            pass
         return self.__sock.family == socket.AF_UNIX
tests/qemu-iotests/147 +7 -0
index d2081df84b..b58455645b 100755
@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
         sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sockfd.connect(unix_socket)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(sockfd.fileno(), True)
+        except AttributeError:
+            pass
+
         result = self.vm.send_fd_scm(str(sockfd.fileno()))
         self.assertEqual(result, 0, 'Failed to send socket FD')
 
author
#28 lore raw

PATCH 7/9 iotests: 'new' module replacement in 169

In reply to Max Reitz

iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.

---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +1 -2
index f243db9955..e5614b159d 100755
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 from iotests import qemu_img
 
 
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + name, lambda self: mc(self))
 
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
Signature · 1 line
2.17.1
Reviewed-by on v1 7/9
#29 lore raw

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

Quoted 22 lines · Max Reitz

iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.

---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +0 -1
index f243db9955..e5614b159d 100755
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 from iotests import qemu_img
 
 

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 146

Quoted 5 lines
tests/qemu-iotests/169 +1 -1
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + name, lambda self: mc(self))

The "lambda self: mc(self)" expression looks weird and unnecessary at first look, but I have just confirmed that this doesn't work:

setattr(klass, 'test_' + name, mc)

Probably because the methodcaller object won't automatically become a instance method after the TestCase class is instantiated.

Quoted 6 lines
tests/qemu-iotests/169 +0 -0
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
#30 lore raw

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

In reply to Max Reitz
Quoted 23 lines · Max Reitz

iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.

---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +0 -1
index f243db9955..e5614b159d 100755
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 from iotests import qemu_img
 
 

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149

Quoted 8 lines
tests/qemu-iotests/169 +1 -1
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + name, lambda self: mc(self))
 
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'

This can be simplified with:

tests/qemu-iotests/169 +3 -7
index e5614b159d..2199f14ae7 100755
@@ -22,7 +22,6 @@ import os
 import iotests
 import time
 import itertools
-import operator
 from iotests import qemu_img
 
 
@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.check_bitmap(self.vm_b, sha256 if persistent else False)
 
 
-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'
 
-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
+    setattr(TestDirtyBitmapMigration, 'test_' + name,
+            lambda self: TestDirtyBitmapMigration.do_test_migration(
+                self, *list(cmb)))
 
 
 if __name__ == '__main__':
#31 lore raw

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

Quoted 58 lines · Cleber Rosa
Quoted 23 lines · Max Reitz

iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.

---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +0 -1
index f243db9955..e5614b159d 100755
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 from iotests import qemu_img
 
 

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149

Quoted 8 lines
tests/qemu-iotests/169 +1 -1
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + name, lambda self: mc(self))
 
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'

This can be simplified with:

tests/qemu-iotests/169 +0 -5
index e5614b159d..2199f14ae7 100755
@@ -22,7 +22,6 @@ import os
 import iotests
 import time
 import itertools
-import operator
 from iotests import qemu_img
 
 
@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.check_bitmap(self.vm_b, sha256 if persistent else False)
 
 
-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
 for cmb in list(itertools.product((True, False), repeat=4)):

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 151

Quoted 10 lines
tests/qemu-iotests/169 +3 -2
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'
 
-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
+    setattr(TestDirtyBitmapMigration, 'test_' + name,
+            lambda self: TestDirtyBitmapMigration.do_test_migration(
+                self, *list(cmb)))

I'm not fond of the long multi-line lambda expression, but I love that you got rid of operator.methodcaller(). :)

However, this doesn't seem to work: *list(cmb) will be evaluated only when the lambda is called, and cmb will be always (False, False, False, False).

I was going to suggest defining a nested function inside inject_test_case() to replace operator.methodcaller(), but then I thought it was not worth it.

Signature · 1 line
Eduardo
#32 lore raw

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

Quoted 73 lines · Eduardo Habkost
Quoted 58 lines · Cleber Rosa
Quoted 23 lines · Max Reitz

iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.

---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +0 -1
index f243db9955..e5614b159d 100755
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 from iotests import qemu_img
 
 

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149

Quoted 8 lines
tests/qemu-iotests/169 +1 -1
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + name, lambda self: mc(self))
 
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'

This can be simplified with:

tests/qemu-iotests/169 +0 -5
index e5614b159d..2199f14ae7 100755
@@ -22,7 +22,6 @@ import os
 import iotests
 import time
 import itertools
-import operator
 from iotests import qemu_img
 
 
@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.check_bitmap(self.vm_b, sha256 if persistent else False)
 
 
-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
 for cmb in list(itertools.product((True, False), repeat=4)):

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 151

Quoted 10 lines
tests/qemu-iotests/169 +3 -2
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'
 
-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
+    setattr(TestDirtyBitmapMigration, 'test_' + name,
+            lambda self: TestDirtyBitmapMigration.do_test_migration(
+                self, *list(cmb)))

I'm not fond of the long multi-line lambda expression, but I love that you got rid of operator.methodcaller(). :)

Quoted 8 lines

However, this doesn't seem to work: *list(cmb) will be evaluated only when the lambda is called, and cmb will be always (False, False, False, False).

I was going to suggest defining a nested function inside inject_test_case() to replace operator.methodcaller(), but then I thought it was not worth it.

You're right! I missed that. Anyway, another possibility:

tests/qemu-iotests/169 +3 -10
index e5614b159d..cd0d9c289c 100755
@@ -22,7 +22,7 @@ import os
 import iotests
 import time
 import itertools
-import operator
+import functools
 from iotests import qemu_img
 
 
@@ -140,20 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.vm_b.launch()
             self.check_bitmap(self.vm_b, sha256 if persistent else False)
 
-
-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
-for cmb in list(itertools.product((True, False), repeat=4)):
+for cmb in itertools.product((True, False), repeat=4):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'
 
-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
-
+    test =

functools.partialmethod(TestDirtyBitmapMigration.do_test_migration,
+ *cmb)
+ setattr(TestDirtyBitmapMigration, 'test_' + name, test)

if __name__ == '__main__':

     iotests.main(supported_fmts=['qcow2'])
author
#33 lore raw

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

Quoted 113 lines · Cleber Rosa
Quoted 73 lines · Eduardo Habkost
Quoted 58 lines · Cleber Rosa
Quoted 23 lines · Max Reitz

iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.

---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +0 -1
index f243db9955..e5614b159d 100755
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 from iotests import qemu_img
 
 

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149

Quoted 8 lines
tests/qemu-iotests/169 +1 -1
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + name, lambda self: mc(self))
 
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'

This can be simplified with:

tests/qemu-iotests/169 +0 -5
index e5614b159d..2199f14ae7 100755
@@ -22,7 +22,6 @@ import os
 import iotests
 import time
 import itertools
-import operator
 from iotests import qemu_img
 
 
@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.check_bitmap(self.vm_b, sha256 if persistent else False)
 
 
-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
 for cmb in list(itertools.product((True, False), repeat=4)):

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 151

Quoted 10 lines
tests/qemu-iotests/169 +3 -2
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'
 
-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
+    setattr(TestDirtyBitmapMigration, 'test_' + name,
+            lambda self: TestDirtyBitmapMigration.do_test_migration(
+                self, *list(cmb)))

I'm not fond of the long multi-line lambda expression, but I love that you got rid of operator.methodcaller(). :)

Quoted 8 lines

However, this doesn't seem to work: *list(cmb) will be evaluated only when the lambda is called, and cmb will be always (False, False, False, False).

I was going to suggest defining a nested function inside inject_test_case() to replace operator.methodcaller(), but then I thought it was not worth it.

You're right! I missed that. Anyway, another possibility:

tests/qemu-iotests/169 +2 -7
index e5614b159d..cd0d9c289c 100755
@@ -22,7 +22,7 @@ import os
 import iotests
 import time
 import itertools
-import operator
+import functools
 from iotests import qemu_img
 
 
@@ -140,20 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.vm_b.launch()
             self.check_bitmap(self.vm_b, sha256 if persistent else False)
 
-
-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
-for cmb in list(itertools.product((True, False), repeat=4)):
+for cmb in itertools.product((True, False), repeat=4):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 152

Quoted 10 lines
tests/qemu-iotests/169 +3 -3
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'
 
-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
-
+    test =
 functools.partialmethod(TestDirtyBitmapMigration.do_test_migration,
+                                   *cmb)
+    setattr(TestDirtyBitmapMigration, 'test_' + name, test)

I'm not sure how that is any simpler, though (apart from the inlining). :-)

I mean, my personal goal is not to touch this beyond what I need to because it's black magic to me anyway, so I'm happy with what works.

Max

#34 lore raw

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

On Fri, Oct 19, 2018 at 11:46:59AM +0200, Max Reitz wrote:
[...]

Quoted 2 lines

I mean, my personal goal is not to touch this beyond what I need to because it's black magic to me anyway, so I'm happy with what works.

I agree with this approach, and your original patch looks good enough to me.

Signature · 1 line
Eduardo
author
#35 lore raw

PATCH 8/9 iotests: Modify imports for Python 3

In reply to Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +6 -2
index 7ca94e9278..a64ea90fb4 100644
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO
+
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        output = StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +5 -2
index d45e2e0a6a..6b2d659dee 100755
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)
Signature · 1 line
2.17.1
#36 lore raw

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

Quoted 53 lines · Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +6 -2
index 7ca94e9278..a64ea90fb4 100644
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO
+
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        output = StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +4 -1
index d45e2e0a6a..6b2d659dee 100755
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 

tests/qemu-iotests/nbd-fault-injector.py def parse_config(config): · line 234

Quoted 8 lines
tests/qemu-iotests/nbd-fault-injector.py +1 -1
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)

This may be a type of culture clash (on my side, due to not enough QEMU culture), but shouldn't this be applied before anything else on this series?

I mean, PATCH 1/9 is supposed to fix the reliability aspects of nbd-fault-injector under Python 3, but without this patch, it won't actually run on Python 3.

Regards,
- Cleber.

#37 lore raw

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

Quoted 68 lines · Cleber Rosa
Quoted 53 lines · Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +6 -2
index 7ca94e9278..a64ea90fb4 100644
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO
+
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        output = StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +4 -1
index d45e2e0a6a..6b2d659dee 100755
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 

tests/qemu-iotests/nbd-fault-injector.py def parse_config(config): · line 234

Quoted 8 lines
tests/qemu-iotests/nbd-fault-injector.py +1 -1
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)

This may be a type of culture clash (on my side, due to not enough QEMU culture), but shouldn't this be applied before anything else on this series?

Quoted 3 lines

I mean, PATCH 1/9 is supposed to fix the reliability aspects of nbd-fault-injector under Python 3, but without this patch, it won't actually run on Python 3.

Both patches are required to make the code run on Python 3, and they don't depend on each other. So I think the order doesn't matter.

But I think the order chosen by Max is slightly more intuitive:
having explicit mentions of Python 3 in the code would be confusing if we didn't fix the compatibility issues on patches 1-7 first.

Signature · 1 line
Eduardo
author
#38 lore raw

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

In reply to Cleber Rosa
Quoted 68 lines · Cleber Rosa
Quoted 53 lines · Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +6 -2
index 7ca94e9278..a64ea90fb4 100644
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO
+
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        output = StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +4 -1
index d45e2e0a6a..6b2d659dee 100755
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 

tests/qemu-iotests/nbd-fault-injector.py def parse_config(config): · line 234

Quoted 8 lines
tests/qemu-iotests/nbd-fault-injector.py +1 -1
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)

This may be a type of culture clash (on my side, due to not enough QEMU culture), but shouldn't this be applied before anything else on this series?

Quoted 3 lines

I mean, PATCH 1/9 is supposed to fix the reliability aspects of nbd-fault-injector under Python 3, but without this patch, it won't actually run on Python 3.

I don't mind, I followed no specific order. Well, patch 9 is patch 9 because it is the largest one, so I didn't want to discourage people before the end of the series. :-)

Max

Reviewed-by on v1 8/9
#39 lore raw

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

In reply to Max Reitz
Quoted 16 lines · Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +0 -0
index 7ca94e9278..a64ea90fb4 100644

tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689

Quoted 8 lines
tests/qemu-iotests/iotests.py +4 -1
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO

Considering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.

But this shouldn't block this workaround, so:

Quoted 39 lines
tests/qemu-iotests/iotests.py +2 -1
+
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        output = StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +5 -2
index d45e2e0a6a..6b2d659dee 100755
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)
Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
#40 lore raw

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

Quoted 18 lines · Eduardo Habkost
Quoted 17 lines · Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +0 -0
index 7ca94e9278..a64ea90fb4 100644
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
Quoted 11 lines

tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689

Quoted 7 lines
tests/qemu-iotests/iotests.py +4 -1
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO

Considering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.

Another possibility, that I find self explanatory:

import io

if sys.version_info.major >= 3:
output = io.StringIO()
else:
output = io.BytesIO()

- Cleber.

Quoted 45 lines

But this shouldn't block this workaround, so:

Quoted 39 lines
tests/qemu-iotests/iotests.py +2 -1
+
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        output = StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
tests/qemu-iotests/nbd-fault-injector.py +5 -2
index d45e2e0a6a..6b2d659dee 100755
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)
Signature · 1 line
2.17.1
#41 lore raw

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

Quoted 33 lines · Cleber Rosa
Quoted 18 lines · Eduardo Habkost
Quoted 17 lines · Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +0 -0
index 7ca94e9278..a64ea90fb4 100644
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
Quoted 11 lines

tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689

Quoted 7 lines
tests/qemu-iotests/iotests.py +4 -1
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO

Considering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.

Quoted 8 lines

Another possibility, that I find self explanatory:

import io

if sys.version_info.major >= 3:
output = io.StringIO()
else:
output = io.BytesIO()

Looks nice and clean.

But I'm not sure all output sent to output when running with Python 2 will be byte strings. What if unittest.TextTestRunner tries to write unicode strings to output?

Signature · 1 line
Eduardo
author
#42 lore raw

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

Quoted 22 lines · Eduardo Habkost
Quoted 21 lines · Cleber Rosa
Quoted 18 lines · Eduardo Habkost
Quoted 17 lines · Max Reitz

There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.

---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +0 -0
index 7ca94e9278..a64ea90fb4 100644
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689

Quoted 7 lines
tests/qemu-iotests/iotests.py +4 -1
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO

Considering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.

Oh, it does exist? I didn't know. O:-)

So I suppose it's because the test runner emits just normal strings, which in 2.x are byte strings; but io's StringIO always expects Unicode strings. StringIO, OTOH, accepts both (and returns Unicode strings once you put a Unicode string into it).

Quoted 9 lines
Quoted 8 lines

Another possibility, that I find self explanatory:

import io

if sys.version_info.major >= 3:
output = io.StringIO()
else:
output = io.BytesIO()

Quoted 5 lines

Looks nice and clean.

But I'm not sure all output sent to output when running with Python 2 will be byte strings. What if unittest.TextTestRunner tries to write unicode strings to output?

Hm. It doesn't look this way to me in practice. And considering that it only really prints a test summary, I don't think there are edge cases where it behaves differently.

So I think using BytesIO() in 2.x is better.

Max

author
#43 lore raw

PATCH 9/9 iotests: Unify log outputs between Python 2 and 3

In reply to Max Reitz

When dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). Second, the order of keys in dicts may differ. Third, especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.

To get around these differences, this patch introduces functions to convert an object to a string that looks the same regardless of the Python version: In Python 2, they decode unicode strings to byte strings (normal str); in Python 3, they encode byte strings to unicode strings (normal str). They also manually convert lists and dicts to strings, which allows sorting the dicts by key, so we are no longer at the mercy of the internal implementation when it comes to how the keys appear in the output.

This changes the output of all tests that use these logging functions.

---
 tests/qemu-iotests/194.out    |  22 +-
 tests/qemu-iotests/202.out    |  12 +-
 tests/qemu-iotests/203.out    |  14 +-
 tests/qemu-iotests/206.out    | 144 +++++-----
 tests/qemu-iotests/207.out    |  52 ++--
 tests/qemu-iotests/208.out    |   8 +-
 tests/qemu-iotests/210.out    |  72 ++---
 tests/qemu-iotests/211.out    |  66 ++---
 tests/qemu-iotests/212.out    | 102 +++----
 tests/qemu-iotests/213.out    | 124 ++++----
 tests/qemu-iotests/216.out    |   4 +-
 tests/qemu-iotests/218.out    |  20 +-
 tests/qemu-iotests/219.out    | 526 +++++++++++++++++-----------------
 tests/qemu-iotests/222.out    |  24 +-
 tests/qemu-iotests/iotests.py |  42 ++-
 15 files changed, 634 insertions(+), 598 deletions(-)
tests/qemu-iotests/194.out +11 -11
index 50ac50da5e..722af888f6 100644
@@ -1,18 +1,18 @@
 Launching VMs...
 Launching NBD server on destination...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Starting `drive-mirror` on source...
-{u'return': {}}
+{'return': {}}
 Waiting for `drive-mirror` to complete...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
+{'data': {'device': 'mirror-job0', 'len': 1073741824, 'offset': 1073741824, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Starting migration...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
+{'return': {}}
+{'data': {'status': 'setup'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'active'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'completed'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Gracefully ending the `drive-mirror` job on source...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'}
+{'return': {}}
+{'data': {'device': 'mirror-job0', 'len': 1073741824, 'offset': 1073741824, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_COMPLETED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Stopping the NBD server on destination...
-{u'return': {}}
+{'return': {}}
tests/qemu-iotests/202.out +6 -6
index d5ea374e17..d133f09fe2 100644
@@ -1,11 +1,11 @@
 Launching VM...
 Adding IOThread...
-{u'return': {}}
+{'return': {}}
 Adding blockdevs...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Setting iothread...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Creating external snapshots...
-{u'return': {}}
+{'return': {}}
tests/qemu-iotests/203.out +7 -7
index 1a11f0975c..b156388a0b 100644
@@ -1,11 +1,11 @@
 Launching VM...
 Setting IOThreads...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Enabling migration QMP events...
-{u'return': {}}
+{'return': {}}
 Starting migration...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
+{'return': {}}
+{'data': {'status': 'setup'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'active'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'completed'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
tests/qemu-iotests/206.out +72 -72

255 lines not on this page · raw message

tests/qemu-iotests/207.out +26 -26

99 lines not on this page · raw message

tests/qemu-iotests/208.out +4 -4
index 3687e9d0dd..38c26457eb 100644
@@ -1,9 +1,9 @@
 Launching VM...
 Starting NBD server...
-{u'return': {}}
+{'return': {}}
 Adding NBD export...
-{u'return': {}}
+{'return': {}}
 Creating external snapshot...
-{u'return': {}}
+{'return': {}}
 Stopping NBD server...
-{u'return': {}}
+{'return': {}}
tests/qemu-iotests/210.out +36 -36

118 lines not on this page · raw message

tests/qemu-iotests/211.out +33 -33

126 lines not on this page · raw message

tests/qemu-iotests/212.out +51 -51

199 lines not on this page · raw message

tests/qemu-iotests/213.out +62 -62

219 lines not on this page · raw message

tests/qemu-iotests/216.out +2 -2
index 45ea857ee1..908f9314ac 100644
@@ -7,8 +7,8 @@ Done
 
 --- Doing COR ---
 
-{u'return': {}}
-{u'return': u''}
+{'return': {}}
+{'return': ''}
 
 --- Checking COR result ---
 
tests/qemu-iotests/218.out +10 -10
index 7dbf78e682..9d270e4834 100644
@@ -4,27 +4,27 @@
 --- force=false ---
 
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 65536, 'speed': 65536, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 --- force=true ---
 
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 65536, 'speed': 65536, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 === Cancel mirror job after convergence ===
 
 --- force=false ---
 
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_COMPLETED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_COMPLETED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 --- force=true ---
 
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_CANCELLED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
tests/qemu-iotests/219.out +263 -263

589 lines not on this page · raw message

tests/qemu-iotests/222.out +12 -12
index 48f336a02b..37455f6c2d 100644
@@ -8,13 +8,13 @@ Done
 
 --- Setting up Fleecing Graph ---
 
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 
 --- Setting up NBD Export ---
 
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 
 --- Sanity Check ---
 
@@ -29,13 +29,13 @@ read -P0 0x3fe0000 64k
 --- Testing COW ---
 
 write -P0xab 0 64k
-{u'return': u''}
+{'return': ''}
 write -P0xad 0x00f8000 64k
-{u'return': u''}
+{'return': ''}
 write -P0x1d 0x2008000 64k
-{u'return': u''}
+{'return': ''}
 write -P0xea 0x3fe0000 64k
-{u'return': u''}
+{'return': ''}
 
 --- Verifying Data ---
 
@@ -49,10 +49,10 @@ read -P0 0x3fe0000 64k
 
 --- Cleanup ---
 
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'drive0', u'type': u'backup', u'speed': 0, u'len': 67108864, u'offset': 393216}, u'event': u'BLOCK_JOB_CANCELLED'}
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'data': {'device': 'drive0', 'len': 67108864, 'offset': 393216, 'speed': 0, 'type': 'backup'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': {}}
+{'return': {}}
 
 --- Confirming writes ---
 
tests/qemu-iotests/iotests.py +39 -3

59 lines not on this page · raw message

Signature · 1 line
2.17.1
#44 lore raw

Re: [Qemu-devel] [PATCH 9/9] iotests: Unify log outputs between Python 2 and 3

Quoted 4 lines · Max Reitz

When dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). [...]

This could be addressed using JSON. See below[1].

Quoted 1 line

[...] Second, the order of keys in dicts may differ. [...]

Can be addressed using json.dumps(..., sort_keys=True).

Quoted 3 lines

[...] Third,
especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.

print() doesn't add a L suffix on Python 2.7 on my system:

Python 2.7.15 (default, May 15 2018, 15:37:31)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

print(99999999999999999999999999999999999999999999999999999999999999999999999999999999999L)

99999999999999999999999999999999999999999999999999999999999999999999999999999999999

So I assume this happens only for QMP commands. It would be addressed if using JSON, too.

Quoted 30 lines

To get around these differences, this patch introduces functions to convert an object to a string that looks the same regardless of the Python version: In Python 2, they decode unicode strings to byte strings (normal str); in Python 3, they encode byte strings to unicode strings (normal str). They also manually convert lists and dicts to strings, which allows sorting the dicts by key, so we are no longer at the mercy of the internal implementation when it comes to how the keys appear in the output.

This changes the output of all tests that use these logging functions.

---
 tests/qemu-iotests/194.out    |  22 +-
 tests/qemu-iotests/202.out    |  12 +-
 tests/qemu-iotests/203.out    |  14 +-
 tests/qemu-iotests/206.out    | 144 +++++-----
 tests/qemu-iotests/207.out    |  52 ++--
 tests/qemu-iotests/208.out    |   8 +-
 tests/qemu-iotests/210.out    |  72 ++---
 tests/qemu-iotests/211.out    |  66 ++---
 tests/qemu-iotests/212.out    | 102 +++----
 tests/qemu-iotests/213.out    | 124 ++++----
 tests/qemu-iotests/216.out    |   4 +-
 tests/qemu-iotests/218.out    |  20 +-
 tests/qemu-iotests/219.out    | 526 +++++++++++++++++-----------------
 tests/qemu-iotests/222.out    |  24 +-
 tests/qemu-iotests/iotests.py |  42 ++-
 15 files changed, 634 insertions(+), 598 deletions(-)

[...]

Quoted 19 lines
tests/qemu-iotests/iotests.py +11 -0
index a64ea90fb4..f8dbc8cc71 100644
@@ -250,10 +250,45 @@ def filter_img_info(output, filename):
         lines.append(line)
     return '\n'.join(lines)
 
+def log_to_string_repr(obj):
+    # Normal Python 3 strings are the unicode strings from Python 2;
+    # and normal Python 2 strings are byte strings in Python 3.  Thus,
+    # we convert bytes -> str in py3 and unicode -> str in py2.
+    if sys.version_info.major >= 3:
+        if type(obj) is bytes:
+            return repr(obj.decode('utf-8'))
+    else:
+        if type(obj) is unicode:
+            return repr(obj.encode('utf-8'))
+        elif type(obj) is long:

tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 273

Quoted 10 lines
tests/qemu-iotests/iotests.py +10 -0
+            return str(obj) # repr() would include an 'L' suffix
+
+    if type(obj) is list:
+        return '[' + ', '.join(map(log_to_string_repr, obj)) + ']'
+    elif type(obj) is dict:
+        return '{' + ', '.join(map(lambda k: log_to_string_repr(k) + ': ' +
+                                             log_to_string_repr(obj[k]),
+                                   sorted(obj.keys()))) + '}'
+    else:
+        return repr(obj)

I assume this function exists only because of QMP logging, correct?

I would just use json.dumps() at qmp_log(), see below[1].

tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 281

Quoted 8 lines
tests/qemu-iotests/iotests.py +8 -0
+
+def log_to_string(obj):
+    if type(obj) is str:
+        return obj
+
+    if sys.version_info.major >= 3:
+        if type(obj) is bytes:
+            return obj.decode('utf-8')

Do you know how many of existing log() calls actually pass a byte string as argument?

tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 284

Quoted 3 lines
tests/qemu-iotests/iotests.py +3 -0
+    else:
+        if type(obj) is unicode:
+            return obj.encode('utf-8')

Is this really necessary? The existing code just calls print(msg) and it works, doesn't it?

Quoted 12 lines
tests/qemu-iotests/iotests.py +4 -1
+
+    return log_to_string_repr(obj)
+
 def log(msg, filters=[]):
     for flt in filters:
         msg = flt(msg)
-    print(msg)
+    print(log_to_string(msg))
 
 class Timeout:
     def __init__(self, seconds, errmsg = "Timeout"):

tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 483

Quoted 10 lines
tests/qemu-iotests/iotests.py +3 -2
@@ -441,10 +476,11 @@ class VM(qtest.QEMUQtestMachine):
         return result
 
     def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
-        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
+        logmsg = "{'execute': '%s', 'arguments': %s}" % \
+            (cmd, log_to_string(kwargs))
         log(logmsg, filters)
         result = self.qmp(cmd, **kwargs)
-        log(str(result), filters)
+        log(log_to_string(result), filters)

[1]

If we're being forced to regenerate all the QMP output in the .out files, what about always using JSON instead of the hack at log_to_string_repr()? i.e.:

tests/qemu-iotests/iotests.py +4 -2
index a64ea90fb4..0b28dc2a65 100644
@@ -441,10 +441,12 @@ class VM(qtest.QEMUQtestMachine):
         return result
 
     def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
-        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
+        logmsg = '{"execute": %s, "arguments": %s}' % \
+                 (json.dumps(cmd, sort_keys=True),
+                  json.dumps(kwargs, sort_keys=True))
         log(logmsg, filters)
         result = self.qmp(cmd, **kwargs)
-        log(str(result), filters)
+        log(json.dumps(result, sort_keys=True), filters)
         return result
 
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Quoted 7 lines
tests/qemu-iotests/iotests.py +0 -0
         return result
 
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
author
#45 lore raw

Re: [Qemu-devel] [PATCH 9/9] iotests: Unify log outputs between Python 2 and 3

Quoted 11 lines · Eduardo Habkost
Quoted 4 lines · Max Reitz

When dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). [...]

This could be addressed using JSON. See below[1].

Quoted 1 line

[...] Second, the order of keys in dicts may differ. [...]

Can be addressed using json.dumps(..., sort_keys=True).

Ah. Nice. :-)

Quoted 6 lines
Quoted 3 lines

[...] Third,
especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.

print() doesn't add a L suffix on Python 2.7 on my system:

Quoted 7 lines

Python 2.7.15 (default, May 15 2018, 15:37:31)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

print(99999999999999999999999999999999999999999999999999999999999999999999999999999999999L)

99999999999999999999999999999999999999999999999999999999999999999999999999999999999

So I assume this happens only for QMP commands.

It happens for dicts:

Quoted 1 line
Quoted 1 line
Quoted 1 line

print 99999999999999999999L

99999999999999999999

Quoted 1 line
Quoted 1 line
Quoted 1 line

print {'foo':99999999999999999999L}

{'foo': 99999999999999999999L}

Quoted 1 line

It would be addressed if using JSON, too.

OK.

Quoted 54 lines
Quoted 29 lines

To get around these differences, this patch introduces functions to convert an object to a string that looks the same regardless of the Python version: In Python 2, they decode unicode strings to byte strings (normal str); in Python 3, they encode byte strings to unicode strings (normal str). They also manually convert lists and dicts to strings, which allows sorting the dicts by key, so we are no longer at the mercy of the internal implementation when it comes to how the keys appear in the output.

This changes the output of all tests that use these logging functions.

---
 tests/qemu-iotests/194.out    |  22 +-
 tests/qemu-iotests/202.out    |  12 +-
 tests/qemu-iotests/203.out    |  14 +-
 tests/qemu-iotests/206.out    | 144 +++++-----
 tests/qemu-iotests/207.out    |  52 ++--
 tests/qemu-iotests/208.out    |   8 +-
 tests/qemu-iotests/210.out    |  72 ++---
 tests/qemu-iotests/211.out    |  66 ++---
 tests/qemu-iotests/212.out    | 102 +++----
 tests/qemu-iotests/213.out    | 124 ++++----
 tests/qemu-iotests/216.out    |   4 +-
 tests/qemu-iotests/218.out    |  20 +-
 tests/qemu-iotests/219.out    | 526 +++++++++++++++++-----------------
 tests/qemu-iotests/222.out    |  24 +-
 tests/qemu-iotests/iotests.py |  42 ++-
 15 files changed, 634 insertions(+), 598 deletions(-)

[...]

Quoted 24 lines
tests/qemu-iotests/iotests.py +16 -0
index a64ea90fb4..f8dbc8cc71 100644
@@ -250,10 +250,45 @@ def filter_img_info(output, filename):
         lines.append(line)
     return '\n'.join(lines)
 
+def log_to_string_repr(obj):
+    # Normal Python 3 strings are the unicode strings from Python 2;
+    # and normal Python 2 strings are byte strings in Python 3.  Thus,
+    # we convert bytes -> str in py3 and unicode -> str in py2.
+    if sys.version_info.major >= 3:
+        if type(obj) is bytes:
+            return repr(obj.decode('utf-8'))
+    else:
+        if type(obj) is unicode:
+            return repr(obj.encode('utf-8'))
+        elif type(obj) is long:
+            return str(obj) # repr() would include an 'L' suffix
+
+    if type(obj) is list:
+        return '[' + ', '.join(map(log_to_string_repr, obj)) + ']'
+    elif type(obj) is dict:
Quoted 8 lines

tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 273

Quoted 5 lines
tests/qemu-iotests/iotests.py +5 -0
+        return '{' + ', '.join(map(lambda k: log_to_string_repr(k) + ': ' +
+                                             log_to_string_repr(obj[k]),
+                                   sorted(obj.keys()))) + '}'
+    else:
+        return repr(obj)

I assume this function exists only because of QMP logging, correct?

In practice probably yes.

Quoted 1 line

I would just use json.dumps() at qmp_log(), see below[1].

However, there is not just qmp_log(), there are places that dump objects directly into log().

Quoted 11 lines

tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 281

Quoted 8 lines
tests/qemu-iotests/iotests.py +8 -0
+
+def log_to_string(obj):
+    if type(obj) is str:
+        return obj
+
+    if sys.version_info.major >= 3:
+        if type(obj) is bytes:
+            return obj.decode('utf-8')

Do you know how many of existing log() calls actually pass a byte string as argument?

Frankly I hope none.

Quoted 6 lines

tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 284

Quoted 3 lines
tests/qemu-iotests/iotests.py +3 -0
+    else:
+        if type(obj) is unicode:
+            return obj.encode('utf-8')

Is this really necessary? The existing code just calls print(msg) and it works, doesn't it?

True. But the main issue is that we still need to return immediately for byte strings and Unicode strings, but the type "bytes" only exists in 3.x and "unicode" only exists in 2.x, i.e. "if type(obj) is unicode" throws an exception in 3.x.

So I have to distinguish between 2.x and 3.x anyway, and I thought to myself that if I distinguished anyway, I might do the conversion to the native type at the same time.

Quoted 36 lines
Quoted 12 lines
tests/qemu-iotests/iotests.py +4 -1
+
+    return log_to_string_repr(obj)
+
 def log(msg, filters=[]):
     for flt in filters:
         msg = flt(msg)
-    print(msg)
+    print(log_to_string(msg))
 
 class Timeout:
     def __init__(self, seconds, errmsg = "Timeout"):

tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 483

Quoted 10 lines
tests/qemu-iotests/iotests.py +3 -2
@@ -441,10 +476,11 @@ class VM(qtest.QEMUQtestMachine):
         return result
 
     def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
-        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
+        logmsg = "{'execute': '%s', 'arguments': %s}" % \
+            (cmd, log_to_string(kwargs))
         log(logmsg, filters)
         result = self.qmp(cmd, **kwargs)
-        log(str(result), filters)
+        log(log_to_string(result), filters)

[1]

If we're being forced to regenerate all the QMP output in the .out files, what about always using JSON instead of the hack at log_to_string_repr()? i.e.:

tests/qemu-iotests/iotests.py +0 -0
index a64ea90fb4..0b28dc2a65 100644
@@ -441,10 +441,12 @@ class VM(qtest.QEMUQtestMachine):
         return result
 

tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 450

Quoted 10 lines
tests/qemu-iotests/iotests.py +4 -2
     def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
-        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
+        logmsg = '{"execute": %s, "arguments": %s}' % \
+                 (json.dumps(cmd, sort_keys=True),
+                  json.dumps(kwargs, sort_keys=True))
         log(logmsg, filters)
         result = self.qmp(cmd, **kwargs)
-        log(str(result), filters)
+        log(json.dumps(result, sort_keys=True), filters)
         return result

Sure, sounds good, thanks. :-)

The only thing is I would allow log() to still accept objects, it should do the conversion to JSON itself (if the object to be logged is a list or dict).

Max

Quoted 10 lines
tests/qemu-iotests/iotests.py +0 -0
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):
 
Quoted 7 lines
tests/qemu-iotests/iotests.py +0 -0
         return result
 
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Signature · 1 line
2.17.1
#46 lore raw
In reply to Max Reitz

Hi Max,

Quoted 10 lines · Max Reitz

This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO class from the 'io' or the 'StringIO' module), but most of the time, it just makes code work for both versions in general.

And when we make the switch to make Python 3 mandatory, we can simply drop the Python 2 branches.

Quoted 10 lines

Max Reitz (9):
iotests: Make nbd-fault-injector flush
iotests: Flush in iotests.py's QemuIoInteractive
iotests: Use Python byte strings where appropriate
iotests: Use // for Python integer division
iotests: Different iterator behavior in Python 3
iotests: Explicitly inherit FDs in Python
iotests: 'new' module replacement in 169
iotests: Modify imports for Python 3
iotests: Unify log outputs between Python 2 and 3

You forgot:

MAINTAINERS: Add myself in the Python scripts section

;)

Quoted 37 lines
 scripts/qemu.py                          |  13 +-
 scripts/qmp/qmp.py                       |   7 +
 scripts/qtest.py                         |   2 +-
 tests/qemu-iotests/040                   |   4 +-
 tests/qemu-iotests/044                   |  20 +-
 tests/qemu-iotests/056                   |   2 +-
 tests/qemu-iotests/065                   |   4 +-
 tests/qemu-iotests/083.out               |   9 +
 tests/qemu-iotests/093                   |  18 +-
 tests/qemu-iotests/124                   |   4 +-
 tests/qemu-iotests/139                   |   2 +-
 tests/qemu-iotests/147                   |   7 +
 tests/qemu-iotests/149                   |  14 +-
 tests/qemu-iotests/151                   |  12 +-
 tests/qemu-iotests/163                   |   8 +-
 tests/qemu-iotests/169                   |   3 +-
 tests/qemu-iotests/194.out               |  22 +-
 tests/qemu-iotests/202.out               |  12 +-
 tests/qemu-iotests/203.out               |  14 +-
 tests/qemu-iotests/206.out               | 144 +++----
 tests/qemu-iotests/207                   |   4 +-
 tests/qemu-iotests/207.out               |  52 +--
 tests/qemu-iotests/208.out               |   8 +-
 tests/qemu-iotests/210.out               |  72 ++--
 tests/qemu-iotests/211.out               |  66 +--
 tests/qemu-iotests/212.out               | 102 ++---
 tests/qemu-iotests/213.out               | 124 +++---
 tests/qemu-iotests/216.out               |   4 +-
 tests/qemu-iotests/218.out               |  20 +-
 tests/qemu-iotests/219.out               | 526 +++++++++++------------
 tests/qemu-iotests/222.out               |  24 +-
 tests/qemu-iotests/iotests.py            |  64 ++-
 tests/qemu-iotests/nbd-fault-injector.py |  12 +-
 tests/qemu-iotests/qcow2.py              |  10 +-
 34 files changed, 745 insertions(+), 664 deletions(-)
author
#47 lore raw
Quoted 24 lines · Philippe Mathieu-Daudé

Hi Max,

Quoted 10 lines · Max Reitz

This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO class from the 'io' or the 'StringIO' module), but most of the time, it just makes code work for both versions in general.

And when we make the switch to make Python 3 mandatory, we can simply drop the Python 2 branches.

Quoted 10 lines

Max Reitz (9):
iotests: Make nbd-fault-injector flush
iotests: Flush in iotests.py's QemuIoInteractive
iotests: Use Python byte strings where appropriate
iotests: Use // for Python integer division
iotests: Different iterator behavior in Python 3
iotests: Explicitly inherit FDs in Python
iotests: 'new' module replacement in 169
iotests: Modify imports for Python 3
iotests: Unify log outputs between Python 2 and 3

Quoted 5 lines

You forgot:

MAINTAINERS: Add myself in the Python scripts section

;)

Aw, please, no. You do know I don't really know Python? ;-)

Max