iotests: Make them work for both Python 2 and 3
cover iotests: Make them work for both Python 2 and 3
This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO class from the 'io' or the 'StringIO' module), but most of the time, it just makes code work for both versions in general.
And when we make the switch to make Python 3 mandatory, we can simply drop the Python 2 branches.
Max Reitz (9):
iotests: Make nbd-fault-injector flush
iotests: Flush in iotests.py's QemuIoInteractive
iotests: Use Python byte strings where appropriate
iotests: Use // for Python integer division
iotests: Different iterator behavior in Python 3
iotests: Explicitly inherit FDs in Python
iotests: 'new' module replacement in 169
iotests: Modify imports for Python 3
iotests: Unify log outputs between Python 2 and 3
scripts/qemu.py | 13 +- scripts/qmp/qmp.py | 7 + scripts/qtest.py | 2 +- tests/qemu-iotests/040 | 4 +- tests/qemu-iotests/044 | 20 +- tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 +- tests/qemu-iotests/083.out | 9 + tests/qemu-iotests/093 | 18 +- tests/qemu-iotests/124 | 4 +- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/147 | 7 + tests/qemu-iotests/149 | 14 +- tests/qemu-iotests/151 | 12 +- tests/qemu-iotests/163 | 8 +- tests/qemu-iotests/169 | 3 +- tests/qemu-iotests/194.out | 22 +- tests/qemu-iotests/202.out | 12 +- tests/qemu-iotests/203.out | 14 +- tests/qemu-iotests/206.out | 144 +++---- tests/qemu-iotests/207 | 4 +- tests/qemu-iotests/207.out | 52 +-- tests/qemu-iotests/208.out | 8 +- tests/qemu-iotests/210.out | 72 ++-- tests/qemu-iotests/211.out | 66 +-- tests/qemu-iotests/212.out | 102 ++--- tests/qemu-iotests/213.out | 124 +++--- tests/qemu-iotests/216.out | 4 +- tests/qemu-iotests/218.out | 20 +- tests/qemu-iotests/219.out | 526 +++++++++++------------ tests/qemu-iotests/222.out | 24 +- tests/qemu-iotests/iotests.py | 64 ++- tests/qemu-iotests/nbd-fault-injector.py | 12 +- tests/qemu-iotests/qcow2.py | 10 +- 34 files changed, 745 insertions(+), 664 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/9 iotests: Make nbd-fault-injector flush
When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.
--- tests/qemu-iotests/083.out | 9 +++++++++ tests/qemu-iotests/nbd-fault-injector.py | 1 + 2 files changed, 10 insertions(+)
tests/qemu-iotests/083.out +9 -0
@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect after neg2 === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error
@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect before request === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error
@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/ === Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error
@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock === Check disconnect after neg2 === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error === Check disconnect 8 neg2 ===
@@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock === Check disconnect before request === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error === Check disconnect after request ===
@@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock === Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error *** done
tests/qemu-iotests/nbd-fault-injector.py +1 -0
@@ -112,6 +112,7 @@ class FaultInjectionSocket(object): if rule.match(event, io): if rule.when == 0 or bufsize is None: print('Closing connection on rule match %s' % rule.name) + self.sock.flush() sys.exit(0) if rule.when != -1: return rule.when
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush
Quoted 4 lines · Max Reitz
When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.
Signed-off-by Max Reitz <[email protected]>
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush
In reply to Max ReitzQuoted 4 lines · Max Reitz
When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.
Signed-off-by Max Reitz <[email protected]>
Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush
In reply to Max ReitzQuoted 8 lines · Max Reitz
When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/083.out | 9 +++++++++ tests/qemu-iotests/nbd-fault-injector.py | 1 + 2 files changed, 10 insertions(+)
I already had a complaint that the error message in 083.out should NOT be printing a message (whether the server is python 2 and auto-flushes, or python 3 and needs an explicit flush, the message itself is pointless, and the test is racy as a result). We may need to revisit this patch when that thread is resolved.
https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html
That said, I'm not opposed to this patch, if it gets iotests to be more useful in the meantime.
Quoted 69 lines
tests/qemu-iotests/083.out +9
-0@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect after neg2 === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect before request === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/ === Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock === Check disconnect after neg2 === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error === Check disconnect 8 neg2 ===@@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock === Check disconnect before request === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error === Check disconnect after request ===@@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock === Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error *** donetests/qemu-iotests/nbd-fault-injector.py +1
-0@@ -112,6 +112,7 @@ class FaultInjectionSocket(object): if rule.match(event, io): if rule.when == 0 or bufsize is None: print('Closing connection on rule match %s' % rule.name) + self.sock.flush() sys.exit(0) if rule.when != -1: return rule.when
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush
Quoted 4 lines · Eric Blake
Quoted 3 lines · Max Reitz
When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.
Quoted 8 lines
Quoted 5 lines
Signed-off-by Max Reitz <[email protected]>---
tests/qemu-iotests/083.out | 9 +++++++++tests/qemu-iotests/nbd-fault-injector.py| 1 +2 files changed, 10 insertions(+)I already had a complaint that the error message in 083.out should NOT be printing a message
You mean the NBD server itself, right?
Quoted 6 lines
(whether the server is python 2 and auto-flushes,
or python 3 and needs an explicit flush, the message itself is pointless, and the test is racy as a result). We may need to revisit this patch when that thread is resolved.https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html
Well, it's not like the flush hurts either way. :-)
Max
Quoted 60 lines
That said, I'm not opposed to this patch, if it gets iotests to be more useful in the meantime.
Quoted 56 lines
tests/qemu-iotests/083.out +0
-0@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo=== Check disconnect after neg2 ===
+Unable to read from socket: Connection reset by peer Connection closed
read failed: Input/output error
@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect before request ===
+Unable to read from socket: Connection reset by peer Connection closed
read failed: Input/output error
@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/ === Check disconnect after neg-classic ===
+Unable to read from socket: Connection reset by peer Connection closed
read failed: Input/output error
@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
=== Check disconnect after neg2 ===
+Unable to read from socket: Connection reset by peer +Connection closed
read failed: Input/output error
=== Check disconnect 8 neg2 ===tests/qemu-iotests/083.out +0
-0@@ -173,6 +178,8 @@ can't open devicenbd+unix:///foo?socket=TEST_DIR/nbd.sock
=== Check disconnect before request === +Unable to read from socket: Connection reset by peer +Connection closed
read failed: Input/output error
=== Check disconnect after request ===tests/qemu-iotests/083.out +0
-0@@ -234,6 +241,8 @@ can't open devicenbd+unix:///?socket=TEST_DIR/nbd.sock
=== Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer +Connection closed
read failed: Input/output error
*** donetests/qemu-iotests/nbd-fault-injector.py +0
-0@@ -112,6 +112,7 @@ class FaultInjectionSocket(object):if
rule.match(event, io):
if rule.when == 0 or bufsize is None:
print('Closing connection on rule match %s' % rule.name)
+self.sock.flush()
sys.exit(0)
if rule.when != -1:
return rule.when
Eric Blake
Reviewed-byRe: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush
Quoted 11 lines · Max Reitz
Quoted 10 lines · Eric Blake
Quoted 8 lines · Max Reitz
When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/083.out | 9 +++++++++ tests/qemu-iotests/nbd-fault-injector.py | 1 + 2 files changed, 10 insertions(+)
Quoted 4 lines
Quoted 2 lines
I already had a complaint that the error message in 083.out should NOT be printing a message
You mean the NBD server itself, right?
Yes, the NBD server should not be printing the redundant error messages that 083.out exposes.
Quoted 9 lines
Quoted 7 lines
(whether the server is python 2 and auto-flushes,
or python 3 and needs an explicit flush, the message itself is pointless, and the test is racy as a result). We may need to revisit this patch when that thread is resolved.https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html
Well, it's not like the flush hurts either way. :-)
True. On that grounds:
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH 2/9 iotests: Flush in iotests.py's QemuIoInteractive
In reply to Max ReitzAfter issuing a command, flush the pipe. This does not change anything in Python 2, but it makes a difference in Python 3.
--- tests/qemu-iotests/iotests.py | 1 + 1 file changed, 1 insertion(+)
tests/qemu-iotests/iotests.py +1 -0
@@ -178,6 +178,7 @@ class QemuIoInteractive: cmd = cmd.strip() assert cmd != 'q' and cmd != 'quit' self._p.stdin.write(cmd + '\n') + self._p.stdin.flush() return self._read_output()
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 2/9] iotests: Flush in iotests.py's QemuIoInteractive
Quoted 4 lines · Max Reitz
After issuing a command, flush the pipe. This does not change anything in Python 2, but it makes a difference in Python 3.
Signed-off-by Max Reitz <[email protected]>
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 2/9] iotests: Flush in iotests.py's QemuIoInteractive
In reply to Max ReitzQuoted 4 lines · Max Reitz
After issuing a command, flush the pipe. This does not change anything in Python 2, but it makes a difference in Python 3.
Signed-off-by Max Reitz <[email protected]>
PATCH 3/9 iotests: Use Python byte strings where appropriate
In reply to Max ReitzSince byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.
On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set univeral_newlines to True so that the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.
--- scripts/qtest.py | 2 +- tests/qemu-iotests/044 | 8 ++++---- tests/qemu-iotests/149 | 8 +++++--- tests/qemu-iotests/207 | 4 ++-- tests/qemu-iotests/iotests.py | 11 +++++++---- tests/qemu-iotests/nbd-fault-injector.py | 4 ++-- tests/qemu-iotests/qcow2.py | 10 +++++----- 7 files changed, 26 insertions(+), 21 deletions(-)
scripts/qtest.py +1 -1
@@ -64,7 +64,7 @@ class QEMUQtestProtocol(object): @param qtest_cmd: qtest command text to be sent """ - self._sock.sendall(qtest_cmd + "\n") + self._sock.sendall((qtest_cmd + "\n").encode('utf-8')) def close(self): self._sock.close()
tests/qemu-iotests/044 +4 -4
@@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): fd.seek(off_reftable) for i in xrange(0, h.refcount_table_clusters): - sector = ''.join(struct.pack('>Q', + sector = b''.join(struct.pack('>Q', off_refblock + i * 64 * 512 + j * 512) for j in xrange(0, 64)) fd.write(sector) # Write the refcount blocks assert(fd.tell() == off_refblock) - sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256)) + sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256)) for block in xrange(0, h.refcount_table_clusters): fd.write(sector) # Write the L1 table assert(fd.tell() == off_l1) assert(off_l2 + 512 * h.l1_size == off_data) - table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) + table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) for j in xrange(0, h.l1_size)) fd.write(table)
@@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): remaining = remaining - 1024 * 512 off = off + 1024 * 512 - table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) + table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) for j in xrange(0, remaining / 512)) fd.write(table)
tests/qemu-iotests/149 +5 -3
@@ -79,7 +79,7 @@ class LUKSConfig(object): def first_password_base64(self): (pw, slot) = self.first_password() - return base64.b64encode(pw) + return base64.b64encode(pw.encode('ascii')).decode('ascii') def active_slots(self): slots = []
@@ -98,7 +98,8 @@ def verify_passwordless_sudo(): proc = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) msg = proc.communicate()[0]
@@ -116,7 +117,8 @@ def cryptsetup(args, password=None): proc = subprocess.Popen(fullargs, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) msg = proc.communicate(password)[0]
tests/qemu-iotests/207 +2 -2
@@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \ md5_key = subprocess.check_output( 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + 'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1', - shell=True).rstrip() + shell=True).rstrip().decode('ascii') vm.launch() blockdev_create(vm, { 'driver': 'ssh',
@@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \ sha1_key = subprocess.check_output( 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + 'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1', - shell=True).rstrip() + shell=True).rstrip().decode('ascii') vm.launch() blockdev_create(vm, { 'driver': 'ssh',
tests/qemu-iotests/iotests.py +7 -4
@@ -104,7 +104,8 @@ def qemu_img_pipe(*args): '''Run qemu-img and return its output''' subp = subprocess.Popen(qemu_img_args + list(args), stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) exitcode = subp.wait() if exitcode < 0: sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
@@ -128,7 +129,8 @@ def qemu_io(*args): '''Run qemu-io and return the stdout data''' args = qemu_io_args + list(args) subp = subprocess.Popen(args, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) exitcode = subp.wait() if exitcode < 0: sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))
@@ -149,7 +151,8 @@ class QemuIoInteractive: self.args = qemu_io_args + list(args) self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) assert self._p.stdout.read(9) == 'qemu-io> ' def close(self):
@@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt): def create_image(name, size): '''Create a fully-allocated raw image with sector markers''' - file = open(name, 'w') + file = open(name, 'wb') i = 0 while i < size: sector = struct.pack('>l504xl', i / 512, i / 512)
tests/qemu-iotests/nbd-fault-injector.py +2 -2
@@ -86,7 +86,7 @@ def recvall(sock, bufsize): raise Exception('unexpected disconnect') chunks.append(chunk) received += len(chunk) - return ''.join(chunks) + return b''.join(chunks) class Rule(object): def __init__(self, name, event, io, when):
@@ -177,7 +177,7 @@ def handle_connection(conn, use_export): req = read_request(conn) if req.type == NBD_CMD_READ: write_reply(conn, 0, req.handle) - conn.send('\0' * req.len, event='data') + conn.send(b'\0' * req.len, event='data') elif req.type == NBD_CMD_WRITE: _ = conn.recv(req.len, event='data') write_reply(conn, 0, req.handle)
tests/qemu-iotests/qcow2.py +5 -5
@@ -10,7 +10,7 @@ class QcowHeaderExtension: def __init__(self, magic, length, data): if length % 8 != 0: padding = 8 - (length % 8) - data += "\0" * padding + data += b"\0" * padding self.magic = magic self.length = length
@@ -103,7 +103,7 @@ class QcowHeader: fd.seek(self.header_length) extensions = self.extensions - extensions.append(QcowHeaderExtension(0, 0, "")) + extensions.append(QcowHeaderExtension(0, 0, b"")) for ex in extensions: buf = struct.pack('>II', ex.magic, ex.length) fd.write(buf)
@@ -137,8 +137,8 @@ class QcowHeader: for ex in self.extensions: data = ex.data[:ex.length] - if all(c in string.printable for c in data): - data = "'%s'" % data + if all(c in string.printable.encode('ascii') for c in data): + data = "'%s'" % data.decode('ascii') else: data = "<binary>"
@@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data): sys.exit(1) h = QcowHeader(fd) - h.extensions.append(QcowHeaderExtension.create(magic, data)) + h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii'))) h.update(fd) def cmd_add_header_ext_stdio(fd, magic):
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate
Quoted 15 lines · Max Reitz
Since byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.
On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set
univeral_newlinesto True so that the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.Signed-off-by Max Reitz <[email protected]>
[...]
tests/qemu-iotests/149 class LUKSConfig(object): · line 82
Quoted 10 lines
tests/qemu-iotests/149 +1
-1@@ -79,7 +79,7 @@ class LUKSConfig(object): def first_password_base64(self): (pw, slot) = self.first_password() - return base64.b64encode(pw) + return base64.b64encode(pw.encode('ascii')).decode('ascii')
Would we want to have a test case for non-ascii passwords in the future? In that case you would probably need to make self.passwords[] contain byte strings instead of text.
In either case, using 'ascii' as the encoding everywhere will ensure the code will not try to be too smart about string encodings if that happens. I like that.
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate
Quoted 23 lines · Eduardo Habkost
Quoted 15 lines · Max Reitz
Since byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.
On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set
univeral_newlinesto True so that the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.Signed-off-by Max Reitz <[email protected]>[...]
Quoted 6 lines
tests/qemu-iotests/149 +0
-0@@ -79,7 +79,7 @@ class LUKSConfig(object):
Quoted 8 lines
tests/qemu-iotests/149 class LUKSConfig(object): · line 82
Quoted 4 lines
tests/qemu-iotests/149 +1
-1def first_password_base64(self): (pw, slot) = self.first_password() - return base64.b64encode(pw) + return base64.b64encode(pw.encode('ascii')).decode('ascii')Would we want to have a test case for non-ascii passwords in the future? In that case you would probably need to make self.passwords[] contain byte strings instead of text.
I remember someone once providing a non-ASCII initial password to some system for me. I remember the system was running Windows, and I was running Linux, so the system expected ISO-8859-1, while I was sending UTF-8.
The moral of the story is that you probably don't want non-ASCII passwords. And if we do want to test them, well, we'll need to decide on an encoding then (or use byte strings, as you suggest).
Quoted 5 lines
In either case, using 'ascii' as the encoding everywhere will ensure the code will not try to be too smart about string encodings if that happens. I like that.
Reviewed-by Eduardo Habkost <[email protected]>
Thanks for reviewing!
Max
Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate
In reply to Max ReitzQuoted 10 lines · Max Reitz
Since byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.
On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set
univeral_newlinesto True so that
'universal_newlines'
Quoted 229 lines
the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.
Signed-off-by Max Reitz <[email protected]>--- scripts/qtest.py | 2 +- tests/qemu-iotests/044 | 8 ++++---- tests/qemu-iotests/149 | 8 +++++--- tests/qemu-iotests/207 | 4 ++-- tests/qemu-iotests/iotests.py | 11 +++++++---- tests/qemu-iotests/nbd-fault-injector.py | 4 ++-- tests/qemu-iotests/qcow2.py | 10 +++++----- 7 files changed, 26 insertions(+), 21 deletions(-)scripts/qtest.py +1
-1@@ -64,7 +64,7 @@ class QEMUQtestProtocol(object): @param qtest_cmd: qtest command text to be sent """ - self._sock.sendall(qtest_cmd + "\n") + self._sock.sendall((qtest_cmd + "\n").encode('utf-8')) def close(self): self._sock.close()tests/qemu-iotests/044 +4
-4@@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): fd.seek(off_reftable) for i in xrange(0, h.refcount_table_clusters): - sector = ''.join(struct.pack('>Q', + sector = b''.join(struct.pack('>Q', off_refblock + i * 64 * 512 + j * 512) for j in xrange(0, 64)) fd.write(sector) # Write the refcount blocks assert(fd.tell() == off_refblock) - sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256)) + sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256)) for block in xrange(0, h.refcount_table_clusters): fd.write(sector) # Write the L1 table assert(fd.tell() == off_l1) assert(off_l2 + 512 * h.l1_size == off_data) - table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) + table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) for j in xrange(0, h.l1_size)) fd.write(table)@@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): remaining = remaining - 1024 * 512 off = off + 1024 * 512 - table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) + table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) for j in xrange(0, remaining / 512)) fd.write(table)tests/qemu-iotests/149 +5
-3@@ -79,7 +79,7 @@ class LUKSConfig(object): def first_password_base64(self): (pw, slot) = self.first_password() - return base64.b64encode(pw) + return base64.b64encode(pw.encode('ascii')).decode('ascii') def active_slots(self): slots = []@@ -98,7 +98,8 @@ def verify_passwordless_sudo(): proc = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) msg = proc.communicate()[0]@@ -116,7 +117,8 @@ def cryptsetup(args, password=None): proc = subprocess.Popen(fullargs, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) msg = proc.communicate(password)[0]tests/qemu-iotests/207 +2
-2@@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \ md5_key = subprocess.check_output( 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + 'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1', - shell=True).rstrip() + shell=True).rstrip().decode('ascii') vm.launch() blockdev_create(vm, { 'driver': 'ssh',@@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \ sha1_key = subprocess.check_output( 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + 'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1', - shell=True).rstrip() + shell=True).rstrip().decode('ascii') vm.launch() blockdev_create(vm, { 'driver': 'ssh',tests/qemu-iotests/iotests.py +7
-4@@ -104,7 +104,8 @@ def qemu_img_pipe(*args): '''Run qemu-img and return its output''' subp = subprocess.Popen(qemu_img_args + list(args), stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) exitcode = subp.wait() if exitcode < 0: sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))@@ -128,7 +129,8 @@ def qemu_io(*args): '''Run qemu-io and return the stdout data''' args = qemu_io_args + list(args) subp = subprocess.Popen(args, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) exitcode = subp.wait() if exitcode < 0: sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))@@ -149,7 +151,8 @@ class QemuIoInteractive: self.args = qemu_io_args + list(args) self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) assert self._p.stdout.read(9) == 'qemu-io> ' def close(self):@@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt): def create_image(name, size): '''Create a fully-allocated raw image with sector markers''' - file = open(name, 'w') + file = open(name, 'wb') i = 0 while i < size: sector = struct.pack('>l504xl', i / 512, i / 512)tests/qemu-iotests/nbd-fault-injector.py +2
-2@@ -86,7 +86,7 @@ def recvall(sock, bufsize): raise Exception('unexpected disconnect') chunks.append(chunk) received += len(chunk) - return ''.join(chunks) + return b''.join(chunks) class Rule(object): def __init__(self, name, event, io, when):@@ -177,7 +177,7 @@ def handle_connection(conn, use_export): req = read_request(conn) if req.type == NBD_CMD_READ: write_reply(conn, 0, req.handle) - conn.send('\0' * req.len, event='data') + conn.send(b'\0' * req.len, event='data') elif req.type == NBD_CMD_WRITE: _ = conn.recv(req.len, event='data') write_reply(conn, 0, req.handle)tests/qemu-iotests/qcow2.py +5
-5@@ -10,7 +10,7 @@ class QcowHeaderExtension: def __init__(self, magic, length, data): if length % 8 != 0: padding = 8 - (length % 8) - data += "\0" * padding + data += b"\0" * padding self.magic = magic self.length = length@@ -103,7 +103,7 @@ class QcowHeader: fd.seek(self.header_length) extensions = self.extensions - extensions.append(QcowHeaderExtension(0, 0, "")) + extensions.append(QcowHeaderExtension(0, 0, b"")) for ex in extensions: buf = struct.pack('>II', ex.magic, ex.length) fd.write(buf)@@ -137,8 +137,8 @@ class QcowHeader: for ex in self.extensions: data = ex.data[:ex.length] - if all(c in string.printable for c in data): - data = "'%s'" % data + if all(c in string.printable.encode('ascii') for c in data): + data = "'%s'" % data.decode('ascii') else: data = "<binary>"@@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data): sys.exit(1) h = QcowHeader(fd) - h.extensions.append(QcowHeaderExtension.create(magic, data)) + h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii'))) h.update(fd) def cmd_add_header_ext_stdio(fd, magic):
PATCH 4/9 iotests: Use // for Python integer division
In reply to Max ReitzIn Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.
--- tests/qemu-iotests/040 | 4 ++-- tests/qemu-iotests/044 | 2 +- tests/qemu-iotests/093 | 18 +++++++++--------- tests/qemu-iotests/149 | 6 +++--- tests/qemu-iotests/151 | 12 ++++++------ tests/qemu-iotests/163 | 2 +- tests/qemu-iotests/iotests.py | 2 +- 7 files changed, 23 insertions(+), 23 deletions(-)
tests/qemu-iotests/040 +2 -2
@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('device_del', id='scsi0') self.assert_qmp(result, 'return', {})
@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('query-block')
tests/qemu-iotests/044 +1 -1
@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining / 512)) + for j in xrange(0, remaining // 512)) fd.write(table)
tests/qemu-iotests/093 +9 -9
@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase): # in. The throttled requests won't be executed until we # advance the virtual clock. rq_size = 512 - rd_nr = max(params['bps'] / rq_size / 2, - params['bps_rd'] / rq_size, - params['iops'] / 2, + rd_nr = max(params['bps'] // rq_size // 2, + params['bps_rd'] // rq_size, + params['iops'] // 2, params['iops_rd']) rd_nr *= seconds * 2 - rd_nr /= ndrives - wr_nr = max(params['bps'] / rq_size / 2, - params['bps_wr'] / rq_size, - params['iops'] / 2, + rd_nr //= ndrives + wr_nr = max(params['bps'] // rq_size // 2, + params['bps_wr'] // rq_size, + params['iops'] // 2, params['iops_wr']) wr_nr *= seconds * 2 - wr_nr /= ndrives + wr_nr //= ndrives # Send I/O requests to all drives for i in range(rd_nr):
@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase): self.configure_throttle(ndrives, settings) # Wait for the bucket to empty so we can do bursts - wait_ns = nsec_per_sec * burst_length * burst_rate / rate + wait_ns = nsec_per_sec * burst_length * burst_rate // rate self.vm.qtest("clock_step %d" % wait_ns) # Test I/O at the max burst rate
tests/qemu-iotests/149 +3 -3
@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False): image_size = 4 * oneTB if qemu_img: iotests.log("# Create image") - qemu_img_create(config, image_size / oneMB) + qemu_img_create(config, image_size // oneMB) else: iotests.log("# Create image") - create_image(config, image_size / oneMB) + create_image(config, image_size // oneMB) lowOffsetMB = 100 - highOffsetMB = 3 * oneTB / oneMB + highOffsetMB = 3 * oneTB // oneMB try: if not qemu_img:
tests/qemu-iotests/151 +6 -6
@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase): 'write -P 1 0 %i' % self.image_len); # Start some background requests - for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset) - for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Start the block job
@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase): self.assert_qmp(result, 'return', {}) # Start some more requests - for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Wait for the READY event
@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase): # the source) should be settled using the active mechanism. # The mirror code itself asserts that the source BDS's dirty # bitmap will stay clean between READY and COMPLETED. - for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) if sync_source_and_target:
tests/qemu-iotests/163 +1 -1
@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): entry_bits = 3 entry_size = 1 << entry_bits l1_mask = 0x00fffffffffffe00 - div_roundup = lambda n, d: (n + d - 1) / d + div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): for x in xrange(0, len(data), n):
tests/qemu-iotests/iotests.py +1 -1
@@ -199,7 +199,7 @@ def create_image(name, size): file = open(name, 'wb') i = 0 while i < size: - sector = struct.pack('>l504xl', i / 512, i / 512) + sector = struct.pack('>l504xl', i // 512, i // 512) file.write(sector) i = i + 512 file.close()
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division
Quoted 5 lines · Max Reitz
In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.
Signed-off-by Max Reitz <[email protected]>
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division
In reply to Max ReitzQuoted 170 lines · Max Reitz
In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/040 | 4 ++-- tests/qemu-iotests/044 | 2 +- tests/qemu-iotests/093 | 18 +++++++++--------- tests/qemu-iotests/149 | 6 +++--- tests/qemu-iotests/151 | 12 ++++++------ tests/qemu-iotests/163 | 2 +- tests/qemu-iotests/iotests.py | 2 +- 7 files changed, 23 insertions(+), 23 deletions(-)tests/qemu-iotests/040 +2
-2@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('device_del', id='scsi0') self.assert_qmp(result, 'return', {})@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('query-block')tests/qemu-iotests/044 +1
-1@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining / 512)) + for j in xrange(0, remaining // 512)) fd.write(table)tests/qemu-iotests/093 +9
-9@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase): # in. The throttled requests won't be executed until we # advance the virtual clock. rq_size = 512 - rd_nr = max(params['bps'] / rq_size / 2, - params['bps_rd'] / rq_size, - params['iops'] / 2, + rd_nr = max(params['bps'] // rq_size // 2, + params['bps_rd'] // rq_size, + params['iops'] // 2, params['iops_rd']) rd_nr *= seconds * 2 - rd_nr /= ndrives - wr_nr = max(params['bps'] / rq_size / 2, - params['bps_wr'] / rq_size, - params['iops'] / 2, + rd_nr //= ndrives + wr_nr = max(params['bps'] // rq_size // 2, + params['bps_wr'] // rq_size, + params['iops'] // 2, params['iops_wr']) wr_nr *= seconds * 2 - wr_nr /= ndrives + wr_nr //= ndrives # Send I/O requests to all drives for i in range(rd_nr):@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase): self.configure_throttle(ndrives, settings) # Wait for the bucket to empty so we can do bursts - wait_ns = nsec_per_sec * burst_length * burst_rate / rate + wait_ns = nsec_per_sec * burst_length * burst_rate // rate self.vm.qtest("clock_step %d" % wait_ns) # Test I/O at the max burst ratetests/qemu-iotests/149 +3
-3@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False): image_size = 4 * oneTB if qemu_img: iotests.log("# Create image") - qemu_img_create(config, image_size / oneMB) + qemu_img_create(config, image_size // oneMB) else: iotests.log("# Create image") - create_image(config, image_size / oneMB) + create_image(config, image_size // oneMB) lowOffsetMB = 100 - highOffsetMB = 3 * oneTB / oneMB + highOffsetMB = 3 * oneTB // oneMB try: if not qemu_img:tests/qemu-iotests/151 +6
-6@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase): 'write -P 1 0 %i' % self.image_len); # Start some background requests - for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset) - for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Start the block job@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase): self.assert_qmp(result, 'return', {}) # Start some more requests - for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Wait for the READY event@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase): # the source) should be settled using the active mechanism. # The mirror code itself asserts that the source BDS's dirty # bitmap will stay clean between READY and COMPLETED. - for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) if sync_source_and_target:tests/qemu-iotests/163 +1
-1@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): entry_bits = 3 entry_size = 1 << entry_bits l1_mask = 0x00fffffffffffe00 - div_roundup = lambda n, d: (n + d - 1) / d + div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): for x in xrange(0, len(data), n):tests/qemu-iotests/iotests.py +0
-0
tests/qemu-iotests/iotests.py def create_image(name, size): · line 205
Quoted 8 lines
tests/qemu-iotests/iotests.py +1
-1@@ -199,7 +199,7 @@ def create_image(name, size): file = open(name, 'wb') i = 0 while i < size: - sector = struct.pack('>l504xl', i / 512, i / 512) + sector = struct.pack('>l504xl', i // 512, i // 512) file.write(sector) i = i + 512 file.close()
Found a few other occurrences:
030:''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))
- Cleber.
Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division
Quoted 184 lines · Cleber Rosa
Quoted 170 lines · Max Reitz
In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/040 | 4 ++-- tests/qemu-iotests/044 | 2 +- tests/qemu-iotests/093 | 18 +++++++++--------- tests/qemu-iotests/149 | 6 +++--- tests/qemu-iotests/151 | 12 ++++++------ tests/qemu-iotests/163 | 2 +- tests/qemu-iotests/iotests.py | 2 +- 7 files changed, 23 insertions(+), 23 deletions(-)tests/qemu-iotests/040 +2
-2@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('device_del', id='scsi0') self.assert_qmp(result, 'return', {})@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('query-block')tests/qemu-iotests/044 +1
-1@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining / 512)) + for j in xrange(0, remaining // 512)) fd.write(table)tests/qemu-iotests/093 +9
-9@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase): # in. The throttled requests won't be executed until we # advance the virtual clock. rq_size = 512 - rd_nr = max(params['bps'] / rq_size / 2, - params['bps_rd'] / rq_size, - params['iops'] / 2, + rd_nr = max(params['bps'] // rq_size // 2, + params['bps_rd'] // rq_size, + params['iops'] // 2, params['iops_rd']) rd_nr *= seconds * 2 - rd_nr /= ndrives - wr_nr = max(params['bps'] / rq_size / 2, - params['bps_wr'] / rq_size, - params['iops'] / 2, + rd_nr //= ndrives + wr_nr = max(params['bps'] // rq_size // 2, + params['bps_wr'] // rq_size, + params['iops'] // 2, params['iops_wr']) wr_nr *= seconds * 2 - wr_nr /= ndrives + wr_nr //= ndrives # Send I/O requests to all drives for i in range(rd_nr):@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase): self.configure_throttle(ndrives, settings) # Wait for the bucket to empty so we can do bursts - wait_ns = nsec_per_sec * burst_length * burst_rate / rate + wait_ns = nsec_per_sec * burst_length * burst_rate // rate self.vm.qtest("clock_step %d" % wait_ns) # Test I/O at the max burst ratetests/qemu-iotests/149 +3
-3@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False): image_size = 4 * oneTB if qemu_img: iotests.log("# Create image") - qemu_img_create(config, image_size / oneMB) + qemu_img_create(config, image_size // oneMB) else: iotests.log("# Create image") - create_image(config, image_size / oneMB) + create_image(config, image_size // oneMB) lowOffsetMB = 100 - highOffsetMB = 3 * oneTB / oneMB + highOffsetMB = 3 * oneTB // oneMB try: if not qemu_img:tests/qemu-iotests/151 +6
-6@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase): 'write -P 1 0 %i' % self.image_len); # Start some background requests - for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset) - for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Start the block job@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase): self.assert_qmp(result, 'return', {}) # Start some more requests - for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Wait for the READY event@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase): # the source) should be settled using the active mechanism. # The mirror code itself asserts that the source BDS's dirty # bitmap will stay clean between READY and COMPLETED. - for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) if sync_source_and_target:tests/qemu-iotests/163 +1
-1@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): entry_bits = 3 entry_size = 1 << entry_bits l1_mask = 0x00fffffffffffe00 - div_roundup = lambda n, d: (n + d - 1) / d + div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): for x in xrange(0, len(data), n):tests/qemu-iotests/iotests.py +0
-0tests/qemu-iotests/iotests.py def create_image(name, size): · line 205
Quoted 8 lines
tests/qemu-iotests/iotests.py +1
-1@@ -199,7 +199,7 @@ def create_image(name, size): file = open(name, 'wb') i = 0 while i < size: - sector = struct.pack('>l504xl', i / 512, i / 512) + sector = struct.pack('>l504xl', i // 512, i // 512) file.write(sector) i = i + 512 file.close()Found a few other occurrences:
Quoted 3 lines
030:''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event)) 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))
Oh, thanks. Right, those just worked because they were converted back to integers thanks to the %d in the format string.
Will fix. Thanks for reviewing!
Max
PATCH 5/9 iotests: Different iterator behavior in Python 3
In reply to Max ReitzIn Python 3, several functions now return iterators instead of lists. This includes range(), items(), map(), and filter(). This means that if we really want a list, we have to wrap those instances with list(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes used xrange() and iteritems() which no longer exist in Python 3. Just change these calls to be range() and items(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).
In one instance, we only wanted the first instance of the result of a filter() call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for with next() wrapped around, which works both in 2.7 and 3.
--- tests/qemu-iotests/044 | 12 ++++++------ tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 ++-- tests/qemu-iotests/124 | 4 ++-- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/163 | 6 +++--- 6 files changed, 15 insertions(+), 15 deletions(-)
tests/qemu-iotests/044 +6 -6
@@ -52,23 +52,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): # Write a refcount table fd.seek(off_reftable) - for i in xrange(0, h.refcount_table_clusters): + for i in range(0, h.refcount_table_clusters): sector = b''.join(struct.pack('>Q', off_refblock + i * 64 * 512 + j * 512) - for j in xrange(0, 64)) + for j in range(0, 64)) fd.write(sector) # Write the refcount blocks assert(fd.tell() == off_refblock) sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256)) - for block in xrange(0, h.refcount_table_clusters): + for block in range(0, h.refcount_table_clusters): fd.write(sector) # Write the L1 table assert(fd.tell() == off_l1) assert(off_l2 + 512 * h.l1_size == off_data) table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) - for j in xrange(0, h.l1_size)) + for j in range(0, h.l1_size)) fd.write(table) # Write the L2 tables
@@ -79,14 +79,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off_data while remaining > 1024 * 512: pytable = list((1 << 63) | off + 512 * j - for j in xrange(0, 1024)) + for j in range(0, 1024)) table = struct.pack('>1024Q', *pytable) fd.write(table) remaining = remaining - 1024 * 512 off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining // 512)) + for j in range(0, remaining // 512)) fd.write(table)
tests/qemu-iotests/056 +1 -1
@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img') def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs): fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt)) optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [fullname, size] iotests.qemu_img(*args)
tests/qemu-iotests/065 +2 -2
@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific): :data.index('')] for field in data: self.assertTrue(re.match('^ {4}[^ ]', field) is not None) - data = map(lambda line: line.strip(), data) + data = list(map(lambda line: line.strip(), data)) self.assertEqual(data, self.human_compare) class TestQMP(TestImageInfoSpecific):
@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific): def test_qmp(self): result = self.vm.qmp('query-block')['return'] - drive = filter(lambda drive: drive['device'] == 'drive0', result)[0] + drive = next(drive for drive in result if drive['device'] == 'drive0') data = drive['inserted']['image']['format-specific'] self.assertEqual(data['type'], iotests.imgfmt) self.assertEqual(data['data'], self.compare)
tests/qemu-iotests/124 +2 -2
@@ -39,7 +39,7 @@ def try_remove(img): def transaction_action(action, **kwargs): return { 'type': action, - 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems()) + 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items()) }
@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): def img_create(self, img, fmt=iotests.imgfmt, size='64M', parent=None, parentFormat=None, **kwargs): optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [img, size] if parent:
tests/qemu-iotests/139 +1 -1
@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase): # Check whether a BlockDriverState exists def checkBlockDriverState(self, node, must_exist = True): result = self.vm.qmp('query-named-block-nodes') - nodes = filter(lambda x: x['node-name'] == node, result['return']) + nodes = list(filter(lambda x: x['node-name'] == node, result['return'])) self.assertLessEqual(len(nodes), 1) self.assertEqual(must_exist, len(nodes) == 1)
tests/qemu-iotests/163 +3 -3
@@ -41,7 +41,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): - for x in xrange(0, len(data), n): + for x in range(0, len(data), n): yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask def check_l1_table(h, l1_data):
@@ -135,8 +135,8 @@ class ShrinkBaseClass(iotests.QMPTestCase): self.image_verify() def test_random_write(self): - offs_list = range(0, size_to_int(self.image_len), - size_to_int(self.chunk_size)) + offs_list = list(range(0, size_to_int(self.image_len), + size_to_int(self.chunk_size))) random.shuffle(offs_list) for offs in offs_list: qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3
Quoted 9 lines · Max Reitz
In Python 3, several functions now return iterators instead of lists. This includes
range(),items(),map(), andfilter(). This means that if we really want a list, we have to wrap those instances withlist(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes usedxrange()anditeritems()which no longer exist in Python 3. Just change these calls to berange()anditems(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).
Quoted 7 lines
In one instance, we only wanted the first instance of the result of a
filter()call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for withnext()wrapped around, which works both in 2.7 and 3.Signed-off-by Max Reitz <[email protected]>
Minor comments below:
Quoted 76 lines
--- tests/qemu-iotests/044 | 12 ++++++------ tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 ++-- tests/qemu-iotests/124 | 4 ++-- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/163 | 6 +++--- 6 files changed, 15 insertions(+), 15 deletions(-)tests/qemu-iotests/044 +6
-6@@ -52,23 +52,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): # Write a refcount table fd.seek(off_reftable) - for i in xrange(0, h.refcount_table_clusters): + for i in range(0, h.refcount_table_clusters): sector = b''.join(struct.pack('>Q', off_refblock + i * 64 * 512 + j * 512) - for j in xrange(0, 64)) + for j in range(0, 64)) fd.write(sector) # Write the refcount blocks assert(fd.tell() == off_refblock) sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256)) - for block in xrange(0, h.refcount_table_clusters): + for block in range(0, h.refcount_table_clusters): fd.write(sector) # Write the L1 table assert(fd.tell() == off_l1) assert(off_l2 + 512 * h.l1_size == off_data) table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) - for j in xrange(0, h.l1_size)) + for j in range(0, h.l1_size)) fd.write(table) # Write the L2 tables@@ -79,14 +79,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off_data while remaining > 1024 * 512: pytable = list((1 << 63) | off + 512 * j - for j in xrange(0, 1024)) + for j in range(0, 1024)) table = struct.pack('>1024Q', *pytable) fd.write(table) remaining = remaining - 1024 * 512 off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining // 512)) + for j in range(0, remaining // 512)) fd.write(table)tests/qemu-iotests/056 +1
-1@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img') def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs): fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt)) optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [fullname, size] iotests.qemu_img(*args)tests/qemu-iotests/065 +0
-0
tests/qemu-iotests/065 class TestQemuImgInfo(TestImageInfoSpecific): · line 62
Quoted 5 lines
tests/qemu-iotests/065 +1
-1@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific): :data.index('')] for field in data: self.assertTrue(re.match('^ {4}[^ ]', field) is not None) - data = map(lambda line: line.strip(), data) + data = list(map(lambda line: line.strip(), data))
I would find this more readable:
data = [line.strip() for line in data]
Not a blocker, though.
tests/qemu-iotests/065 class TestQMP(TestImageInfoSpecific): · line 83
Quoted 9 lines
tests/qemu-iotests/065 +1
-1self.assertEqual(data, self.human_compare) class TestQMP(TestImageInfoSpecific):@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific): def test_qmp(self): result = self.vm.qmp('query-block')['return'] - drive = filter(lambda drive: drive['device'] == 'drive0', result)[0] + drive = next(drive for drive in result if drive['device'] == 'drive0')
I didn't understand what you meant by "single-line for" on the commit message, until I saw the generator expression here. :)
This will raise an exception if there's no drive0 in the list, but that was already true on the original code.
Quoted 30 lines
tests/qemu-iotests/065 +0
-0data = drive['inserted']['image']['format-specific'] self.assertEqual(data['type'], iotests.imgfmt) self.assertEqual(data['data'], self.compare)tests/qemu-iotests/124 +2
-2@@ -39,7 +39,7 @@ def try_remove(img): def transaction_action(action, **kwargs): return { 'type': action, - 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems()) + 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items()) }@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): def img_create(self, img, fmt=iotests.imgfmt, size='64M', parent=None, parentFormat=None, **kwargs): optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [img, size] if parent:tests/qemu-iotests/139 +0
-0
tests/qemu-iotests/139 class TestBlockdevDel(iotests.QMPTestCase): · line 54
Quoted 5 lines
tests/qemu-iotests/139 +1
-1@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase): # Check whether a BlockDriverState exists def checkBlockDriverState(self, node, must_exist = True): result = self.vm.qmp('query-named-block-nodes') - nodes = filter(lambda x: x['node-name'] == node, result['return']) + nodes = list(filter(lambda x: x['node-name'] == node, result['return']))
I'd prefer a list expression:
nodes = [x for x in result['return'] if x['node-name'] == node]
Also not a blocker.
Quoted 30 lines
tests/qemu-iotests/139 +0
-0self.assertLessEqual(len(nodes), 1) self.assertEqual(must_exist, len(nodes) == 1)tests/qemu-iotests/163 +3
-3@@ -41,7 +41,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): - for x in xrange(0, len(data), n): + for x in range(0, len(data), n): yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask def check_l1_table(h, l1_data):@@ -135,8 +135,8 @@ class ShrinkBaseClass(iotests.QMPTestCase): self.image_verify() def test_random_write(self): - offs_list = range(0, size_to_int(self.image_len), - size_to_int(self.chunk_size)) + offs_list = list(range(0, size_to_int(self.image_len), + size_to_int(self.chunk_size))) random.shuffle(offs_list) for offs in offs_list: qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3
Quoted 21 lines · Eduardo Habkost
Quoted 9 lines · Max Reitz
In Python 3, several functions now return iterators instead of lists. This includes
range(),items(),map(), andfilter(). This means that if we really want a list, we have to wrap those instances withlist(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes usedxrange()anditeritems()which no longer exist in Python 3. Just change these calls to berange()anditems(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).Quoted 7 lines
In one instance, we only wanted the first instance of the result of a
filter()call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for withnext()wrapped around, which works both in 2.7 and 3.Signed-off-by Max Reitz <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Minor comments below:
[...]
Quoted 13 lines
tests/qemu-iotests/065 class TestQemuImgInfo(TestImageInfoSpecific): · line 62
Quoted 10 lines
tests/qemu-iotests/065 +1
-1@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific): :data.index('')] for field in data: self.assertTrue(re.match('^ {4}[^ ]', field) is not None) - data = map(lambda line: line.strip(), data) + data = list(map(lambda line: line.strip(), data))I would find this more readable:
Quoted 3 lines
data = [
line.strip()for line in data]Not a blocker, though.
Yes, I agree, that looks better.
Quoted 5 lines
Quoted 5 lines
tests/qemu-iotests/065 +0
-0self.assertEqual(data, self.human_compare) class TestQMP(TestImageInfoSpecific):@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
Quoted 7 lines
tests/qemu-iotests/065 class TestQMP(TestImageInfoSpecific): · line 83
Quoted 4 lines
tests/qemu-iotests/065 +1
-1def test_qmp(self): result = self.vm.qmp('query-block')['return'] - drive = filter(lambda drive: drive['device'] == 'drive0', result)[0] + drive = next(drive for drive in result if drive['device'] == 'drive0')I didn't understand what you meant by "single-line for" on the commit message, until I saw the generator expression here. :)
Maybe I should at least put quotes around the "for", because the combination of "for with" really makes it difficult to read...
Quoted 42 lines
This will raise an exception if there's no drive0 in the list, but that was already true on the original code.
Quoted 30 lines
tests/qemu-iotests/065 +0
-0data = drive['inserted']['image']['format-specific'] self.assertEqual(data['type'], iotests.imgfmt) self.assertEqual(data['data'], self.compare)tests/qemu-iotests/124 +2
-2@@ -39,7 +39,7 @@ def try_remove(img): def transaction_action(action, **kwargs): return { 'type': action, - 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems()) + 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items()) }@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): def img_create(self, img, fmt=iotests.imgfmt, size='64M', parent=None, parentFormat=None, **kwargs): optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [img, size] if parent:tests/qemu-iotests/139 +0
-0tests/qemu-iotests/139 class TestBlockdevDel(iotests.QMPTestCase): · line 54
Quoted 5 lines
tests/qemu-iotests/139 +1
-1@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase): # Check whether a BlockDriverState exists def checkBlockDriverState(self, node, must_exist = True): result = self.vm.qmp('query-named-block-nodes') - nodes = filter(lambda x: x['node-name'] == node, result['return']) + nodes = list(filter(lambda x: x['node-name'] == node, result['return']))I'd prefer a list expression:
Quoted 3 lines
nodes = [x for x in result['return'] if x['node-name'] == node]Also not a blocker.
I don't mind either way, so why not.
Max
Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3
In reply to Max ReitzQuoted 16 lines · Max Reitz
In Python 3, several functions now return iterators instead of lists. This includes
range(),items(),map(), andfilter(). This means that if we really want a list, we have to wrap those instances withlist(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes usedxrange()anditeritems()which no longer exist in Python 3. Just change these calls to berange()anditems(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).In one instance, we only wanted the first instance of the result of a
filter()call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for withnext()wrapped around, which works both in 2.7 and 3.Signed-off-by Max Reitz <[email protected]>
Quoted 9 lines
--- tests/qemu-iotests/044 | 12 ++++++------ tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 ++-- tests/qemu-iotests/124 | 4 ++-- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/163 | 6 +++--- 6 files changed, 15 insertions(+), 15 deletions(-)
You have 2 files here which use xrange (which is a manageable size, and whose occurrences involve a moderate size of items) to also consider:
if sys.version_info.major == 2:
range = xrange
Defaulting to the Python 3 names, but behaving the same across Python 2 and 3.
To do the same for dict.iteritems() => dict.items() requires a lot more code, so I'd stay away from it for now. Also, it looks like most of those dicts are small in size (**kwargs and the like).
Other than that suggestion,
Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3
Quoted 29 lines · Cleber Rosa
Quoted 16 lines · Max Reitz
In Python 3, several functions now return iterators instead of lists. This includes
range(),items(),map(), andfilter(). This means that if we really want a list, we have to wrap those instances withlist(). On the other hand, sometimes we do just want an iterator, in which case we have sometimes usedxrange()anditeritems()which no longer exist in Python 3. Just change these calls to berange()anditems(), which costs a bit of performance in Python 2, but will do the right thing in Python 3 (which is what is important).In one instance, we only wanted the first instance of the result of a
filter()call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to a single-line for withnext()wrapped around, which works both in 2.7 and 3.Signed-off-by Max Reitz <[email protected]>Quoted 9 lines
--- tests/qemu-iotests/044 | 12 ++++++------ tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 ++-- tests/qemu-iotests/124 | 4 ++-- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/163 | 6 +++--- 6 files changed, 15 insertions(+), 15 deletions(-)
Quoted 5 lines
You have 2 files here which use xrange (which is a manageable size, and whose occurrences involve a moderate size of items) to also consider:
if sys.version_info.major == 2:
range = xrange
I don't think it's necessary, but it's so easy to grep for "sys.version_info" once we want to completely switch to Python 3 that I can't find good arguments against it. O:-)
Max
Quoted 11 lines
Defaulting to the Python 3 names, but behaving the same across Python 2 and 3.
To do the same for
dict.iteritems()=>dict.items()requires a lot more code, so I'd stay away from it for now. Also, it looks like most of those dicts are small in size (**kwargs and the like).Other than that suggestion,
Reviewed-by Cleber Rosa <[email protected]>
PATCH 6/9 iotests: Explicitly inherit FDs in Python
In reply to Max ReitzPython 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if close_fds is explicitly set to False.
Adhere to this by setting close_fds to False when working with subprocesses that may want to inherit FDs, and by trying to set_inheritable() on FDs that we do want to bequeath to them.
--- scripts/qemu.py | 13 +++++++++++-- scripts/qmp/qmp.py | 7 +++++++ tests/qemu-iotests/147 | 7 +++++++ 3 files changed, 25 insertions(+), 2 deletions(-)
scripts/qemu.py +11 -2
@@ -142,10 +142,18 @@ class QEMUMachine(object): if opts: options.append(opts) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(fd, True) + except AttributeError: + pass + self._args.append('-add-fd') self._args.append(','.join(options)) return self + # The caller needs to make sure the FD is inheritable def send_fd_scm(self, fd_file_path): # In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()
@@ -159,7 +167,7 @@ class QEMUMachine(object): "%s" % fd_file_path] devnull = open(os.path.devnull, 'rb') proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, close_fds=False) output = proc.communicate()[0] if output: LOG.debug(output)
@@ -280,7 +288,8 @@ class QEMUMachine(object): stdin=devnull, stdout=self._qemu_log_file, stderr=subprocess.STDOUT, - shell=False) + shell=False, + close_fds=False) self._post_launch() def wait(self):
scripts/qmp/qmp.py +7 -0
@@ -10,6 +10,7 @@ import json import errno +import os import socket import logging
@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object): return self.__sock.fileno() def is_scm_available(self): + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(self.get_sock_fd(), True) + except AttributeError: + pass return self.__sock.family == socket.AF_UNIX
tests/qemu-iotests/147 +7 -0
@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase): sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sockfd.connect(unix_socket) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(sockfd.fileno(), True) + except AttributeError: + pass + result = self.vm.send_fd_scm(str(sockfd.fileno())) self.assertEqual(result, 0, 'Failed to send socket FD')
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python
Quoted 21 lines · Max Reitz
Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if
close_fdsis explicitly set to False.Adhere to this by setting
close_fdsto False when working with subprocesses that may want to inherit FDs, and by trying toset_inheritable()on FDs that we do want to bequeath to them.Signed-off-by Max Reitz <[email protected]>--- scripts/qemu.py | 13 +++++++++++-- scripts/qmp/qmp.py | 7 +++++++ tests/qemu-iotests/147 | 7 +++++++ 3 files changed, 25 insertions(+), 2 deletions(-)scripts/qemu.py +0
-0
scripts/qemu.py class QEMUMachine(object): · line 151
Quoted 10 lines
scripts/qemu.py +7
-0@@ -142,10 +142,18 @@ class QEMUMachine(object): if opts: options.append(opts) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(fd, True) + except AttributeError: + pass +
This is add_fd(), so calling set_inheritable() automatically here makes sense.
Quoted 40 lines
scripts/qemu.py +4
-2self._args.append('-add-fd') self._args.append(','.join(options)) return self + # The caller needs to make sure the FD is inheritable def send_fd_scm(self, fd_file_path): # In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()@@ -159,7 +167,7 @@ class QEMUMachine(object): "%s" % fd_file_path] devnull = open(os.path.devnull, 'rb') proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, close_fds=False) output = proc.communicate()[0] if output: LOG.debug(output)@@ -280,7 +288,8 @@ class QEMUMachine(object): stdin=devnull, stdout=self._qemu_log_file, stderr=subprocess.STDOUT, - shell=False) + shell=False, + close_fds=False) self._post_launch() def wait(self):scripts/qmp/qmp.py +1
-0@@ -10,6 +10,7 @@ import json import errno +import os import socket import logging
scripts/qmp/qmp.py class QEMUMonitorProtocol(object): · line 262
Quoted 9 lines
scripts/qmp/qmp.py +6
-0@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object): return self.__sock.fileno() def is_scm_available(self): + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(self.get_sock_fd(), True) + except AttributeError: + pass
Why did you decide to place this code inside is_scm_available()?
For reference, this is the only caller of is_scm_available():
def send_fd_scm(self, fd_file_path):
# In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()
...
In addition to making a method called is_*() have an unexpected side-effect, the method won't be called at all if running with debugging disabled.
I suggest simply placing the os.set_inheritable() call inside send_fd_scm(), as close as possible to the subprocess.Popen() call.
Quoted 6 lines
scripts/qmp/qmp.py +0
-0return self.__sock.family == socket.AF_UNIXtests/qemu-iotests/147 +0
-0
tests/qemu-iotests/147 class BuiltinNBD(NBDBlockdevAddBase): · line 238
Quoted 10 lines
tests/qemu-iotests/147 +7
-0@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase): sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sockfd.connect(unix_socket) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(sockfd.fileno(), True) + except AttributeError: + pass +
Why not make send_fd_scm() responsible for calling os.set_inheritable(), making this hunk unnecessary?
Quoted 6 lines
tests/qemu-iotests/147 +0
-0result = self.vm.send_fd_scm(str(sockfd.fileno())) self.assertEqual(result, 0, 'Failed to send socket FD')Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python
Quoted 90 lines · Eduardo Habkost
Quoted 21 lines · Max Reitz
Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if
close_fdsis explicitly set to False.Adhere to this by setting
close_fdsto False when working with subprocesses that may want to inherit FDs, and by trying toset_inheritable()on FDs that we do want to bequeath to them.Signed-off-by Max Reitz <[email protected]>--- scripts/qemu.py | 13 +++++++++++-- scripts/qmp/qmp.py | 7 +++++++ tests/qemu-iotests/147 | 7 +++++++ 3 files changed, 25 insertions(+), 2 deletions(-)scripts/qemu.py +0
-0scripts/qemu.py class QEMUMachine(object): · line 151
Quoted 10 lines
scripts/qemu.py +7
-0@@ -142,10 +142,18 @@ class QEMUMachine(object): if opts: options.append(opts) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(fd, True) + except AttributeError: + pass +This is
add_fd(), so callingset_inheritable()automatically here makes sense.Quoted 40 lines
scripts/qemu.py +4
-2self._args.append('-add-fd') self._args.append(','.join(options)) return self + # The caller needs to make sure the FD is inheritable def send_fd_scm(self, fd_file_path): # In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()@@ -159,7 +167,7 @@ class QEMUMachine(object): "%s" % fd_file_path] devnull = open(os.path.devnull, 'rb') proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, close_fds=False) output = proc.communicate()[0] if output: LOG.debug(output)@@ -280,7 +288,8 @@ class QEMUMachine(object): stdin=devnull, stdout=self._qemu_log_file, stderr=subprocess.STDOUT, - shell=False) + shell=False, + close_fds=False) self._post_launch() def wait(self):scripts/qmp/qmp.py +1
-0@@ -10,6 +10,7 @@ import json import errno +import os import socket import loggingscripts/qmp/qmp.py class QEMUMonitorProtocol(object): · line 262
Quoted 9 lines
scripts/qmp/qmp.py +6
-0@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object): return self.__sock.fileno() def is_scm_available(self): + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(self.get_sock_fd(), True) + except AttributeError: + passWhy did you decide to place this code inside
is_scm_available()?For reference, this is the only caller of
is_scm_available():
Quoted 7 lines
def
send_fd_scm(self, fd_file_path):
# Iniotest.py, the qmp should always use unix socket. assertself._qmp.is_scm_available()
...In addition to making a method called is_*() have an unexpected side-effect,
True. My idea was that a function that asks for SCM to be available might as well make it available.
Quoted 2 lines
the method won't be called at all if running with
debugging disabled.
Well, I sure hope we don't disable debugging in the iotests. We use assert quite a number of times there.
On the other hand, someone might want to use this outside of the iotests but I don't even know whether that works, considering the SCM helper program is part of the iotests.
Quoted 3 lines
I suggest simply placing the
os.set_inheritable()call insidesend_fd_scm(), as close as possible to thesubprocess.Popen()call.
Yes, I think you're right. I didn't want to put it into send_fd_scm(), because that method is only there to send some FD over QMP/SCM; it isn't really supposed to send the QMP socket FD somewhere. But sending something over QMP/SCM is different from bequeathing something to a child process (the socket_scm_helper). And since send_fd_scm() needs to bequeath the QMP socket FD to that helper, it should be responsible for making it inheritable.
And maybe even more importantly, whether the socket allows for SCM really has nothing to do with whether it's inheritable. So it's actually just wrong to put it here.
Quoted 9 lines
Quoted 9 lines
scripts/qmp/qmp.py +0
-0return self.__sock.family == socket.AF_UNIXtests/qemu-iotests/147 +0
-0@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase): sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sockfd.connect(unix_socket)
Quoted 10 lines
tests/qemu-iotests/147 class BuiltinNBD(NBDBlockdevAddBase): · line 238
Quoted 7 lines
tests/qemu-iotests/147 +7
-0+ # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(sockfd.fileno(), True) + except AttributeError: + pass +Why not make
send_fd_scm()responsible for callingos.set_inheritable(), making this hunk unnecessary?
My idea was: Because send_fd_scm() takes a string and not an integer. The socket_scm_helper takes either an FD or a path, so send_fd_scm() accepts either.
But now I realize that if we pass a path, we don't need to make the FD inheritable. So send_fd_scm() can check whether it's supposed to pass an FD, and if so, make it inheritable, yes.
Max
Quoted 7 lines
Quoted 6 lines
tests/qemu-iotests/147 +0
-0result = self.vm.send_fd_scm(str(sockfd.fileno())) self.assertEqual(result, 0, 'Failed to send socket FD')Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python
In reply to Max ReitzQuoted 5 lines · Max Reitz
Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if
close_fdsis explicitly set to False.
It's actually a change that was introduced in 3.4:
https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors
Quoted 16 lines
Adhere to this by setting
close_fdsto False when working with subprocesses that may want to inherit FDs, and by trying toset_inheritable()on FDs that we do want to bequeath to them.Signed-off-by Max Reitz <[email protected]>--- scripts/qemu.py | 13 +++++++++++-- scripts/qmp/qmp.py | 7 +++++++ tests/qemu-iotests/147 | 7 +++++++ 3 files changed, 25 insertions(+), 2 deletions(-)scripts/qemu.py +0
-0
scripts/qemu.py class QEMUMachine(object): · line 147
Quoted 6 lines
scripts/qemu.py +3
-0@@ -142,10 +142,18 @@ class QEMUMachine(object): if opts: options.append(opts) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try:
Version should be 3.4 here as well.
scripts/qemu.py class QEMUMachine(object): · line 151
Quoted 4 lines
scripts/qemu.py +4
-0+ os.set_inheritable(fd, True) + except AttributeError: + pass +
Doing hasattr(os, "set_inheritable") is cheaper.
- Cleber.
Quoted 69 lines
scripts/qemu.py +4
-2self._args.append('-add-fd') self._args.append(','.join(options)) return self + # The caller needs to make sure the FD is inheritable def send_fd_scm(self, fd_file_path): # In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()@@ -159,7 +167,7 @@ class QEMUMachine(object): "%s" % fd_file_path] devnull = open(os.path.devnull, 'rb') proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, close_fds=False) output = proc.communicate()[0] if output: LOG.debug(output)@@ -280,7 +288,8 @@ class QEMUMachine(object): stdin=devnull, stdout=self._qemu_log_file, stderr=subprocess.STDOUT, - shell=False) + shell=False, + close_fds=False) self._post_launch() def wait(self):scripts/qmp/qmp.py +7
-0@@ -10,6 +10,7 @@ import json import errno +import os import socket import logging@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object): return self.__sock.fileno() def is_scm_available(self): + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(self.get_sock_fd(), True) + except AttributeError: + pass return self.__sock.family == socket.AF_UNIXtests/qemu-iotests/147 +7
-0@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase): sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sockfd.connect(unix_socket) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(sockfd.fileno(), True) + except AttributeError: + pass + result = self.vm.send_fd_scm(str(sockfd.fileno())) self.assertEqual(result, 0, 'Failed to send socket FD')
Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python
Quoted 9 lines · Cleber Rosa
Quoted 5 lines · Max Reitz
Python 3.2 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if
close_fdsis explicitly set to False.
Quoted 3 lines
It's actually a change that was introduced in 3.4:
https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors
Oops, don't know how I got that wrong.
Quoted 19 lines
Quoted 19 lines
Adhere to this by setting
close_fdsto False when working with subprocesses that may want to inherit FDs, and by trying toset_inheritable()on FDs that we do want to bequeath to them.Signed-off-by Max Reitz <[email protected]>--- scripts/qemu.py | 13 +++++++++++-- scripts/qmp/qmp.py | 7 +++++++ tests/qemu-iotests/147 | 7 +++++++ 3 files changed, 25 insertions(+), 2 deletions(-)scripts/qemu.py +0
-0@@ -142,10 +142,18 @@ class QEMUMachine(object): if opts: options.append(opts)
Quoted 5 lines
scripts/qemu.py class QEMUMachine(object): · line 147
Quoted 3 lines
scripts/qemu.py +3
-0+ # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try:Version should be 3.4 here as well.
Yes, will fix everywhere.
Quoted 8 lines
scripts/qemu.py class QEMUMachine(object): · line 151
Quoted 4 lines
scripts/qemu.py +4
-0+ os.set_inheritable(fd, True) + except AttributeError: + pass +Doing
hasattr(os, "set_inheritable")is cheaper.- Cleber.
Nice, I'll use that then.
Max
Quoted 70 lines
Quoted 69 lines
scripts/qemu.py +4
-2self._args.append('-add-fd') self._args.append(','.join(options)) return self + # The caller needs to make sure the FD is inheritable def send_fd_scm(self, fd_file_path): # In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available()@@ -159,7 +167,7 @@ class QEMUMachine(object): "%s" % fd_file_path] devnull = open(os.path.devnull, 'rb') proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, close_fds=False) output = proc.communicate()[0] if output: LOG.debug(output)@@ -280,7 +288,8 @@ class QEMUMachine(object): stdin=devnull, stdout=self._qemu_log_file, stderr=subprocess.STDOUT, - shell=False) + shell=False, + close_fds=False) self._post_launch() def wait(self):scripts/qmp/qmp.py +7
-0@@ -10,6 +10,7 @@ import json import errno +import os import socket import logging@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object): return self.__sock.fileno() def is_scm_available(self): + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(self.get_sock_fd(), True) + except AttributeError: + pass return self.__sock.family == socket.AF_UNIXtests/qemu-iotests/147 +7
-0@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase): sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sockfd.connect(unix_socket) + # This did not exist before 3.2, but since then it is + # mandatory for our purpose + try: + os.set_inheritable(sockfd.fileno(), True) + except AttributeError: + pass + result = self.vm.send_fd_scm(str(sockfd.fileno())) self.assertEqual(result, 0, 'Failed to send socket FD')
PATCH 7/9 iotests: 'new' module replacement in 169
In reply to Max Reitziotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
--- tests/qemu-iotests/169 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +1 -2
@@ -23,7 +23,6 @@ import iotests import time import itertools import operator -import new from iotests import qemu_img
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): def inject_test_case(klass, name, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass)) + setattr(klass, 'test_' + name, lambda self: mc(self)) for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_'
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169
Quoted 22 lines · Max Reitz
iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/169 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)tests/qemu-iotests/169 +0
-1@@ -23,7 +23,6 @@ import iotests import time import itertools import operator -import new from iotests import qemu_img
tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 146
Quoted 5 lines
tests/qemu-iotests/169 +1
-1@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): def inject_test_case(klass, name, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass)) + setattr(klass, 'test_' + name, lambda self: mc(self))
The "lambda self: mc(self)" expression looks weird and unnecessary at first look, but I have just confirmed that this doesn't work:
setattr(klass, 'test_' + name, mc)
Probably because the methodcaller object won't automatically become a instance method after the TestCase class is instantiated.
Quoted 6 lines
tests/qemu-iotests/169 +0
-0for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_'Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169
In reply to Max ReitzQuoted 23 lines · Max Reitz
iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/169 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)tests/qemu-iotests/169 +0
-1@@ -23,7 +23,6 @@ import iotests import time import itertools import operator -import new from iotests import qemu_img
tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149
Quoted 8 lines
tests/qemu-iotests/169 +1
-1@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): def inject_test_case(klass, name, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass)) + setattr(klass, 'test_' + name, lambda self: mc(self)) for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_'
This can be simplified with:
tests/qemu-iotests/169 +3 -7
@@ -22,7 +22,6 @@ import os import iotests import time import itertools -import operator from iotests import qemu_img
@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.check_bitmap(self.vm_b, sha256 if persistent else False) -def inject_test_case(klass, name, method, *args, **kwargs): - mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, lambda self: mc(self)) - for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_' name += ('_' if cmb[1] else '_not_') + 'migbitmap_' name += '_online' if cmb[2] else '_offline' name += '_shared' if cmb[3] else '_nonshared' - inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration', - *list(cmb)) + setattr(TestDirtyBitmapMigration, 'test_' + name, + lambda self: TestDirtyBitmapMigration.do_test_migration( + self, *list(cmb))) if __name__ == '__main__':
Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169
Quoted 58 lines · Cleber Rosa
Quoted 23 lines · Max Reitz
iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/169 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)tests/qemu-iotests/169 +0
-1@@ -23,7 +23,6 @@ import iotests import time import itertools import operator -import new from iotests import qemu_imgtests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149
Quoted 8 lines
tests/qemu-iotests/169 +1
-1@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): def inject_test_case(klass, name, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass)) + setattr(klass, 'test_' + name, lambda self: mc(self)) for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_'This can be simplified with:
tests/qemu-iotests/169 +0
-5@@ -22,7 +22,6 @@ import os import iotests import time import itertools -import operator from iotests import qemu_img@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.check_bitmap(self.vm_b, sha256 if persistent else False) -def inject_test_case(klass, name, method, *args, **kwargs): - mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, lambda self: mc(self)) - for cmb in list(itertools.product((True, False), repeat=4)):
tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 151
Quoted 10 lines
tests/qemu-iotests/169 +3
-2name = ('_' if cmb[0] else '_not_') + 'persistent_' name += ('_' if cmb[1] else '_not_') + 'migbitmap_' name += '_online' if cmb[2] else '_offline' name += '_shared' if cmb[3] else '_nonshared' - inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration', - *list(cmb)) + setattr(TestDirtyBitmapMigration, 'test_' + name, + lambda self: TestDirtyBitmapMigration.do_test_migration( + self, *list(cmb)))
I'm not fond of the long multi-line lambda expression, but I love that you got rid of operator.methodcaller(). :)
However, this doesn't seem to work: *list(cmb) will be evaluated only when the lambda is called, and cmb will be always (False, False, False, False).
I was going to suggest defining a nested function inside inject_test_case() to replace operator.methodcaller(), but then I thought it was not worth it.
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169
Quoted 73 lines · Eduardo Habkost
Quoted 58 lines · Cleber Rosa
Quoted 23 lines · Max Reitz
iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/169 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)tests/qemu-iotests/169 +0
-1@@ -23,7 +23,6 @@ import iotests import time import itertools import operator -import new from iotests import qemu_imgtests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149
Quoted 8 lines
tests/qemu-iotests/169 +1
-1@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): def inject_test_case(klass, name, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass)) + setattr(klass, 'test_' + name, lambda self: mc(self)) for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_'This can be simplified with:
tests/qemu-iotests/169 +0
-5@@ -22,7 +22,6 @@ import os import iotests import time import itertools -import operator from iotests import qemu_img@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.check_bitmap(self.vm_b, sha256 if persistent else False) -def inject_test_case(klass, name, method, *args, **kwargs): - mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, lambda self: mc(self)) - for cmb in list(itertools.product((True, False), repeat=4)):tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 151
Quoted 10 lines
tests/qemu-iotests/169 +3
-2name = ('_' if cmb[0] else '_not_') + 'persistent_' name += ('_' if cmb[1] else '_not_') + 'migbitmap_' name += '_online' if cmb[2] else '_offline' name += '_shared' if cmb[3] else '_nonshared' - inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration', - *list(cmb)) + setattr(TestDirtyBitmapMigration, 'test_' + name, + lambda self: TestDirtyBitmapMigration.do_test_migration( + self, *list(cmb)))I'm not fond of the long multi-line lambda expression, but I love that you got rid of
operator.methodcaller(). :)
Quoted 8 lines
However, this doesn't seem to work:
*list(cmb)will be evaluated only when the lambda is called, andcmbwill be always(False, False, False, False).I was going to suggest defining a nested function inside
inject_test_case()to replaceoperator.methodcaller(), but then I thought it was not worth it.
You're right! I missed that. Anyway, another possibility:
tests/qemu-iotests/169 +3 -10
@@ -22,7 +22,7 @@ import os import iotests import time import itertools -import operator +import functools from iotests import qemu_img
@@ -140,20 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.vm_b.launch() self.check_bitmap(self.vm_b, sha256 if persistent else False) - -def inject_test_case(klass, name, method, *args, **kwargs): - mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, lambda self: mc(self)) - -for cmb in list(itertools.product((True, False), repeat=4)): +for cmb in itertools.product((True, False), repeat=4): name = ('_' if cmb[0] else '_not_') + 'persistent_' name += ('_' if cmb[1] else '_not_') + 'migbitmap_' name += '_online' if cmb[2] else '_offline' name += '_shared' if cmb[3] else '_nonshared' - inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration', - *list(cmb)) - + test =
functools.partialmethod(TestDirtyBitmapMigration.do_test_migration,
+ *cmb)
+ setattr(TestDirtyBitmapMigration, 'test_' + name, test)
if __name__ == '__main__':
iotests.main(supported_fmts=['qcow2'])
Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169
Quoted 113 lines · Cleber Rosa
Quoted 73 lines · Eduardo Habkost
Quoted 58 lines · Cleber Rosa
Quoted 23 lines · Max Reitz
iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/169 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)tests/qemu-iotests/169 +0
-1@@ -23,7 +23,6 @@ import iotests import time import itertools import operator -import new from iotests import qemu_imgtests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 149
Quoted 8 lines
tests/qemu-iotests/169 +1
-1@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): def inject_test_case(klass, name, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass)) + setattr(klass, 'test_' + name, lambda self: mc(self)) for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_'This can be simplified with:
tests/qemu-iotests/169 +0
-5@@ -22,7 +22,6 @@ import os import iotests import time import itertools -import operator from iotests import qemu_img@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.check_bitmap(self.vm_b, sha256 if persistent else False) -def inject_test_case(klass, name, method, *args, **kwargs): - mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, lambda self: mc(self)) - for cmb in list(itertools.product((True, False), repeat=4)):tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 151
Quoted 10 lines
tests/qemu-iotests/169 +3
-2name = ('_' if cmb[0] else '_not_') + 'persistent_' name += ('_' if cmb[1] else '_not_') + 'migbitmap_' name += '_online' if cmb[2] else '_offline' name += '_shared' if cmb[3] else '_nonshared' - inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration', - *list(cmb)) + setattr(TestDirtyBitmapMigration, 'test_' + name, + lambda self: TestDirtyBitmapMigration.do_test_migration( + self, *list(cmb)))I'm not fond of the long multi-line lambda expression, but I love that you got rid of
operator.methodcaller(). :)Quoted 8 lines
However, this doesn't seem to work:
*list(cmb)will be evaluated only when the lambda is called, andcmbwill be always(False, False, False, False).I was going to suggest defining a nested function inside
inject_test_case()to replaceoperator.methodcaller(), but then I thought it was not worth it.You're right! I missed that. Anyway, another possibility:
tests/qemu-iotests/169 +2
-7@@ -22,7 +22,7 @@ import os import iotests import time import itertools -import operator +import functools from iotests import qemu_img@@ -140,20 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.vm_b.launch() self.check_bitmap(self.vm_b, sha256 if persistent else False) - -def inject_test_case(klass, name, method, *args, **kwargs): - mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, lambda self: mc(self)) - -for cmb in list(itertools.product((True, False), repeat=4)): +for cmb in itertools.product((True, False), repeat=4): name = ('_' if cmb[0] else '_not_') + 'persistent_' name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 152
Quoted 10 lines
tests/qemu-iotests/169 +3
-3name += '_online' if cmb[2] else '_offline' name += '_shared' if cmb[3] else '_nonshared' - inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration', - *list(cmb)) - + test = functools.partialmethod(TestDirtyBitmapMigration.do_test_migration, + *cmb) + setattr(TestDirtyBitmapMigration, 'test_' + name, test)
I'm not sure how that is any simpler, though (apart from the inlining). :-)
I mean, my personal goal is not to touch this beyond what I need to because it's black magic to me anyway, so I'm happy with what works.
Max
Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169
On Fri, Oct 19, 2018 at 11:46:59AM +0200, Max Reitz wrote:
[...]
Quoted 2 lines
I mean, my personal goal is not to touch this beyond what I need to because it's black magic to me anyway, so I'm happy with what works.
I agree with this approach, and your original patch looks good enough to me.
Signature · 1 line
Eduardo
PATCH 8/9 iotests: Modify imports for Python 3
In reply to Max ReitzThere are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +6 -2
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], # We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIO + if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + output = StringIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
tests/qemu-iotests/nbd-fault-injector.py +5 -2
@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3
Quoted 53 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)tests/qemu-iotests/iotests.py +6
-2@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], # We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIO + if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + output = StringIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))tests/qemu-iotests/nbd-fault-injector.py +4
-1@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
tests/qemu-iotests/nbd-fault-injector.py def parse_config(config): · line 234
Quoted 8 lines
tests/qemu-iotests/nbd-fault-injector.py +1
-1@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)
This may be a type of culture clash (on my side, due to not enough QEMU culture), but shouldn't this be applied before anything else on this series?
I mean, PATCH 1/9 is supposed to fix the reliability aspects of nbd-fault-injector under Python 3, but without this patch, it won't actually run on Python 3.
Regards,
- Cleber.
Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3
Quoted 68 lines · Cleber Rosa
Quoted 53 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)tests/qemu-iotests/iotests.py +6
-2@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], # We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIO + if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + output = StringIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))tests/qemu-iotests/nbd-fault-injector.py +4
-1@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GBtests/qemu-iotests/nbd-fault-injector.py def parse_config(config): · line 234
Quoted 8 lines
tests/qemu-iotests/nbd-fault-injector.py +1
-1@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)This may be a type of culture clash (on my side, due to not enough QEMU culture), but shouldn't this be applied before anything else on this series?
Quoted 3 lines
I mean, PATCH 1/9 is supposed to fix the reliability aspects of nbd-fault-injector under Python 3, but without this patch, it won't actually run on Python 3.
Both patches are required to make the code run on Python 3, and they don't depend on each other. So I think the order doesn't matter.
But I think the order chosen by Max is slightly more intuitive:
having explicit mentions of Python 3 in the code would be confusing if we didn't fix the compatibility issues on patches 1-7 first.
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3
In reply to Cleber RosaQuoted 68 lines · Cleber Rosa
Quoted 53 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)tests/qemu-iotests/iotests.py +6
-2@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], # We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIO + if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + output = StringIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))tests/qemu-iotests/nbd-fault-injector.py +4
-1@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GBtests/qemu-iotests/nbd-fault-injector.py def parse_config(config): · line 234
Quoted 8 lines
tests/qemu-iotests/nbd-fault-injector.py +1
-1@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)This may be a type of culture clash (on my side, due to not enough QEMU culture), but shouldn't this be applied before anything else on this series?
Quoted 3 lines
I mean, PATCH 1/9 is supposed to fix the reliability aspects of nbd-fault-injector under Python 3, but without this patch, it won't actually run on Python 3.
I don't mind, I followed no specific order. Well, patch 9 is patch 9 because it is the largest one, so I didn't want to discourage people before the end of the series. :-)
Max
Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3
In reply to Max ReitzQuoted 16 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)tests/qemu-iotests/iotests.py +0
-0
tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689
Quoted 8 lines
tests/qemu-iotests/iotests.py +4
-1@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], # We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIO
Considering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.
But this shouldn't block this workaround, so:
Quoted 39 lines
tests/qemu-iotests/iotests.py +2
-1+ if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + output = StringIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))tests/qemu-iotests/nbd-fault-injector.py +5
-2@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3
Quoted 18 lines · Eduardo Habkost
Quoted 17 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)tests/qemu-iotests/iotests.py +0
-0@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
Quoted 11 lines
tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689
Quoted 7 lines
tests/qemu-iotests/iotests.py +4
-1# We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIOConsidering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.
Another possibility, that I find self explanatory:
import io
if sys.version_info.major >= 3:
output = io.StringIO()
else:
output = io.BytesIO()
- Cleber.
Quoted 45 lines
But this shouldn't block this workaround, so:
Reviewed-by Eduardo Habkost <[email protected]>Quoted 39 lines
tests/qemu-iotests/iotests.py +2
-1+ if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + output = StringIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))tests/qemu-iotests/nbd-fault-injector.py +5
-2@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3
Quoted 33 lines · Cleber Rosa
Quoted 18 lines · Eduardo Habkost
Quoted 17 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)tests/qemu-iotests/iotests.py +0
-0@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],Quoted 11 lines
tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689
Quoted 7 lines
tests/qemu-iotests/iotests.py +4
-1# We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIOConsidering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.
Quoted 8 lines
Another possibility, that I find self explanatory:
import io
if sys.version_info.major >= 3:
output =io.StringIO()
else:
output =io.BytesIO()
Looks nice and clean.
But I'm not sure all output sent to output when running with Python 2 will be byte strings. What if unittest.TextTestRunner tries to write unicode strings to output?
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3
Quoted 22 lines · Eduardo Habkost
Quoted 21 lines · Cleber Rosa
Quoted 18 lines · Eduardo Habkost
Quoted 17 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there. The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 8 ++++++-- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 11 insertions(+), 4 deletions(-)tests/qemu-iotests/iotests.py +0
-0@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 689
Quoted 7 lines
tests/qemu-iotests/iotests.py +4
-1# We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + if sys.version_info.major >= 3: + from io import StringIO + else: + from StringIO import StringIOConsidering that io.StringIO exists on Python 2.7, a comment explaining why exactly it doesn't work would be nice.
Oh, it does exist? I didn't know. O:-)
So I suppose it's because the test runner emits just normal strings, which in 2.x are byte strings; but io's StringIO always expects Unicode strings. StringIO, OTOH, accepts both (and returns Unicode strings once you put a Unicode string into it).
Quoted 9 lines
Quoted 8 lines
Another possibility, that I find self explanatory:
import io
if sys.version_info.major >= 3:
output =io.StringIO()
else:
output =io.BytesIO()
Quoted 5 lines
Looks nice and clean.
But I'm not sure all output sent to
outputwhen running with Python 2 will be byte strings. What ifunittest.TextTestRunnertries to write unicode strings tooutput?
Hm. It doesn't look this way to me in practice. And considering that it only really prints a test summary, I don't think there are edge cases where it behaves differently.
So I think using BytesIO() in 2.x is better.
Max
PATCH 9/9 iotests: Unify log outputs between Python 2 and 3
In reply to Max ReitzWhen dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). Second, the order of keys in dicts may differ. Third, especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.
To get around these differences, this patch introduces functions to convert an object to a string that looks the same regardless of the Python version: In Python 2, they decode unicode strings to byte strings (normal str); in Python 3, they encode byte strings to unicode strings (normal str). They also manually convert lists and dicts to strings, which allows sorting the dicts by key, so we are no longer at the mercy of the internal implementation when it comes to how the keys appear in the output.
This changes the output of all tests that use these logging functions.
--- tests/qemu-iotests/194.out | 22 +- tests/qemu-iotests/202.out | 12 +- tests/qemu-iotests/203.out | 14 +- tests/qemu-iotests/206.out | 144 +++++----- tests/qemu-iotests/207.out | 52 ++-- tests/qemu-iotests/208.out | 8 +- tests/qemu-iotests/210.out | 72 ++--- tests/qemu-iotests/211.out | 66 ++--- tests/qemu-iotests/212.out | 102 +++---- tests/qemu-iotests/213.out | 124 ++++---- tests/qemu-iotests/216.out | 4 +- tests/qemu-iotests/218.out | 20 +- tests/qemu-iotests/219.out | 526 +++++++++++++++++----------------- tests/qemu-iotests/222.out | 24 +- tests/qemu-iotests/iotests.py | 42 ++- 15 files changed, 634 insertions(+), 598 deletions(-)
tests/qemu-iotests/194.out +11 -11
@@ -1,18 +1,18 @@ Launching VMs... Launching NBD server on destination... -{u'return': {}} -{u'return': {}} +{'return': {}} +{'return': {}} Starting `drive-mirror` on source... -{u'return': {}} +{'return': {}} Waiting for `drive-mirror` to complete... -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'} +{'data': {'device': 'mirror-job0', 'len': 1073741824, 'offset': 1073741824, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} Starting migration... -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'} +{'return': {}} +{'data': {'status': 'setup'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} +{'data': {'status': 'active'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} +{'data': {'status': 'completed'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} Gracefully ending the `drive-mirror` job on source... -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'} +{'return': {}} +{'data': {'device': 'mirror-job0', 'len': 1073741824, 'offset': 1073741824, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_COMPLETED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} Stopping the NBD server on destination... -{u'return': {}} +{'return': {}}
tests/qemu-iotests/202.out +6 -6
@@ -1,11 +1,11 @@ Launching VM... Adding IOThread... -{u'return': {}} +{'return': {}} Adding blockdevs... -{u'return': {}} -{u'return': {}} +{'return': {}} +{'return': {}} Setting iothread... -{u'return': {}} -{u'return': {}} +{'return': {}} +{'return': {}} Creating external snapshots... -{u'return': {}} +{'return': {}}
tests/qemu-iotests/203.out +7 -7
@@ -1,11 +1,11 @@ Launching VM... Setting IOThreads... -{u'return': {}} -{u'return': {}} +{'return': {}} +{'return': {}} Enabling migration QMP events... -{u'return': {}} +{'return': {}} Starting migration... -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'} +{'return': {}} +{'data': {'status': 'setup'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} +{'data': {'status': 'active'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} +{'data': {'status': 'completed'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
tests/qemu-iotests/206.out +72 -72
255 lines not on this page · raw message
tests/qemu-iotests/207.out +26 -26
99 lines not on this page · raw message
tests/qemu-iotests/208.out +4 -4
@@ -1,9 +1,9 @@ Launching VM... Starting NBD server... -{u'return': {}} +{'return': {}} Adding NBD export... -{u'return': {}} +{'return': {}} Creating external snapshot... -{u'return': {}} +{'return': {}} Stopping NBD server... -{u'return': {}} +{'return': {}}
tests/qemu-iotests/210.out +36 -36
118 lines not on this page · raw message
tests/qemu-iotests/211.out +33 -33
126 lines not on this page · raw message
tests/qemu-iotests/212.out +51 -51
199 lines not on this page · raw message
tests/qemu-iotests/213.out +62 -62
219 lines not on this page · raw message
tests/qemu-iotests/216.out +2 -2
@@ -7,8 +7,8 @@ Done --- Doing COR --- -{u'return': {}} -{u'return': u''} +{'return': {}} +{'return': ''} --- Checking COR result ---
tests/qemu-iotests/218.out +10 -10
@@ -4,27 +4,27 @@ --- force=false --- Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'} +{'return': {}} +{'data': {'device': 'mirror', 'len': 1048576, 'offset': 65536, 'speed': 65536, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} --- force=true --- Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'} +{'return': {}} +{'data': {'device': 'mirror', 'len': 1048576, 'offset': 65536, 'speed': 65536, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} === Cancel mirror job after convergence === --- force=false --- -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'} +{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_COMPLETED'} +{'return': {}} +{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_COMPLETED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} --- force=true --- -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'} +{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_CANCELLED'} +{'return': {}} +{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
tests/qemu-iotests/219.out +263 -263
589 lines not on this page · raw message
tests/qemu-iotests/222.out +12 -12
@@ -8,13 +8,13 @@ Done --- Setting up Fleecing Graph --- -{u'return': {}} -{u'return': {}} +{'return': {}} +{'return': {}} --- Setting up NBD Export --- -{u'return': {}} -{u'return': {}} +{'return': {}} +{'return': {}} --- Sanity Check ---
@@ -29,13 +29,13 @@ read -P0 0x3fe0000 64k --- Testing COW --- write -P0xab 0 64k -{u'return': u''} +{'return': ''} write -P0xad 0x00f8000 64k -{u'return': u''} +{'return': ''} write -P0x1d 0x2008000 64k -{u'return': u''} +{'return': ''} write -P0xea 0x3fe0000 64k -{u'return': u''} +{'return': ''} --- Verifying Data ---
@@ -49,10 +49,10 @@ read -P0 0x3fe0000 64k --- Cleanup --- -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'drive0', u'type': u'backup', u'speed': 0, u'len': 67108864, u'offset': 393216}, u'event': u'BLOCK_JOB_CANCELLED'} -{u'return': {}} -{u'return': {}} +{'return': {}} +{'data': {'device': 'drive0', 'len': 67108864, 'offset': 393216, 'speed': 0, 'type': 'backup'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}} +{'return': {}} +{'return': {}} --- Confirming writes ---
tests/qemu-iotests/iotests.py +39 -3
59 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH 9/9] iotests: Unify log outputs between Python 2 and 3
Quoted 4 lines · Max Reitz
When dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). [...]
This could be addressed using JSON. See below[1].
Quoted 1 line
[...] Second, the order of keys in dicts may differ. [...]
Can be addressed using json.dumps(..., sort_keys=True).
Quoted 3 lines
[...] Third,
especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.
print() doesn't add a L suffix on Python 2.7 on my system:
Python 2.7.15 (default, May 15 2018, 15:37:31)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Quoted 1 line
Quoted 1 line
Quoted 1 line
print(99999999999999999999999999999999999999999999999999999999999999999999999999999999999L)
99999999999999999999999999999999999999999999999999999999999999999999999999999999999
So I assume this happens only for QMP commands. It would be addressed if using JSON, too.
Quoted 30 lines
To get around these differences, this patch introduces functions to convert an object to a string that looks the same regardless of the Python version: In Python 2, they decode unicode strings to byte strings (normal str); in Python 3, they encode byte strings to unicode strings (normal str). They also manually convert lists and dicts to strings, which allows sorting the dicts by key, so we are no longer at the mercy of the internal implementation when it comes to how the keys appear in the output.
This changes the output of all tests that use these logging functions.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/194.out | 22 +- tests/qemu-iotests/202.out | 12 +- tests/qemu-iotests/203.out | 14 +- tests/qemu-iotests/206.out | 144 +++++----- tests/qemu-iotests/207.out | 52 ++-- tests/qemu-iotests/208.out | 8 +- tests/qemu-iotests/210.out | 72 ++--- tests/qemu-iotests/211.out | 66 ++--- tests/qemu-iotests/212.out | 102 +++---- tests/qemu-iotests/213.out | 124 ++++---- tests/qemu-iotests/216.out | 4 +- tests/qemu-iotests/218.out | 20 +- tests/qemu-iotests/219.out | 526 +++++++++++++++++----------------- tests/qemu-iotests/222.out | 24 +- tests/qemu-iotests/iotests.py | 42 ++- 15 files changed, 634 insertions(+), 598 deletions(-)
[...]
Quoted 19 lines
tests/qemu-iotests/iotests.py +11
-0@@ -250,10 +250,45 @@ def filter_img_info(output, filename): lines.append(line) return '\n'.join(lines) +def log_to_string_repr(obj): + # Normal Python 3 strings are the unicode strings from Python 2; + # and normal Python 2 strings are byte strings in Python 3. Thus, + # we convert bytes -> str in py3 and unicode -> str in py2. + if sys.version_info.major >= 3: + if type(obj) is bytes: + return repr(obj.decode('utf-8')) + else: + if type(obj) is unicode: + return repr(obj.encode('utf-8')) + elif type(obj) is long:
tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 273
Quoted 10 lines
tests/qemu-iotests/iotests.py +10
-0+ return str(obj) # repr() would include an 'L' suffix + + if type(obj) is list: + return '[' + ', '.join(map(log_to_string_repr, obj)) + ']' + elif type(obj) is dict: + return '{' + ', '.join(map(lambda k: log_to_string_repr(k) + ': ' + + log_to_string_repr(obj[k]), + sorted(obj.keys()))) + '}' + else: + return repr(obj)
I assume this function exists only because of QMP logging, correct?
I would just use json.dumps() at qmp_log(), see below[1].
tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 281
Quoted 8 lines
tests/qemu-iotests/iotests.py +8
-0+ +def log_to_string(obj): + if type(obj) is str: + return obj + + if sys.version_info.major >= 3: + if type(obj) is bytes: + return obj.decode('utf-8')
Do you know how many of existing log() calls actually pass a byte string as argument?
tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 284
Quoted 3 lines
tests/qemu-iotests/iotests.py +3
-0+ else: + if type(obj) is unicode: + return obj.encode('utf-8')
Is this really necessary? The existing code just calls print(msg) and it works, doesn't it?
Quoted 12 lines
tests/qemu-iotests/iotests.py +4
-1+ + return log_to_string_repr(obj) + def log(msg, filters=[]): for flt in filters: msg = flt(msg) - print(msg) + print(log_to_string(msg)) class Timeout: def __init__(self, seconds, errmsg = "Timeout"):
tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 483
Quoted 10 lines
tests/qemu-iotests/iotests.py +3
-2@@ -441,10 +476,11 @@ class VM(qtest.QEMUQtestMachine): return result def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs): - logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs) + logmsg = "{'execute': '%s', 'arguments': %s}" % \ + (cmd, log_to_string(kwargs)) log(logmsg, filters) result = self.qmp(cmd, **kwargs) - log(str(result), filters) + log(log_to_string(result), filters)
[1]
If we're being forced to regenerate all the QMP output in the .out files, what about always using JSON instead of the hack at log_to_string_repr()? i.e.:
tests/qemu-iotests/iotests.py +4 -2
@@ -441,10 +441,12 @@ class VM(qtest.QEMUQtestMachine): return result def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs): - logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs) + logmsg = '{"execute": %s, "arguments": %s}' % \ + (json.dumps(cmd, sort_keys=True), + json.dumps(kwargs, sort_keys=True)) log(logmsg, filters) result = self.qmp(cmd, **kwargs) - log(str(result), filters) + log(json.dumps(result, sort_keys=True), filters) return result def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Quoted 7 lines
tests/qemu-iotests/iotests.py +0
-0return result def run_job(self, job, auto_finalize=True, auto_dismiss=False):Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 9/9] iotests: Unify log outputs between Python 2 and 3
Quoted 11 lines · Eduardo Habkost
Quoted 4 lines · Max Reitz
When dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). [...]
This could be addressed using JSON. See below[1].
Quoted 1 line
[...] Second, the order of keys in dicts may differ. [...]
Can be addressed using
json.dumps(..., sort_keys=True).
Ah. Nice. :-)
Quoted 6 lines
Quoted 3 lines
[...] Third,
especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.
print()doesn't add a L suffix on Python 2.7 on my system:
Quoted 7 lines
Python 2.7.15 (default, May 15 2018, 15:37:31)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Quoted 1 line
Quoted 1 line
Quoted 1 line
print(99999999999999999999999999999999999999999999999999999999999999999999999999999999999L)
99999999999999999999999999999999999999999999999999999999999999999999999999999999999
So I assume this happens only for QMP commands.
It happens for dicts:
Quoted 1 line
Quoted 1 line
Quoted 1 line
print 99999999999999999999L
99999999999999999999
Quoted 1 line
Quoted 1 line
Quoted 1 line
print {'foo':99999999999999999999L}
{'foo': 99999999999999999999L}
Quoted 1 line
It would be addressed if using JSON, too.
OK.
Quoted 54 lines
Quoted 29 lines
To get around these differences, this patch introduces functions to convert an object to a string that looks the same regardless of the Python version: In Python 2, they decode unicode strings to byte strings (normal str); in Python 3, they encode byte strings to unicode strings (normal str). They also manually convert lists and dicts to strings, which allows sorting the dicts by key, so we are no longer at the mercy of the internal implementation when it comes to how the keys appear in the output.
This changes the output of all tests that use these logging functions.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/194.out | 22 +- tests/qemu-iotests/202.out | 12 +- tests/qemu-iotests/203.out | 14 +- tests/qemu-iotests/206.out | 144 +++++----- tests/qemu-iotests/207.out | 52 ++-- tests/qemu-iotests/208.out | 8 +- tests/qemu-iotests/210.out | 72 ++--- tests/qemu-iotests/211.out | 66 ++--- tests/qemu-iotests/212.out | 102 +++---- tests/qemu-iotests/213.out | 124 ++++---- tests/qemu-iotests/216.out | 4 +- tests/qemu-iotests/218.out | 20 +- tests/qemu-iotests/219.out | 526 +++++++++++++++++----------------- tests/qemu-iotests/222.out | 24 +- tests/qemu-iotests/iotests.py | 42 ++- 15 files changed, 634 insertions(+), 598 deletions(-)[...]
Quoted 24 lines
tests/qemu-iotests/iotests.py +16
-0@@ -250,10 +250,45 @@ def filter_img_info(output, filename): lines.append(line) return '\n'.join(lines) +def log_to_string_repr(obj): + # Normal Python 3 strings are the unicode strings from Python 2; + # and normal Python 2 strings are byte strings in Python 3. Thus, + # we convert bytes -> str in py3 and unicode -> str in py2. + if sys.version_info.major >= 3: + if type(obj) is bytes: + return repr(obj.decode('utf-8')) + else: + if type(obj) is unicode: + return repr(obj.encode('utf-8')) + elif type(obj) is long: + return str(obj) # repr() would include an 'L' suffix + + if type(obj) is list: + return '[' + ', '.join(map(log_to_string_repr, obj)) + ']' + elif type(obj) is dict:
Quoted 8 lines
tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 273
Quoted 5 lines
tests/qemu-iotests/iotests.py +5
-0+ return '{' + ', '.join(map(lambda k: log_to_string_repr(k) + ': ' + + log_to_string_repr(obj[k]), + sorted(obj.keys()))) + '}' + else: + return repr(obj)I assume this function exists only because of QMP logging, correct?
In practice probably yes.
Quoted 1 line
I would just use
json.dumps()atqmp_log(), see below[1].
However, there is not just qmp_log(), there are places that dump objects directly into log().
Quoted 11 lines
tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 281
Quoted 8 lines
tests/qemu-iotests/iotests.py +8
-0+ +def log_to_string(obj): + if type(obj) is str: + return obj + + if sys.version_info.major >= 3: + if type(obj) is bytes: + return obj.decode('utf-8')Do you know how many of existing
log()calls actually pass a byte string as argument?
Frankly I hope none.
Quoted 6 lines
tests/qemu-iotests/iotests.py def filter_img_info(output, filename): · line 284
Quoted 3 lines
tests/qemu-iotests/iotests.py +3
-0+ else: + if type(obj) is unicode: + return obj.encode('utf-8')Is this really necessary? The existing code just calls print(msg) and it works, doesn't it?
True. But the main issue is that we still need to return immediately for byte strings and Unicode strings, but the type "bytes" only exists in 3.x and "unicode" only exists in 2.x, i.e. "if type(obj) is unicode" throws an exception in 3.x.
So I have to distinguish between 2.x and 3.x anyway, and I thought to myself that if I distinguished anyway, I might do the conversion to the native type at the same time.
Quoted 36 lines
Quoted 12 lines
tests/qemu-iotests/iotests.py +4
-1+ + return log_to_string_repr(obj) + def log(msg, filters=[]): for flt in filters: msg = flt(msg) - print(msg) + print(log_to_string(msg)) class Timeout: def __init__(self, seconds, errmsg = "Timeout"):tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 483
Quoted 10 lines
tests/qemu-iotests/iotests.py +3
-2@@ -441,10 +476,11 @@ class VM(qtest.QEMUQtestMachine): return result def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs): - logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs) + logmsg = "{'execute': '%s', 'arguments': %s}" % \ + (cmd, log_to_string(kwargs)) log(logmsg, filters) result = self.qmp(cmd, **kwargs) - log(str(result), filters) + log(log_to_string(result), filters)[1]
If we're being forced to regenerate all the QMP output in the .out files, what about always using JSON instead of the hack at
log_to_string_repr()? i.e.:tests/qemu-iotests/iotests.py +0
-0@@ -441,10 +441,12 @@ class VM(qtest.QEMUQtestMachine): return result
tests/qemu-iotests/iotests.py class VM(qtest.QEMUQtestMachine): · line 450
Quoted 10 lines
tests/qemu-iotests/iotests.py +4
-2def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs): - logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs) + logmsg = '{"execute": %s, "arguments": %s}' % \ + (json.dumps(cmd, sort_keys=True), + json.dumps(kwargs, sort_keys=True)) log(logmsg, filters) result = self.qmp(cmd, **kwargs) - log(str(result), filters) + log(json.dumps(result, sort_keys=True), filters) return result
Sure, sounds good, thanks. :-)
The only thing is I would allow log() to still accept objects, it should do the conversion to JSON itself (if the object to be logged is a list or dict).
Max
Quoted 10 lines
tests/qemu-iotests/iotests.py +0
-0def run_job(self, job, auto_finalize=True, auto_dismiss=False):Quoted 7 lines
tests/qemu-iotests/iotests.py +0
-0return result def run_job(self, job, auto_finalize=True, auto_dismiss=False):Signature · 1 line
2.17.1
Hi Max,
Quoted 10 lines · Max Reitz
This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO class from the 'io' or the 'StringIO' module), but most of the time, it just makes code work for both versions in general.
And when we make the switch to make Python 3 mandatory, we can simply drop the Python 2 branches.
Quoted 10 lines
Max Reitz (9):
iotests: Make nbd-fault-injector flush
iotests: Flush in iotests.py's QemuIoInteractive
iotests: Use Python byte strings where appropriate
iotests: Use // for Python integer division
iotests: Different iterator behavior in Python 3
iotests: Explicitly inherit FDs in Python
iotests: 'new' module replacement in 169
iotests: Modify imports for Python 3
iotests: Unify log outputs between Python 2 and 3
You forgot:
MAINTAINERS: Add myself in the Python scripts section
;)
Quoted 37 lines
scripts/qemu.py | 13 +- scripts/qmp/qmp.py | 7 + scripts/qtest.py | 2 +- tests/qemu-iotests/040 | 4 +- tests/qemu-iotests/044 | 20 +- tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 +- tests/qemu-iotests/083.out | 9 + tests/qemu-iotests/093 | 18 +- tests/qemu-iotests/124 | 4 +- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/147 | 7 + tests/qemu-iotests/149 | 14 +- tests/qemu-iotests/151 | 12 +- tests/qemu-iotests/163 | 8 +- tests/qemu-iotests/169 | 3 +- tests/qemu-iotests/194.out | 22 +- tests/qemu-iotests/202.out | 12 +- tests/qemu-iotests/203.out | 14 +- tests/qemu-iotests/206.out | 144 +++---- tests/qemu-iotests/207 | 4 +- tests/qemu-iotests/207.out | 52 +-- tests/qemu-iotests/208.out | 8 +- tests/qemu-iotests/210.out | 72 ++-- tests/qemu-iotests/211.out | 66 +-- tests/qemu-iotests/212.out | 102 ++--- tests/qemu-iotests/213.out | 124 +++--- tests/qemu-iotests/216.out | 4 +- tests/qemu-iotests/218.out | 20 +- tests/qemu-iotests/219.out | 526 +++++++++++------------ tests/qemu-iotests/222.out | 24 +- tests/qemu-iotests/iotests.py | 64 ++- tests/qemu-iotests/nbd-fault-injector.py | 12 +- tests/qemu-iotests/qcow2.py | 10 +- 34 files changed, 745 insertions(+), 664 deletions(-)
Quoted 24 lines · Philippe Mathieu-Daudé
Hi Max,
Quoted 10 lines · Max Reitz
This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO class from the 'io' or the 'StringIO' module), but most of the time, it just makes code work for both versions in general.
And when we make the switch to make Python 3 mandatory, we can simply drop the Python 2 branches.
Quoted 10 lines
Max Reitz (9):
iotests: Make nbd-fault-injector flush
iotests: Flush in iotests.py's QemuIoInteractive
iotests: Use Python byte strings where appropriate
iotests: Use // for Python integer division
iotests: Different iterator behavior in Python 3
iotests: Explicitly inherit FDs in Python
iotests: 'new' module replacement in 169
iotests: Modify imports for Python 3
iotests: Unify log outputs between Python 2 and 3
Quoted 5 lines
You forgot:
MAINTAINERS: Add myself in the Python scripts section;)
Aw, please, no. You do know I don't really know Python? ;-)
Max