iotests: Make them work for both Python 2 and 3
v2 cover iotests: Make them work for both Python 2 and 3
This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO or the BytesIO class from 'io' for use with the test runner), but most of the time, it just makes code work for both versions in general.
And when we make the switch to make Python 3 mandatory, we can simply drop the Python 2 branches.
v2:
- Patch 3: Commit message fix [Philippe]
- Patch 4: Really convert all places that use / [Cleber]
- Patch 5:
- Use list comprehension expressions where that makes sense [Eduardo]
- Make 'range' an alias for 'xrange' in 044 and 163 [Cleber]
- Patch 6:
- %s/3\.2/3.4/g [Cleber]
- Use hasattr() instead of catching AttributeError [Cleber]
- Call set_inheritable in send_fd_scm() for both the QMP socket FD and the FD to be passed (if any) [Eduardo]
- Patch 7: Kept as-is, because it's just the minimal amount of change
- Patch 8: Use io.BytesIO in Python 2 [Cleber]
- Patch 9: Use json.dumps(..., sort_keys=True) instead of homegrown stringification function [Eduardo]
(Also kept the order, because one vote against, one vote for, so I go for what's simplest for me. O:-))
git-backport-diff against v1:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/9:[----] [--] 'iotests: Make nbd-fault-injector flush' 002/9:[----] [--] 'iotests: Flush in iotests.py's QemuIoInteractive' 003/9:[----] [--] 'iotests: Use Python byte strings where appropriate' 004/9:[0014] [FC] 'iotests: Use // for Python integer division' 005/9:[0013] [FC] 'iotests: Different iterator behavior in Python 3' 006/9:[0049] [FC] 'iotests: Explicitly inherit FDs in Python' 007/9:[----] [--] 'iotests: 'new' module replacement in 169' 008/9:[0017] [FC] 'iotests: Modify imports for Python 3' 009/9:[1644] [FC] 'iotests: Unify log outputs between Python 2 and 3'
Max Reitz (9):
iotests: Make nbd-fault-injector flush
iotests: Flush in iotests.py's QemuIoInteractive
iotests: Use Python byte strings where appropriate
iotests: Use // for Python integer division
iotests: Different iterator behavior in Python 3
iotests: Explicitly inherit FDs in Python
iotests: 'new' module replacement in 169
iotests: Modify imports for Python 3
iotests: Unify log outputs between Python 2 and 3
scripts/qemu.py | 34 +- scripts/qtest.py | 2 +- tests/qemu-iotests/030 | 2 +- tests/qemu-iotests/040 | 4 +- tests/qemu-iotests/041 | 4 +- tests/qemu-iotests/044 | 24 +- tests/qemu-iotests/045 | 2 +- tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 +- tests/qemu-iotests/083.out | 9 + tests/qemu-iotests/093 | 18 +- tests/qemu-iotests/124 | 4 +- tests/qemu-iotests/136 | 2 +- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/147 | 2 +- tests/qemu-iotests/149 | 14 +- tests/qemu-iotests/151 | 12 +- tests/qemu-iotests/163 | 13 +- tests/qemu-iotests/169 | 3 +- tests/qemu-iotests/194.out | 22 +- tests/qemu-iotests/202.out | 12 +- tests/qemu-iotests/203.out | 14 +- tests/qemu-iotests/206.out | 218 +++++----- tests/qemu-iotests/207 | 6 +- tests/qemu-iotests/207.out | 72 ++-- tests/qemu-iotests/208.out | 8 +- tests/qemu-iotests/210.out | 94 ++-- tests/qemu-iotests/211.out | 102 ++--- tests/qemu-iotests/212.out | 174 ++++---- tests/qemu-iotests/213.out | 182 ++++---- tests/qemu-iotests/216.out | 4 +- tests/qemu-iotests/218.out | 20 +- tests/qemu-iotests/219.out | 526 +++++++++++------------ tests/qemu-iotests/222.out | 24 +- tests/qemu-iotests/iotests.py | 37 +- tests/qemu-iotests/nbd-fault-injector.py | 12 +- tests/qemu-iotests/qcow2.py | 10 +- tests/qemu-iotests/qed.py | 6 +- 38 files changed, 879 insertions(+), 821 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/9 iotests: Make nbd-fault-injector flush
When closing a connection, make the nbd-fault-injector flush the socket. Without this, the output is a bit unreliable with Python 3.
--- tests/qemu-iotests/083.out | 9 +++++++++ tests/qemu-iotests/nbd-fault-injector.py | 1 + 2 files changed, 10 insertions(+)
tests/qemu-iotests/083.out +9 -0
@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect after neg2 === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error
@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo === Check disconnect before request === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error
@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/ === Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer Connection closed read failed: Input/output error
@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock === Check disconnect after neg2 === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error === Check disconnect 8 neg2 ===
@@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock === Check disconnect before request === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error === Check disconnect after request ===
@@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock === Check disconnect after neg-classic === +Unable to read from socket: Connection reset by peer +Connection closed read failed: Input/output error *** done
tests/qemu-iotests/nbd-fault-injector.py +1 -0
@@ -112,6 +112,7 @@ class FaultInjectionSocket(object): if rule.match(event, io): if rule.when == 0 or bufsize is None: print('Closing connection on rule match %s' % rule.name) + self.sock.flush() sys.exit(0) if rule.when != -1: return rule.when
Signature · 1 line
2.17.1
PATCH v2 2/9 iotests: Flush in iotests.py's QemuIoInteractive
In reply to Max ReitzAfter issuing a command, flush the pipe. This does not change anything in Python 2, but it makes a difference in Python 3.
--- tests/qemu-iotests/iotests.py | 1 + 1 file changed, 1 insertion(+)
tests/qemu-iotests/iotests.py +1 -0
@@ -178,6 +178,7 @@ class QemuIoInteractive: cmd = cmd.strip() assert cmd != 'q' and cmd != 'quit' self._p.stdin.write(cmd + '\n') + self._p.stdin.flush() return self._read_output()
Signature · 1 line
2.17.1
PATCH v2 3/9 iotests: Use Python byte strings where appropriate
In reply to Max ReitzSince byte strings are no longer the default in Python 3, we have to explicitly use them where we need to, which is mostly when working with structures. It also means that we need to open a file in binary mode when we want to use structures.
On the other hand, we have to accomodate for the fact that some functions (still) work with byte strings but we want to use unicode strings (in Python 3 at least, and it does not matter in Python 2). This includes base64 encoding, but it is most notable when working with the subprocess module: Either we set universal_newlines to True so that the default streams are opened in text mode (hence this parameter is aliased as "text" as of 3.7), or, if that is not possible, we have to decode the output to a normal string.
--- scripts/qtest.py | 2 +- tests/qemu-iotests/044 | 8 ++++---- tests/qemu-iotests/149 | 8 +++++--- tests/qemu-iotests/207 | 4 ++-- tests/qemu-iotests/iotests.py | 11 +++++++---- tests/qemu-iotests/nbd-fault-injector.py | 4 ++-- tests/qemu-iotests/qcow2.py | 10 +++++----- 7 files changed, 26 insertions(+), 21 deletions(-)
scripts/qtest.py +1 -1
@@ -64,7 +64,7 @@ class QEMUQtestProtocol(object): @param qtest_cmd: qtest command text to be sent """ - self._sock.sendall(qtest_cmd + "\n") + self._sock.sendall((qtest_cmd + "\n").encode('utf-8')) def close(self): self._sock.close()
tests/qemu-iotests/044 +4 -4
@@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): fd.seek(off_reftable) for i in xrange(0, h.refcount_table_clusters): - sector = ''.join(struct.pack('>Q', + sector = b''.join(struct.pack('>Q', off_refblock + i * 64 * 512 + j * 512) for j in xrange(0, 64)) fd.write(sector) # Write the refcount blocks assert(fd.tell() == off_refblock) - sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256)) + sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256)) for block in xrange(0, h.refcount_table_clusters): fd.write(sector) # Write the L1 table assert(fd.tell() == off_l1) assert(off_l2 + 512 * h.l1_size == off_data) - table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) + table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) for j in xrange(0, h.l1_size)) fd.write(table)
@@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): remaining = remaining - 1024 * 512 off = off + 1024 * 512 - table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) + table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) for j in xrange(0, remaining / 512)) fd.write(table)
tests/qemu-iotests/149 +5 -3
@@ -79,7 +79,7 @@ class LUKSConfig(object): def first_password_base64(self): (pw, slot) = self.first_password() - return base64.b64encode(pw) + return base64.b64encode(pw.encode('ascii')).decode('ascii') def active_slots(self): slots = []
@@ -98,7 +98,8 @@ def verify_passwordless_sudo(): proc = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) msg = proc.communicate()[0]
@@ -116,7 +117,8 @@ def cryptsetup(args, password=None): proc = subprocess.Popen(fullargs, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) msg = proc.communicate(password)[0]
tests/qemu-iotests/207 +2 -2
@@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \ md5_key = subprocess.check_output( 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + 'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1', - shell=True).rstrip() + shell=True).rstrip().decode('ascii') vm.launch() blockdev_create(vm, { 'driver': 'ssh',
@@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \ sha1_key = subprocess.check_output( 'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' + 'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1', - shell=True).rstrip() + shell=True).rstrip().decode('ascii') vm.launch() blockdev_create(vm, { 'driver': 'ssh',
tests/qemu-iotests/iotests.py +7 -4
@@ -104,7 +104,8 @@ def qemu_img_pipe(*args): '''Run qemu-img and return its output''' subp = subprocess.Popen(qemu_img_args + list(args), stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) exitcode = subp.wait() if exitcode < 0: sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
@@ -128,7 +129,8 @@ def qemu_io(*args): '''Run qemu-io and return the stdout data''' args = qemu_io_args + list(args) subp = subprocess.Popen(args, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) exitcode = subp.wait() if exitcode < 0: sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))
@@ -149,7 +151,8 @@ class QemuIoInteractive: self.args = qemu_io_args + list(args) self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, + universal_newlines=True) assert self._p.stdout.read(9) == 'qemu-io> ' def close(self):
@@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt): def create_image(name, size): '''Create a fully-allocated raw image with sector markers''' - file = open(name, 'w') + file = open(name, 'wb') i = 0 while i < size: sector = struct.pack('>l504xl', i / 512, i / 512)
tests/qemu-iotests/nbd-fault-injector.py +2 -2
@@ -86,7 +86,7 @@ def recvall(sock, bufsize): raise Exception('unexpected disconnect') chunks.append(chunk) received += len(chunk) - return ''.join(chunks) + return b''.join(chunks) class Rule(object): def __init__(self, name, event, io, when):
@@ -177,7 +177,7 @@ def handle_connection(conn, use_export): req = read_request(conn) if req.type == NBD_CMD_READ: write_reply(conn, 0, req.handle) - conn.send('\0' * req.len, event='data') + conn.send(b'\0' * req.len, event='data') elif req.type == NBD_CMD_WRITE: _ = conn.recv(req.len, event='data') write_reply(conn, 0, req.handle)
tests/qemu-iotests/qcow2.py +5 -5
@@ -10,7 +10,7 @@ class QcowHeaderExtension: def __init__(self, magic, length, data): if length % 8 != 0: padding = 8 - (length % 8) - data += "\0" * padding + data += b"\0" * padding self.magic = magic self.length = length
@@ -103,7 +103,7 @@ class QcowHeader: fd.seek(self.header_length) extensions = self.extensions - extensions.append(QcowHeaderExtension(0, 0, "")) + extensions.append(QcowHeaderExtension(0, 0, b"")) for ex in extensions: buf = struct.pack('>II', ex.magic, ex.length) fd.write(buf)
@@ -137,8 +137,8 @@ class QcowHeader: for ex in self.extensions: data = ex.data[:ex.length] - if all(c in string.printable for c in data): - data = "'%s'" % data + if all(c in string.printable.encode('ascii') for c in data): + data = "'%s'" % data.decode('ascii') else: data = "<binary>"
@@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data): sys.exit(1) h = QcowHeader(fd) - h.extensions.append(QcowHeaderExtension.create(magic, data)) + h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii'))) h.update(fd) def cmd_add_header_ext_stdio(fd, magic):
Signature · 1 line
2.17.1
PATCH v2 4/9 iotests: Use // for Python integer division
In reply to Max ReitzIn Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.
--- tests/qemu-iotests/030 | 2 +- tests/qemu-iotests/040 | 4 ++-- tests/qemu-iotests/041 | 4 ++-- tests/qemu-iotests/044 | 2 +- tests/qemu-iotests/093 | 18 +++++++++--------- tests/qemu-iotests/136 | 2 +- tests/qemu-iotests/149 | 6 +++--- tests/qemu-iotests/151 | 12 ++++++------ tests/qemu-iotests/163 | 2 +- tests/qemu-iotests/iotests.py | 2 +- tests/qemu-iotests/qed.py | 6 +++--- 11 files changed, 30 insertions(+), 30 deletions(-)
tests/qemu-iotests/030 +1 -1
@@ -521,7 +521,7 @@ new_state = "2" state = "2" event = "%s" new_state = "1" -''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event)) +''' % (event, errno, self.STREAM_BUFFER_SIZE // 512, event, event)) file.close() class TestEIO(TestErrors):
tests/qemu-iotests/040 +2 -2
@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('device_del', id='scsi0') self.assert_qmp(result, 'return', {})
@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase): self.assert_no_active_block_jobs() result = self.vm.qmp('block-commit', device='drive0', top=mid_img, - base=backing_img, speed=(self.image_len / 4)) + base=backing_img, speed=(self.image_len // 4)) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('query-block')
tests/qemu-iotests/041 +2 -2
@@ -404,7 +404,7 @@ new_state = "2" state = "2" event = "%s" new_state = "1" -''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event)) +''' % (event, errno, self.MIRROR_GRANULARITY // 512, event, event)) file.close() def setUp(self):
@@ -569,7 +569,7 @@ new_state = "2" state = "2" event = "%s" new_state = "1" -''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event)) +''' % (event, errno, self.MIRROR_GRANULARITY // 512, event, event)) file.close() def setUp(self):
tests/qemu-iotests/044 +1 -1
@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining / 512)) + for j in xrange(0, remaining // 512)) fd.write(table)
tests/qemu-iotests/093 +9 -9
@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase): # in. The throttled requests won't be executed until we # advance the virtual clock. rq_size = 512 - rd_nr = max(params['bps'] / rq_size / 2, - params['bps_rd'] / rq_size, - params['iops'] / 2, + rd_nr = max(params['bps'] // rq_size // 2, + params['bps_rd'] // rq_size, + params['iops'] // 2, params['iops_rd']) rd_nr *= seconds * 2 - rd_nr /= ndrives - wr_nr = max(params['bps'] / rq_size / 2, - params['bps_wr'] / rq_size, - params['iops'] / 2, + rd_nr //= ndrives + wr_nr = max(params['bps'] // rq_size // 2, + params['bps_wr'] // rq_size, + params['iops'] // 2, params['iops_wr']) wr_nr *= seconds * 2 - wr_nr /= ndrives + wr_nr //= ndrives # Send I/O requests to all drives for i in range(rd_nr):
@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase): self.configure_throttle(ndrives, settings) # Wait for the bucket to empty so we can do bursts - wait_ns = nsec_per_sec * burst_length * burst_rate / rate + wait_ns = nsec_per_sec * burst_length * burst_rate // rate self.vm.qtest("clock_step %d" % wait_ns) # Test I/O at the max burst rate
tests/qemu-iotests/136 +1 -1
@@ -24,7 +24,7 @@ import os interval_length = 10 nsec_per_sec = 1000000000 -op_latency = nsec_per_sec / 1000 # See qtest_latency_ns in accounting.c +op_latency = nsec_per_sec // 1000 # See qtest_latency_ns in accounting.c bad_sector = 8192 bad_offset = bad_sector * 512 blkdebug_file = os.path.join(iotests.test_dir, 'blkdebug.conf')
tests/qemu-iotests/149 +3 -3
@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False): image_size = 4 * oneTB if qemu_img: iotests.log("# Create image") - qemu_img_create(config, image_size / oneMB) + qemu_img_create(config, image_size // oneMB) else: iotests.log("# Create image") - create_image(config, image_size / oneMB) + create_image(config, image_size // oneMB) lowOffsetMB = 100 - highOffsetMB = 3 * oneTB / oneMB + highOffsetMB = 3 * oneTB // oneMB try: if not qemu_img:
tests/qemu-iotests/151 +6 -6
@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase): 'write -P 1 0 %i' % self.image_len); # Start some background requests - for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset) - for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024): + for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Start the block job
@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase): self.assert_qmp(result, 'return', {}) # Start some more requests - for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024): + for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) # Wait for the READY event
@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase): # the source) should be settled using the active mechanism. # The mirror code itself asserts that the source BDS's dirty # bitmap will stay clean between READY and COMPLETED. - for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset) - for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024): + for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024): self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset) if sync_source_and_target:
tests/qemu-iotests/163 +1 -1
@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): entry_bits = 3 entry_size = 1 << entry_bits l1_mask = 0x00fffffffffffe00 - div_roundup = lambda n, d: (n + d - 1) / d + div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): for x in xrange(0, len(data), n):
tests/qemu-iotests/iotests.py +1 -1
@@ -199,7 +199,7 @@ def create_image(name, size): file = open(name, 'wb') i = 0 while i < size: - sector = struct.pack('>l504xl', i / 512, i / 512) + sector = struct.pack('>l504xl', i // 512, i // 512) file.write(sector) i = i + 512 file.close()
tests/qemu-iotests/qed.py +3 -3
@@ -80,7 +80,7 @@ class QED(object): def load_l1_table(self): self.l1_table = self.read_table(self.header['l1_table_offset']) - self.table_nelems = self.header['table_size'] * self.header['cluster_size'] / table_elem_size + self.table_nelems = self.header['table_size'] * self.header['cluster_size'] // table_elem_size def write_table(self, offset, table): s = ''.join(pack_table_elem(x) for x in table)
@@ -167,14 +167,14 @@ def cmd_zero_cluster(qed, pos, *args): n = int(args[0]) for i in xrange(n): - l1_index = pos / qed.header['cluster_size'] / len(qed.l1_table) + l1_index = pos // qed.header['cluster_size'] // len(qed.l1_table) if qed.l1_table[l1_index] == 0: err('no l2 table allocated') l2_offset = qed.l1_table[l1_index] l2_table = qed.read_table(l2_offset) - l2_index = (pos / qed.header['cluster_size']) % len(qed.l1_table) + l2_index = (pos // qed.header['cluster_size']) % len(qed.l1_table) l2_table[l2_index] = 1 # zero the data cluster qed.write_table(l2_offset, l2_table) pos += qed.header['cluster_size']
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 4/9] iotests: Use // for Python integer division
Quoted 5 lines · Max Reitz
In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.
Signed-off-by Max Reitz <[email protected]>
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH v2 4/9] iotests: Use // for Python integer division
In reply to Max ReitzQuoted 5 lines · Max Reitz
In Python 3, / is always a floating-point division. We usually do not want this, and as Python 2.7 understands // as well, change all integer divisions to use that.
Signed-off-by Max Reitz <[email protected]>
PATCH v2 5/9 iotests: Different iterator behavior in Python 3
In reply to Max ReitzIn Python 3, several functions now return iterators instead of lists. This includes range(), items(), map(), and filter(). This means that if we really want a list, we have to wrap those instances with list(). But then again, the two instances where this is the case for map() and filter(), there are shorter expressions which work without either function.
On the other hand, sometimes we do just want an iterator, in which case we have sometimes used xrange() and iteritems() which no longer exist in Python 3. Just change these calls to be range() and items(), works in both Python 2 and 3, and is really what we want in 3 (which is what matters). But because it is so simple to do (and to find and remove once we completely switch to Python 3), make range() be an alias for xrange() in the two affected tests (044 and 163).
In one instance, we only wanted the first instance of the result of a filter() call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to use list comprehension with a next() wrapped around, which works both in 2.7 and 3.
--- tests/qemu-iotests/044 | 16 ++++++++++------ tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 ++-- tests/qemu-iotests/124 | 4 ++-- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/163 | 11 +++++++---- 6 files changed, 23 insertions(+), 16 deletions(-)
tests/qemu-iotests/044 +10 -6
@@ -26,6 +26,10 @@ import iotests from iotests import qemu_img, qemu_img_verbose, qemu_io import struct import subprocess +import sys + +if sys.version_info.major == 2: + range = xrange test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -52,23 +56,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): # Write a refcount table fd.seek(off_reftable) - for i in xrange(0, h.refcount_table_clusters): + for i in range(0, h.refcount_table_clusters): sector = b''.join(struct.pack('>Q', off_refblock + i * 64 * 512 + j * 512) - for j in xrange(0, 64)) + for j in range(0, 64)) fd.write(sector) # Write the refcount blocks assert(fd.tell() == off_refblock) sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256)) - for block in xrange(0, h.refcount_table_clusters): + for block in range(0, h.refcount_table_clusters): fd.write(sector) # Write the L1 table assert(fd.tell() == off_l1) assert(off_l2 + 512 * h.l1_size == off_data) table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) - for j in xrange(0, h.l1_size)) + for j in range(0, h.l1_size)) fd.write(table) # Write the L2 tables
@@ -79,14 +83,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off_data while remaining > 1024 * 512: pytable = list((1 << 63) | off + 512 * j - for j in xrange(0, 1024)) + for j in range(0, 1024)) table = struct.pack('>1024Q', *pytable) fd.write(table) remaining = remaining - 1024 * 512 off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining // 512)) + for j in range(0, remaining // 512)) fd.write(table)
tests/qemu-iotests/056 +1 -1
@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img') def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs): fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt)) optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [fullname, size] iotests.qemu_img(*args)
tests/qemu-iotests/065 +2 -2
@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific): :data.index('')] for field in data: self.assertTrue(re.match('^ {4}[^ ]', field) is not None) - data = map(lambda line: line.strip(), data) + data = [line.strip() for line in data] self.assertEqual(data, self.human_compare) class TestQMP(TestImageInfoSpecific):
@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific): def test_qmp(self): result = self.vm.qmp('query-block')['return'] - drive = filter(lambda drive: drive['device'] == 'drive0', result)[0] + drive = next(drive for drive in result if drive['device'] == 'drive0') data = drive['inserted']['image']['format-specific'] self.assertEqual(data['type'], iotests.imgfmt) self.assertEqual(data['data'], self.compare)
tests/qemu-iotests/124 +2 -2
@@ -39,7 +39,7 @@ def try_remove(img): def transaction_action(action, **kwargs): return { 'type': action, - 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems()) + 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items()) }
@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): def img_create(self, img, fmt=iotests.imgfmt, size='64M', parent=None, parentFormat=None, **kwargs): optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [img, size] if parent:
tests/qemu-iotests/139 +1 -1
@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase): # Check whether a BlockDriverState exists def checkBlockDriverState(self, node, must_exist = True): result = self.vm.qmp('query-named-block-nodes') - nodes = filter(lambda x: x['node-name'] == node, result['return']) + nodes = [x for x in result['return'] if x['node-name'] == node] self.assertLessEqual(len(nodes), 1) self.assertEqual(must_exist, len(nodes) == 1)
tests/qemu-iotests/163 +7 -4
@@ -18,9 +18,12 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. # -import os, random, iotests, struct, qcow2 +import os, random, iotests, struct, qcow2, sys from iotests import qemu_img, qemu_io, image_size +if sys.version_info.major == 2: + range = xrange + test_img = os.path.join(iotests.test_dir, 'test.img') check_img = os.path.join(iotests.test_dir, 'check.img')
@@ -41,7 +44,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): - for x in xrange(0, len(data), n): + for x in range(0, len(data), n): yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask def check_l1_table(h, l1_data):
@@ -135,8 +138,8 @@ class ShrinkBaseClass(iotests.QMPTestCase): self.image_verify() def test_random_write(self): - offs_list = range(0, size_to_int(self.image_len), - size_to_int(self.chunk_size)) + offs_list = list(range(0, size_to_int(self.image_len), + size_to_int(self.chunk_size))) random.shuffle(offs_list) for offs in offs_list: qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 5/9] iotests: Different iterator behavior in Python 3
Quoted 15 lines · Max Reitz
In Python 3, several functions now return iterators instead of lists. This includes
range(),items(),map(), andfilter(). This means that if we really want a list, we have to wrap those instances withlist(). But then again, the two instances where this is the case formap()andfilter(), there are shorter expressions which work without either function.On the other hand, sometimes we do just want an iterator, in which case we have sometimes used
xrange()anditeritems()which no longer exist in Python 3. Just change these calls to berange()anditems(), works in both Python 2 and 3, and is really what we want in 3 (which is what matters). But because it is so simple to do (and to find and remove once we completely switch to Python 3), makerange()be an alias forxrange()in the two affected tests (044 and 163).
Quoted 5 lines
In one instance, we only wanted the first instance of the result of a
filter()call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to use list comprehension with anext()wrapped around, which works both in 2.7 and 3.
Nit: the expression you put inside next(...) is not a list comprehension; it's a generator expression. A list comprehension expression would generate the full list in advance before you get the first element.
It would be OK to rewrite the expression using an actual list comprehension:
drive = [drive for drive in result if drive['device'] == 'drive0'][0]
But the solution you chose is OK, too.
Quoted 182 lines
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/044 | 16 ++++++++++------ tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/065 | 4 ++-- tests/qemu-iotests/124 | 4 ++-- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/163 | 11 +++++++---- 6 files changed, 23 insertions(+), 16 deletions(-)tests/qemu-iotests/044 +10
-6@@ -26,6 +26,10 @@ import iotests from iotests import qemu_img, qemu_img_verbose, qemu_io import struct import subprocess +import sys + +if sys.version_info.major == 2: + range = xrange test_img = os.path.join(iotests.test_dir, 'test.img')@@ -52,23 +56,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): # Write a refcount table fd.seek(off_reftable) - for i in xrange(0, h.refcount_table_clusters): + for i in range(0, h.refcount_table_clusters): sector = b''.join(struct.pack('>Q', off_refblock + i * 64 * 512 + j * 512) - for j in xrange(0, 64)) + for j in range(0, 64)) fd.write(sector) # Write the refcount blocks assert(fd.tell() == off_refblock) sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256)) - for block in xrange(0, h.refcount_table_clusters): + for block in range(0, h.refcount_table_clusters): fd.write(sector) # Write the L1 table assert(fd.tell() == off_l1) assert(off_l2 + 512 * h.l1_size == off_data) table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j) - for j in xrange(0, h.l1_size)) + for j in range(0, h.l1_size)) fd.write(table) # Write the L2 tables@@ -79,14 +83,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): off = off_data while remaining > 1024 * 512: pytable = list((1 << 63) | off + 512 * j - for j in xrange(0, 1024)) + for j in range(0, 1024)) table = struct.pack('>1024Q', *pytable) fd.write(table) remaining = remaining - 1024 * 512 off = off + 1024 * 512 table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j) - for j in xrange(0, remaining // 512)) + for j in range(0, remaining // 512)) fd.write(table)tests/qemu-iotests/056 +1
-1@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img') def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs): fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt)) optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [fullname, size] iotests.qemu_img(*args)tests/qemu-iotests/065 +2
-2@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific): :data.index('')] for field in data: self.assertTrue(re.match('^ {4}[^ ]', field) is not None) - data = map(lambda line: line.strip(), data) + data = [line.strip() for line in data] self.assertEqual(data, self.human_compare) class TestQMP(TestImageInfoSpecific):@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific): def test_qmp(self): result = self.vm.qmp('query-block')['return'] - drive = filter(lambda drive: drive['device'] == 'drive0', result)[0] + drive = next(drive for drive in result if drive['device'] == 'drive0') data = drive['inserted']['image']['format-specific'] self.assertEqual(data['type'], iotests.imgfmt) self.assertEqual(data['data'], self.compare)tests/qemu-iotests/124 +2
-2@@ -39,7 +39,7 @@ def try_remove(img): def transaction_action(action, **kwargs): return { 'type': action, - 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems()) + 'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items()) }@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): def img_create(self, img, fmt=iotests.imgfmt, size='64M', parent=None, parentFormat=None, **kwargs): optargs = [] - for k,v in kwargs.iteritems(): + for k,v in kwargs.items(): optargs = optargs + ['-o', '%s=%s' % (k,v)] args = ['create', '-f', fmt] + optargs + [img, size] if parent:tests/qemu-iotests/139 +1
-1@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase): # Check whether a BlockDriverState exists def checkBlockDriverState(self, node, must_exist = True): result = self.vm.qmp('query-named-block-nodes') - nodes = filter(lambda x: x['node-name'] == node, result['return']) + nodes = [x for x in result['return'] if x['node-name'] == node] self.assertLessEqual(len(nodes), 1) self.assertEqual(must_exist, len(nodes) == 1)tests/qemu-iotests/163 +7
-4@@ -18,9 +18,12 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. # -import os, random, iotests, struct, qcow2 +import os, random, iotests, struct, qcow2, sys from iotests import qemu_img, qemu_io, image_size +if sys.version_info.major == 2: + range = xrange + test_img = os.path.join(iotests.test_dir, 'test.img') check_img = os.path.join(iotests.test_dir, 'check.img')@@ -41,7 +44,7 @@ class ShrinkBaseClass(iotests.QMPTestCase): div_roundup = lambda n, d: (n + d - 1) // d def split_by_n(data, n): - for x in xrange(0, len(data), n): + for x in range(0, len(data), n): yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask def check_l1_table(h, l1_data):@@ -135,8 +138,8 @@ class ShrinkBaseClass(iotests.QMPTestCase): self.image_verify() def test_random_write(self): - offs_list = range(0, size_to_int(self.image_len), - size_to_int(self.chunk_size)) + offs_list = list(range(0, size_to_int(self.image_len), + size_to_int(self.chunk_size))) random.shuffle(offs_list) for offs in offs_list: qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH v2 5/9] iotests: Different iterator behavior in Python 3
In reply to Max ReitzQuoted 15 lines · Max Reitz
In Python 3, several functions now return iterators instead of lists. This includes
range(),items(),map(), andfilter(). This means that if we really want a list, we have to wrap those instances withlist(). But then again, the two instances where this is the case formap()andfilter(), there are shorter expressions which work without either function.On the other hand, sometimes we do just want an iterator, in which case we have sometimes used
xrange()anditeritems()which no longer exist in Python 3. Just change these calls to berange()anditems(), works in both Python 2 and 3, and is really what we want in 3 (which is what matters). But because it is so simple to do (and to find and remove once we completely switch to Python 3), makerange()be an alias forxrange()in the two affected tests (044 and 163).
Quoted 7 lines
In one instance, we only wanted the first instance of the result of a
filter()call. Instead of using next(filter()) which would work only in Python 3, or list(filter())[0] which would work everywhere but is a bit weird, this instance is changed to use list comprehension with anext()wrapped around, which works both in 2.7 and 3.Signed-off-by Max Reitz <[email protected]>
PATCH v2 6/9 iotests: Explicitly inherit FDs in Python
In reply to Max ReitzPython 3.4 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if close_fds is explicitly set to False.
Adhere to this by setting close_fds to False when working with subprocesses that may want to inherit FDs, and by trying to set_inheritable() on FDs that we do want to bequeath to them.
--- scripts/qemu.py | 34 +++++++++++++++++++++++++++++----- tests/qemu-iotests/045 | 2 +- tests/qemu-iotests/147 | 2 +- 3 files changed, 31 insertions(+), 7 deletions(-)
scripts/qemu.py +29 -5
@@ -142,11 +142,19 @@ class QEMUMachine(object): if opts: options.append(opts) + # This did not exist before 3.4, but since then it is + # mandatory for our purpose + if hasattr(os, 'set_inheritable'): + os.set_inheritable(fd, True) + self._args.append('-add-fd') self._args.append(','.join(options)) return self - def send_fd_scm(self, fd_file_path): + # Exactly one of fd and file_path must be given. + # (If it is file_path, the helper will open that file and pass its + # own fd) + def send_fd_scm(self, fd=None, file_path=None): # In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available() if self._socket_scm_helper is None:
@@ -154,12 +162,27 @@ class QEMUMachine(object): if not os.path.exists(self._socket_scm_helper): raise QEMUMachineError("%s does not exist" % self._socket_scm_helper) + + # This did not exist before 3.4, but since then it is + # mandatory for our purpose + if hasattr(os, 'set_inheritable'): + os.set_inheritable(self._qmp.get_sock_fd(), True) + if fd is not None: + os.set_inheritable(fd, True) + fd_param = ["%s" % self._socket_scm_helper, - "%d" % self._qmp.get_sock_fd(), - "%s" % fd_file_path] + "%d" % self._qmp.get_sock_fd()] + + if file_path is not None: + assert fd is None + fd_param.append(file_path) + else: + assert fd is not None + fd_param.append(str(fd)) + devnull = open(os.path.devnull, 'rb') proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, close_fds=False) output = proc.communicate()[0] if output: LOG.debug(output)
@@ -280,7 +303,8 @@ class QEMUMachine(object): stdin=devnull, stdout=self._qemu_log_file, stderr=subprocess.STDOUT, - shell=False) + shell=False, + close_fds=False) self._post_launch() def wait(self):
tests/qemu-iotests/045 +1 -1
@@ -140,7 +140,7 @@ class TestSCMFd(iotests.QMPTestCase): os.remove(image0) def _send_fd_by_SCM(self): - ret = self.vm.send_fd_scm(image0) + ret = self.vm.send_fd_scm(file_path=image0) self.assertEqual(ret, 0, 'Failed to send fd with UNIX SCM') def test_add_fd(self):
tests/qemu-iotests/147 +1 -1
@@ -229,7 +229,7 @@ class BuiltinNBD(NBDBlockdevAddBase): sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sockfd.connect(unix_socket) - result = self.vm.send_fd_scm(str(sockfd.fileno())) + result = self.vm.send_fd_scm(fd=sockfd.fileno()) self.assertEqual(result, 0, 'Failed to send socket FD') result = self.vm.qmp('getfd', fdname='nbd-fifo')
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 6/9] iotests: Explicitly inherit FDs in Python
Quoted 42 lines · Max Reitz
Python 3.4 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if
close_fdsis explicitly set to False.Adhere to this by setting
close_fdsto False when working with subprocesses that may want to inherit FDs, and by trying toset_inheritable()on FDs that we do want to bequeath to them.Signed-off-by Max Reitz <[email protected]>--- scripts/qemu.py | 34 +++++++++++++++++++++++++++++----- tests/qemu-iotests/045 | 2 +- tests/qemu-iotests/147 | 2 +- 3 files changed, 31 insertions(+), 7 deletions(-)scripts/qemu.py +9
-1@@ -142,11 +142,19 @@ class QEMUMachine(object): if opts: options.append(opts) + # This did not exist before 3.4, but since then it is + # mandatory for our purpose + if hasattr(os, 'set_inheritable'): + os.set_inheritable(fd, True) + self._args.append('-add-fd') self._args.append(','.join(options)) return self - def send_fd_scm(self, fd_file_path): + # Exactly one of fd and file_path must be given. + # (If it is file_path, the helper will open that file and pass its + # own fd) + def send_fd_scm(self, fd=None, file_path=None): # In iotest.py, the qmp should always use unix socket. assert self._qmp.is_scm_available() if self._socket_scm_helper is None:
scripts/qemu.py class QEMUMachine(object): · line 170
Quoted 9 lines
scripts/qemu.py +6
-0@@ -154,12 +162,27 @@ class QEMUMachine(object): if not os.path.exists(self._socket_scm_helper): raise QEMUMachineError("%s does not exist" % self._socket_scm_helper) + + # This did not exist before 3.4, but since then it is + # mandatory for our purpose + if hasattr(os, 'set_inheritable'): + os.set_inheritable(self._qmp.get_sock_fd(), True) + if fd is not None:
I was going to suggest keeping the existing function parameter, and using:
isinstance(fd_file_path, int)
But your solution makes callers more explicit. This seems to be a good thing.
Quoted 60 lines
scripts/qemu.py +14
-4+ os.set_inheritable(fd, True) + fd_param = ["%s" % self._socket_scm_helper, - "%d" % self._qmp.get_sock_fd(), - "%s" % fd_file_path] + "%d" % self._qmp.get_sock_fd()] + + if file_path is not None: + assert fd is None + fd_param.append(file_path) + else: + assert fd is not None + fd_param.append(str(fd)) + devnull = open(os.path.devnull, 'rb') proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, close_fds=False) output = proc.communicate()[0] if output: LOG.debug(output)@@ -280,7 +303,8 @@ class QEMUMachine(object): stdin=devnull, stdout=self._qemu_log_file, stderr=subprocess.STDOUT, - shell=False) + shell=False, + close_fds=False) self._post_launch() def wait(self):tests/qemu-iotests/045 +1
-1@@ -140,7 +140,7 @@ class TestSCMFd(iotests.QMPTestCase): os.remove(image0) def _send_fd_by_SCM(self): - ret = self.vm.send_fd_scm(image0) + ret = self.vm.send_fd_scm(file_path=image0) self.assertEqual(ret, 0, 'Failed to send fd with UNIX SCM') def test_add_fd(self):tests/qemu-iotests/147 +1
-1@@ -229,7 +229,7 @@ class BuiltinNBD(NBDBlockdevAddBase): sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sockfd.connect(unix_socket) - result = self.vm.send_fd_scm(str(sockfd.fileno())) + result = self.vm.send_fd_scm(fd=sockfd.fileno()) self.assertEqual(result, 0, 'Failed to send socket FD') result = self.vm.qmp('getfd', fdname='nbd-fifo')Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH v2 6/9] iotests: Explicitly inherit FDs in Python
In reply to Max ReitzQuoted 10 lines · Max Reitz
Python 3.4 introduced the inheritable attribute for FDs. At the same time, it changed the default so that all FDs are not inheritable by default, that only inheritable FDs are inherited to subprocesses, and only if
close_fdsis explicitly set to False.Adhere to this by setting
close_fdsto False when working with subprocesses that may want to inherit FDs, and by trying toset_inheritable()on FDs that we do want to bequeath to them.Signed-off-by Max Reitz <[email protected]>
PATCH v2 7/9 iotests: 'new' module replacement in 169
In reply to Max Reitziotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
--- tests/qemu-iotests/169 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
tests/qemu-iotests/169 +1 -2
@@ -23,7 +23,6 @@ import iotests import time import itertools import operator -import new from iotests import qemu_img
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): def inject_test_case(klass, name, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass)) + setattr(klass, 'test_' + name, lambda self: mc(self)) for cmb in list(itertools.product((True, False), repeat=4)): name = ('_' if cmb[0] else '_not_') + 'persistent_'
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 7/9] iotests: 'new' module replacement in 169
Quoted 6 lines · Max Reitz
iotest 169 uses the 'new' module to add methods to a class. This module no longer exists in Python 3. Instead, we can use a lambda. Best of all, this works in 2.7 just as well.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>
PATCH v2 8/9 iotests: Modify imports for Python 3
In reply to Max ReitzThere are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there (and use the BytesIO class for Python 2). The other is the ConfigParser, which has just been renamed to configparser.
--- tests/qemu-iotests/iotests.py | 13 +++++++++---- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 14 insertions(+), 6 deletions(-)
tests/qemu-iotests/iotests.py +9 -4
@@ -29,6 +29,7 @@ import json import signal import logging import atexit +import io sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts')) import qtest
@@ -681,15 +682,19 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], verify_platform(supported_oses) verify_cache_mode(supported_cache_modes) - # We need to filter out the time taken from the output so that qemu-iotest - # can reliably diff the results against master output. - import StringIO if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + # We need to filter out the time taken from the output so that + # qemu-iotest can reliably diff the results against master output. + if sys.version_info.major >= 3: + output = io.StringIO() + else: + # StringIO() is for unicode strings, which is not what + # 2.x's test runner emits. + output = io.BytesIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
tests/qemu-iotests/nbd-fault-injector.py +5 -2
@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 8/9] iotests: Modify imports for Python 3
Quoted 33 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there (and use the BytesIO class for Python 2). The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/iotests.py | 13 +++++++++---- tests/qemu-iotests/nbd-fault-injector.py | 7 +++++-- 2 files changed, 14 insertions(+), 6 deletions(-)tests/qemu-iotests/iotests.py +1
-3@@ -29,6 +29,7 @@ import json import signal import logging import atexit +import io sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts')) import qtest@@ -681,15 +682,19 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], verify_platform(supported_oses) verify_cache_mode(supported_cache_modes) - # We need to filter out the time taken from the output so that qemu-iotest - # can reliably diff the results against master output. - import StringIO if debug: output = sys.stdout
tests/qemu-iotests/iotests.py def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], · line 695
Quoted 10 lines
tests/qemu-iotests/iotests.py +6
-1verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + # We need to filter out the time taken from the output so that + # qemu-iotest can reliably diff the results against master output. + if sys.version_info.major >= 3: + output = io.StringIO() + else: + # StringIO() is for unicode strings, which is not what
Nit: I would change the comment to say "io.StringIO" instead of "StringIO", to avoid confusion with StringIO.StringIO.
Not a big deal, so:
Quoted 33 lines
tests/qemu-iotests/iotests.py +2
-0+ # 2.x's test runner emits. + output = io.BytesIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))tests/qemu-iotests/nbd-fault-injector.py +5
-2@@ -48,7 +48,10 @@ import sys import socket import struct import collections -import ConfigParser +if sys.version_info.major >= 3: + import configparser +else: + import ConfigParser as configparser FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB@@ -225,7 +228,7 @@ def parse_config(config): return rules def load_rules(filename): - config = ConfigParser.RawConfigParser() + config = configparser.RawConfigParser() with open(filename, 'rt') as f: config.readfp(f, filename) return parse_config(config)Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH v2 8/9] iotests: Modify imports for Python 3
In reply to Max ReitzQuoted 7 lines · Max Reitz
There are two imports that need to be modified when running the iotests under Python 3: One is StringIO, which no longer exists; instead, the StringIO class comes from the io module, so import it from there (and use the BytesIO class for Python 2). The other is the ConfigParser, which has just been renamed to configparser.
Signed-off-by Max Reitz <[email protected]>
PATCH v2 9/9 iotests: Unify log outputs between Python 2 and 3
In reply to Max ReitzWhen dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). Second, the order of keys in dicts may differ. Third, especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.
We can get around all of these differences by dumping objects (lists and dicts) in a language-independent format, namely JSON. The JSON generator even allows emitting dicts with their keys sorted alphabetically.
This changes the output of all tests that use these logging functions (dict keys are ordered now, strings in dicts are now enclosed in double quotes instead of single quotes, the 'L' suffix of large integers is dropped, and "true" and "false" are now in lower case).
The quote change necessitates a small change to a filter used in test 207.
--- tests/qemu-iotests/194.out | 22 +- tests/qemu-iotests/202.out | 12 +- tests/qemu-iotests/203.out | 14 +- tests/qemu-iotests/206.out | 218 +++++++------- tests/qemu-iotests/207 | 2 +- tests/qemu-iotests/207.out | 72 ++--- tests/qemu-iotests/208.out | 8 +- tests/qemu-iotests/210.out | 94 +++--- tests/qemu-iotests/211.out | 102 +++---- tests/qemu-iotests/212.out | 174 +++++------ tests/qemu-iotests/213.out | 182 ++++++------ tests/qemu-iotests/216.out | 4 +- tests/qemu-iotests/218.out | 20 +- tests/qemu-iotests/219.out | 526 +++++++++++++++++----------------- tests/qemu-iotests/222.out | 24 +- tests/qemu-iotests/iotests.py | 10 +- 16 files changed, 744 insertions(+), 740 deletions(-)
tests/qemu-iotests/194.out +11 -11
@@ -1,18 +1,18 @@ Launching VMs... Launching NBD server on destination... -{u'return': {}} -{u'return': {}} +{"return": {}} +{"return": {}} Starting `drive-mirror` on source... -{u'return': {}} +{"return": {}} Waiting for `drive-mirror` to complete... -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'} +{"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} Starting migration... -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'} +{"return": {}} +{"data": {"status": "setup"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"data": {"status": "active"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} Gracefully ending the `drive-mirror` job on source... -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'} +{"return": {}} +{"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} Stopping the NBD server on destination... -{u'return': {}} +{"return": {}}
tests/qemu-iotests/202.out +6 -6
@@ -1,11 +1,11 @@ Launching VM... Adding IOThread... -{u'return': {}} +{"return": {}} Adding blockdevs... -{u'return': {}} -{u'return': {}} +{"return": {}} +{"return": {}} Setting iothread... -{u'return': {}} -{u'return': {}} +{"return": {}} +{"return": {}} Creating external snapshots... -{u'return': {}} +{"return": {}}
tests/qemu-iotests/203.out +7 -7
@@ -1,11 +1,11 @@ Launching VM... Setting IOThreads... -{u'return': {}} -{u'return': {}} +{"return": {}} +{"return": {}} Enabling migration QMP events... -{u'return': {}} +{"return": {}} Starting migration... -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'} +{"return": {}} +{"data": {"status": "setup"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"data": {"status": "active"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
tests/qemu-iotests/206.out +109 -109
293 lines not on this page · raw message
tests/qemu-iotests/207 +1 -1
@@ -28,7 +28,7 @@ iotests.verify_image_format(supported_fmts=['raw']) iotests.verify_protocol(supported=['ssh']) def filter_hash(msg): - return re.sub("'hash': '[0-9a-f]+'", "'hash': HASH", msg) + return re.sub('"hash": "[0-9a-f]+"', '"hash": HASH', msg) def blockdev_create(vm, options): result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
tests/qemu-iotests/207.out +36 -36
109 lines not on this page · raw message
tests/qemu-iotests/208.out +4 -4
@@ -1,9 +1,9 @@ Launching VM... Starting NBD server... -{u'return': {}} +{"return": {}} Adding NBD export... -{u'return': {}} +{"return": {}} Creating external snapshot... -{u'return': {}} +{"return": {}} Stopping NBD server... -{u'return': {}} +{"return": {}}
tests/qemu-iotests/210.out +47 -47
130 lines not on this page · raw message
tests/qemu-iotests/211.out +51 -51
146 lines not on this page · raw message
tests/qemu-iotests/212.out +87 -87
238 lines not on this page · raw message
tests/qemu-iotests/213.out +91 -91
250 lines not on this page · raw message
tests/qemu-iotests/216.out +2 -2
@@ -7,8 +7,8 @@ Done --- Doing COR --- -{u'return': {}} -{u'return': u''} +{"return": {}} +{"return": ""} --- Checking COR result ---
tests/qemu-iotests/218.out +10 -10
@@ -4,27 +4,27 @@ --- force=false --- Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'} +{"return": {}} +{"data": {"device": "mirror", "len": 1048576, "offset": 65536, "speed": 65536, "type": "mirror"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} --- force=true --- Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'} +{"return": {}} +{"data": {"device": "mirror", "len": 1048576, "offset": 65536, "speed": 65536, "type": "mirror"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} === Cancel mirror job after convergence === --- force=false --- -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'} +{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_COMPLETED'} +{"return": {}} +{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} --- force=true --- -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'} +{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} Cancelling job -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_CANCELLED'} +{"return": {}} +{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
tests/qemu-iotests/219.out +263 -263
589 lines not on this page · raw message
tests/qemu-iotests/222.out +12 -12
@@ -8,13 +8,13 @@ Done --- Setting up Fleecing Graph --- -{u'return': {}} -{u'return': {}} +{"return": {}} +{"return": {}} --- Setting up NBD Export --- -{u'return': {}} -{u'return': {}} +{"return": {}} +{"return": {}} --- Sanity Check ---
@@ -29,13 +29,13 @@ read -P0 0x3fe0000 64k --- Testing COW --- write -P0xab 0 64k -{u'return': u''} +{"return": ""} write -P0xad 0x00f8000 64k -{u'return': u''} +{"return": ""} write -P0x1d 0x2008000 64k -{u'return': u''} +{"return": ""} write -P0xea 0x3fe0000 64k -{u'return': u''} +{"return": ""} --- Verifying Data ---
@@ -49,10 +49,10 @@ read -P0 0x3fe0000 64k --- Cleanup --- -{u'return': {}} -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'drive0', u'type': u'backup', u'speed': 0, u'len': 67108864, u'offset': 393216}, u'event': u'BLOCK_JOB_CANCELLED'} -{u'return': {}} -{u'return': {}} +{"return": {}} +{"data": {"device": "drive0", "len": 67108864, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"return": {}} +{"return": {}} --- Confirming writes ---
tests/qemu-iotests/iotests.py +7 -3
@@ -254,7 +254,10 @@ def filter_img_info(output, filename): def log(msg, filters=[]): for flt in filters: msg = flt(msg) - print(msg) + if type(msg) is dict or type(msg) is list: + print(json.dumps(msg, sort_keys=True)) + else: + print(msg) class Timeout: def __init__(self, seconds, errmsg = "Timeout"):
@@ -442,10 +445,11 @@ class VM(qtest.QEMUQtestMachine): return result def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs): - logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs) + logmsg = '{"execute": "%s", "arguments": %s}' % \ + (cmd, json.dumps(kwargs, sort_keys=True)) log(logmsg, filters) result = self.qmp(cmd, **kwargs) - log(str(result), filters) + log(json.dumps(result, sort_keys=True), filters) return result def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 9/9] iotests: Unify log outputs between Python 2 and 3
Quoted 19 lines · Max Reitz
When dumping an object into the log, there are differences between Python 2 and 3. First, unicode strings are prefixed by 'u' in Python 2 (they are no longer in 3, because unicode strings are the default there). Second, the order of keys in dicts may differ. Third, especially long numbers are longs in Python 2 and thus get an 'L' suffix, which does not happen in Python 3.
We can get around all of these differences by dumping objects (lists and dicts) in a language-independent format, namely JSON. The JSON generator even allows emitting dicts with their keys sorted alphabetically.
This changes the output of all tests that use these logging functions (dict keys are ordered now, strings in dicts are now enclosed in double quotes instead of single quotes, the 'L' suffix of large integers is dropped, and "true" and "false" are now in lower case).
The quote change necessitates a small change to a filter used in test 207.
Quoted 2 lines
Suggested-by Eduardo Habkost <[email protected]>Signed-off-by Max Reitz <[email protected]>
Quoted 6 lines · Max Reitz
This series prepares the iotests to work with both Python 2 and 3. In some places, it adds version-specific code and decides what to do based on the version (for instance, whether to import the StringIO or the BytesIO class from 'io' for use with the test runner), but most of the time, it just makes code work for both versions in general.
Tested on both Python 2 and 3.