iotests: Dump QCOW2 dirty bitmaps metadata
v5 cover iotests: Dump QCOW2 dirty bitmaps metadata
Hi all!
Here is my suggestion to substitute only first three patches :) of Andrey's [PATCH v3 0/6] iotests: Dump QCOW2 dirty bitmaps metadata
v5:
01: - typo in commit message
- fix over-79 lines
02: new, specify GPL license
03: specify GPL license
04-06: add Andrey's r-b
07: - grammar in commit message
- add Andrey's r-b
08: - add copyright
- note u8 and u16 in a comment
- add doc for __init__ of Qcow2Struct
- add Andrey's r-b
09-10: add Andrey's r-b
11: - note about current fd position in __init__ doc of
QcowHeaderExtension
- add Andrey's r-b
12: - keep magic in hex
- use lower-case for magic constants
13: add Andrey's r-b
Tags up-refactor-qcow2.py-v4 and up-refactor-qcow2.py-v4 may be found
here: https://src.openvz.org/users/vsementsov/repos/qemu/browse
(git remote: https://src.openvz.org/scm/~vsementsov/qemu.git)
Plan for continuation:
- add type hints
- support compression_type field
- dump more about bitmaps
Vladimir Sementsov-Ogievskiy (13):
qcow2.py: python style fixes
qcow2.py: add licensing blurb
qcow2.py: move qcow2 format classes to separate module qcow2_format.py: drop new line printing at end of dump() qcow2_format.py: use tuples instead of lists for fields qcow2_format.py: use modern string formatting qcow2_format.py: use strings to specify c-type of struct fields qcow2_format.py: separate generic functionality of structure classes qcow2_format.py: add field-formatting class qcow2_format.py: QcowHeaderExtension: add dump method
qcow2_format: refactor QcowHeaderExtension as a subclass of
Qcow2Struct
qcow2: QcowHeaderExtension print names for extension magics qcow2_format.py: dump bitmaps header extension
tests/qemu-iotests/031.out | 22 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/061.out | 14 +- tests/qemu-iotests/291 | 4 + tests/qemu-iotests/291.out | 33 ++++ tests/qemu-iotests/qcow2.py | 218 +++++----------------- tests/qemu-iotests/qcow2_format.py | 286 +++++++++++++++++++++++++++++ 7 files changed, 393 insertions(+), 188 deletions(-) create mode 100644 tests/qemu-iotests/qcow2_format.py
Signature · 1 line
2.21.0
PATCH v5 1/13 qcow2.py: python style fixes
Fix flake8 complaints. Leave the only chunk of lines over 79 characters:
initialization of cmds variable. Leave it for another day, when it should be refactored to utilize argparse instead of hand-written parsing.
--- tests/qemu-iotests/qcow2.py | 95 ++++++++++++++++++++++--------------- 1 file changed, 56 insertions(+), 39 deletions(-)
tests/qemu-iotests/qcow2.py +56 -39
@@ -4,6 +4,7 @@ import sys import struct import string + class QcowHeaderExtension: def __init__(self, magic, length, data):
@@ -11,14 +12,15 @@ class QcowHeaderExtension: padding = 8 - (length % 8) data += b"\0" * padding - self.magic = magic + self.magic = magic self.length = length - self.data = data + self.data = data @classmethod def create(cls, magic, data): return QcowHeaderExtension(magic, len(data), data) + class QcowHeader: uint32_t = 'I'
@@ -26,27 +28,27 @@ class QcowHeader: fields = [ # Version 2 header fields - [ uint32_t, '%#x', 'magic' ], - [ uint32_t, '%d', 'version' ], - [ uint64_t, '%#x', 'backing_file_offset' ], - [ uint32_t, '%#x', 'backing_file_size' ], - [ uint32_t, '%d', 'cluster_bits' ], - [ uint64_t, '%d', 'size' ], - [ uint32_t, '%d', 'crypt_method' ], - [ uint32_t, '%d', 'l1_size' ], - [ uint64_t, '%#x', 'l1_table_offset' ], - [ uint64_t, '%#x', 'refcount_table_offset' ], - [ uint32_t, '%d', 'refcount_table_clusters' ], - [ uint32_t, '%d', 'nb_snapshots' ], - [ uint64_t, '%#x', 'snapshot_offset' ], + [uint32_t, '%#x', 'magic'], + [uint32_t, '%d', 'version'], + [uint64_t, '%#x', 'backing_file_offset'], + [uint32_t, '%#x', 'backing_file_size'], + [uint32_t, '%d', 'cluster_bits'], + [uint64_t, '%d', 'size'], + [uint32_t, '%d', 'crypt_method'], + [uint32_t, '%d', 'l1_size'], + [uint64_t, '%#x', 'l1_table_offset'], + [uint64_t, '%#x', 'refcount_table_offset'], + [uint32_t, '%d', 'refcount_table_clusters'], + [uint32_t, '%d', 'nb_snapshots'], + [uint64_t, '%#x', 'snapshot_offset'], # Version 3 header fields - [ uint64_t, 'mask', 'incompatible_features' ], - [ uint64_t, 'mask', 'compatible_features' ], - [ uint64_t, 'mask', 'autoclear_features' ], - [ uint32_t, '%d', 'refcount_order' ], - [ uint32_t, '%d', 'header_length' ], - ]; + [uint64_t, 'mask', 'incompatible_features'], + [uint64_t, 'mask', 'compatible_features'], + [uint64_t, 'mask', 'autoclear_features'], + [uint32_t, '%d', 'refcount_order'], + [uint32_t, '%d', 'header_length'], + ] fmt = '>' + ''.join(field[0] for field in fields)
@@ -59,7 +61,7 @@ class QcowHeader: header = struct.unpack(QcowHeader.fmt, buf) self.__dict__ = dict((field[2], header[i]) - for i, field in enumerate(QcowHeader.fields)) + for i, field in enumerate(QcowHeader.fields)) self.set_defaults() self.cluster_size = 1 << self.cluster_bits
@@ -96,7 +98,8 @@ class QcowHeader: else: padded = (length + 7) & ~7 data = fd.read(padded) - self.extensions.append(QcowHeaderExtension(magic, length, data)) + self.extensions.append(QcowHeaderExtension(magic, length, + data)) def update_extensions(self, fd):
@@ -108,14 +111,13 @@ class QcowHeader: fd.write(buf) fd.write(ex.data) - if self.backing_file != None: + if self.backing_file is not None: self.backing_file_offset = fd.tell() fd.write(self.backing_file) if fd.tell() > self.cluster_size: raise Exception("I think I just broke the image...") - def update(self, fd): header_bytes = self.header_length
@@ -163,19 +165,21 @@ def cmd_dump_header(fd): h.dump() h.dump_extensions() + def cmd_dump_header_exts(fd): h = QcowHeader(fd) h.dump_extensions() + def cmd_set_header(fd, name, value): try: value = int(value, 0) - except: + except ValueError: print("'%s' is not a valid number" % value) sys.exit(1) fields = (field[2] for field in QcowHeader.fields) - if not name in fields: + if name not in fields: print("'%s' is not a known header field" % name) sys.exit(1)
@@ -183,25 +187,29 @@ def cmd_set_header(fd, name, value): h.__dict__[name] = value h.update(fd) + def cmd_add_header_ext(fd, magic, data): try: magic = int(magic, 0) - except: + except ValueError: print("'%s' is not a valid magic number" % magic) sys.exit(1) h = QcowHeader(fd) - h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii'))) + h.extensions.append(QcowHeaderExtension.create(magic, + data.encode('ascii'))) h.update(fd) + def cmd_add_header_ext_stdio(fd, magic): data = sys.stdin.read() cmd_add_header_ext(fd, magic, data) + def cmd_del_header_ext(fd, magic): try: magic = int(magic, 0) - except: + except ValueError: print("'%s' is not a valid magic number" % magic) sys.exit(1)
@@ -219,12 +227,13 @@ def cmd_del_header_ext(fd, magic): h.update(fd) + def cmd_set_feature_bit(fd, group, bit): try: bit = int(bit, 0) if bit < 0 or bit >= 64: raise ValueError - except: + except ValueError: print("'%s' is not a valid bit number in range [0, 64)" % bit) sys.exit(1)
@@ -236,21 +245,27 @@ def cmd_set_feature_bit(fd, group, bit): elif group == 'autoclear': h.autoclear_features |= 1 << bit else: - print("'%s' is not a valid group, try 'incompatible', 'compatible', or 'autoclear'" % group) + print("'%s' is not a valid group, try " + "'incompatible', 'compatible', or 'autoclear'" % group) sys.exit(1) h.update(fd) + cmds = [ - [ 'dump-header', cmd_dump_header, 0, 'Dump image header and header extensions' ], - [ 'dump-header-exts', cmd_dump_header_exts, 0, 'Dump image header extensions' ], - [ 'set-header', cmd_set_header, 2, 'Set a field in the header'], - [ 'add-header-ext', cmd_add_header_ext, 2, 'Add a header extension' ], - [ 'add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin' ], - [ 'del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension' ], - [ 'set-feature-bit', cmd_set_feature_bit, 2, 'Set a feature bit'], + ['dump-header', cmd_dump_header, 0, + 'Dump image header and header extensions'], + ['dump-header-exts', cmd_dump_header_exts, 0, + 'Dump image header extensions'], + ['set-header', cmd_set_header, 2, 'Set a field in the header'], + ['add-header-ext', cmd_add_header_ext, 2, 'Add a header extension'], + ['add-header-ext-stdio', cmd_add_header_ext_stdio, 1, + 'Add a header extension, data from stdin'], + ['del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension'], + ['set-feature-bit', cmd_set_feature_bit, 2, 'Set a feature bit'], ] + def main(filename, cmd, args): fd = open(filename, "r+b") try:
@@ -267,6 +282,7 @@ def main(filename, cmd, args): finally: fd.close() + def usage(): print("Usage: %s <file> <cmd> [<arg>, ...]" % sys.argv[0]) print("")
@@ -274,6 +290,7 @@ def usage(): for name, handler, num_args, desc in cmds: print(" %-20s - %s" % (name, desc)) + if __name__ == '__main__': if len(sys.argv) < 3: usage()
Signature · 1 line
2.21.0
Re: [PATCH v5 01/13] qcow2.py: python style fixes
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Fix flake8 complaints. Leave the only chunk of lines over 79 characters:
Stale sentence since you fixed that; on my Fedora 32 machine, with flake8 3.7.7, this patch silences all warnings.
Quoted 8 lines
initialization of cmds variable. Leave it for another day, when it should be refactored to utilize argparse instead of hand-written parsing.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/qcow2.py | 95 ++++++++++++++++++++++--------------- 1 file changed, 56 insertions(+), 39 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v5 2/13 qcow2.py: add licensing blurb
In reply to Vladimir Sementsov-OgievskiyAdd classic heading, which is missing here. Keep copyright place empty, for anyone who have added (or will add) some intellectual property here.
--- tests/qemu-iotests/qcow2.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
tests/qemu-iotests/qcow2.py +16 -0
@@ -1,4 +1,20 @@ #!/usr/bin/env python3 +# +# Manipulations with qcow2 image +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# import sys import struct
Signature · 1 line
2.21.0
Re: [PATCH v5 02/13] qcow2.py: add licensing blurb
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Add classic heading, which is missing here. Keep copyright place empty, for anyone who have added (or will add) some intellectual property here.
It's not so much intellectual property (since that term is at odds with open source), but authorship rights.
Looking at git history, the file has been touched by:
Kevin Wolf
Stefan Hajnoczi (while at IBM)
Eduardo Habkost
Max Reitz
Philippe Mathieu-Daudé
Paolo Bonzini
where Stefan was the only contributor without a redhat.com address at the time. So if anything, a Red Hat copyright is most likely; but you are also correct that it is incorrect to add a copyright line on someone else's behalf without their permission.
Quoted 11 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/qcow2.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)tests/qemu-iotests/qcow2.py +0
-0
tests/qemu-iotests/qcow2.py line 4
Quoted 4 lines
tests/qemu-iotests/qcow2.py +3
-0@@ -1,4 +1,20 @@ #!/usr/bin/env python3 +# +# Manipulations with qcow2 image +#
I've cc'd all prior authors; if Kevin agrees, and unless anyone speaks up to the contrary, I'm willing to add:
# Copyright (C) 2012 Red Hat, Inc.
for Kevin's initial contribution, without worrying about subsequent contributions.
Quoted 7 lines
tests/qemu-iotests/qcow2.py +6
-0+# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful,
tests/qemu-iotests/qcow2.py line 20
Quoted 10 lines
tests/qemu-iotests/qcow2.py +7
-0+# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# import sys import struct
Adding a copyright line could be a followup patch, so in the meantime, making what was previously an implicit license now explicit is fine even if it is odd to assert GPL without also asserting Copyright.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v5 3/13 qcow2.py: move qcow2 format classes to separate module
In reply to Vladimir Sementsov-OgievskiyWe are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code.
--- tests/qemu-iotests/qcow2.py | 160 +------------------------- tests/qemu-iotests/qcow2_format.py | 173 +++++++++++++++++++++++++++++ 2 files changed, 177 insertions(+), 156 deletions(-) create mode 100644 tests/qemu-iotests/qcow2_format.py
tests/qemu-iotests/qcow2.py +4 -156
@@ -17,163 +17,11 @@ # import sys -import struct -import string - - -class QcowHeaderExtension: - - def __init__(self, magic, length, data): - if length % 8 != 0: - padding = 8 - (length % 8) - data += b"\0" * padding - - self.magic = magic - self.length = length - self.data = data - - @classmethod - def create(cls, magic, data): - return QcowHeaderExtension(magic, len(data), data) - - -class QcowHeader: - - uint32_t = 'I' - uint64_t = 'Q' - - fields = [ - # Version 2 header fields - [uint32_t, '%#x', 'magic'], - [uint32_t, '%d', 'version'], - [uint64_t, '%#x', 'backing_file_offset'], - [uint32_t, '%#x', 'backing_file_size'], - [uint32_t, '%d', 'cluster_bits'], - [uint64_t, '%d', 'size'], - [uint32_t, '%d', 'crypt_method'], - [uint32_t, '%d', 'l1_size'], - [uint64_t, '%#x', 'l1_table_offset'], - [uint64_t, '%#x', 'refcount_table_offset'], - [uint32_t, '%d', 'refcount_table_clusters'], - [uint32_t, '%d', 'nb_snapshots'], - [uint64_t, '%#x', 'snapshot_offset'], - - # Version 3 header fields - [uint64_t, 'mask', 'incompatible_features'], - [uint64_t, 'mask', 'compatible_features'], - [uint64_t, 'mask', 'autoclear_features'], - [uint32_t, '%d', 'refcount_order'], - [uint32_t, '%d', 'header_length'], - ] - - fmt = '>' + ''.join(field[0] for field in fields) - - def __init__(self, fd): - - buf_size = struct.calcsize(QcowHeader.fmt) - - fd.seek(0) - buf = fd.read(buf_size) - - header = struct.unpack(QcowHeader.fmt, buf) - self.__dict__ = dict((field[2], header[i]) - for i, field in enumerate(QcowHeader.fields)) - - self.set_defaults() - self.cluster_size = 1 << self.cluster_bits - - fd.seek(self.header_length) - self.load_extensions(fd) - - if self.backing_file_offset: - fd.seek(self.backing_file_offset) - self.backing_file = fd.read(self.backing_file_size) - else: - self.backing_file = None - - def set_defaults(self): - if self.version == 2: - self.incompatible_features = 0 - self.compatible_features = 0 - self.autoclear_features = 0 - self.refcount_order = 4 - self.header_length = 72 - - def load_extensions(self, fd): - self.extensions = [] - - if self.backing_file_offset != 0: - end = min(self.cluster_size, self.backing_file_offset) - else: - end = self.cluster_size - - while fd.tell() < end: - (magic, length) = struct.unpack('>II', fd.read(8)) - if magic == 0: - break - else: - padded = (length + 7) & ~7 - data = fd.read(padded) - self.extensions.append(QcowHeaderExtension(magic, length, - data)) - - def update_extensions(self, fd): - - fd.seek(self.header_length) - extensions = self.extensions - extensions.append(QcowHeaderExtension(0, 0, b"")) - for ex in extensions: - buf = struct.pack('>II', ex.magic, ex.length) - fd.write(buf) - fd.write(ex.data) - - if self.backing_file is not None: - self.backing_file_offset = fd.tell() - fd.write(self.backing_file) - - if fd.tell() > self.cluster_size: - raise Exception("I think I just broke the image...") - - def update(self, fd): - header_bytes = self.header_length - - self.update_extensions(fd) - - fd.seek(0) - header = tuple(self.__dict__[f] for t, p, f in QcowHeader.fields) - buf = struct.pack(QcowHeader.fmt, *header) - buf = buf[0:header_bytes-1] - fd.write(buf) - - def dump(self): - for f in QcowHeader.fields: - value = self.__dict__[f[2]] - if f[1] == 'mask': - bits = [] - for bit in range(64): - if value & (1 << bit): - bits.append(bit) - value_str = str(bits) - else: - value_str = f[1] % value - - print("%-25s" % f[2], value_str) - print("") - - def dump_extensions(self): - for ex in self.extensions: - - data = ex.data[:ex.length] - if all(c in string.printable.encode('ascii') for c in data): - data = "'%s'" % data.decode('ascii') - else: - data = "<binary>" - print("Header extension:") - print("%-25s %#x" % ("magic", ex.magic)) - print("%-25s %d" % ("length", ex.length)) - print("%-25s %s" % ("data", data)) - print("") +from qcow2_format import ( + QcowHeader, + QcowHeaderExtension +) def cmd_dump_header(fd):
tests/qemu-iotests/qcow2_format.py new file +173 -0
173 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH v5 03/13] qcow2.py: move qcow2 format classes to separate module
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
We are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/qcow2.py | 160 +------------------------- tests/qemu-iotests/qcow2_format.py | 173 +++++++++++++++++++++++++++++ 2 files changed, 177 insertions(+), 156 deletions(-)create mode 100644
tests/qemu-iotests/qcow2_format.py+++
b/tests/qemu-iotests/qcow2_format.py
Review line 2
Quoted 2 lines
(quoted diff) +2
-0@@ -0,0 +1,173 @@ +# Library for manipulations with qcow2 image +#
Any copyright line added in 2/13 should also be added here. Again, that can be a followup for authorship reasons if we decide what copyright line is best.
Otherwise, this is a clean code motion patch where the difference in line length is the boilerplate header and the change to imports.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v5 4/13 qcow2_format.py: drop new line printing at end of dump()
In reply to Vladimir Sementsov-OgievskiyThis will simplify further conversion. To compensate, print this empty line directly in cmd_dump_header().
--- tests/qemu-iotests/qcow2.py | 1 + tests/qemu-iotests/qcow2_format.py | 1 - 2 files changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/qcow2.py +1 -0
@@ -27,6 +27,7 @@ from qcow2_format import ( def cmd_dump_header(fd): h = QcowHeader(fd) h.dump() + print() h.dump_extensions()
tests/qemu-iotests/qcow2_format.py +0 -1
@@ -155,7 +155,6 @@ class QcowHeader: value_str = f[1] % value print("%-25s" % f[2], value_str) - print("") def dump_extensions(self): for ex in self.extensions:
Signature · 1 line
2.21.0
PATCH v5 5/13 qcow2_format.py: use tuples instead of lists for fields
In reply to Vladimir Sementsov-OgievskiyNo need in lists: it's a constant variable.
--- tests/qemu-iotests/qcow2_format.py | 40 +++++++++++++++--------------- 1 file changed, 20 insertions(+), 20 deletions(-)
tests/qemu-iotests/qcow2_format.py +20 -20
@@ -39,29 +39,29 @@ class QcowHeader: uint32_t = 'I' uint64_t = 'Q' - fields = [ + fields = ( # Version 2 header fields - [uint32_t, '%#x', 'magic'], - [uint32_t, '%d', 'version'], - [uint64_t, '%#x', 'backing_file_offset'], - [uint32_t, '%#x', 'backing_file_size'], - [uint32_t, '%d', 'cluster_bits'], - [uint64_t, '%d', 'size'], - [uint32_t, '%d', 'crypt_method'], - [uint32_t, '%d', 'l1_size'], - [uint64_t, '%#x', 'l1_table_offset'], - [uint64_t, '%#x', 'refcount_table_offset'], - [uint32_t, '%d', 'refcount_table_clusters'], - [uint32_t, '%d', 'nb_snapshots'], - [uint64_t, '%#x', 'snapshot_offset'], + (uint32_t, '%#x', 'magic'), + (uint32_t, '%d', 'version'), + (uint64_t, '%#x', 'backing_file_offset'), + (uint32_t, '%#x', 'backing_file_size'), + (uint32_t, '%d', 'cluster_bits'), + (uint64_t, '%d', 'size'), + (uint32_t, '%d', 'crypt_method'), + (uint32_t, '%d', 'l1_size'), + (uint64_t, '%#x', 'l1_table_offset'), + (uint64_t, '%#x', 'refcount_table_offset'), + (uint32_t, '%d', 'refcount_table_clusters'), + (uint32_t, '%d', 'nb_snapshots'), + (uint64_t, '%#x', 'snapshot_offset'), # Version 3 header fields - [uint64_t, 'mask', 'incompatible_features'], - [uint64_t, 'mask', 'compatible_features'], - [uint64_t, 'mask', 'autoclear_features'], - [uint32_t, '%d', 'refcount_order'], - [uint32_t, '%d', 'header_length'], - ] + (uint64_t, 'mask', 'incompatible_features'), + (uint64_t, 'mask', 'compatible_features'), + (uint64_t, 'mask', 'autoclear_features'), + (uint32_t, '%d', 'refcount_order'), + (uint32_t, '%d', 'header_length'), + ) fmt = '>' + ''.join(field[0] for field in fields)
Signature · 1 line
2.21.0
PATCH v5 6/13 qcow2_format.py: use modern string formatting
In reply to Vladimir Sementsov-OgievskiyUse .format and f-strings instead of old %style. Also, the file uses both '' and "" quotes, for consistency let's use '', except for cases when we need '' inside the string (use "" to avoid extra escaping).
--- tests/qemu-iotests/qcow2_format.py | 54 +++++++++++++++--------------- 1 file changed, 27 insertions(+), 27 deletions(-)
tests/qemu-iotests/qcow2_format.py +27 -27
@@ -23,7 +23,7 @@ class QcowHeaderExtension: def __init__(self, magic, length, data): if length % 8 != 0: padding = 8 - (length % 8) - data += b"\0" * padding + data += b'\0' * padding self.magic = magic self.length = length
@@ -41,26 +41,26 @@ class QcowHeader: fields = ( # Version 2 header fields - (uint32_t, '%#x', 'magic'), - (uint32_t, '%d', 'version'), - (uint64_t, '%#x', 'backing_file_offset'), - (uint32_t, '%#x', 'backing_file_size'), - (uint32_t, '%d', 'cluster_bits'), - (uint64_t, '%d', 'size'), - (uint32_t, '%d', 'crypt_method'), - (uint32_t, '%d', 'l1_size'), - (uint64_t, '%#x', 'l1_table_offset'), - (uint64_t, '%#x', 'refcount_table_offset'), - (uint32_t, '%d', 'refcount_table_clusters'), - (uint32_t, '%d', 'nb_snapshots'), - (uint64_t, '%#x', 'snapshot_offset'), + (uint32_t, '{:#x}', 'magic'), + (uint32_t, '{}', 'version'), + (uint64_t, '{:#x}', 'backing_file_offset'), + (uint32_t, '{:#x}', 'backing_file_size'), + (uint32_t, '{}', 'cluster_bits'), + (uint64_t, '{}', 'size'), + (uint32_t, '{}', 'crypt_method'), + (uint32_t, '{}', 'l1_size'), + (uint64_t, '{:#x}', 'l1_table_offset'), + (uint64_t, '{:#x}', 'refcount_table_offset'), + (uint32_t, '{}', 'refcount_table_clusters'), + (uint32_t, '{}', 'nb_snapshots'), + (uint64_t, '{:#x}', 'snapshot_offset'), # Version 3 header fields (uint64_t, 'mask', 'incompatible_features'), (uint64_t, 'mask', 'compatible_features'), (uint64_t, 'mask', 'autoclear_features'), - (uint32_t, '%d', 'refcount_order'), - (uint32_t, '%d', 'header_length'), + (uint32_t, '{}', 'refcount_order'), + (uint32_t, '{}', 'header_length'), ) fmt = '>' + ''.join(field[0] for field in fields)
@@ -118,7 +118,7 @@ class QcowHeader: fd.seek(self.header_length) extensions = self.extensions - extensions.append(QcowHeaderExtension(0, 0, b"")) + extensions.append(QcowHeaderExtension(0, 0, b'')) for ex in extensions: buf = struct.pack('>II', ex.magic, ex.length) fd.write(buf)
@@ -129,7 +129,7 @@ class QcowHeader: fd.write(self.backing_file) if fd.tell() > self.cluster_size: - raise Exception("I think I just broke the image...") + raise Exception('I think I just broke the image...') def update(self, fd): header_bytes = self.header_length
@@ -152,21 +152,21 @@ class QcowHeader: bits.append(bit) value_str = str(bits) else: - value_str = f[1] % value + value_str = f[1].format(value) - print("%-25s" % f[2], value_str) + print(f'{f[2]:<25} {value_str}') def dump_extensions(self): for ex in self.extensions: data = ex.data[:ex.length] if all(c in string.printable.encode('ascii') for c in data): - data = "'%s'" % data.decode('ascii') + data = f"'{ data.decode('ascii') }'" else: - data = "<binary>" + data = '<binary>' - print("Header extension:") - print("%-25s %#x" % ("magic", ex.magic)) - print("%-25s %d" % ("length", ex.length)) - print("%-25s %s" % ("data", data)) - print("") + print('Header extension:') + print(f'{"magic":<25} {ex.magic:#x}') + print(f'{"length":<25} {ex.length}') + print(f'{"data":<25} {data}') + print()
Signature · 1 line
2.21.0
PATCH v5 7/13 qcow2_format.py: use strings to specify c-type of struct fields
In reply to Vladimir Sementsov-OgievskiyWe are going to move field-parsing to super-class, this will be simpler with simple string specifiers instead of variables.
For some reason, python doesn't allow the definition of ctypes variable in the class alongside fields: it would not be available then for use by the 'for' operator. Don't worry: ctypes will be moved to metaclass soon.
--- tests/qemu-iotests/qcow2_format.py | 50 +++++++++++++++++------------- 1 file changed, 28 insertions(+), 22 deletions(-)
tests/qemu-iotests/qcow2_format.py +28 -22
@@ -34,36 +34,42 @@ class QcowHeaderExtension: return QcowHeaderExtension(magic, len(data), data) -class QcowHeader: +# Mapping from c types to python struct format +ctypes = { + 'u8': 'B', + 'u16': 'H', + 'u32': 'I', + 'u64': 'Q' +} + - uint32_t = 'I' - uint64_t = 'Q' +class QcowHeader: fields = ( # Version 2 header fields - (uint32_t, '{:#x}', 'magic'), - (uint32_t, '{}', 'version'), - (uint64_t, '{:#x}', 'backing_file_offset'), - (uint32_t, '{:#x}', 'backing_file_size'), - (uint32_t, '{}', 'cluster_bits'), - (uint64_t, '{}', 'size'), - (uint32_t, '{}', 'crypt_method'), - (uint32_t, '{}', 'l1_size'), - (uint64_t, '{:#x}', 'l1_table_offset'), - (uint64_t, '{:#x}', 'refcount_table_offset'), - (uint32_t, '{}', 'refcount_table_clusters'), - (uint32_t, '{}', 'nb_snapshots'), - (uint64_t, '{:#x}', 'snapshot_offset'), + ('u32', '{:#x}', 'magic'), + ('u32', '{}', 'version'), + ('u64', '{:#x}', 'backing_file_offset'), + ('u32', '{:#x}', 'backing_file_size'), + ('u32', '{}', 'cluster_bits'), + ('u64', '{}', 'size'), + ('u32', '{}', 'crypt_method'), + ('u32', '{}', 'l1_size'), + ('u64', '{:#x}', 'l1_table_offset'), + ('u64', '{:#x}', 'refcount_table_offset'), + ('u32', '{}', 'refcount_table_clusters'), + ('u32', '{}', 'nb_snapshots'), + ('u64', '{:#x}', 'snapshot_offset'), # Version 3 header fields - (uint64_t, 'mask', 'incompatible_features'), - (uint64_t, 'mask', 'compatible_features'), - (uint64_t, 'mask', 'autoclear_features'), - (uint32_t, '{}', 'refcount_order'), - (uint32_t, '{}', 'header_length'), + ('u64', 'mask', 'incompatible_features'), + ('u64', 'mask', 'compatible_features'), + ('u64', 'mask', 'autoclear_features'), + ('u32', '{}', 'refcount_order'), + ('u32', '{}', 'header_length'), ) - fmt = '>' + ''.join(field[0] for field in fields) + fmt = '>' + ''.join(ctypes[f[0]] for f in fields) def __init__(self, fd):
Signature · 1 line
2.21.0
PATCH v5 8/13 qcow2_format.py: separate generic functionality of structure classes
In reply to Vladimir Sementsov-OgievskiyWe are going to introduce more Qcow2 structure types, defined like QcowHeader. Move generic functionality into base class to be reused for further structure classes.
--- tests/qemu-iotests/qcow2_format.py | 101 +++++++++++++++++++---------- 1 file changed, 66 insertions(+), 35 deletions(-)
tests/qemu-iotests/qcow2_format.py +66 -35
@@ -1,5 +1,7 @@ # Library for manipulations with qcow2 image # +# Copyright (c) 2020 Virtuozzo International GmbH. +# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or
@@ -18,6 +20,68 @@ import struct import string +class Qcow2StructMeta(type): + + # Mapping from c types to python struct format + ctypes = { + 'u8': 'B', + 'u16': 'H', + 'u32': 'I', + 'u64': 'Q' + } + + def __init__(self, name, bases, attrs): + if 'fields' in attrs: + self.fmt = '>' + ''.join(self.ctypes[f[0]] for f in self.fields) + + +class Qcow2Struct(metaclass=Qcow2StructMeta): + + """Qcow2Struct: base class for qcow2 data structures + + Successors should define fields class variable, which is: list of tuples, + each of three elements: + - c-type (one of 'u8', 'u16', 'u32', 'u64') + - format (format_spec to use with .format() when dump or 'mask' to dump + bitmasks) + - field name + """ + + def __init__(self, fd=None, offset=None, data=None): + """ + Two variants: + 1. Specify data. fd and offset must be None. + 2. Specify fd and offset, data must be None. offset may be omitted + in this case, than current position of fd is used. + """ + if data is None: + assert fd is not None + buf_size = struct.calcsize(self.fmt) + if offset is not None: + fd.seek(offset) + data = fd.read(buf_size) + else: + assert fd is None and offset is None + + values = struct.unpack(self.fmt, data) + self.__dict__ = dict((field[2], values[i]) + for i, field in enumerate(self.fields)) + + def dump(self): + for f in self.fields: + value = self.__dict__[f[2]] + if f[1] == 'mask': + bits = [] + for bit in range(64): + if value & (1 << bit): + bits.append(bit) + value_str = str(bits) + else: + value_str = f[1].format(value) + + print('{:<25} {}'.format(f[2], value_str)) + + class QcowHeaderExtension: def __init__(self, magic, length, data):
@@ -34,16 +98,7 @@ class QcowHeaderExtension: return QcowHeaderExtension(magic, len(data), data) -# Mapping from c types to python struct format -ctypes = { - 'u8': 'B', - 'u16': 'H', - 'u32': 'I', - 'u64': 'Q' -} - - -class QcowHeader: +class QcowHeader(Qcow2Struct): fields = ( # Version 2 header fields
@@ -69,18 +124,8 @@ class QcowHeader: ('u32', '{}', 'header_length'), ) - fmt = '>' + ''.join(ctypes[f[0]] for f in fields) - def __init__(self, fd): - - buf_size = struct.calcsize(QcowHeader.fmt) - - fd.seek(0) - buf = fd.read(buf_size) - - header = struct.unpack(QcowHeader.fmt, buf) - self.__dict__ = dict((field[2], header[i]) - for i, field in enumerate(QcowHeader.fields)) + super().__init__(fd=fd, offset=0) self.set_defaults() self.cluster_size = 1 << self.cluster_bits
@@ -148,20 +193,6 @@ class QcowHeader: buf = buf[0:header_bytes-1] fd.write(buf) - def dump(self): - for f in QcowHeader.fields: - value = self.__dict__[f[2]] - if f[1] == 'mask': - bits = [] - for bit in range(64): - if value & (1 << bit): - bits.append(bit) - value_str = str(bits) - else: - value_str = f[1].format(value) - - print(f'{f[2]:<25} {value_str}') - def dump_extensions(self): for ex in self.extensions:
Signature · 1 line
2.21.0
Re: [PATCH v5 08/13] qcow2_format.py: separate generic functionality of structure classes
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
We are going to introduce more Qcow2 structure types, defined like QcowHeader. Move generic functionality into base class to be reused for further structure classes.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Andrey Shinkevich <[email protected]>--- tests/qemu-iotests/qcow2_format.py | 101 +++++++++++++++++++---------- 1 file changed, 66 insertions(+), 35 deletions(-)tests/qemu-iotests/qcow2_format.py +0
-0
tests/qemu-iotests/qcow2_format.py line 3
Quoted 3 lines
tests/qemu-iotests/qcow2_format.py +1
-0@@ -1,5 +1,7 @@ # Library for manipulations with qcow2 image # +# Copyright (c) 2020 Virtuozzo International GmbH.
Deferring the addition of this line until this point in the series is fine. Depending on the answer I get to adding any other copyright line in 2/ and 3/13, or if that is deferred to later, we can figure out the merge conflict by keeping both.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v5 9/13 qcow2_format.py: add field-formatting class
In reply to Vladimir Sementsov-OgievskiyAllow formatter class in structure definition instead of hacking with 'mask'. This will simplify further introduction of new formatters.
--- tests/qemu-iotests/qcow2_format.py | 35 +++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 10 deletions(-)
tests/qemu-iotests/qcow2_format.py +25 -10
@@ -20,6 +20,25 @@ import struct import string +class Qcow2Field: + + def __init__(self, value): + self.value = value + + def __str__(self): + return str(self.value) + + +class Flags64(Qcow2Field): + + def __str__(self): + bits = [] + for bit in range(64): + if self.value & (1 << bit): + bits.append(bit) + return str(bits) + + class Qcow2StructMeta(type): # Mapping from c types to python struct format
@@ -70,14 +89,10 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): def dump(self): for f in self.fields: value = self.__dict__[f[2]] - if f[1] == 'mask': - bits = [] - for bit in range(64): - if value & (1 << bit): - bits.append(bit) - value_str = str(bits) - else: + if isinstance(f[1], str): value_str = f[1].format(value) + else: + value_str = str(f[1](value)) print('{:<25} {}'.format(f[2], value_str))
@@ -117,9 +132,9 @@ class QcowHeader(Qcow2Struct): ('u64', '{:#x}', 'snapshot_offset'), # Version 3 header fields - ('u64', 'mask', 'incompatible_features'), - ('u64', 'mask', 'compatible_features'), - ('u64', 'mask', 'autoclear_features'), + ('u64', Flags64, 'incompatible_features'), + ('u64', Flags64, 'compatible_features'), + ('u64', Flags64, 'autoclear_features'), ('u32', '{}', 'refcount_order'), ('u32', '{}', 'header_length'), )
Signature · 1 line
2.21.0
PATCH v5 10/13 qcow2_format.py: QcowHeaderExtension: add dump method
In reply to Vladimir Sementsov-OgievskiyObviously, for-loop body in dump_extensions should be the dump method of extension.
--- tests/qemu-iotests/qcow2_format.py | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
tests/qemu-iotests/qcow2_format.py +12 -10
@@ -108,6 +108,17 @@ class QcowHeaderExtension: self.length = length self.data = data + def dump(self): + data = self.data[:self.length] + if all(c in string.printable.encode('ascii') for c in data): + data = f"'{ data.decode('ascii') }'" + else: + data = '<binary>' + + print(f'{"magic":<25} {self.magic:#x}') + print(f'{"length":<25} {self.length}') + print(f'{"data":<25} {data}') + @classmethod def create(cls, magic, data): return QcowHeaderExtension(magic, len(data), data)
@@ -210,15 +221,6 @@ class QcowHeader(Qcow2Struct): def dump_extensions(self): for ex in self.extensions: - - data = ex.data[:ex.length] - if all(c in string.printable.encode('ascii') for c in data): - data = f"'{ data.decode('ascii') }'" - else: - data = '<binary>' - print('Header extension:') - print(f'{"magic":<25} {ex.magic:#x}') - print(f'{"length":<25} {ex.length}') - print(f'{"data":<25} {data}') + ex.dump() print()
Signature · 1 line
2.21.0
PATCH v5 11/13 qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct
In reply to Vladimir Sementsov-OgievskiyOnly two fields we can parse by generic code, but that is better than nothing. Keep further refactoring of variable-length fields for another day.
--- tests/qemu-iotests/qcow2_format.py | 53 +++++++++++++++++++++--------- 1 file changed, 37 insertions(+), 16 deletions(-)
tests/qemu-iotests/qcow2_format.py +37 -16
@@ -97,16 +97,41 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): print('{:<25} {}'.format(f[2], value_str)) -class QcowHeaderExtension: +class QcowHeaderExtension(Qcow2Struct): - def __init__(self, magic, length, data): - if length % 8 != 0: - padding = 8 - (length % 8) - data += b'\0' * padding + fields = ( + ('u32', '{:#x}', 'magic'), + ('u32', '{}', 'length') + # length bytes of data follows + # then padding to next multiply of 8 + ) - self.magic = magic - self.length = length - self.data = data + def __init__(self, magic=None, length=None, data=None, fd=None): + """ + Support both loading from fd and creation from user data. + For fd-based creation current position in a file will be used to read + the data. + + This should be somehow refactored and functionality should be moved to + superclass (to allow creation of any qcow2 struct), but then, fields + of variable length (data here) should be supported in base class + somehow. So, it's a TODO. We'll see how to properly refactor this when + we have more qcow2 structures. + """ + if fd is None: + assert all(v is not None for v in (magic, length, data)) + self.magic = magic + self.length = length + if length % 8 != 0: + padding = 8 - (length % 8) + data += b'\0' * padding + self.data = data + else: + assert all(v is None for v in (magic, length, data)) + super().__init__(fd=fd) + padded = (self.length + 7) & ~7 + self.data = fd.read(padded) + assert self.data is not None def dump(self): data = self.data[:self.length]
@@ -115,8 +140,7 @@ class QcowHeaderExtension: else: data = '<binary>' - print(f'{"magic":<25} {self.magic:#x}') - print(f'{"length":<25} {self.length}') + super().dump() print(f'{"data":<25} {data}') @classmethod
@@ -182,14 +206,11 @@ class QcowHeader(Qcow2Struct): end = self.cluster_size while fd.tell() < end: - (magic, length) = struct.unpack('>II', fd.read(8)) - if magic == 0: + ext = QcowHeaderExtension(fd=fd) + if ext.magic == 0: break else: - padded = (length + 7) & ~7 - data = fd.read(padded) - self.extensions.append(QcowHeaderExtension(magic, length, - data)) + self.extensions.append(ext) def update_extensions(self, fd):
Signature · 1 line
2.21.0
PATCH v5 12/13 qcow2: QcowHeaderExtension print names for extension magics
In reply to Vladimir Sementsov-Ogievskiy--- tests/qemu-iotests/031.out | 22 +++++++++++----------- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++- 4 files changed, 36 insertions(+), 21 deletions(-)
tests/qemu-iotests/031.out +11 -11
@@ -25,7 +25,7 @@ refcount_order 4 header_length 72 Header extension: -magic 0x12345678 +magic 0x12345678 (<unknown>) length 31 data 'This is a test header extension'
@@ -53,7 +53,7 @@ refcount_order 4 header_length 72 Header extension: -magic 0x12345678 +magic 0x12345678 (<unknown>) length 31 data 'This is a test header extension'
@@ -81,12 +81,12 @@ refcount_order 4 header_length 72 Header extension: -magic 0xe2792aca +magic 0xe2792aca (Backing format) length 11 data 'host_device' Header extension: -magic 0x12345678 +magic 0x12345678 (<unknown>) length 31 data 'This is a test header extension'
@@ -116,12 +116,12 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary> Header extension: -magic 0x12345678 +magic 0x12345678 (<unknown>) length 31 data 'This is a test header extension'
@@ -149,12 +149,12 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary> Header extension: -magic 0x12345678 +magic 0x12345678 (<unknown>) length 31 data 'This is a test header extension'
@@ -182,17 +182,17 @@ refcount_order 4 header_length 112 Header extension: -magic 0xe2792aca +magic 0xe2792aca (Backing format) length 11 data 'host_device' Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary> Header extension: -magic 0x12345678 +magic 0x12345678 (<unknown>) length 31 data 'This is a test header extension'
tests/qemu-iotests/036.out +2 -2
@@ -25,7 +25,7 @@ incompatible_features [] compatible_features [] autoclear_features [63] Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
@@ -37,7 +37,7 @@ incompatible_features [] compatible_features [] autoclear_features [] Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
tests/qemu-iotests/061.out +7 -7
@@ -25,7 +25,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
@@ -83,7 +83,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
@@ -139,7 +139,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
@@ -194,7 +194,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
@@ -263,7 +263,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
@@ -325,7 +325,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
@@ -354,7 +354,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data <binary>
tests/qemu-iotests/qcow2_format.py +16 -1
@@ -39,6 +39,12 @@ class Flags64(Qcow2Field): return str(bits) +class Enum(Qcow2Field): + + def __str__(self): + return f'{self.value:#x} ({self.mapping.get(self.value, "<unknown>")})' + + class Qcow2StructMeta(type): # Mapping from c types to python struct format
@@ -99,8 +105,17 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): class QcowHeaderExtension(Qcow2Struct): + class Magic(Enum): + mapping = { + 0xe2792aca: 'Backing format', + 0x6803f857: 'Feature table', + 0x0537be77: 'Crypto header', + 0x23852875: 'Bitmaps', + 0x44415441: 'Data file' + } + fields = ( - ('u32', '{:#x}', 'magic'), + ('u32', Magic, 'magic'), ('u32', '{}', 'length') # length bytes of data follows # then padding to next multiply of 8
Signature · 1 line
2.21.0
Re: [PATCH v5 12/13] qcow2: QcowHeaderExtension print names for extension magics
________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Saturday, June 6, 2020 11:18 AM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Andrey Shinkevich <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>
Subject: [PATCH v5 12/13] qcow2: QcowHeaderExtension print names for extension magics
--- tests/qemu-iotests/031.out | 22 +++++++++++----------- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++- 4 files changed, 36 insertions(+), 21 deletions(-)
tests/qemu-iotests/qcow2_format.py +4 -0
@@ -39,6 +39,12 @@ class Flags64(Qcow2Field): return str(bits) +class Enum(Qcow2Field): + + def __str__(self): + return f'{self.value:#x} ({self.mapping.get(self.value, "<unknown>")})'
I've got the error E0100: invalid syntax [pylama]
Andrey
Re: [PATCH v5 12/13] qcow2: QcowHeaderExtension print names for extension magics
________________________________
From: Andrey Shinkevich <[email protected]>
Sent: Monday, June 8, 2020 8:48 AM
To: Vladimir Sementsov-Ogievskiy <[email protected]>; [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>
Subject: Re: [PATCH v5 12/13] qcow2: QcowHeaderExtension print names for extension magics
________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Saturday, June 6, 2020 11:18 AM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Andrey Shinkevich <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>
Subject: [PATCH v5 12/13] qcow2: QcowHeaderExtension print names for extension magics
--- tests/qemu-iotests/031.out | 22 +++++++++++----------- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++- 4 files changed, 36 insertions(+), 21 deletions(-)
tests/qemu-iotests/qcow2_format.py +4 -0
@@ -39,6 +39,12 @@ class Flags64(Qcow2Field): return str(bits) +class Enum(Qcow2Field): + + def __str__(self): + return f'{self.value:#x} ({self.mapping.get(self.value, "<unknown>")})'
I've got the error E0100: invalid syntax [pylama]
Andrey
The script itself works. Probably, my python syntax checker is obsolete.
Andrey
Re: [PATCH v5 12/13] qcow2: QcowHeaderExtension print names for extension magics
In reply to Vladimir Sementsov-OgievskiyQuoted 9 lines · Vladimir Sementsov-Ogievskiy
Suggested-by Andrey Shinkevich <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/031.out | 22 +++++++++++----------- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++- 4 files changed, 36 insertions(+), 21 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v5 13/13 qcow2_format.py: dump bitmaps header extension
In reply to Vladimir Sementsov-OgievskiyAdd class for bitmap extension and dump its fields. Further work is to dump bitmap directory.
Test new functionality inside 291 iotest.
--- tests/qemu-iotests/291 | 4 +++ tests/qemu-iotests/291.out | 33 +++++++++++++++++++++++ tests/qemu-iotests/qcow2_format.py | 42 +++++++++++++++++++++++------- 3 files changed, 70 insertions(+), 9 deletions(-)
tests/qemu-iotests/291 +4 -0
@@ -62,6 +62,8 @@ $QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io $QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1 $QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2 $QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +echo "Check resulting qcow2 header extensions:" +$PYTHON qcow2.py "$TEST_IMG" dump-header-exts echo echo "=== Bitmap preservation not possible to non-qcow2 ==="
@@ -88,6 +90,8 @@ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 $QEMU_IMG bitmap --remove --image-opts \ driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +echo "Check resulting qcow2 header extensions:" +$PYTHON qcow2.py "$TEST_IMG" dump-header-exts echo echo "=== Check bitmap contents ==="
tests/qemu-iotests/291.out +33 -0
@@ -14,6 +14,25 @@ wrote 1048576/1048576 bytes at offset 1048576 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 1048576/1048576 bytes at offset 2097152 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Check resulting qcow2 header extensions: +Header extension: +magic 3799591626 (Backing format) +length 5 +data 'qcow2' + +Header extension: +magic 1745090647 (Feature table) +length 336 +data <binary> + +Header extension: +magic 595929205 (Bitmaps) +length 24 +nb_bitmaps 2 +reserved32 0 +bitmap_directory_size 0x40 +bitmap_directory_offset 0x510000 + === Bitmap preservation not possible to non-qcow2 ===
@@ -65,6 +84,20 @@ Format specific information: granularity: 65536 refcount bits: 16 corrupt: false +Check resulting qcow2 header extensions: +Header extension: +magic 1745090647 (Feature table) +length 336 +data <binary> + +Header extension: +magic 595929205 (Bitmaps) +length 24 +nb_bitmaps 3 +reserved32 0 +bitmap_directory_size 0x60 +bitmap_directory_offset 0x520000 + === Check bitmap contents ===
tests/qemu-iotests/qcow2_format.py +33 -9
69 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH v5 13/13] qcow2_format.py: dump bitmaps header extension
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Add class for bitmap extension and dump its fields. Further work is to dump bitmap directory.
Test new functionality inside 291 iotest.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Andrey Shinkevich <[email protected]>---
tests/qemu-iotests/291 $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 · line 94
Quoted 5 lines
tests/qemu-iotests/291 +2
-0@@ -88,6 +90,8 @@ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 $QEMU_IMG bitmap --remove --image-opts \ driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +echo "Check resulting qcow2 header extensions:" +$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
Context conflict with my pending patch to resolve an issue reported by Kevin:
https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg02057.html
but that is easy enough to resolve.
I'll be queuing this series through my bitmaps tree, hopefully with a pull request on Tuesday.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v5 13/13] qcow2_format.py: dump bitmaps header extension
Quoted 19 lines · Eric Blake
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Add class for bitmap extension and dump its fields. Further work is to dump bitmap directory.
Test new functionality inside 291 iotest.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Andrey Shinkevich <[email protected]>---
tests/qemu-iotests/291 +0
-0@@ -88,6 +90,8 @@ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 $QEMU_IMG bitmap --remove --image-opts \Quoted 4 lines
driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp $QEMU_IMG info "$TEST_IMG" |
_filter_img_info--format-specific +echo "Check resulting qcow2 header extensions:" +$PYTHONqcow2.py"$TEST_IMG" dump-header-extsContext conflict with my pending patch to resolve an issue reported by Kevin:
Quoted 6 lines
https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg02057.html
but that is easy enough to resolve.
I'll be queuing this series through my bitmaps tree, hopefully with a pull request on Tuesday.
Great! Thanks!
Signature · 2 lines
Best regards, Vladimir