Skip to contentSkip to conversation

iotests: Dump QCOW2 dirty bitmaps metadata

PATCH Superseded Vladimir Sementsov-Ogievskiy 4 participants 39 messages Jun 4, 2020 – Jun 8, 2020

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover iotests: Dump QCOW2 dirty bitmaps metadata

Hi all!

Here is my suggestion to substitute only first three patches :) of Andrey's [PATCH v3 0/6] iotests: Dump QCOW2 dirty bitmaps metadata

so, I called it v4 for convenience.

What is here:
1. First, update code style
2. Next, try to refactor in a manner which will make adding new data structures simple (look at Qcow2BitmapExt class in last patch)

I think, next step is to add type hints. Then add more structures. And, anyway, at some point we should move it into python/ directory (at least qcow2_format.py lib)

Vladimir Sementsov-Ogievskiy (12):
qcow2.py: python style fixes
qcow2.py: move qcow2 format classes to separate module qcow2_format.py: drop new line printing at end of dump() qcow2_format.py: use tuples instead of lists for fields qcow2_format.py: use modern string formatting qcow2_format.py: use strings to specify c-type of struct fields qcow2_format.py: separate generic functionality of structure classes qcow2_format.py: add field-formatting class qcow2_format.py: QcowHeaderExtension: add dump method
qcow2_format: refactor QcowHeaderExtension as a subclass of
Qcow2Struct
qcow2: QcowHeaderExtension print names for extension magics qcow2_format.py: dump bitmaps header extension

 tests/qemu-iotests/031.out         |  22 +--
 tests/qemu-iotests/036.out         |   4 +-
 tests/qemu-iotests/061.out         |  14 +-
 tests/qemu-iotests/291             |   4 +
 tests/qemu-iotests/291.out         |  33 ++++
 tests/qemu-iotests/qcow2.py        | 200 ++++------------------
 tests/qemu-iotests/qcow2_format.py | 260 +++++++++++++++++++++++++++++
 7 files changed, 348 insertions(+), 189 deletions(-)
 create mode 100644 tests/qemu-iotests/qcow2_format.py
Signature · 1 line
2.21.0
#1 lore raw

PATCH v4 1/12 qcow2.py: python style fixes

Fix flake8 complains. Leave the only chunk of lines over 79 characters:
initialization of cmds variable. Leave it for another day, when it should be refactored to utilize argparse instead of hand-written parsing.

---
 tests/qemu-iotests/qcow2.py | 92 +++++++++++++++++++++----------------
 1 file changed, 53 insertions(+), 39 deletions(-)
tests/qemu-iotests/qcow2.py +53 -39
index 94a07b2f6f..539f5a186b 100755
@@ -4,6 +4,7 @@ import sys
 import struct
 import string
 
+
 class QcowHeaderExtension:
 
     def __init__(self, magic, length, data):
@@ -11,14 +12,15 @@ class QcowHeaderExtension:
             padding = 8 - (length % 8)
             data += b"\0" * padding
 
-        self.magic  = magic
+        self.magic = magic
         self.length = length
-        self.data   = data
+        self.data = data
 
     @classmethod
     def create(cls, magic, data):
         return QcowHeaderExtension(magic, len(data), data)
 
+
 class QcowHeader:
 
     uint32_t = 'I'
@@ -26,27 +28,27 @@ class QcowHeader:
 
     fields = [
         # Version 2 header fields
-        [ uint32_t, '%#x',  'magic' ],
-        [ uint32_t, '%d',   'version' ],
-        [ uint64_t, '%#x',  'backing_file_offset' ],
-        [ uint32_t, '%#x',  'backing_file_size' ],
-        [ uint32_t, '%d',   'cluster_bits' ],
-        [ uint64_t, '%d',   'size' ],
-        [ uint32_t, '%d',   'crypt_method' ],
-        [ uint32_t, '%d',   'l1_size' ],
-        [ uint64_t, '%#x',  'l1_table_offset' ],
-        [ uint64_t, '%#x',  'refcount_table_offset' ],
-        [ uint32_t, '%d',   'refcount_table_clusters' ],
-        [ uint32_t, '%d',   'nb_snapshots' ],
-        [ uint64_t, '%#x',  'snapshot_offset' ],
+        [uint32_t, '%#x',  'magic'],
+        [uint32_t, '%d',   'version'],
+        [uint64_t, '%#x',  'backing_file_offset'],
+        [uint32_t, '%#x',  'backing_file_size'],
+        [uint32_t, '%d',   'cluster_bits'],
+        [uint64_t, '%d',   'size'],
+        [uint32_t, '%d',   'crypt_method'],
+        [uint32_t, '%d',   'l1_size'],
+        [uint64_t, '%#x',  'l1_table_offset'],
+        [uint64_t, '%#x',  'refcount_table_offset'],
+        [uint32_t, '%d',   'refcount_table_clusters'],
+        [uint32_t, '%d',   'nb_snapshots'],
+        [uint64_t, '%#x',  'snapshot_offset'],
 
         # Version 3 header fields
-        [ uint64_t, 'mask', 'incompatible_features' ],
-        [ uint64_t, 'mask', 'compatible_features' ],
-        [ uint64_t, 'mask', 'autoclear_features' ],
-        [ uint32_t, '%d',   'refcount_order' ],
-        [ uint32_t, '%d',   'header_length' ],
-    ];
+        [uint64_t, 'mask', 'incompatible_features'],
+        [uint64_t, 'mask', 'compatible_features'],
+        [uint64_t, 'mask', 'autoclear_features'],
+        [uint32_t, '%d',   'refcount_order'],
+        [uint32_t, '%d',   'header_length'],
+    ]
 
     fmt = '>' + ''.join(field[0] for field in fields)
 
@@ -59,7 +61,7 @@ class QcowHeader:
 
         header = struct.unpack(QcowHeader.fmt, buf)
         self.__dict__ = dict((field[2], header[i])
-            for i, field in enumerate(QcowHeader.fields))
+                             for i, field in enumerate(QcowHeader.fields))
 
         self.set_defaults()
         self.cluster_size = 1 << self.cluster_bits
@@ -96,7 +98,8 @@ class QcowHeader:
             else:
                 padded = (length + 7) & ~7
                 data = fd.read(padded)
-                self.extensions.append(QcowHeaderExtension(magic, length, data))
+                self.extensions.append(QcowHeaderExtension(magic, length,
+                                                           data))
 
     def update_extensions(self, fd):
 
@@ -108,14 +111,13 @@ class QcowHeader:
             fd.write(buf)
             fd.write(ex.data)
 
-        if self.backing_file != None:
+        if self.backing_file is not None:
             self.backing_file_offset = fd.tell()
             fd.write(self.backing_file)
 
         if fd.tell() > self.cluster_size:
             raise Exception("I think I just broke the image...")
 
-
     def update(self, fd):
         header_bytes = self.header_length
 
@@ -163,19 +165,21 @@ def cmd_dump_header(fd):
     h.dump()
     h.dump_extensions()
 
+
 def cmd_dump_header_exts(fd):
     h = QcowHeader(fd)
     h.dump_extensions()
 
+
 def cmd_set_header(fd, name, value):
     try:
         value = int(value, 0)
-    except:
+    except ValueError:
         print("'%s' is not a valid number" % value)
         sys.exit(1)
 
     fields = (field[2] for field in QcowHeader.fields)
-    if not name in fields:
+    if name not in fields:
         print("'%s' is not a known header field" % name)
         sys.exit(1)
 
@@ -183,25 +187,29 @@ def cmd_set_header(fd, name, value):
     h.__dict__[name] = value
     h.update(fd)
 
+
 def cmd_add_header_ext(fd, magic, data):
     try:
         magic = int(magic, 0)
-    except:
+    except ValueError:
         print("'%s' is not a valid magic number" % magic)
         sys.exit(1)
 
     h = QcowHeader(fd)
-    h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii')))
+    h.extensions.append(QcowHeaderExtension.create(magic,
+                                                   data.encode('ascii')))
     h.update(fd)
 
+
 def cmd_add_header_ext_stdio(fd, magic):
     data = sys.stdin.read()
     cmd_add_header_ext(fd, magic, data)
 
+
 def cmd_del_header_ext(fd, magic):
     try:
         magic = int(magic, 0)
-    except:
+    except ValueError:
         print("'%s' is not a valid magic number" % magic)
         sys.exit(1)
 
@@ -219,12 +227,13 @@ def cmd_del_header_ext(fd, magic):
 
     h.update(fd)
 
+
 def cmd_set_feature_bit(fd, group, bit):
     try:
         bit = int(bit, 0)
         if bit < 0 or bit >= 64:
             raise ValueError
-    except:
+    except ValueError:
         print("'%s' is not a valid bit number in range [0, 64)" % bit)
         sys.exit(1)
 
@@ -236,21 +245,24 @@ def cmd_set_feature_bit(fd, group, bit):
     elif group == 'autoclear':
         h.autoclear_features |= 1 << bit
     else:
-        print("'%s' is not a valid group, try 'incompatible', 'compatible', or 'autoclear'" % group)
+        print("'%s' is not a valid group, try "
+              "'incompatible', 'compatible', or 'autoclear'" % group)
         sys.exit(1)
 
     h.update(fd)
 
+
 cmds = [
-    [ 'dump-header',          cmd_dump_header,          0, 'Dump image header and header extensions' ],
-    [ 'dump-header-exts',     cmd_dump_header_exts,     0, 'Dump image header extensions' ],
-    [ 'set-header',           cmd_set_header,           2, 'Set a field in the header'],
-    [ 'add-header-ext',       cmd_add_header_ext,       2, 'Add a header extension' ],
-    [ 'add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin' ],
-    [ 'del-header-ext',       cmd_del_header_ext,       1, 'Delete a header extension' ],
-    [ 'set-feature-bit',      cmd_set_feature_bit,      2, 'Set a feature bit'],
+    ['dump-header',          cmd_dump_header,          0, 'Dump image header and header extensions'],
+    ['dump-header-exts',     cmd_dump_header_exts,     0, 'Dump image header extensions'],
+    ['set-header',           cmd_set_header,           2, 'Set a field in the header'],
+    ['add-header-ext',       cmd_add_header_ext,       2, 'Add a header extension'],
+    ['add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin'],
+    ['del-header-ext',       cmd_del_header_ext,       1, 'Delete a header extension'],
+    ['set-feature-bit',      cmd_set_feature_bit,      2, 'Set a feature bit'],
 ]
 
+
 def main(filename, cmd, args):
     fd = open(filename, "r+b")
     try:
@@ -267,6 +279,7 @@ def main(filename, cmd, args):
     finally:
         fd.close()
 
+
 def usage():
     print("Usage: %s <file> <cmd> [<arg>, ...]" % sys.argv[0])
     print("")
@@ -274,6 +287,7 @@ def usage():
     for name, handler, num_args, desc in cmds:
         print("    %-20s - %s" % (name, desc))
 
+
 if __name__ == '__main__':
     if len(sys.argv) < 3:
         usage()
Signature · 1 line
2.21.0
Reviewed-by on v4 1/12
#2 lore raw

Re: [PATCH v4 01/12] qcow2.py: python style fixes

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 01/12] qcow2.py: python style fixes

Fix flake8 complains. Leave the only chunk of lines over 79 characters:
initialization of cmds variable. Leave it for another day, when it should be refactored to utilize argparse instead of hand-written parsing.

---
 tests/qemu-iotests/qcow2.py | 92 +++++++++++++++++++++----------------
 1 file changed, 53 insertions(+), 39 deletions(-)
#3 lore raw

Re: [PATCH v4 01/12] qcow2.py: python style fixes

In reply to Vladimir Sementsov-Ogievskiy
Quoted 1 line · Vladimir Sementsov-Ogievskiy

Fix flake8 complains. Leave the only chunk of lines over 79 characters:

complaints

Quoted 11 lines

initialization of cmds variable. Leave it for another day, when it should be refactored to utilize argparse instead of hand-written parsing.

---
  tests/qemu-iotests/qcow2.py | 92 +++++++++++++++++++++----------------
  1 file changed, 53 insertions(+), 39 deletions(-)

cmds = [

tests/qemu-iotests/qcow2.py def cmd_set_feature_bit(fd, group, bit): · line 256

Quoted 8 lines
tests/qemu-iotests/qcow2.py +1 -7
-    [ 'dump-header',          cmd_dump_header,          0, 'Dump image header and header extensions' ],
-    [ 'dump-header-exts',     cmd_dump_header_exts,     0, 'Dump image header extensions' ],
-    [ 'set-header',           cmd_set_header,           2, 'Set a field in the header'],
-    [ 'add-header-ext',       cmd_add_header_ext,       2, 'Add a header extension' ],
-    [ 'add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin' ],
-    [ 'del-header-ext',       cmd_del_header_ext,       1, 'Delete a header extension' ],
-    [ 'set-feature-bit',      cmd_set_feature_bit,      2, 'Set a feature bit'],
+    ['dump-header',          cmd_dump_header,          0, 'Dump image header and header extensions'],

I know you mentioned argparse as a later refactoring, but is it worth reflowing the table in the meantime?

['dump-header', cmd_dump_header, 0,
'Dump image header and header extensions'],
[...
'...'],

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#4 lore raw

Re: [PATCH v4 01/12] qcow2.py: python style fixes

Quoted 25 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy

Fix flake8 complains. Leave the only chunk of lines over 79 characters:

complaints

Quoted 10 lines

initialization of cmds variable. Leave it for another day, when it should be refactored to utilize argparse instead of hand-written parsing.

---
  tests/qemu-iotests/qcow2.py | 92 +++++++++++++++++++++----------------

  1 file changed, 53 insertions(+), 39 deletions(-)
Quoted 9 lines

  cmds = [
-    [ 'dump-header',          cmd_dump_header,          0, 'Dump image header and header extensions' ], -    [ 'dump-header-exts',     cmd_dump_header_exts,     0, 'Dump image header extensions' ], -    [ 'set-header',           cmd_set_header,           2, 'Set a field in the header'], -    [ 'add-header-ext',       cmd_add_header_ext,       2, 'Add a header extension' ], -    [ 'add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin' ], -    [ 'del-header-ext',       cmd_del_header_ext,       1, 'Delete a header extension' ], -    [ 'set-feature-bit',      cmd_set_feature_bit,      2, 'Set a feature bit'],
+    ['dump-header',          cmd_dump_header,          0, 'Dump image header and header extensions'],

Quoted 7 lines

I know you mentioned argparse as a later refactoring, but is it worth reflowing the table in the meantime?

['dump-header', cmd_dump_header, 0,
 'Dump image header and header extensions'],
[...
 '...'],

Yes, now I think it worth doing, patchew emails never makes series more beautiful.

Signature · 2 lines
Best regards,
Vladimir
#5 lore raw

PATCH v4 2/12 qcow2.py: move qcow2 format classes to separate module

In reply to Vladimir Sementsov-Ogievskiy

We are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code.

---
 tests/qemu-iotests/qcow2.py        | 161 +----------------------------
 tests/qemu-iotests/qcow2_format.py | 157 ++++++++++++++++++++++++++++
 2 files changed, 161 insertions(+), 157 deletions(-)
 create mode 100644 tests/qemu-iotests/qcow2_format.py
tests/qemu-iotests/qcow2.py +4 -157
index 539f5a186b..d9c41668fd 100755
@@ -1,163 +1,10 @@
 #!/usr/bin/env python3
-
 import sys
-import struct
-import string
-
-
-class QcowHeaderExtension:
-
-    def __init__(self, magic, length, data):
-        if length % 8 != 0:
-            padding = 8 - (length % 8)
-            data += b"\0" * padding
-
-        self.magic = magic
-        self.length = length
-        self.data = data
-
-    @classmethod
-    def create(cls, magic, data):
-        return QcowHeaderExtension(magic, len(data), data)
-
-
-class QcowHeader:
-
-    uint32_t = 'I'
-    uint64_t = 'Q'
-
-    fields = [
-        # Version 2 header fields
-        [uint32_t, '%#x',  'magic'],
-        [uint32_t, '%d',   'version'],
-        [uint64_t, '%#x',  'backing_file_offset'],
-        [uint32_t, '%#x',  'backing_file_size'],
-        [uint32_t, '%d',   'cluster_bits'],
-        [uint64_t, '%d',   'size'],
-        [uint32_t, '%d',   'crypt_method'],
-        [uint32_t, '%d',   'l1_size'],
-        [uint64_t, '%#x',  'l1_table_offset'],
-        [uint64_t, '%#x',  'refcount_table_offset'],
-        [uint32_t, '%d',   'refcount_table_clusters'],
-        [uint32_t, '%d',   'nb_snapshots'],
-        [uint64_t, '%#x',  'snapshot_offset'],
-
-        # Version 3 header fields
-        [uint64_t, 'mask', 'incompatible_features'],
-        [uint64_t, 'mask', 'compatible_features'],
-        [uint64_t, 'mask', 'autoclear_features'],
-        [uint32_t, '%d',   'refcount_order'],
-        [uint32_t, '%d',   'header_length'],
-    ]
-
-    fmt = '>' + ''.join(field[0] for field in fields)
-
-    def __init__(self, fd):
-
-        buf_size = struct.calcsize(QcowHeader.fmt)
-
-        fd.seek(0)
-        buf = fd.read(buf_size)
-
-        header = struct.unpack(QcowHeader.fmt, buf)
-        self.__dict__ = dict((field[2], header[i])
-                             for i, field in enumerate(QcowHeader.fields))
-
-        self.set_defaults()
-        self.cluster_size = 1 << self.cluster_bits
-
-        fd.seek(self.header_length)
-        self.load_extensions(fd)
-
-        if self.backing_file_offset:
-            fd.seek(self.backing_file_offset)
-            self.backing_file = fd.read(self.backing_file_size)
-        else:
-            self.backing_file = None
-
-    def set_defaults(self):
-        if self.version == 2:
-            self.incompatible_features = 0
-            self.compatible_features = 0
-            self.autoclear_features = 0
-            self.refcount_order = 4
-            self.header_length = 72
-
-    def load_extensions(self, fd):
-        self.extensions = []
-
-        if self.backing_file_offset != 0:
-            end = min(self.cluster_size, self.backing_file_offset)
-        else:
-            end = self.cluster_size
-
-        while fd.tell() < end:
-            (magic, length) = struct.unpack('>II', fd.read(8))
-            if magic == 0:
-                break
-            else:
-                padded = (length + 7) & ~7
-                data = fd.read(padded)
-                self.extensions.append(QcowHeaderExtension(magic, length,
-                                                           data))
-
-    def update_extensions(self, fd):
-
-        fd.seek(self.header_length)
-        extensions = self.extensions
-        extensions.append(QcowHeaderExtension(0, 0, b""))
-        for ex in extensions:
-            buf = struct.pack('>II', ex.magic, ex.length)
-            fd.write(buf)
-            fd.write(ex.data)
-
-        if self.backing_file is not None:
-            self.backing_file_offset = fd.tell()
-            fd.write(self.backing_file)
-
-        if fd.tell() > self.cluster_size:
-            raise Exception("I think I just broke the image...")
-
-    def update(self, fd):
-        header_bytes = self.header_length
-
-        self.update_extensions(fd)
-
-        fd.seek(0)
-        header = tuple(self.__dict__[f] for t, p, f in QcowHeader.fields)
-        buf = struct.pack(QcowHeader.fmt, *header)
-        buf = buf[0:header_bytes-1]
-        fd.write(buf)
-
-    def dump(self):
-        for f in QcowHeader.fields:
-            value = self.__dict__[f[2]]
-            if f[1] == 'mask':
-                bits = []
-                for bit in range(64):
-                    if value & (1 << bit):
-                        bits.append(bit)
-                value_str = str(bits)
-            else:
-                value_str = f[1] % value
-
-            print("%-25s" % f[2], value_str)
-        print("")
-
-    def dump_extensions(self):
-        for ex in self.extensions:
-
-            data = ex.data[:ex.length]
-            if all(c in string.printable.encode('ascii') for c in data):
-                data = "'%s'" % data.decode('ascii')
-            else:
-                data = "<binary>"
 
-            print("Header extension:")
-            print("%-25s %#x" % ("magic", ex.magic))
-            print("%-25s %d" % ("length", ex.length))
-            print("%-25s %s" % ("data", data))
-            print("")
+from qcow2_format import (
+    QcowHeader,
+    QcowHeaderExtension
+)
 
 
 def cmd_dump_header(fd):
tests/qemu-iotests/qcow2_format.py new file +157 -0

157 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v4 2/12
#6 lore raw

Re: [PATCH v4 02/12] qcow2.py: move qcow2 format classes to separate module

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 02/12] qcow2.py: move qcow2 format classes to separate module

We are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code.

---
 tests/qemu-iotests/qcow2.py        | 161 +----------------------------
 tests/qemu-iotests/qcow2_format.py | 157 ++++++++++++++++++++++++++++
 2 files changed, 161 insertions(+), 157 deletions(-)
 create mode 100644 tests/qemu-iotests/qcow2_format.py
#7 lore raw

Re: [PATCH v4 02/12] qcow2.py: move qcow2 format classes to separate module

In reply to Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Vladimir Sementsov-Ogievskiy

We are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code.

---
  tests/qemu-iotests/qcow2.py        | 161 +----------------------------
  tests/qemu-iotests/qcow2_format.py | 157 ++++++++++++++++++++++++++++
  2 files changed, 161 insertions(+), 157 deletions(-)

create mode 100644 tests/qemu-iotests/qcow2_format.py

tests/qemu-iotests/qcow2.py +0 -0
index 539f5a186b..d9c41668fd 100755

tests/qemu-iotests/qcow2.py line 2

Quoted 3 lines
tests/qemu-iotests/qcow2.py +0 -1
@@ -1,163 +1,10 @@
  #!/usr/bin/env python3
-
  import sys

Pre-existing: no copyright blurb on the old file...

tests/qemu-iotests/qcow2.py line 3

Quoted 5 lines
tests/qemu-iotests/qcow2.py +4 -0
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -0,0 +1,157 @@
+import struct
+import string
+

It would be nice to fix that, and have one on the new file as well.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#8 lore raw

Re: [PATCH v4 02/12] qcow2.py: move qcow2 format classes to separate module

Quoted 22 lines · Eric Blake
Quoted 17 lines · Vladimir Sementsov-Ogievskiy

We are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code.

---
  tests/qemu-iotests/qcow2.py        | 161 +---------------------------- tests/qemu-iotests/qcow2_format.py | 157 ++++++++++++++++++++++++++++

  2 files changed, 161 insertions(+), 157 deletions(-)

  create mode 100644 tests/qemu-iotests/qcow2_format.py

tests/qemu-iotests/qcow2.py +0 -1
index 539f5a186b..d9c41668fd 100755
@@ -1,163 +1,10 @@
   #!/usr/bin/env python3
-
Quoted 1 line

  import sys

Pre-existing: no copyright blurb on the old file...

Quoted 8 lines

tests/qemu-iotests/qcow2.py line 3

Quoted 5 lines

+++ b/tests/qemu-iotests/qcow2_format.py

tests/qemu-iotests/qcow2.py +3 -0
@@ -0,0 +1,157 @@
+import struct
+import string
+

It would be nice to fix that, and have one on the new file as well.

Yes, I thought about this.. But what to add? Seems OK to add Virtuozzo copyright and common "This program is free software; you can redist... GPL ..." header to qcow2_format.py, as we are going to rewrite and improve it a lot. But what to add to qcow2.py, to the part which is kept mostly unchanged?

Signature · 2 lines
Best regards,
Vladimir
#9 lore raw

Re: [PATCH v4 02/12] qcow2.py: move qcow2 format classes to separate module

Quoted 26 lines · Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Eric Blake
Quoted 11 lines · Vladimir Sementsov-Ogievskiy

We are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code.

---

+++ b/tests/qemu-iotests/qcow2.py

(quoted diff) +0 -1
@@ -1,163 +1,10 @@
   #!/usr/bin/env python3
-

  import sys

Pre-existing: no copyright blurb on the old file...

Quoted 8 lines

tests/qemu-iotests/qcow2_format.py line 3

Quoted 5 lines

+++ b/tests/qemu-iotests/qcow2_format.py

tests/qemu-iotests/qcow2_format.py new file +3 -0
@@ -0,0 +1,157 @@
+import struct
+import string
+

It would be nice to fix that, and have one on the new file as well.

Quoted 1 line

Yes, I thought about this.. But what to add? Seems OK to add Virtuozzo copyright and common "This program is free software; you can redist... GPL ..." header to qcow2_format.py, as we are going to rewrite and improve it a lot. But what to add to qcow2.py, to the part which is kept mostly unchanged?

Late answer, since I see you've already posted v5, but any file without an explicit header is implicitly GPLv2+ by virtue of the top-level LICENSE file.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#10 lore raw

PATCH v4 3/12 qcow2_format.py: drop new line printing at end of dump()

In reply to Vladimir Sementsov-Ogievskiy

This will simplify further conversion. To compensate, print this empty line directly in cmd_dump_header().

---
 tests/qemu-iotests/qcow2.py        | 1 +
 tests/qemu-iotests/qcow2_format.py | 1 -
 2 files changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/qcow2.py +1 -0
index d9c41668fd..79db81b040 100755
@@ -10,6 +10,7 @@ from qcow2_format import (
 def cmd_dump_header(fd):
     h = QcowHeader(fd)
     h.dump()
+    print()
     h.dump_extensions()
 
 
tests/qemu-iotests/qcow2_format.py +0 -1
index c7270a0a6e..99e5248e73 100644
@@ -139,7 +139,6 @@ class QcowHeader:
                 value_str = f[1] % value
 
             print("%-25s" % f[2], value_str)
-        print("")
 
     def dump_extensions(self):
         for ex in self.extensions:
Signature · 1 line
2.21.0
Reviewed-by on v4 3/12
#11 lore raw

Re: [PATCH v4 03/12] qcow2_format.py: drop new line printing at end of dump()

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 03/12] qcow2_format.py: drop new line printing at end of dump()

This will simplify further conversion. To compensate, print this empty line directly in cmd_dump_header().

---
 tests/qemu-iotests/qcow2.py        | 1 +
 tests/qemu-iotests/qcow2_format.py | 1 -
 2 files changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/qcow2.py +1 -0
index d9c41668fd..79db81b040 100755
@@ -10,6 +10,7 @@ from qcow2_format import (
 def cmd_dump_header(fd):
     h = QcowHeader(fd)
     h.dump()
+    print()
     h.dump_extensions()
 
 
tests/qemu-iotests/qcow2_format.py +0 -1
index c7270a0a6e..99e5248e73 100644
@@ -139,7 +139,6 @@ class QcowHeader:
                 value_str = f[1] % value
 
             print("%-25s" % f[2], value_str)
-        print("")
 
     def dump_extensions(self):
         for ex in self.extensions:

--
2.21.0

#12 lore raw

PATCH v4 4/12 qcow2_format.py: use tuples instead of lists for fields

In reply to Vladimir Sementsov-Ogievskiy

No need in lists: it's a constant variable.

---
 tests/qemu-iotests/qcow2_format.py | 40 +++++++++++++++---------------
 1 file changed, 20 insertions(+), 20 deletions(-)
tests/qemu-iotests/qcow2_format.py +20 -20
index 99e5248e73..5d242c4aa4 100644
@@ -23,29 +23,29 @@ class QcowHeader:
     uint32_t = 'I'
     uint64_t = 'Q'
 
-    fields = [
+    fields = (
         # Version 2 header fields
-        [uint32_t, '%#x',  'magic'],
-        [uint32_t, '%d',   'version'],
-        [uint64_t, '%#x',  'backing_file_offset'],
-        [uint32_t, '%#x',  'backing_file_size'],
-        [uint32_t, '%d',   'cluster_bits'],
-        [uint64_t, '%d',   'size'],
-        [uint32_t, '%d',   'crypt_method'],
-        [uint32_t, '%d',   'l1_size'],
-        [uint64_t, '%#x',  'l1_table_offset'],
-        [uint64_t, '%#x',  'refcount_table_offset'],
-        [uint32_t, '%d',   'refcount_table_clusters'],
-        [uint32_t, '%d',   'nb_snapshots'],
-        [uint64_t, '%#x',  'snapshot_offset'],
+        (uint32_t, '%#x',  'magic'),
+        (uint32_t, '%d',   'version'),
+        (uint64_t, '%#x',  'backing_file_offset'),
+        (uint32_t, '%#x',  'backing_file_size'),
+        (uint32_t, '%d',   'cluster_bits'),
+        (uint64_t, '%d',   'size'),
+        (uint32_t, '%d',   'crypt_method'),
+        (uint32_t, '%d',   'l1_size'),
+        (uint64_t, '%#x',  'l1_table_offset'),
+        (uint64_t, '%#x',  'refcount_table_offset'),
+        (uint32_t, '%d',   'refcount_table_clusters'),
+        (uint32_t, '%d',   'nb_snapshots'),
+        (uint64_t, '%#x',  'snapshot_offset'),
 
         # Version 3 header fields
-        [uint64_t, 'mask', 'incompatible_features'],
-        [uint64_t, 'mask', 'compatible_features'],
-        [uint64_t, 'mask', 'autoclear_features'],
-        [uint32_t, '%d',   'refcount_order'],
-        [uint32_t, '%d',   'header_length'],
-    ]
+        (uint64_t, 'mask', 'incompatible_features'),
+        (uint64_t, 'mask', 'compatible_features'),
+        (uint64_t, 'mask', 'autoclear_features'),
+        (uint32_t, '%d',   'refcount_order'),
+        (uint32_t, '%d',   'header_length'),
+    )
 
     fmt = '>' + ''.join(field[0] for field in fields)
 
Signature · 1 line
2.21.0
Reviewed-by on v4 4/12
#13 lore raw

Re: [PATCH v4 04/12] qcow2_format.py: use tuples instead of lists for fields

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 04/12] qcow2_format.py: use tuples instead of lists for fields

No need in lists: it's a constant variable.

---
 tests/qemu-iotests/qcow2_format.py | 40 +++++++++++++++---------------
 1 file changed, 20 insertions(+), 20 deletions(-)
#14 lore raw

Re: [PATCH v4 04/12] qcow2_format.py: use tuples instead of lists for fields

In reply to Vladimir Sementsov-Ogievskiy
Quoted 12 lines · Vladimir Sementsov-Ogievskiy

No need in lists: it's a constant variable.

---
  tests/qemu-iotests/qcow2_format.py | 40 +++++++++++++++---------------
  1 file changed, 20 insertions(+), 20 deletions(-)

# Version 3 header fields

tests/qemu-iotests/qcow2_format.py +0 -2
-        [uint64_t, 'mask', 'incompatible_features'],
-        [uint64_t, 'mask', 'compatible_features'],

tests/qemu-iotests/qcow2_format.py class QcowHeader: · line 48

Quoted 10 lines
tests/qemu-iotests/qcow2_format.py +6 -4
-        [uint64_t, 'mask', 'autoclear_features'],
-        [uint32_t, '%d',   'refcount_order'],
-        [uint32_t, '%d',   'header_length'],
-    ]
+        (uint64_t, 'mask', 'incompatible_features'),
+        (uint64_t, 'mask', 'compatible_features'),
+        (uint64_t, 'mask', 'autoclear_features'),
+        (uint32_t, '%d',   'refcount_order'),
+        (uint32_t, '%d',   'header_length'),
+    )

Not for this patch, but noticing it since we're here: should this be taught to list the optional compression_type field that we recently added after header_length?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#15 lore raw

Re: [PATCH v4 04/12] qcow2_format.py: use tuples instead of lists for fields

Quoted 23 lines · Eric Blake
Quoted 8 lines · Vladimir Sementsov-Ogievskiy

No need in lists: it's a constant variable.

---
  tests/qemu-iotests/qcow2_format.py | 40 +++++++++++++++---------------

  1 file changed, 20 insertions(+), 20 deletions(-)
Quoted 13 lines

          # Version 3 header fields
-        [uint64_t, 'mask', 'incompatible_features'], -        [uint64_t, 'mask', 'compatible_features'], -        [uint64_t, 'mask', 'autoclear_features'], -        [uint32_t, '%d',   'refcount_order'], -        [uint32_t, '%d',   'header_length'], -    ]
+        (uint64_t, 'mask', 'incompatible_features'), +        (uint64_t, 'mask', 'compatible_features'), +        (uint64_t, 'mask', 'autoclear_features'), +        (uint32_t, '%d',   'refcount_order'), +        (uint32_t, '%d',   'header_length'), +    )

Quoted 2 lines

Not for this patch, but noticing it since we're here: should this be taught to list the optional compression_type field that we recently added after header_length?

Good note. And we can use it in corresponding iotest about compression_type. We just should carefully check, how header_length field is handled and fix if needed. I think, Andrey or I will make an additional patch.

Signature · 2 lines
Best regards,
Vladimir
#16 lore raw

PATCH v4 5/12 qcow2_format.py: use modern string formatting

In reply to Vladimir Sementsov-Ogievskiy

Use .format and f-strings instead of old %style. Also, the file uses both '' and "" quotes, for consistency let's use '', except for cases when we need '' inside the string (use "" to avoid extra escaping).

---
 tests/qemu-iotests/qcow2_format.py | 54 +++++++++++++++---------------
 1 file changed, 27 insertions(+), 27 deletions(-)
tests/qemu-iotests/qcow2_format.py +27 -27
index 5d242c4aa4..80e7c3f81d 100644
@@ -7,7 +7,7 @@ class QcowHeaderExtension:
     def __init__(self, magic, length, data):
         if length % 8 != 0:
             padding = 8 - (length % 8)
-            data += b"\0" * padding
+            data += b'\0' * padding
 
         self.magic = magic
         self.length = length
@@ -25,26 +25,26 @@ class QcowHeader:
 
     fields = (
         # Version 2 header fields
-        (uint32_t, '%#x',  'magic'),
-        (uint32_t, '%d',   'version'),
-        (uint64_t, '%#x',  'backing_file_offset'),
-        (uint32_t, '%#x',  'backing_file_size'),
-        (uint32_t, '%d',   'cluster_bits'),
-        (uint64_t, '%d',   'size'),
-        (uint32_t, '%d',   'crypt_method'),
-        (uint32_t, '%d',   'l1_size'),
-        (uint64_t, '%#x',  'l1_table_offset'),
-        (uint64_t, '%#x',  'refcount_table_offset'),
-        (uint32_t, '%d',   'refcount_table_clusters'),
-        (uint32_t, '%d',   'nb_snapshots'),
-        (uint64_t, '%#x',  'snapshot_offset'),
+        (uint32_t, '{:#x}', 'magic'),
+        (uint32_t, '{}', 'version'),
+        (uint64_t, '{:#x}', 'backing_file_offset'),
+        (uint32_t, '{:#x}', 'backing_file_size'),
+        (uint32_t, '{}', 'cluster_bits'),
+        (uint64_t, '{}', 'size'),
+        (uint32_t, '{}', 'crypt_method'),
+        (uint32_t, '{}', 'l1_size'),
+        (uint64_t, '{:#x}', 'l1_table_offset'),
+        (uint64_t, '{:#x}', 'refcount_table_offset'),
+        (uint32_t, '{}', 'refcount_table_clusters'),
+        (uint32_t, '{}', 'nb_snapshots'),
+        (uint64_t, '{:#x}', 'snapshot_offset'),
 
         # Version 3 header fields
         (uint64_t, 'mask', 'incompatible_features'),
         (uint64_t, 'mask', 'compatible_features'),
         (uint64_t, 'mask', 'autoclear_features'),
-        (uint32_t, '%d',   'refcount_order'),
-        (uint32_t, '%d',   'header_length'),
+        (uint32_t, '{}', 'refcount_order'),
+        (uint32_t, '{}', 'header_length'),
     )
 
     fmt = '>' + ''.join(field[0] for field in fields)
@@ -102,7 +102,7 @@ class QcowHeader:
 
         fd.seek(self.header_length)
         extensions = self.extensions
-        extensions.append(QcowHeaderExtension(0, 0, b""))
+        extensions.append(QcowHeaderExtension(0, 0, b''))
         for ex in extensions:
             buf = struct.pack('>II', ex.magic, ex.length)
             fd.write(buf)
@@ -113,7 +113,7 @@ class QcowHeader:
             fd.write(self.backing_file)
 
         if fd.tell() > self.cluster_size:
-            raise Exception("I think I just broke the image...")
+            raise Exception('I think I just broke the image...')
 
     def update(self, fd):
         header_bytes = self.header_length
@@ -136,21 +136,21 @@ class QcowHeader:
                         bits.append(bit)
                 value_str = str(bits)
             else:
-                value_str = f[1] % value
+                value_str = f[1].format(value)
 
-            print("%-25s" % f[2], value_str)
+            print(f'{f[2]:<25} {value_str}')
 
     def dump_extensions(self):
         for ex in self.extensions:
 
             data = ex.data[:ex.length]
             if all(c in string.printable.encode('ascii') for c in data):
-                data = "'%s'" % data.decode('ascii')
+                data = f"'{ data.decode('ascii') }'"
             else:
-                data = "<binary>"
+                data = '<binary>'
 
-            print("Header extension:")
-            print("%-25s %#x" % ("magic", ex.magic))
-            print("%-25s %d" % ("length", ex.length))
-            print("%-25s %s" % ("data", data))
-            print("")
+            print('Header extension:')
+            print(f'{"magic":<25} {ex.magic:#x}')
+            print(f'{"length":<25} {ex.length}')
+            print(f'{"data":<25} {data}')
+            print()
Signature · 1 line
2.21.0
Reviewed-by on v4 5/12
#17 lore raw

Re: [PATCH v4 05/12] qcow2_format.py: use modern string formatting

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 05/12] qcow2_format.py: use modern string formatting

Use .format and f-strings instead of old %style. Also, the file uses both '' and "" quotes, for consistency let's use '', except for cases when we need '' inside the string (use "" to avoid extra escaping).

---
 tests/qemu-iotests/qcow2_format.py | 54 +++++++++++++++---------------
 1 file changed, 27 insertions(+), 27 deletions(-)
#18 lore raw

PATCH v4 6/12 qcow2_format.py: use strings to specify c-type of struct fields

In reply to Vladimir Sementsov-Ogievskiy

We are going to move field-parsing to super-class, this will be simpler with simple string specifiers instead of variables.

For some reason python doesn't allow to define ctypes in class too, as well as fields: it's not available than in 'for' operator. Don't worry:
ctypes will be moved to metaclass soon.

---
 tests/qemu-iotests/qcow2_format.py | 50 +++++++++++++++++-------------
 1 file changed, 28 insertions(+), 22 deletions(-)
tests/qemu-iotests/qcow2_format.py +28 -22
index 80e7c3f81d..1fd9473b7f 100644
@@ -18,36 +18,42 @@ class QcowHeaderExtension:
         return QcowHeaderExtension(magic, len(data), data)
 
 
-class QcowHeader:
+# Mapping from c types to python struct format
+ctypes = {
+    'u8': 'B',
+    'u16': 'H',
+    'u32': 'I',
+    'u64': 'Q'
+}
+
 
-    uint32_t = 'I'
-    uint64_t = 'Q'
+class QcowHeader:
 
     fields = (
         # Version 2 header fields
-        (uint32_t, '{:#x}', 'magic'),
-        (uint32_t, '{}', 'version'),
-        (uint64_t, '{:#x}', 'backing_file_offset'),
-        (uint32_t, '{:#x}', 'backing_file_size'),
-        (uint32_t, '{}', 'cluster_bits'),
-        (uint64_t, '{}', 'size'),
-        (uint32_t, '{}', 'crypt_method'),
-        (uint32_t, '{}', 'l1_size'),
-        (uint64_t, '{:#x}', 'l1_table_offset'),
-        (uint64_t, '{:#x}', 'refcount_table_offset'),
-        (uint32_t, '{}', 'refcount_table_clusters'),
-        (uint32_t, '{}', 'nb_snapshots'),
-        (uint64_t, '{:#x}', 'snapshot_offset'),
+        ('u32', '{:#x}', 'magic'),
+        ('u32', '{}', 'version'),
+        ('u64', '{:#x}', 'backing_file_offset'),
+        ('u32', '{:#x}', 'backing_file_size'),
+        ('u32', '{}', 'cluster_bits'),
+        ('u64', '{}', 'size'),
+        ('u32', '{}', 'crypt_method'),
+        ('u32', '{}', 'l1_size'),
+        ('u64', '{:#x}', 'l1_table_offset'),
+        ('u64', '{:#x}', 'refcount_table_offset'),
+        ('u32', '{}', 'refcount_table_clusters'),
+        ('u32', '{}', 'nb_snapshots'),
+        ('u64', '{:#x}', 'snapshot_offset'),
 
         # Version 3 header fields
-        (uint64_t, 'mask', 'incompatible_features'),
-        (uint64_t, 'mask', 'compatible_features'),
-        (uint64_t, 'mask', 'autoclear_features'),
-        (uint32_t, '{}', 'refcount_order'),
-        (uint32_t, '{}', 'header_length'),
+        ('u64', 'mask', 'incompatible_features'),
+        ('u64', 'mask', 'compatible_features'),
+        ('u64', 'mask', 'autoclear_features'),
+        ('u32', '{}', 'refcount_order'),
+        ('u32', '{}', 'header_length'),
     )
 
-    fmt = '>' + ''.join(field[0] for field in fields)
+    fmt = '>' + ''.join(ctypes[f[0]] for f in fields)
 
     def __init__(self, fd):
 
Signature · 1 line
2.21.0
Reviewed-by on v4 6/12
#19 lore raw

Re: [PATCH v4 06/12] qcow2_format.py: use strings to specify c-type of struct fields

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 06/12] qcow2_format.py: use strings to specify c-type of struct fields

We are going to move field-parsing to super-class, this will be simpler with simple string specifiers instead of variables.

For some reason python doesn't allow to define ctypes in class too, as well as fields: it's not available than in 'for' operator. Don't worry:
ctypes will be moved to metaclass soon.

---
 tests/qemu-iotests/qcow2_format.py | 50 +++++++++++++++++-------------
 1 file changed, 28 insertions(+), 22 deletions(-)
#20 lore raw

Re: [PATCH v4 06/12] qcow2_format.py: use strings to specify c-type of struct fields

In reply to Vladimir Sementsov-Ogievskiy
Quoted 6 lines · Vladimir Sementsov-Ogievskiy

We are going to move field-parsing to super-class, this will be simpler with simple string specifiers instead of variables.

For some reason python doesn't allow to define ctypes in class too, as well as fields: it's not available than in 'for' operator. Don't worry:
ctypes will be moved to metaclass soon.

Grammar suggestion:
For some reason, python doesn't allow the definition of ctypes in the class alongside fields: it would not be available then for use by the 'for' operator.

Quoted 6 lines
---
  tests/qemu-iotests/qcow2_format.py | 50 +++++++++++++++++-------------
  1 file changed, 28 insertions(+), 22 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#21 lore raw

PATCH v4 7/12 qcow2_format.py: separate generic functionality of structure classes

In reply to Vladimir Sementsov-Ogievskiy

We are going to introduce more Qcow2 structure types, defined like QcowHeader. Move generic functionality into base class to be reused for further structure classes.

---
 tests/qemu-iotests/qcow2_format.py | 93 +++++++++++++++++++-----------
 1 file changed, 58 insertions(+), 35 deletions(-)
tests/qemu-iotests/qcow2_format.py +58 -35
index 1fd9473b7f..d71f578377 100644
@@ -2,6 +2,62 @@ import struct
 import string
 
 
+class Qcow2StructMeta(type):
+
+    # Mapping from c types to python struct format
+    ctypes = {
+        'u8': 'B',
+        'u16': 'H',
+        'u32': 'I',
+        'u64': 'Q'
+    }
+
+    def __init__(self, name, bases, attrs):
+        if 'fields' in attrs:
+            self.fmt = '>' + ''.join(self.ctypes[f[0]] for f in self.fields)
+
+
+class Qcow2Struct(metaclass=Qcow2StructMeta):
+
+    """Qcow2Struct: base class for qcow2 data structures
+
+    Successors should define fields class variable, which is: list of tuples,
+    each of three elements:
+        - c-type (one of 'u32', 'u64')
+        - format (format_spec to use with .format() when dump or 'mask' to dump
+                  bitmasks)
+        - field name
+    """
+
+    def __init__(self, fd=None, offset=None, data=None):
+        if data is None:
+            assert fd is not None
+            buf_size = struct.calcsize(self.fmt)
+            if offset is not None:
+                fd.seek(offset)
+            data = fd.read(buf_size)
+        else:
+            assert fd is None and offset is None
+
+        values = struct.unpack(self.fmt, data)
+        self.__dict__ = dict((field[2], values[i])
+                             for i, field in enumerate(self.fields))
+
+    def dump(self):
+        for f in self.fields:
+            value = self.__dict__[f[2]]
+            if f[1] == 'mask':
+                bits = []
+                for bit in range(64):
+                    if value & (1 << bit):
+                        bits.append(bit)
+                value_str = str(bits)
+            else:
+                value_str = f[1].format(value)
+
+            print('{:<25} {}'.format(f[2], value_str))
+
+
 class QcowHeaderExtension:
 
     def __init__(self, magic, length, data):
@@ -18,16 +74,7 @@ class QcowHeaderExtension:
         return QcowHeaderExtension(magic, len(data), data)
 
 
-# Mapping from c types to python struct format
-ctypes = {
-    'u8': 'B',
-    'u16': 'H',
-    'u32': 'I',
-    'u64': 'Q'
-}
-
-
-class QcowHeader:
+class QcowHeader(Qcow2Struct):
 
     fields = (
         # Version 2 header fields
@@ -53,18 +100,8 @@ class QcowHeader:
         ('u32', '{}', 'header_length'),
     )
 
-    fmt = '>' + ''.join(ctypes[f[0]] for f in fields)
-
     def __init__(self, fd):
-
-        buf_size = struct.calcsize(QcowHeader.fmt)
-
-        fd.seek(0)
-        buf = fd.read(buf_size)
-
-        header = struct.unpack(QcowHeader.fmt, buf)
-        self.__dict__ = dict((field[2], header[i])
-                             for i, field in enumerate(QcowHeader.fields))
+        super().__init__(fd=fd, offset=0)
 
         self.set_defaults()
         self.cluster_size = 1 << self.cluster_bits
@@ -132,20 +169,6 @@ class QcowHeader:
         buf = buf[0:header_bytes-1]
         fd.write(buf)
 
-    def dump(self):
-        for f in QcowHeader.fields:
-            value = self.__dict__[f[2]]
-            if f[1] == 'mask':
-                bits = []
-                for bit in range(64):
-                    if value & (1 << bit):
-                        bits.append(bit)
-                value_str = str(bits)
-            else:
-                value_str = f[1].format(value)
-
-            print(f'{f[2]:<25} {value_str}')
-
     def dump_extensions(self):
         for ex in self.extensions:
 
Signature · 1 line
2.21.0
Reviewed-by on v4 7/12
#22 lore raw

Re: [PATCH v4 07/12] qcow2_format.py: separate generic functionality of structure classes

Two more int types in the comment?

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 07/12] qcow2_format.py: separate generic functionality of structure classes

We are going to introduce more Qcow2 structure types, defined like QcowHeader. Move generic functionality into base class to be reused for further structure classes.

---
 tests/qemu-iotests/qcow2_format.py | 93 +++++++++++++++++++-----------
 1 file changed, 58 insertions(+), 35 deletions(-)
tests/qemu-iotests/qcow2_format.py +8 -0
index 1fd9473b7f..d71f578377 100644
@@ -2,6 +2,62 @@ import struct
 
+
+class Qcow2Struct(metaclass=Qcow2StructMeta):
+
+    """Qcow2Struct: base class for qcow2 data structures
+
+    Successors should define fields class variable, which is: list of tuples,
+    each of three elements:
+        - c-type (one of 'u32', 'u64')
 

u8, u16 ?

#23 lore raw

Re: [PATCH v4 07/12] qcow2_format.py: separate generic functionality of structure classes

Quoted 34 lines · Andrey Shinkevich

Two more int types in the comment?

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*From:* Vladimir Sementsov-Ogievskiy <[email protected]>
*Sent:* Thursday, June 4, 2020 8:41 PM
*To:* [email protected] <[email protected]>
*Cc:* [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]> *Subject:* [PATCH v4 07/12] qcow2_format.py: separate generic functionality of structure classes
We are going to introduce more Qcow2 structure types, defined like
QcowHeader. Move generic functionality into base class to be reused for
further structure classes.

---
  tests/qemu-iotests/qcow2_format.py | 93 +++++++++++++++++++-----------
  1 file changed, 58 insertions(+), 35 deletions(-)
tests/qemu-iotests/qcow2_format.py +8 -0
index 1fd9473b7f..d71f578377 100644
@@ -2,6 +2,62 @@ import struct
 
+
+class Qcow2Struct(metaclass=Qcow2StructMeta):
+
+    """Qcow2Struct: base class for qcow2 data structures
+
+    Successors should define fields class variable, which is: list of tuples,
+    each of three elements:
+        - c-type (one of 'u32', 'u64')
 
Quoted 1 line

u8, u16 ?

yes

Signature · 2 lines
Best regards,
Vladimir
#24 lore raw

PATCH v4 8/12 qcow2_format.py: add field-formatting class

In reply to Vladimir Sementsov-Ogievskiy

Allow formatter class in structure definition instead of hacking with 'mask'. This will simplify further introduction of new formatters.

---
 tests/qemu-iotests/qcow2_format.py | 35 +++++++++++++++++++++---------
 1 file changed, 25 insertions(+), 10 deletions(-)
tests/qemu-iotests/qcow2_format.py +25 -10
index d71f578377..ceb09bde5a 100644
@@ -2,6 +2,25 @@ import struct
 import string
 
 
+class Qcow2Field:
+
+    def __init__(self, value):
+        self.value = value
+
+    def __str__(self):
+        return str(self.value)
+
+
+class Flags64(Qcow2Field):
+
+    def __str__(self):
+        bits = []
+        for bit in range(64):
+            if self.value & (1 << bit):
+                bits.append(bit)
+        return str(bits)
+
+
 class Qcow2StructMeta(type):
 
     # Mapping from c types to python struct format
@@ -46,14 +65,10 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
     def dump(self):
         for f in self.fields:
             value = self.__dict__[f[2]]
-            if f[1] == 'mask':
-                bits = []
-                for bit in range(64):
-                    if value & (1 << bit):
-                        bits.append(bit)
-                value_str = str(bits)
-            else:
+            if isinstance(f[1], str):
                 value_str = f[1].format(value)
+            else:
+                value_str = str(f[1](value))
 
             print('{:<25} {}'.format(f[2], value_str))
 
@@ -93,9 +108,9 @@ class QcowHeader(Qcow2Struct):
         ('u64', '{:#x}', 'snapshot_offset'),
 
         # Version 3 header fields
-        ('u64', 'mask', 'incompatible_features'),
-        ('u64', 'mask', 'compatible_features'),
-        ('u64', 'mask', 'autoclear_features'),
+        ('u64', Flags64, 'incompatible_features'),
+        ('u64', Flags64, 'compatible_features'),
+        ('u64', Flags64, 'autoclear_features'),
         ('u32', '{}', 'refcount_order'),
         ('u32', '{}', 'header_length'),
     )
Signature · 1 line
2.21.0
Reviewed-by on v4 8/12
#25 lore raw

Re: [PATCH v4 08/12] qcow2_format.py: add field-formatting class

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 08/12] qcow2_format.py: add field-formatting class

Allow formatter class in structure definition instead of hacking with 'mask'. This will simplify further introduction of new formatters.

---
 tests/qemu-iotests/qcow2_format.py | 35 +++++++++++++++++++++---------
 1 file changed, 25 insertions(+), 10 deletions(-)
#26 lore raw

PATCH v4 9/12 qcow2_format.py: QcowHeaderExtension: add dump method

In reply to Vladimir Sementsov-Ogievskiy

Obviously, for-loop body in dump_extensions should be the dump method of extension.

---
 tests/qemu-iotests/qcow2_format.py | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
tests/qemu-iotests/qcow2_format.py +12 -10
index ceb09bde5a..ffc7c35b18 100644
@@ -84,6 +84,17 @@ class QcowHeaderExtension:
         self.length = length
         self.data = data
 
+    def dump(self):
+        data = self.data[:self.length]
+        if all(c in string.printable.encode('ascii') for c in data):
+            data = f"'{ data.decode('ascii') }'"
+        else:
+            data = '<binary>'
+
+        print(f'{"magic":<25} {self.magic:#x}')
+        print(f'{"length":<25} {self.length}')
+        print(f'{"data":<25} {data}')
+
     @classmethod
     def create(cls, magic, data):
         return QcowHeaderExtension(magic, len(data), data)
@@ -186,15 +197,6 @@ class QcowHeader(Qcow2Struct):
 
     def dump_extensions(self):
         for ex in self.extensions:
-
-            data = ex.data[:ex.length]
-            if all(c in string.printable.encode('ascii') for c in data):
-                data = f"'{ data.decode('ascii') }'"
-            else:
-                data = '<binary>'
-
             print('Header extension:')
-            print(f'{"magic":<25} {ex.magic:#x}')
-            print(f'{"length":<25} {ex.length}')
-            print(f'{"data":<25} {data}')
+            ex.dump()
             print()
Signature · 1 line
2.21.0
Reviewed-by on v4 9/12
#27 lore raw

Re: [PATCH v4 09/12] qcow2_format.py: QcowHeaderExtension: add dump method

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 09/12] qcow2_format.py: QcowHeaderExtension: add dump method

Obviously, for-loop body in dump_extensions should be the dump method of extension.

---
 tests/qemu-iotests/qcow2_format.py | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
#28 lore raw

PATCH v4 10/12 qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct

In reply to Vladimir Sementsov-Ogievskiy

Only two fields we can parse by generic code, but that is better than nothing. Keep further refactoring of variable-length fields for another day.

---
 tests/qemu-iotests/qcow2_format.py | 51 ++++++++++++++++++++----------
 1 file changed, 35 insertions(+), 16 deletions(-)
tests/qemu-iotests/qcow2_format.py +35 -16
index ffc7c35b18..4683b1e436 100644
@@ -73,16 +73,39 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
             print('{:<25} {}'.format(f[2], value_str))
 
 
-class QcowHeaderExtension:
+class QcowHeaderExtension(Qcow2Struct):
 
-    def __init__(self, magic, length, data):
-        if length % 8 != 0:
-            padding = 8 - (length % 8)
-            data += b'\0' * padding
+    fields = (
+        ('u32', '{:#x}', 'magic'),
+        ('u32', '{}', 'length')
+        # length bytes of data follows
+        # then padding to next multiply of 8
+    )
 
-        self.magic = magic
-        self.length = length
-        self.data = data
+    def __init__(self, magic=None, length=None, data=None, fd=None):
+        """
+        Support both loading from fd and creation from user data.
+
+        This should be somehow refactored and functionality should be moved to
+        superclass (to allow creation of any qcow2 struct), but then, fields
+        of variable length (data here) should be supported in base class
+        somehow. So, it's a TODO. We'll see how to properly refactor this when
+        we have more qcow2 structures.
+        """
+        if fd is None:
+            assert all(v is not None for v in (magic, length, data))
+            self.magic = magic
+            self.length = length
+            if length % 8 != 0:
+                padding = 8 - (length % 8)
+                data += b'\0' * padding
+            self.data = data
+        else:
+            assert all(v is None for v in (magic, length, data))
+            super().__init__(fd=fd)
+            padded = (self.length + 7) & ~7
+            self.data = fd.read(padded)
+            assert self.data is not None
 
     def dump(self):
         data = self.data[:self.length]
@@ -91,8 +114,7 @@ class QcowHeaderExtension:
         else:
             data = '<binary>'
 
-        print(f'{"magic":<25} {self.magic:#x}')
-        print(f'{"length":<25} {self.length}')
+        super().dump()
         print(f'{"data":<25} {data}')
 
     @classmethod
@@ -158,14 +180,11 @@ class QcowHeader(Qcow2Struct):
             end = self.cluster_size
 
         while fd.tell() < end:
-            (magic, length) = struct.unpack('>II', fd.read(8))
-            if magic == 0:
+            ext = QcowHeaderExtension(fd=fd)
+            if ext.magic == 0:
                 break
             else:
-                padded = (length + 7) & ~7
-                data = fd.read(padded)
-                self.extensions.append(QcowHeaderExtension(magic, length,
-                                                           data))
+                self.extensions.append(ext)
 
     def update_extensions(self, fd):
 
Signature · 1 line
2.21.0
Reviewed-by on v4 10/12
#29 lore raw

Re: [PATCH v4 10/12] qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct

I'd add a comment that correct current file offset is the responsibility of a caller.

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 10/12] qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct

Only two fields we can parse by generic code, but that is better than nothing. Keep further refactoring of variable-length fields for another day.

---
 tests/qemu-iotests/qcow2_format.py | 51 ++++++++++++++++++++----------
 1 file changed, 35 insertions(+), 16 deletions(-)
#30 lore raw

PATCH v4 11/12 qcow2: QcowHeaderExtension print names for extension magics

In reply to Vladimir Sementsov-Ogievskiy
---
 tests/qemu-iotests/031.out         | 22 +++++++++++-----------
 tests/qemu-iotests/036.out         |  4 ++--
 tests/qemu-iotests/061.out         | 14 +++++++-------
 tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++-
 4 files changed, 36 insertions(+), 21 deletions(-)
tests/qemu-iotests/031.out +11 -11
index 5a4beda6a2..e0225f4247 100644
@@ -25,7 +25,7 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -53,7 +53,7 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -81,12 +81,12 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0xe2792aca
+magic                     3799591626 (Backing format)
 length                    11
 data                      'host_device'
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -116,12 +116,12 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -149,12 +149,12 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -182,17 +182,17 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0xe2792aca
+magic                     3799591626 (Backing format)
 length                    11
 data                      'host_device'
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
tests/qemu-iotests/036.out +2 -2
index e409acf60e..6cf892123a 100644
@@ -25,7 +25,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        [63]
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -37,7 +37,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
tests/qemu-iotests/061.out +7 -7
index a51ad1b5ba..7afd2be9d9 100644
@@ -25,7 +25,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -83,7 +83,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -139,7 +139,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -194,7 +194,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -263,7 +263,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -325,7 +325,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -354,7 +354,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
tests/qemu-iotests/qcow2_format.py +16 -1
index 4683b1e436..caa190f7b1 100644
@@ -21,6 +21,12 @@ class Flags64(Qcow2Field):
         return str(bits)
 
 
+class Enum(Qcow2Field):
+
+    def __str__(self):
+        return f'{self.value} ({self.mapping.get(self.value, "<unknown>")})'
+
+
 class Qcow2StructMeta(type):
 
     # Mapping from c types to python struct format
@@ -75,8 +81,17 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
 
 class QcowHeaderExtension(Qcow2Struct):
 
+    class Magic(Enum):
+        mapping = {
+            0xE2792ACA: 'Backing format',
+            0x6803f857: 'Feature table',
+            0x0537be77: 'Crypto header',
+            0x23852875: 'Bitmaps',
+            0x44415441: 'Data file'
+        }
+
     fields = (
-        ('u32', '{:#x}', 'magic'),
+        ('u32', Magic, 'magic'),
         ('u32', '{}', 'length')
         # length bytes of data follows
         # then padding to next multiply of 8
Signature · 1 line
2.21.0
Reviewed-by on v4 11/12
#31 lore raw

Re: [PATCH v4 11/12] qcow2: QcowHeaderExtension print names for extension magics

s/0xE2792ACA/0xe2792aca/
(as suggested by Eric)

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 11/12] qcow2: QcowHeaderExtension print names for extension magics

---
 tests/qemu-iotests/031.out         | 22 +++++++++++-----------
 tests/qemu-iotests/036.out         |  4 ++--
 tests/qemu-iotests/061.out         | 14 +++++++-------
 tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++-
 4 files changed, 36 insertions(+), 21 deletions(-)
tests/qemu-iotests/031.out +11 -11
index 5a4beda6a2..e0225f4247 100644
@@ -25,7 +25,7 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -53,7 +53,7 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -81,12 +81,12 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0xe2792aca
+magic                     3799591626 (Backing format)
 length                    11
 data                      'host_device'
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -116,12 +116,12 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -149,12 +149,12 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -182,17 +182,17 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0xe2792aca
+magic                     3799591626 (Backing format)
 length                    11
 data                      'host_device'
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
tests/qemu-iotests/036.out +2 -2
index e409acf60e..6cf892123a 100644
@@ -25,7 +25,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        [63]
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -37,7 +37,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
tests/qemu-iotests/061.out +7 -7
index a51ad1b5ba..7afd2be9d9 100644
@@ -25,7 +25,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -83,7 +83,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -139,7 +139,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -194,7 +194,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -263,7 +263,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -325,7 +325,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
@@ -354,7 +354,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
 length                    336
 data                      <binary>
 
tests/qemu-iotests/qcow2_format.py +16 -1
index 4683b1e436..caa190f7b1 100644
@@ -21,6 +21,12 @@ class Flags64(Qcow2Field):
         return str(bits)
 
 
+class Enum(Qcow2Field):
+
+    def __str__(self):
+        return f'{self.value} ({self.mapping.get(self.value, "<unknown>")})'
+
+
 class Qcow2StructMeta(type):
 
     # Mapping from c types to python struct format
@@ -75,8 +81,17 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
 
 class QcowHeaderExtension(Qcow2Struct):
 
+    class Magic(Enum):
+        mapping = {
+            0xE2792ACA: 'Backing format',
+            0x6803f857: 'Feature table',
+            0x0537be77: 'Crypto header',
+            0x23852875: 'Bitmaps',
+            0x44415441: 'Data file'
+        }
+
     fields = (
-        ('u32', '{:#x}', 'magic'),
+        ('u32', Magic, 'magic'),
         ('u32', '{}', 'length')
         # length bytes of data follows
         # then padding to next multiply of 8

--
2.21.0

#32 lore raw

Re: [PATCH v4 11/12] qcow2: QcowHeaderExtension print names for extension magics

In reply to Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Vladimir Sementsov-Ogievskiy
---
  tests/qemu-iotests/031.out         | 22 +++++++++++-----------
  tests/qemu-iotests/036.out         |  4 ++--
  tests/qemu-iotests/061.out         | 14 +++++++-------
  tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++-
  4 files changed, 36 insertions(+), 21 deletions(-)
tests/qemu-iotests/031.out +0 -0
index 5a4beda6a2..e0225f4247 100644

tests/qemu-iotests/031.out refcount_order 4 · line 28

Quoted 5 lines
tests/qemu-iotests/031.out +1 -1
@@ -25,7 +25,7 @@ refcount_order            4
  header_length             72
 
  Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)

Oops, I've broken hex number. I think it should be kept hex, even with name.

Quoted 204 lines
tests/qemu-iotests/031.out +10 -10
  length                    31
  data                      'This is a test header extension'
 
@@ -53,7 +53,7 @@ refcount_order            4
  header_length             72
 
  Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
  length                    31
  data                      'This is a test header extension'
 
@@ -81,12 +81,12 @@ refcount_order            4
  header_length             72
 
  Header extension:
-magic                     0xe2792aca
+magic                     3799591626 (Backing format)
  length                    11
  data                      'host_device'
 
  Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
  length                    31
  data                      'This is a test header extension'
 
@@ -116,12 +116,12 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
  Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
  length                    31
  data                      'This is a test header extension'
 
@@ -149,12 +149,12 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
  Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
  length                    31
  data                      'This is a test header extension'
 
@@ -182,17 +182,17 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0xe2792aca
+magic                     3799591626 (Backing format)
  length                    11
  data                      'host_device'
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
  Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)
  length                    31
  data                      'This is a test header extension'
 
tests/qemu-iotests/036.out +2 -2
index e409acf60e..6cf892123a 100644
@@ -25,7 +25,7 @@ incompatible_features     []
  compatible_features       []
  autoclear_features        [63]
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
@@ -37,7 +37,7 @@ incompatible_features     []
  compatible_features       []
  autoclear_features        []
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
tests/qemu-iotests/061.out +7 -7
index a51ad1b5ba..7afd2be9d9 100644
@@ -25,7 +25,7 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
@@ -83,7 +83,7 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
@@ -139,7 +139,7 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
@@ -194,7 +194,7 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
@@ -263,7 +263,7 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
@@ -325,7 +325,7 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
@@ -354,7 +354,7 @@ refcount_order            4
  header_length             112
 
  Header extension:
-magic                     0x6803f857
+magic                     1745090647 (Feature table)
  length                    336
  data                      <binary>
 
tests/qemu-iotests/qcow2_format.py +16 -1
index 4683b1e436..caa190f7b1 100644
@@ -21,6 +21,12 @@ class Flags64(Qcow2Field):
          return str(bits)
 
 
+class Enum(Qcow2Field):
+
+    def __str__(self):
+        return f'{self.value} ({self.mapping.get(self.value, "<unknown>")})'
+
+
  class Qcow2StructMeta(type):
 
      # Mapping from c types to python struct format
@@ -75,8 +81,17 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
 
  class QcowHeaderExtension(Qcow2Struct):
 
+    class Magic(Enum):
+        mapping = {
+            0xE2792ACA: 'Backing format',
+            0x6803f857: 'Feature table',
+            0x0537be77: 'Crypto header',
+            0x23852875: 'Bitmaps',
+            0x44415441: 'Data file'
+        }
+
      fields = (
-        ('u32', '{:#x}', 'magic'),
+        ('u32', Magic, 'magic'),
          ('u32', '{}', 'length')
          # length bytes of data follows
          # then padding to next multiply of 8
Signature · 2 lines
Best regards,
Vladimir
#33 lore raw

Re: [PATCH v4 11/12] qcow2: QcowHeaderExtension print names for extension magics

Yes, but hex(int) returns str.

Andrey

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Friday, June 5, 2020 7:26 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: Re: [PATCH v4 11/12] qcow2: QcowHeaderExtension print names for extension magics

Quoted 14 lines · Vladimir Sementsov-Ogievskiy
---
  tests/qemu-iotests/031.out         | 22 +++++++++++-----------
  tests/qemu-iotests/036.out         |  4 ++--
  tests/qemu-iotests/061.out         | 14 +++++++-------
  tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++-
  4 files changed, 36 insertions(+), 21 deletions(-)
tests/qemu-iotests/031.out +0 -0
index 5a4beda6a2..e0225f4247 100644

tests/qemu-iotests/031.out refcount_order 4 · line 28

Quoted 5 lines
tests/qemu-iotests/031.out +1 -1
@@ -25,7 +25,7 @@ refcount_order            4
  header_length             72
 
  Header extension:
-magic                     0x12345678
+magic                     305419896 (<unknown>)

Oops, I've broken hex number. I think it should be kept hex, even with name.

#34 lore raw

PATCH v4 12/12 qcow2_format.py: dump bitmaps header extension

In reply to Vladimir Sementsov-Ogievskiy

Add class for bitmap extension and dump its fields. Further work is to dump bitmap directory.

Test new functionality inside 291 iotest.

---
 tests/qemu-iotests/291             |  4 +++
 tests/qemu-iotests/291.out         | 33 +++++++++++++++++++++++
 tests/qemu-iotests/qcow2_format.py | 42 +++++++++++++++++++++++-------
 3 files changed, 70 insertions(+), 9 deletions(-)
tests/qemu-iotests/291 +4 -0
index 3ca83b9cd1..e0cffc7cb1 100755
@@ -62,6 +62,8 @@ $QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
 $QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1
 $QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2
 $QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+echo "Check resulting qcow2 header extensions:"
+$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
 
 echo
 echo "=== Bitmap preservation not possible to non-qcow2 ==="
@@ -88,6 +90,8 @@ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0
 $QEMU_IMG bitmap --remove --image-opts \
     driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
 $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+echo "Check resulting qcow2 header extensions:"
+$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
 
 echo
 echo "=== Check bitmap contents ==="
tests/qemu-iotests/291.out +33 -0
index 8c62017567..1d4f9cd96d 100644
@@ -14,6 +14,25 @@ wrote 1048576/1048576 bytes at offset 1048576
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1048576/1048576 bytes at offset 2097152
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Check resulting qcow2 header extensions:
+Header extension:
+magic                     3799591626 (Backing format)
+length                    5
+data                      'qcow2'
+
+Header extension:
+magic                     1745090647 (Feature table)
+length                    336
+data                      <binary>
+
+Header extension:
+magic                     595929205 (Bitmaps)
+length                    24
+nb_bitmaps                2
+reserved32                0
+bitmap_directory_size     0x40
+bitmap_directory_offset   0x510000
+
 
 === Bitmap preservation not possible to non-qcow2 ===
 
@@ -65,6 +84,20 @@ Format specific information:
             granularity: 65536
     refcount bits: 16
     corrupt: false
+Check resulting qcow2 header extensions:
+Header extension:
+magic                     1745090647 (Feature table)
+length                    336
+data                      <binary>
+
+Header extension:
+magic                     595929205 (Bitmaps)
+length                    24
+nb_bitmaps                3
+reserved32                0
+bitmap_directory_size     0x60
+bitmap_directory_offset   0x520000
+
 
 === Check bitmap contents ===
 
tests/qemu-iotests/qcow2_format.py +33 -9

69 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v4 12/12
#35 lore raw

Re: [PATCH v4 12/12] qcow2_format.py: dump bitmaps header extension

________________________________
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Sent: Thursday, June 4, 2020 8:41 PM
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Denis Lunev <[email protected]>; Vladimir Sementsov-Ogievskiy <[email protected]>; Andrey Shinkevich <[email protected]>
Subject: [PATCH v4 12/12] qcow2_format.py: dump bitmaps header extension

Add class for bitmap extension and dump its fields. Further work is to dump bitmap directory.

Test new functionality inside 291 iotest.

---
 tests/qemu-iotests/291             |  4 +++
 tests/qemu-iotests/291.out         | 33 +++++++++++++++++++++++
 tests/qemu-iotests/qcow2_format.py | 42 +++++++++++++++++++++++-------
 3 files changed, 70 insertions(+), 9 deletions(-)
#36 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Message-id: [email protected]
Subject: [PATCH v4 00/12] iotests: Dump QCOW2 dirty bitmaps metadata
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
ff3d0e0 qcow2_format.py: dump bitmaps header extension
1ca4b4b qcow2: QcowHeaderExtension print names for extension magics 4894f85 qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct 5d2d3ea qcow2_format.py: QcowHeaderExtension: add dump method 7848f52 qcow2_format.py: add field-formatting class
4f0473f qcow2_format.py: separate generic functionality of structure classes 78964d2 qcow2_format.py: use strings to specify c-type of struct fields 492b753 qcow2_format.py: use modern string formatting
7fef737 qcow2_format.py: use tuples instead of lists for fields 8fae4ab qcow2_format.py: drop new line printing at end of dump() c0b4e4b qcow2.py: move qcow2 format classes to separate module 32668ba qcow2.py: python style fixes

=== OUTPUT BEGIN ===
1/12 Checking commit 32668ba94cf1 (qcow2.py: python style fixes)
ERROR: line over 90 characters
#219: FILE: tests/qemu-iotests/qcow2.py:256:
+ ['dump-header', cmd_dump_header, 0, 'Dump image header and header extensions'],

WARNING: line over 80 characters
#220: FILE: tests/qemu-iotests/qcow2.py:257:
+ ['dump-header-exts', cmd_dump_header_exts, 0, 'Dump image header extensions'],

WARNING: line over 80 characters
#221: FILE: tests/qemu-iotests/qcow2.py:258:
+ ['set-header', cmd_set_header, 2, 'Set a field in the header'],

WARNING: line over 80 characters
#222: FILE: tests/qemu-iotests/qcow2.py:259:
+ ['add-header-ext', cmd_add_header_ext, 2, 'Add a header extension'],

ERROR: line over 90 characters
#223: FILE: tests/qemu-iotests/qcow2.py:260:
+ ['add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin'],

WARNING: line over 80 characters
#224: FILE: tests/qemu-iotests/qcow2.py:261:
+ ['del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension'],

total: 2 errors, 4 warnings, 217 lines checked

Patch 1/12 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

2/12 Checking commit c0b4e4bcc692 (qcow2.py: move qcow2 format classes to separate module)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #186:
new file mode 100644

total: 0 errors, 1 warnings, 324 lines checked

Patch 2/12 has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
3/12 Checking commit 8fae4ab6b058 (qcow2_format.py: drop new line printing at end of dump()) 4/12 Checking commit 7fef73761e63 (qcow2_format.py: use tuples instead of lists for fields) 5/12 Checking commit 492b753d310e (qcow2_format.py: use modern string formatting)
6/12 Checking commit 78964d206cab (qcow2_format.py: use strings to specify c-type of struct fields) 7/12 Checking commit 4f0473f3cf7f (qcow2_format.py: separate generic functionality of structure classes) 8/12 Checking commit 7848f5283863 (qcow2_format.py: add field-formatting class)
9/12 Checking commit 5d2d3ea44d93 (qcow2_format.py: QcowHeaderExtension: add dump method) 10/12 Checking commit 4894f859e63d (qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct) 11/12 Checking commit 1ca4b4b795d8 (qcow2: QcowHeaderExtension print names for extension magics) 12/12 Checking commit ff3d0e0f471a (qcow2_format.py: dump bitmaps header extension)
=== OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#37 lore raw
Quoted 58 lines · 04.06.2020 22:14, [email protected] wrote:

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Message-id: [email protected]
Subject: [PATCH v4 00/12] iotests: Dump QCOW2 dirty bitmaps metadata
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
ff3d0e0 qcow2_format.py: dump bitmaps header extension
1ca4b4b qcow2: QcowHeaderExtension print names for extension magics 4894f85 qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct 5d2d3ea qcow2_format.py: QcowHeaderExtension: add dump method 7848f52 qcow2_format.py: add field-formatting class
4f0473f qcow2_format.py: separate generic functionality of structure classes 78964d2 qcow2_format.py: use strings to specify c-type of struct fields 492b753 qcow2_format.py: use modern string formatting
7fef737 qcow2_format.py: use tuples instead of lists for fields 8fae4ab qcow2_format.py: drop new line printing at end of dump() c0b4e4b qcow2.py: move qcow2 format classes to separate module 32668ba qcow2.py: python style fixes

=== OUTPUT BEGIN ===
1/12 Checking commit 32668ba94cf1 (qcow2.py: python style fixes)
ERROR: line over 90 characters
#219: FILE: tests/qemu-iotests/qcow2.py:256:
+ ['dump-header', cmd_dump_header, 0, 'Dump image header and header extensions'],

WARNING: line over 80 characters
#220: FILE: tests/qemu-iotests/qcow2.py:257:
+ ['dump-header-exts', cmd_dump_header_exts, 0, 'Dump image header extensions'],

WARNING: line over 80 characters
#221: FILE: tests/qemu-iotests/qcow2.py:258:
+ ['set-header', cmd_set_header, 2, 'Set a field in the header'],

WARNING: line over 80 characters
#222: FILE: tests/qemu-iotests/qcow2.py:259:
+ ['add-header-ext', cmd_add_header_ext, 2, 'Add a header extension'],

ERROR: line over 90 characters
#223: FILE: tests/qemu-iotests/qcow2.py:260:
+ ['add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin'],

Quoted 7 lines

WARNING: line over 80 characters
#224: FILE: tests/qemu-iotests/qcow2.py:261:
+ ['del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension'],

total: 2 errors, 4 warnings, 217 lines checked

Patch 1/12 has style problems, please review. If any of these errors

Preexisting. And it will look worse if wrap lines.

Signature · 2 lines
Best regards,
Vladimir
#38 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 7 lines · Vladimir Sementsov-Ogievskiy

Hi all!

Here is my suggestion to substitute only first three patches :) of Andrey's [PATCH v3 0/6] iotests: Dump QCOW2 dirty bitmaps metadata

so, I called it v4 for convenience.

Quoted 8 lines

What is here:
1. First, update code style
2. Next, try to refactor in a manner which will make adding new data structures simple (look at Qcow2BitmapExt class in last patch)

I think, next step is to add type hints. Then add more structures. And, anyway, at some point we should move it into python/ directory (at least qcow2_format.py lib)

My python reviewing skills are weak, but in general this series makes sense. I had enough comments that you're probably better off spinning a v5, but it looks like it is probably close enough that I can include this in my next bitmaps pull request, and we can work on rebasing the remainder of Andrey's patches on top of it.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org