Skip to contentSkip to conversation

spec: add qcow2 bitmaps extension specification

PATCH Superseded Vladimir Sementsov-Ogievskiy 5 participants 22 messages Jan 11, 2016 – Jan 25, 2016

v7 of 22 Superseded Latest is v22 →

lore
Share & tools
lore
#0 lore raw

PATCH v7 1/1 spec: add qcow2 bitmaps extension specification

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

 docs/specs/qcow2.txt | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +171 -1
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                     write to an image with unknown auto-clear features if it
                     clears the respective bits from this field first.
 
-                    Bits 0-63:  Reserved (set to 0)
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for the bitmaps
+                                extension data.
+
+                                It is an error if this bit is set without the
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present but this
+                                bit is unset, the bitmaps extension data is
+                                inconsistent.
+
+                    Bits 1-63:  Reserved (set to 0)
 
          96 -  99:  refcount_order
                     Describes the width of a reference count block entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the following:
                         0x00000000 - End of the header extension area
                         0xE2792ACA - Backing file format name
                         0x6803f857 - Feature name table
+                        0x23852875 - Bitmaps extension
                         other      - Unknown header extension, can be safely
                                      ignored
 
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                     terminated if it has full length)
 
 
+== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap directory
+                   starts. Must be aligned to a cluster boundary.
+
+
 == Host cluster management ==
 
 qcow2 manages the allocation of host clusters by maintaining a reference count
@@ -360,3 +400,133 @@ Snapshot table entry:
 
         variable:   Padding to round up the snapshot table entry size to the
                     next multiple of 8.
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the virtual
+                         disk by any application that would write to this qcow2
+                         file (including writes, snapshot switching, etc.). The
+                         type of this bitmap must be 'dirty tracking bitmap'.
+
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support granularity_bits
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the image
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values greater than
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.
+
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.
Signature · 1 line
2.1.4
#1 lore raw
Quoted 243 lines · Vladimir Sementsov-Ogievskiy

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

 docs/specs/qcow2.txt | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +161 -1
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                     write to an image with unknown auto-clear features if it
                     clears the respective bits from this field first.
 
-                    Bits 0-63:  Reserved (set to 0)
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for the bitmaps
+                                extension data.
+
+                                It is an error if this bit is set without the
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present but this
+                                bit is unset, the bitmaps extension data is
+                                inconsistent.
+
+                    Bits 1-63:  Reserved (set to 0)
 
          96 -  99:  refcount_order
                     Describes the width of a reference count block entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the following:
                         0x00000000 - End of the header extension area
                         0xE2792ACA - Backing file format name
                         0x6803f857 - Feature name table
+                        0x23852875 - Bitmaps extension
                         other      - Unknown header extension, can be safely
                                      ignored
 
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                     terminated if it has full length)
 
 
+== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap directory
+                   starts. Must be aligned to a cluster boundary.
+
+
 == Host cluster management ==
 
 qcow2 manages the allocation of host clusters by maintaining a reference count
@@ -360,3 +400,133 @@ Snapshot table entry:
 
         variable:   Padding to round up the snapshot table entry size to the
                     next multiple of 8.
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the virtual
+                         disk by any application that would write to this qcow2
+                         file (including writes, snapshot switching, etc.). The
+                         type of this bitmap must be 'dirty tracking bitmap'.
+
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support granularity_bits
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the image
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values greater than
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.
+
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see
+above).
+

docs/specs/qcow2.txt Snapshot table entry: · line 532

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

s/multiply/multiple/.

Looks good otherwise, pending the results of our discussion about what to do about the extra_data_size / type-specific data fields.

Thanks!

Signature · 1 line
—js
#2 lore raw
Quoted 257 lines · John Snow
Quoted 243 lines · Vladimir Sementsov-Ogievskiy

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +161 -1
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                      write to an image with unknown auto-clear features if it
                      clears the respective bits from this field first.
 
-                    Bits 0-63:  Reserved (set to 0)
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for the bitmaps
+                                extension data.
+
+                                It is an error if this bit is set without the
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present but this
+                                bit is unset, the bitmaps extension data is
+                                inconsistent.
+
+                    Bits 1-63:  Reserved (set to 0)
 
           96 -  99:  refcount_order
                      Describes the width of a reference count block entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the following:
                          0x00000000 - End of the header extension area
                          0xE2792ACA - Backing file format name
                          0x6803f857 - Feature name table
+                        0x23852875 - Bitmaps extension
                          other      - Unknown header extension, can be safely
                                       ignored
 
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                      terminated if it has full length)
 
 
+== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap directory
+                   starts. Must be aligned to a cluster boundary.
+
+
  == Host cluster management ==
 
  qcow2 manages the allocation of host clusters by maintaining a reference count
@@ -360,3 +400,133 @@ Snapshot table entry:
 
          variable:   Padding to round up the snapshot table entry size to the
                      next multiple of 8.
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the virtual
+                         disk by any application that would write to this qcow2
+                         file (including writes, snapshot switching, etc.). The
+                         type of this bitmap must be 'dirty tracking bitmap'.
+
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support granularity_bits
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the image
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values greater than
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.
+
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see
+above).
+

docs/specs/qcow2.txt Snapshot table entry: · line 532

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

s/multiply/multiple/.

Quoted 5 lines

Looks good otherwise, pending the results of our discussion about what to do about the extra_data_size / type-specific data fields.

Thanks!

should we re-submit? Is this version good enough to be accepted? :)

Den

#3 lore raw
Quoted 306 lines · Denis V. Lunev
Quoted 305 lines · John Snow
Quoted 296 lines · Vladimir Sementsov-Ogievskiy

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 172
++++++++++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +0 -0
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                      write to an image with unknown auto-clear

features if it
clears the respective bits from this field first.
- Bits 0-63: Reserved (set to 0)

docs/specs/qcow2.txt +2 -0
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for

the bitmaps

docs/specs/qcow2.txt +3 -0
+                                extension data.
+
+                                It is an error if this bit is set

without the

docs/specs/qcow2.txt +3 -0
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present

but this
+ bit is unset, the bitmaps extension
data is

docs/specs/qcow2.txt +3 -0
+                                inconsistent.
+
+                    Bits 1-63:  Reserved (set to 0)
             96 -  99:  refcount_order
                      Describes the width of a reference count block

entry (width

docs/specs/qcow2.txt +0 -0
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like

the following:

                          0x00000000 - End of the header extension area
                          0xE2792ACA - Backing file format name
                          0x6803f857 - Feature name table

+ 0x23852875 - Bitmaps extension
other - Unknown header extension, can
be safely
ignored
@@ -166,6 +178,34 @@ the header extension data. Each entry look
like this:
terminated if it has full length)
+== Bitmaps extension ==

docs/specs/qcow2.txt +2 -0
+
+The bitmaps extension is an optional header extension. It provides

the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some

docs/specs/qcow2.txt +4 -0
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features

above.

docs/specs/qcow2.txt +5 -0
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image.

Must be

docs/specs/qcow2.txt +3 -0
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535

bitmaps per

docs/specs/qcow2.txt +4 -0
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the

cumulative

docs/specs/qcow2.txt +4 -0
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap

directory

docs/specs/qcow2.txt +3 -0
+                   starts. Must be aligned to a cluster boundary.
+
+
  == Host cluster management ==
    qcow2 manages the allocation of host clusters by maintaining a

reference count

docs/specs/qcow2.txt +0 -0
@@ -360,3 +400,133 @@ Snapshot table entry:
            variable:   Padding to round up the snapshot table entry

size to the
next multiple of 8.

docs/specs/qcow2.txt +5 -0
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to

store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

docs/specs/qcow2.txt +6 -0
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory

entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These

docs/specs/qcow2.txt +6 -0
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap

table
+ (described below) for the bitmap starts. Must be
aligned to

docs/specs/qcow2.txt +4 -0
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the

bitmap.

docs/specs/qcow2.txt +9 -0
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the

virtual
+ disk by any application that would write to
this qcow2
+ file (including writes, snapshot switching,
etc.). The
+ type of this bitmap must be 'dirty tracking
bitmap'.

docs/specs/qcow2.txt +14 -0
+
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support

granularity_bits

docs/specs/qcow2.txt +6 -0
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the

image

docs/specs/qcow2.txt +6 -0
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values

greater than

docs/specs/qcow2.txt +6 -0
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined,

extra_data_size is

docs/specs/qcow2.txt +5 -0
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.
+
+        variable:   The name of the bitmap (not null terminated).

Must be
+ unique among all bitmap names within the bitmaps
extension.

docs/specs/qcow2.txt +2 -0
+
+        variable:   Padding to round up the bitmap directory entry

size to the

docs/specs/qcow2.txt +6 -0
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the

mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

docs/specs/qcow2.txt +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap

directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image

docs/specs/qcow2.txt +5 -0
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are

non-zero.

docs/specs/qcow2.txt +7 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be

aligned to
+ a cluster boundary. If the offset is 0, the
cluster is
+ unallocated; in that case, bit 0 determines how
this

docs/specs/qcow2.txt +7 -0
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters,

described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into

docs/specs/qcow2.txt +7 -0
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are

zero (see

docs/specs/qcow2.txt +3 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's

granularity, the

docs/specs/qcow2.txt +7 -0
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size
Quoted 5 lines

then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

s/multiply/multiple/.

Quoted 8 lines
Quoted 5 lines

Looks good otherwise, pending the results of our discussion about what to do about the extra_data_size / type-specific data fields.

Thanks!

should we re-submit? Is this version good enough to be accepted? :)

Den

For my money, I'm happy -- though depending on how Vladimir wishes to use his type-specific data field, it may require some clarification. (I had a question over whether or not we needed versioning for this field if the plan was to add type-specific data to a type later, or if the extra data was always defined directly alongside a new type.)

The 2.6 window isn't close to over yet, though, so I would personally be OK with follow-up patches.

Max, Eric, any further input?

--js

#4 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 21 lines · Vladimir Sementsov-Ogievskiy

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

docs/specs/qcow2.txt +1 -0
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                     terminated if it has full length)
 
 
+== Bitmaps extension ==
 

docs/specs/qcow2.txt the header extension data. Each entry look like this: · line 192

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.

Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps), could we run into an image that has so many directory entries as to make the directory itself spill past 4G? But I don't think it is likely, so I can live with your choice.

Quoted 16 lines
docs/specs/qcow2.txt +16 -0
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.

docs/specs/qcow2.txt Snapshot table entry: · line 429

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.

Should this be the size in bytes, instead of the number of entries? But at least the entries are fixed width of 8 bytes each, so this lets you get a bitmap table up to 32G bytes rather than just 4G in size. (Let's see here - if we have 32G bytes in the bitmap table, that means 4G clusters occupied by the bitmap itself; in the worst case of 512-byte clusters and granularity 0, that is a maximum bitmap size of 2T describing 16T of virtual guest image; with larger cluster size and/or larger granularity, we cover a lot more virtual guest space with less bitmap size; so I guess we aren't too worried about running out of space?).

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 7 lines
docs/specs/qcow2.txt +7 -0
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

I'd write this as:
variable: extra_data
Type-specific extra data for the bitmap, occupying extra_data_size bytes.

docs/specs/qcow2.txt Snapshot table entry: · line 482

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.

Should we require the padding to be all NUL bytes? (We aren't consistent on whether we require that for other locations of padding in the spec, so that could be a followup patch).

docs/specs/qcow2.txt Snapshot table entry: · line 488

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

Possible wording tweak:
Bitmaps are stored using a one-level structure (as opposed to the two-level structures for refcounts and guest cluster mapping), and are used for the mapping of bitmap data to host clusters

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)

Does 'Entry' still need to be capitalized?

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.

docs/specs/qcow2.txt Snapshot table entry: · line 506

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.

Possible wording tweak:
s/when read from/during reads/.

Quoted 14 lines
docs/specs/qcow2.txt +14 -0
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see

docs/specs/qcow2.txt Snapshot table entry: · line 530

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last

s/multiply of cluster size/multiple of the cluster size,/

Quoted 3 lines
docs/specs/qcow2.txt +2 -0
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#5 lore raw
Quoted 22 lines · Eric Blake
Quoted 20 lines · Vladimir Sementsov-Ogievskiy

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

docs/specs/qcow2.txt +1 -0
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                     terminated if it has full length)
 
 
+== Bitmaps extension ==
Quoted 16 lines

docs/specs/qcow2.txt the header extension data. Each entry look like this: · line 202

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.

Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps), could we run into an image that has so many directory entries as to make the directory itself spill past 4G? But I don't think it is likely, so I can live with your choice.

"We'll never need this!"

I hope someone in 2082 is reading this right now and is quite angry.

(But really, I can't foresee needing this many per each drive -- and if we do, we have external storage mechanisms in development to handle such wild cases.)

Quoted 20 lines
Quoted 20 lines
docs/specs/qcow2.txt +20 -0
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
Quoted 8 lines

docs/specs/qcow2.txt Snapshot table entry: · line 429

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.

Should this be the size in bytes, instead of the number of entries? But

For what benefit? We can calculate either from the other, and this gives us a better resolution.

Quoted 9 lines

at least the entries are fixed width of 8 bytes each, so this lets you get a bitmap table up to 32G bytes rather than just 4G in size. (Let's see here - if we have 32G bytes in the bitmap table, that means 4G clusters occupied by the bitmap itself; in the worst case of 512-byte clusters and granularity 0, that is a maximum bitmap size of 2T describing 16T of virtual guest image; with larger cluster size and/or larger granularity, we cover a lot more virtual guest space with less bitmap size; so I guess we aren't too worried about running out of space?).

Yes, worst case of g=0 and cluster size of 512 bytes, we can get 2T bitmaps describing 16T of virtual data.

"default case" of 64K clusters and 64K granularity: 256TiB bitmaps describing ... let's see ... if my math is right, 128EiB?

We're probably fine :)

(Cue future space-person from 2159 wondering how I could have ever been so naive. Sorry, future space-person!)

Quoted 88 lines

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 7 lines
docs/specs/qcow2.txt +7 -0
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

I'd write this as:
variable: extra_data
Type-specific extra data for the bitmap, occupying extra_data_size bytes.

docs/specs/qcow2.txt Snapshot table entry: · line 482

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.

Should we require the padding to be all NUL bytes? (We aren't consistent on whether we require that for other locations of padding in the spec, so that could be a followup patch).

docs/specs/qcow2.txt Snapshot table entry: · line 488

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

Possible wording tweak:
Bitmaps are stored using a one-level structure (as opposed to the two-level structures for refcounts and guest cluster mapping), and are used for the mapping of bitmap data to host clusters

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)

Does 'Entry' still need to be capitalized?

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.

docs/specs/qcow2.txt Snapshot table entry: · line 506

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.

Possible wording tweak:
s/when read from/during reads/.

Quoted 14 lines
docs/specs/qcow2.txt +14 -0
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see

docs/specs/qcow2.txt Snapshot table entry: · line 530

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last

s/multiply of cluster size/multiple of the cluster size,/

Quoted 4 lines

docs/specs/qcow2.txt Snapshot table entry: · line 532

Quoted 3 lines
docs/specs/qcow2.txt +2 -0
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

Thanks!

--js

#6 lore raw
Quoted 37 lines · John Snow
Quoted 19 lines · Eric Blake
Quoted 18 lines · Vladimir Sementsov-Ogievskiy

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

docs/specs/qcow2.txt +0 -0
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                      terminated if it has full length)
 
 
Quoted 16 lines

docs/specs/qcow2.txt the header extension data. Each entry look like this: · line 202

Quoted 11 lines
docs/specs/qcow2.txt +11 -0
+== Bitmaps extension ==
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.

Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps), could we run into an image that has so many directory entries as to make the directory itself spill past 4G? But I don't think it is likely, so I can live with your choice.

Quoted 7 lines

"We'll never need this!"

I hope someone in 2082 is reading this right now and is quite angry.

(But really, I can't foresee needing this many per each drive -- and if we do, we have external storage mechanisms in development to handle such wild cases.)

Let's change it to 8 bytes, and add 4bytes reserved field to keep 8b alignment.

Quoted 43 lines
Quoted 20 lines
Quoted 20 lines
docs/specs/qcow2.txt +19 -0
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
Quoted 8 lines

docs/specs/qcow2.txt Snapshot table entry: · line 429

Quoted 7 lines
docs/specs/qcow2.txt +7 -0
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.

Should this be the size in bytes, instead of the number of entries? But

For what benefit? We can calculate either from the other, and this gives us a better resolution.

Quoted 9 lines

at least the entries are fixed width of 8 bytes each, so this lets you get a bitmap table up to 32G bytes rather than just 4G in size. (Let's see here - if we have 32G bytes in the bitmap table, that means 4G clusters occupied by the bitmap itself; in the worst case of 512-byte clusters and granularity 0, that is a maximum bitmap size of 2T describing 16T of virtual guest image; with larger cluster size and/or larger granularity, we cover a lot more virtual guest space with less bitmap size; so I guess we aren't too worried about running out of space?).

Yes, worst case of g=0 and cluster size of 512 bytes, we can get 2T bitmaps describing 16T of virtual data.

Quoted 7 lines

"default case" of 64K clusters and 64K granularity: 256TiB bitmaps describing ... let's see ... if my math is right, 128EiB?

We're probably fine :)

(Cue future space-person from 2159 wondering how I could have ever been so naive. Sorry, future space-person!)

it is just like l1_size from qcow2 header.

36 - 39: l1_size
Number of entries in the active L1 table

Quoted 12 lines
Quoted 12 lines

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 8 lines
docs/specs/qcow2.txt +7 -0
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

I'd write this as:
variable: extra_data
Type-specific extra data for the bitmap, occupying extra_data_size bytes.

Ok.. then, similar should be done for the name.

About extra data and our discussion of versioning it. For now, we don't have concrete cases and requirements for this data, we are only foresee, that it may be needed in future, and waste time in assumptions about what kind of data will it be..

Will it be really type specific, i.e. bitmap type strictly define its structure? Or it will contain self-defined fields, some of them will be optional, some of them will depend on bitmap type or something like this? We have no concrete cases and strategy of using this data... We need not versioning for now, as we have no versions).

The only question, that should be answered, what should do the software, if it find extra_data of unknown format. (currently any extra data is unknown).

variants:

1) abort (or, more precisely: "image file must not be written") (like incompatible feature)
2) delete extra data (like autoclear feature)
3) ignore (like compatible feature)

We can use two bits of flags field to specify the behavior.

===========================================

Or, the other way, we should fix extra data format like this:

Extra data contains several fields of the following format:

4b: field magic
1b: flags: specifies abort/delete/ignore behavior if the field is unknown 1b: field name size - for error message if the field is unknown 2b: field size
var: field name
var: field data
var: padding

- where I've seen it...

=================

And the last option:

We can select abort behavior for now for any extra data, i.e. just remove "variable extra_data" and make field extra_data_size 'reserved must be zero'.

================

So, lets finally choose something and go on.

Note: for ways 1 and 3, we should in future define extra data in the way that gives a software clear method to understand, is this extra data known or unknown for it. I.e. it should have some magic number, defining format, or version, or structure like in way 2.

Quoted 10 lines
Quoted 10 lines

docs/specs/qcow2.txt Snapshot table entry: · line 482

Quoted 7 lines
docs/specs/qcow2.txt +6 -0
+
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.

Should we require the padding to be all NUL bytes? (We aren't consistent on whether we require that for other locations of padding in the spec, so that could be a followup patch).

Let's require this, I do not see any shortcomings of this, only benefits.

Quoted 11 lines
Quoted 11 lines

docs/specs/qcow2.txt Snapshot table entry: · line 488

Quoted 7 lines
docs/specs/qcow2.txt +6 -0
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

Possible wording tweak:
Bitmaps are stored using a one-level structure (as opposed to the two-level structures for refcounts and guest cluster mapping), and are used for the mapping of bitmap data to host clusters

No, this is wrong.. Who are used? Bitmaps are stored and are used? No, not bitmaps, structures are used.

Quoted 4 lines
Quoted 4 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)

Does 'Entry' still need to be capitalized?

It's a mistake, thanks.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
docs/specs/qcow2.txt +10 -0
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
Quoted 8 lines
Quoted 8 lines

docs/specs/qcow2.txt Snapshot table entry: · line 506

Quoted 6 lines
docs/specs/qcow2.txt +6 -0
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.

Possible wording tweak:
s/when read from/during reads/.

ok

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
docs/specs/qcow2.txt +17 -0
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
Quoted 8 lines
Quoted 8 lines

docs/specs/qcow2.txt Snapshot table entry: · line 530

Quoted 7 lines
docs/specs/qcow2.txt +7 -0
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last

s/multiply of cluster size/multiple of the cluster size,/

ok, thanks

Quoted 7 lines
Quoted 4 lines

docs/specs/qcow2.txt Snapshot table entry: · line 532

Quoted 4 lines
docs/specs/qcow2.txt +2 -0
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

Thanks!

--js

Please, let's decide finally about extra data, than I'll reroll it and, I hope, it will be committed, to make it possible to continue work on persistence series. About extra data, I'm ready to accept any variant, strictly defining, what software should do with unknown extra data.

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
#7 lore raw
Quoted 305 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · John Snow
Quoted 28 lines · Eric Blake
Quoted 27 lines · Vladimir Sementsov-Ogievskiy

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

(quoted diff) +0 -0
@@ -166,6 +178,34 @@ the header extension data. Each entry look like

this:
terminated if it has full length)
+== Bitmaps extension ==

(quoted diff) +2 -0
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image.

Must be

docs/specs/qcow2.txt +3 -0
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535

bitmaps per

(quoted diff) +2 -0
+                   image.
+
Quoted 9 lines
Quoted 4 lines
(quoted diff) +2 -0
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the

cumulative
+ size of all (nb_bitmaps) bitmap headers.

Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps), could we run into an image that has so many directory entries as to make the directory itself spill past 4G? But I don't think it is likely, so I can live with your choice.

Quoted 7 lines

"We'll never need this!"

I hope someone in 2082 is reading this right now and is quite angry.

(But really, I can't foresee needing this many per each drive -- and if we do, we have external storage mechanisms in development to handle such wild cases.)

Let's change it to 8 bytes, and add 4bytes reserved field to keep 8b alignment.

Quoted 55 lines
Quoted 31 lines
Quoted 31 lines
docs/specs/qcow2.txt +4 -0
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to

store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

(quoted diff) +2 -0
+
+Note: all bitmaps are related to the virtual disk stored in this

image.

docs/specs/qcow2.txt +4 -0
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory

entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These

docs/specs/qcow2.txt +6 -0
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap

table

Quoted 8 lines
Quoted 7 lines

+ (described below) for the bitmap starts. Must
be aligned to

docs/specs/qcow2.txt +4 -0
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the

bitmap.

Should this be the size in bytes, instead of the number of entries? But

For what benefit? We can calculate either from the other, and this gives us a better resolution.

Quoted 10 lines

at least the entries are fixed width of 8 bytes each, so this lets you get a bitmap table up to 32G bytes rather than just 4G in size. (Let's see here - if we have 32G bytes in the bitmap table, that means 4G clusters occupied by the bitmap itself; in the worst case of 512-byte clusters and granularity 0, that is a maximum bitmap size of 2T describing 16T of virtual guest image; with larger cluster size and/or larger granularity, we cover a lot more virtual guest space with less bitmap size; so I guess we aren't too worried about running out of space?).

Yes, worst case of g=0 and cluster size of 512 bytes, we can get 2T bitmaps describing 16T of virtual data.

Quoted 7 lines

"default case" of 64K clusters and 64K granularity: 256TiB bitmaps describing ... let's see ... if my math is right, 128EiB?

We're probably fine :)

(Cue future space-person from 2159 wondering how I could have ever been so naive. Sorry, future space-person!)

it is just like l1_size from qcow2 header.

36 - 39: l1_size
Number of entries in the active L1 table

Quoted 13 lines
Quoted 13 lines

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 9 lines
docs/specs/qcow2.txt +4 -0
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined,

extra_data_size is

docs/specs/qcow2.txt +3 -0
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

I'd write this as:
variable: extra_data
Type-specific extra data for the bitmap, occupying extra_data_size bytes.

Ok.. then, similar should be done for the name.

About extra data and our discussion of versioning it. For now, we don't have concrete cases and requirements for this data, we are only foresee, that it may be needed in future, and waste time in assumptions about what kind of data will it be..

Will it be really type specific, i.e. bitmap type strictly define its structure? Or it will contain self-defined fields, some of them will be optional, some of them will depend on bitmap type or something like this? We have no concrete cases and strategy of using this data... We need not versioning for now, as we have no versions).

The only question, that should be answered, what should do the software, if it find extra_data of unknown format. (currently any extra data is unknown).

variants:

1) abort (or, more precisely: "image file must not be written") (like incompatible feature)
2) delete extra data (like autoclear feature)
3) ignore (like compatible feature)

We can use two bits of flags field to specify the behavior.

===========================================

Or, the other way, we should fix extra data format like this:

Extra data contains several fields of the following format:

4b: field magic
1b: flags: specifies abort/delete/ignore behavior if the field is unknown 1b: field name size - for error message if the field is unknown 2b: field size
var: field name
var: field data
var: padding

- where I've seen it...

=================

And the last option:

We can select abort behavior for now for any extra data, i.e. just remove "variable extra_data" and make field extra_data_size 'reserved must be zero'.

================

So, lets finally choose something and go on.

Note: for ways 1 and 3, we should in future define extra data in the way that gives a software clear method to understand, is this extra data known or unknown for it. I.e. it should have some magic number, defining format, or version, or structure like in way 2.

Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
(quoted diff) +2 -0
+
+        variable:   The name of the bitmap (not null terminated).

Must be
+ unique among all bitmap names within the
bitmaps extension.

(quoted diff) +2 -0
+
+        variable:   Padding to round up the bitmap directory entry

size to the
+ next multiple of 8.

Should we require the padding to be all NUL bytes? (We aren't consistent on whether we require that for other locations of padding in the spec, so that could be a followup patch).

Let's require this, I do not see any shortcomings of this, only benefits.

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
docs/specs/qcow2.txt +4 -0
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to

two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

Possible wording tweak:
Bitmaps are stored using a one-level structure (as opposed to the two-level structures for refcounts and guest cluster mapping), and are used for the mapping of bitmap data to host clusters

No, this is wrong.. Who are used? Bitmaps are stored and are used? No, not bitmaps, structures are used.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
(quoted diff) +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap

directory Entry)

Does 'Entry' still need to be capitalized?

It's a mistake, thanks.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

+and may use multiple clusters, however, it must be contiguous in the image

docs/specs/qcow2.txt +5 -0
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are

non-zero.

docs/specs/qcow2.txt +7 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

aligned to
+ a cluster boundary. If the offset is 0, the
cluster is
+ unallocated; in that case, bit 0 determines how
this
+ cluster should be treated when read from.

Possible wording tweak:
s/when read from/during reads/.

ok

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
docs/specs/qcow2.txt +6 -0
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters,

described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into

docs/specs/qcow2.txt +7 -0
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are

zero (see

docs/specs/qcow2.txt +3 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's

granularity, the
+bit offset into the bitmap can be calculated like this:

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
docs/specs/qcow2.txt +6 -0
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size

then the last

s/multiply of cluster size/multiple of the cluster size,/

ok, thanks

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

Thanks!

Quoted 8 lines
Quoted 1 line

--js

Please, let's decide finally about extra data, than I'll reroll it and, I hope, it will be committed, to make it possible to continue work on persistence series. About extra data, I'm ready to accept any variant, strictly defining, what software should do with unknown extra data.

I discussed this with Eric Blake on IRC briefly, and I mentioned I was concerned that we didn't specify a format at all for the extra data. Eric felt that it was not unusual to leave a space for future expansion and that as we haven't used it yet, we don't need to solidify it.

He also felt it would be unusual to stipulate the format of data that we don't even intend to use yet.

In short, I'm being too proactive.

A commit message mention that, should anyone wish to expand the type-specific data in the future that adding a 2-byte version as the first field in extra data would probably be sufficient, and we can worry about the spec wording later. It is fine to assume for now that if extra_data_size is 0 that the version/format of the data is "v0" and that does not limit our future expansion.

Sound good? Sorry for the red tape.
--John Snow

#8 lore raw
Quoted 15 lines · John Snow
Quoted 6 lines

Please, let's decide finally about extra data, than I'll reroll it and, I hope, it will be committed, to make it possible to continue work on persistence series. About extra data, I'm ready to accept any variant, strictly defining, what software should do with unknown extra data.

I discussed this with Eric Blake on IRC briefly, and I mentioned I was concerned that we didn't specify a format at all for the extra data. Eric felt that it was not unusual to leave a space for future expansion and that as we haven't used it yet, we don't need to solidify it.

He also felt it would be unusual to stipulate the format of data that we don't even intend to use yet.

Quoted 8 lines

In short, I'm being too proactive.

A commit message mention that, should anyone wish to expand the type-specific data in the future that adding a 2-byte version as the first field in extra data would probably be sufficient, and we can worry about the spec wording later. It is fine to assume for now that if extra_data_size is 0 that the version/format of the data is "v0" and that does not limit our future expansion.

Or put another way:

I'm just fine if our initial implementation provides sufficient information for us to completely parse the file even when the file is generated by a newer qemu (we have a length, so we know how far to skip to find the next entry), while at the same time throwing up our hands if the length is non-zero (we won't read the bitmap at all, because we don't know if the non-zero extra_data contains instructions that would change how to interpret the data) or even prevent writes (if the bitmap entry is marked automatic, we must refuse any write that would requiring an update to the bitmap because we don't know how to write to a bitmap while correctly preserving semantics of those extra_data bytes). We have enough room for future extension, and that's good enough for now; the commit message can document that we thought about the future, without having to actually nail down in the spec what the future will actually do.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#9 lore raw
Quoted 2 lines · Eric Blake

preserving semantics of those extra_data bytes). We have enough room for future extension, and that's good e

Ok, so, what should go to the spec? Current wording is ok? Just delete "Type-specific":

+
+ 20 - 23: extra_data_size
+ Size of type-specific extra data.
+
+ For now, as no extra data is defined, extra_data_size is
+ reserved and must be zero.
+
+ variable: Extra data for the bitmap.
+

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
#10 lore raw

Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 7 lines · Eric Blake
Quoted 2 lines

preserving semantics of those extra_data bytes). We have enough room for future extension, and that's good e

Ok, so, what should go to the spec? Current wording is ok? Just delete "Type-specific":

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Extra data for the bitmap.

Please be explicit that if extra_data_size is non-zero, the bitmap must not be used (i.e. specify the incompatible-feature-bit-like behaviour).

Kevin

#11 lore raw
Quoted 7 lines · Kevin Wolf

Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines · Eric Blake
Quoted 2 lines

preserving semantics of those extra_data bytes). We have enough room for future extension, and that's good e

Ok, so, what should go to the spec? Current wording is ok? Just delete "Type-specific":

Quoted 10 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Extra data for the bitmap.

Please be explicit that if extra_data_size is non-zero, the bitmap must not be used (i.e. specify the incompatible-feature-bit-like behaviour).

It is not enough. If there are some unknown extra data, then just ignoring this bitmap may lead to its inconsistency. So, if it is non-zero, the whole image should not be written. (real incompatible-feature-bit behavior).

Quoted 2 lines

Kevin

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
#12 lore raw

Am 25.01.2016 um 11:15 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 19 lines · Kevin Wolf
Quoted 7 lines

Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines · Eric Blake
Quoted 2 lines

preserving semantics of those extra_data bytes). We have enough room for future extension, and that's good e

Ok, so, what should go to the spec? Current wording is ok? Just delete "Type-specific":

Quoted 10 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Extra data for the bitmap.

Please be explicit that if extra_data_size is non-zero, the bitmap must not be used (i.e. specify the incompatible-feature-bit-like behaviour).

Quoted 4 lines

It is not enough. If there are some unknown extra data, then just ignoring this bitmap may lead to its inconsistency. So, if it is non-zero, the whole image should not be written. (real incompatible-feature-bit behavior).

Don't we generally ignore all bitmaps until a user actively tries to make use of it? Of course, with the 'auto' flag set, just ignoring the bitmap isn't possible, but I think in all other cases it should be.

If we ever add another type of bitmaps that doesn't have the 'auto' flag set, but is still automatically used, so that the image as a whole must become read-only without the bitmap, we can still add a normal incompatible feature flag. But I think it's more likely that we add extra_data that doesn't prevent use of the image, so we should have a way to express that.

Kevin

#13 lore raw
Quoted 27 lines · Kevin Wolf

Am 25.01.2016 um 11:15 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 18 lines · Kevin Wolf
Quoted 7 lines

Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines · Eric Blake
Quoted 2 lines

preserving semantics of those extra_data bytes). We have enough room for future extension, and that's good e

Ok, so, what should go to the spec? Current wording is ok? Just delete "Type-specific":

Quoted 10 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Extra data for the bitmap.

Please be explicit that if extra_data_size is non-zero, the bitmap must not be used (i.e. specify the incompatible-feature-bit-like behaviour).

Quoted 4 lines

It is not enough. If there are some unknown extra data, then just ignoring this bitmap may lead to its inconsistency. So, if it is non-zero, the whole image should not be written. (real incompatible-feature-bit behavior).

Don't we generally ignore all bitmaps until a user actively tries to make use of it? Of course, with the 'auto' flag set, just ignoring the bitmap isn't possible, but I think in all other cases it should be.

Quoted 8 lines

If we ever add another type of bitmaps that doesn't have the 'auto' flag set, but is still automatically used, so that the image as a whole must become read-only without the bitmap, we can still add a normal incompatible feature flag. But I think it's more likely that we add extra_data that doesn't prevent use of the image, so we should have a way to express that.

Kevin

let's add one flag:

2: extra_data_compatible
This flag is meaningful when extra data is unknown for the software. If it is set, the bitmap may be used as usual, extra data must
be left as is.
If it is unset, the bitmap must not be used, but left as is with
extra data.

Ok? And, if there will be incompatible features we will use normal incompatible feature flag.

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
#14 lore raw
In reply to Eric Blake

Am 18.01.2016 um 22:16 hat Eric Blake geschrieben:

Quoted 28 lines · John Snow
Quoted 16 lines
Quoted 7 lines

Please, let's decide finally about extra data, than I'll reroll it and, I hope, it will be committed, to make it possible to continue work on persistence series. About extra data, I'm ready to accept any variant, strictly defining, what software should do with unknown extra data.

I discussed this with Eric Blake on IRC briefly, and I mentioned I was concerned that we didn't specify a format at all for the extra data. Eric felt that it was not unusual to leave a space for future expansion and that as we haven't used it yet, we don't need to solidify it.

He also felt it would be unusual to stipulate the format of data that we don't even intend to use yet.

Quoted 8 lines

In short, I'm being too proactive.

A commit message mention that, should anyone wish to expand the type-specific data in the future that adding a 2-byte version as the first field in extra data would probably be sufficient, and we can worry about the spec wording later. It is fine to assume for now that if extra_data_size is 0 that the version/format of the data is "v0" and that does not limit our future expansion.

Or put another way:

Quoted 10 lines

I'm just fine if our initial implementation provides sufficient information for us to completely parse the file even when the file is generated by a newer qemu (we have a length, so we know how far to skip to find the next entry), while at the same time throwing up our hands if the length is non-zero (we won't read the bitmap at all, because we don't know if the non-zero extra_data contains instructions that would change how to interpret the data) or even prevent writes (if the bitmap entry is marked automatic, we must refuse any write that would requiring an update to the bitmap because we don't know how to write to a bitmap while correctly preserving semantics of those extra_data bytes).

Can we assume that the extra_data doesn't contain references to clusters? Otherwise we need to forbid 'qemu-img check -r leaks' when there is unknown extra_data.

FWIW, this assumption is already made for snapshots, so it seems okay to make it here as well. But we could be explicit about it.

Kevin

#15 lore raw
Quoted 37 lines · Kevin Wolf

Am 18.01.2016 um 22:16 hat Eric Blake geschrieben:

Quoted 26 lines · John Snow
Quoted 15 lines
Quoted 7 lines

Please, let's decide finally about extra data, than I'll reroll it and, I hope, it will be committed, to make it possible to continue work on persistence series. About extra data, I'm ready to accept any variant, strictly defining, what software should do with unknown extra data.

I discussed this with Eric Blake on IRC briefly, and I mentioned I was concerned that we didn't specify a format at all for the extra data. Eric felt that it was not unusual to leave a space for future expansion and that as we haven't used it yet, we don't need to solidify it.

He also felt it would be unusual to stipulate the format of data that we don't even intend to use yet.

Quoted 8 lines

In short, I'm being too proactive.

A commit message mention that, should anyone wish to expand the type-specific data in the future that adding a 2-byte version as the first field in extra data would probably be sufficient, and we can worry about the spec wording later. It is fine to assume for now that if extra_data_size is 0 that the version/format of the data is "v0" and that does not limit our future expansion.

Or put another way:

Quoted 10 lines

I'm just fine if our initial implementation provides sufficient information for us to completely parse the file even when the file is generated by a newer qemu (we have a length, so we know how far to skip to find the next entry), while at the same time throwing up our hands if the length is non-zero (we won't read the bitmap at all, because we don't know if the non-zero extra_data contains instructions that would change how to interpret the data) or even prevent writes (if the bitmap entry is marked automatic, we must refuse any write that would requiring an update to the bitmap because we don't know how to write to a bitmap while correctly preserving semantics of those extra_data bytes).

Quoted 6 lines

Can we assume that the extra_data doesn't contain references to clusters? Otherwise we need to forbid 'qemu-img check -r leaks' when there is unknown extra_data.

FWIW, this assumption is already made for snapshots, so it seems okay to make it here as well. But we could be explicit about it.

ok, will do.

Quoted 2 lines

Kevin

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
#16 lore raw
In reply to Vladimir Sementsov-Ogievskiy

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 60 lines

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

 docs/specs/qcow2.txt | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +0 -1
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                     write to an image with unknown auto-clear features if it
                     clears the respective bits from this field first.
 
-                    Bits 0-63:  Reserved (set to 0)

docs/specs/qcow2.txt in the description of a field. · line 115

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for the bitmaps
+                                extension data.
+
+                                It is an error if this bit is set without the
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present but this
+                                bit is unset, the bitmaps extension data is
+                                inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is potentially inconsistent".

Quoted 14 lines
docs/specs/qcow2.txt +3 -0
+
+                    Bits 1-63:  Reserved (set to 0)
 
          96 -  99:  refcount_order
                     Describes the width of a reference count block entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the following:
                         0x00000000 - End of the header extension area
                         0xE2792ACA - Backing file format name
                         0x6803f857 - Feature name table
+                        0x23852875 - Bitmaps extension
                         other      - Unknown header extension, can be safely
                                      ignored
 

docs/specs/qcow2.txt the header extension data. Each entry look like this: · line 186

Quoted 9 lines
docs/specs/qcow2.txt +6 -0
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                     terminated if it has full length)
 
 
+== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never really defines what a dirty tracking bitmap even contains. It has a few hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies they track guest clusters rather than host clusters.

Quoted 25 lines
docs/specs/qcow2.txt +21 -0
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap directory
+                   starts. Must be aligned to a cluster boundary.
+
+
 == Host cluster management ==
 
 qcow2 manages the allocation of host clusters by maintaining a reference count

docs/specs/qcow2.txt Snapshot table entry: · line 408

Quoted 9 lines
docs/specs/qcow2.txt +6 -0
@@ -360,3 +400,133 @@ Snapshot table entry:
 
         variable:   Padding to round up the snapshot table entry size to the
                     next multiple of 8.
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

s/related/related to/

Quoted 22 lines
docs/specs/qcow2.txt +22 -0
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.
+
+        12 - 15:    flags

docs/specs/qcow2.txt Snapshot table entry: · line 441

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the virtual
+                         disk by any application that would write to this qcow2
+                         file (including writes, snapshot switching, etc.). The
+                         type of this bitmap must be 'dirty tracking bitmap'.

This suggests that we can represent snapshot switching in a dirty tracking bitmap. Which is almost impossible if we track guest clusters, except if loading a snapshot should mean that all bits in the bitmap are set. However, that feels a bit useless and dirty tracking across snapshots doesn't seem to make a whole lot of sense anyway.

Maybe what would make more sense is that a bitmap is tied to a specific snapshot or bitmaps are included in a snapshot, so that you can revert to the dirty status as it was when the snapshot was taken.

Or, if you really meant, that the tracking words on a host cluster level, what clusters would be included in the bitmap? Would only the virtual disk be included? VM state? Any metadata?

It seems we definitely need a new section on dirty tracking bitmaps that describes what the bitmap actually means and how it's supposed to work with snapshots. I guess we could also talk about how it works with other changes to the image like resizing.

Quoted 24 lines
docs/specs/qcow2.txt +24 -0
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support granularity_bits
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the image
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    Note: Qemu currently doesn't support values greater than
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

We talked about this in the other subthread.

Quoted 45 lines
docs/specs/qcow2.txt +45 -0
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see
+above).
+

docs/specs/qcow2.txt Snapshot table entry: · line 532

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

In which order are the bits stored in the bitmap?

Kevin

#17 lore raw
Quoted 186 lines · Kevin Wolf

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 60 lines

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +0 -1
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                      write to an image with unknown auto-clear features if it
                      clears the respective bits from this field first.
 
-                    Bits 0-63:  Reserved (set to 0)

docs/specs/qcow2.txt in the description of a field. · line 115

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for the bitmaps
+                                extension data.
+
+                                It is an error if this bit is set without the
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present but this
+                                bit is unset, the bitmaps extension data is
+                                inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is potentially inconsistent".

Quoted 14 lines
docs/specs/qcow2.txt +3 -0
+
+                    Bits 1-63:  Reserved (set to 0)
 
           96 -  99:  refcount_order
                      Describes the width of a reference count block entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the following:
                          0x00000000 - End of the header extension area
                          0xE2792ACA - Backing file format name
                          0x6803f857 - Feature name table
+                        0x23852875 - Bitmaps extension
                          other      - Unknown header extension, can be safely
                                       ignored
 

docs/specs/qcow2.txt the header extension data. Each entry look like this: · line 186

Quoted 9 lines
docs/specs/qcow2.txt +6 -0
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
                      terminated if it has full length)
 
 
+== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never really defines what a dirty tracking bitmap even contains. It has a few hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies they track guest clusters rather than host clusters.

Quoted 25 lines
docs/specs/qcow2.txt +21 -0
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image. Must be
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535 bitmaps per
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the cumulative
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap directory
+                   starts. Must be aligned to a cluster boundary.
+
+
  == Host cluster management ==
 
  qcow2 manages the allocation of host clusters by maintaining a reference count

docs/specs/qcow2.txt Snapshot table entry: · line 408

Quoted 9 lines
docs/specs/qcow2.txt +6 -0
@@ -360,3 +400,133 @@ Snapshot table entry:
 
          variable:   Padding to round up the snapshot table entry size to the
                      next multiple of 8.
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

s/related/related to/

Quoted 22 lines
docs/specs/qcow2.txt +22 -0
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the bitmap.
+
+        12 - 15:    flags

docs/specs/qcow2.txt Snapshot table entry: · line 441

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the virtual
+                         disk by any application that would write to this qcow2
+                         file (including writes, snapshot switching, etc.). The
+                         type of this bitmap must be 'dirty tracking bitmap'.

This suggests that we can represent snapshot switching in a dirty tracking bitmap. Which is almost impossible if we track guest clusters, except if loading a snapshot should mean that all bits in the bitmap are set. However, that feels a bit useless and dirty tracking across snapshots doesn't seem to make a whole lot of sense anyway.

Maybe what would make more sense is that a bitmap is tied to a specific snapshot or bitmaps are included in a snapshot, so that you can revert to the dirty status as it was when the snapshot was taken.

Quoted 8 lines

Or, if you really meant, that the tracking words on a host cluster level, what clusters would be included in the bitmap? Would only the virtual disk be included? VM state? Any metadata?

It seems we definitely need a new section on dirty tracking bitmaps that describes what the bitmap actually means and how it's supposed to work with snapshots. I guess we could also talk about how it works with other changes to the image like resizing.

Bitmaps track guest clusters.
Backup is not related to snapshots, so, I think set all bits in the dirty bitmap on snapshot switch is ok. Of course we can't just ignore snapshot switch, as it changes how user (and backup) sees the virtual disk. Deleting the bitmap on snapshot switch is not good option too I think.

Bitmap size is defined to be equal to virtual disk size. So on resize, the one who resize must resize the bitmap too, to maintain accordance of the image file and the spec.

I'll think about such section, ok.

Quoted 85 lines
Quoted 25 lines
docs/specs/qcow2.txt +24 -0
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support granularity_bits
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the image
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 10 lines
docs/specs/qcow2.txt +10 -0
+                    Note: Qemu currently doesn't support values greater than
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined, extra_data_size is
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

We talked about this in the other subthread.

Quoted 48 lines
docs/specs/qcow2.txt +48 -0
+        variable:   The name of the bitmap (not null terminated). Must be
+                    unique among all bitmap names within the bitmaps extension.
+
+        variable:   Padding to round up the bitmap directory entry size to the
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the mapping of
+bitmap data to host clusters. This structure is called the bitmap table.
+
+Each bitmap table has a variable size (stored in the bitmap directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are non-zero.
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be aligned to
+                    a cluster boundary. If the offset is 0, the cluster is
+                    unallocated; in that case, bit 0 determines how this
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters, described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are zero (see
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
+bit offset into the bitmap can be calculated like this:
+
Quoted 8 lines

docs/specs/qcow2.txt Snapshot table entry: · line 532

Quoted 7 lines
docs/specs/qcow2.txt +7 -0
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

In which order are the bits stored in the bitmap?

What do you mean?

Quoted 2 lines

Kevin

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
#18 lore raw
Quoted 351 lines · Vladimir Sementsov-Ogievskiy
Quoted 213 lines · Kevin Wolf

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 71 lines

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 172
++++++++++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +0 -0
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                      write to an image with unknown auto-clear

features if it
clears the respective bits from this field first.
- Bits 0-63: Reserved (set to 0)

docs/specs/qcow2.txt +2 -0
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for

the bitmaps

docs/specs/qcow2.txt +3 -0
+                                extension data.
+
+                                It is an error if this bit is set

without the

docs/specs/qcow2.txt +3 -0
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present
Quoted 4 lines

but this
+ bit is unset, the bitmaps extension
data is
+ inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is potentially inconsistent".

Quoted 20 lines
docs/specs/qcow2.txt +2 -0
+
+                    Bits 1-63:  Reserved (set to 0)
             96 -  99:  refcount_order
                      Describes the width of a reference count block

entry (width

docs/specs/qcow2.txt +0 -0
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like

the following:

                          0x00000000 - End of the header extension area
                          0xE2792ACA - Backing file format name
                          0x6803f857 - Feature name table

+ 0x23852875 - Bitmaps extension
other - Unknown header extension, can
be safely
ignored
@@ -166,6 +178,34 @@ the header extension data. Each entry look
like this:
terminated if it has full length)
+== Bitmaps extension ==

docs/specs/qcow2.txt +2 -0
+
+The bitmaps extension is an optional header extension. It provides
Quoted 6 lines

the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never really defines what a dirty tracking bitmap even contains. It has a few hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies they track guest clusters rather than host clusters.

Quoted 38 lines
docs/specs/qcow2.txt +2 -0
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features

above.

docs/specs/qcow2.txt +5 -0
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image.

Must be

docs/specs/qcow2.txt +3 -0
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535

bitmaps per

docs/specs/qcow2.txt +4 -0
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the

cumulative

docs/specs/qcow2.txt +4 -0
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap

directory

docs/specs/qcow2.txt +3 -0
+                   starts. Must be aligned to a cluster boundary.
+
+
  == Host cluster management ==
    qcow2 manages the allocation of host clusters by maintaining a

reference count

docs/specs/qcow2.txt +0 -0
@@ -360,3 +400,133 @@ Snapshot table entry:
            variable:   Padding to round up the snapshot table entry

size to the
next multiple of 8.

docs/specs/qcow2.txt +5 -0
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to
Quoted 3 lines

store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

s/related/related to/

Quoted 37 lines
docs/specs/qcow2.txt +5 -0
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory

entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These

docs/specs/qcow2.txt +6 -0
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap

table
+ (described below) for the bitmap starts. Must be
aligned to

docs/specs/qcow2.txt +4 -0
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the

bitmap.

docs/specs/qcow2.txt +9 -0
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the
Quoted 7 lines

virtual
+ disk by any application that would write to
this qcow2
+ file (including writes, snapshot switching,
etc.). The
+ type of this bitmap must be 'dirty tracking
bitmap'.

This suggests that we can represent snapshot switching in a dirty tracking bitmap. Which is almost impossible if we track guest clusters, except if loading a snapshot should mean that all bits in the bitmap are set. However, that feels a bit useless and dirty tracking across snapshots doesn't seem to make a whole lot of sense anyway.

Maybe what would make more sense is that a bitmap is tied to a specific snapshot or bitmaps are included in a snapshot, so that you can revert to the dirty status as it was when the snapshot was taken.

Quoted 8 lines

Or, if you really meant, that the tracking words on a host cluster level, what clusters would be included in the bitmap? Would only the virtual disk be included? VM state? Any metadata?

It seems we definitely need a new section on dirty tracking bitmaps that describes what the bitmap actually means and how it's supposed to work with snapshots. I guess we could also talk about how it works with other changes to the image like resizing.

Bitmaps track guest clusters.
Backup is not related to snapshots, so, I think set all bits in the dirty bitmap on snapshot switch is ok. Of course we can't just ignore snapshot switch, as it changes how user (and backup) sees the virtual disk. Deleting the bitmap on snapshot switch is not good option too I think.

Bitmap size is defined to be equal to virtual disk size. So on resize, the one who resize must resize the bitmap too, to maintain accordance of the image file and the spec.

I'll think about such section, ok.

Quoted 106 lines
Quoted 36 lines
docs/specs/qcow2.txt +13 -0
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support

granularity_bits

docs/specs/qcow2.txt +6 -0
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the

image

docs/specs/qcow2.txt +6 -0
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values

greater than

docs/specs/qcow2.txt +6 -0
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined,

extra_data_size is

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 3 lines
docs/specs/qcow2.txt +3 -0
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

We talked about this in the other subthread.

Quoted 65 lines

+ variable: The name of the bitmap (not null terminated).
Must be
+ unique among all bitmap names within the bitmaps
extension.

docs/specs/qcow2.txt +2 -0
+
+        variable:   Padding to round up the bitmap directory entry

size to the

docs/specs/qcow2.txt +6 -0
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the

mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

docs/specs/qcow2.txt +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap

directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image

docs/specs/qcow2.txt +5 -0
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are

non-zero.

docs/specs/qcow2.txt +7 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be

aligned to
+ a cluster boundary. If the offset is 0, the
cluster is
+ unallocated; in that case, bit 0 determines how
this

docs/specs/qcow2.txt +7 -0
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters,

described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into

docs/specs/qcow2.txt +7 -0
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are

zero (see

docs/specs/qcow2.txt +3 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's

granularity, the

docs/specs/qcow2.txt +4 -0
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
Quoted 8 lines
Quoted 7 lines
docs/specs/qcow2.txt +3 -0
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size

then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

In which order are the bits stored in the bitmap?

Quoted 1 line

What do you mean?

He means BE or LE. You can intuit it from the formula, but it's not explicitly stated, I think.

Quoted 5 lines
Quoted 3 lines

Kevin

Signature · 1 line
—js
#19 lore raw
Quoted 350 lines · John Snow
Quoted 348 lines · Vladimir Sementsov-Ogievskiy
Quoted 213 lines · Kevin Wolf

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 71 lines

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

   docs/specs/qcow2.txt | 172
++++++++++++++++++++++++++++++++++++++++++++++++++-
   1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +0 -0
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                       write to an image with unknown auto-clear

features if it
clears the respective bits from this field first.
- Bits 0-63: Reserved (set to 0)

docs/specs/qcow2.txt +2 -0
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for

the bitmaps

docs/specs/qcow2.txt +3 -0
+                                extension data.
+
+                                It is an error if this bit is set

without the

docs/specs/qcow2.txt +3 -0
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present
Quoted 4 lines

but this
+ bit is unset, the bitmaps extension
data is
+ inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is potentially inconsistent".

Quoted 20 lines
docs/specs/qcow2.txt +2 -0
+
+                    Bits 1-63:  Reserved (set to 0)
              96 -  99:  refcount_order
                       Describes the width of a reference count block

entry (width

docs/specs/qcow2.txt +0 -0
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like

the following:

                           0x00000000 - End of the header extension area
                           0xE2792ACA - Backing file format name
                           0x6803f857 - Feature name table

+ 0x23852875 - Bitmaps extension
other - Unknown header extension, can
be safely
ignored
@@ -166,6 +178,34 @@ the header extension data. Each entry look
like this:
terminated if it has full length)
+== Bitmaps extension ==

docs/specs/qcow2.txt +2 -0
+
+The bitmaps extension is an optional header extension. It provides
Quoted 6 lines

the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never really defines what a dirty tracking bitmap even contains. It has a few hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies they track guest clusters rather than host clusters.

Quoted 38 lines
docs/specs/qcow2.txt +2 -0
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features

above.

docs/specs/qcow2.txt +5 -0
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image.

Must be

docs/specs/qcow2.txt +3 -0
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535

bitmaps per

docs/specs/qcow2.txt +4 -0
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the

cumulative

docs/specs/qcow2.txt +4 -0
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap

directory

docs/specs/qcow2.txt +3 -0
+                   starts. Must be aligned to a cluster boundary.
+
+
   == Host cluster management ==
     qcow2 manages the allocation of host clusters by maintaining a

reference count

docs/specs/qcow2.txt +0 -0
@@ -360,3 +400,133 @@ Snapshot table entry:
             variable:   Padding to round up the snapshot table entry

size to the
next multiple of 8.

docs/specs/qcow2.txt +5 -0
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to
Quoted 3 lines

store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

s/related/related to/

Quoted 37 lines
docs/specs/qcow2.txt +5 -0
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory

entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These

docs/specs/qcow2.txt +6 -0
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap

table
+ (described below) for the bitmap starts. Must be
aligned to

docs/specs/qcow2.txt +4 -0
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the

bitmap.

docs/specs/qcow2.txt +9 -0
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the
Quoted 7 lines

virtual
+ disk by any application that would write to
this qcow2
+ file (including writes, snapshot switching,
etc.). The
+ type of this bitmap must be 'dirty tracking
bitmap'.

This suggests that we can represent snapshot switching in a dirty tracking bitmap. Which is almost impossible if we track guest clusters, except if loading a snapshot should mean that all bits in the bitmap are set. However, that feels a bit useless and dirty tracking across snapshots doesn't seem to make a whole lot of sense anyway.

Maybe what would make more sense is that a bitmap is tied to a specific snapshot or bitmaps are included in a snapshot, so that you can revert to the dirty status as it was when the snapshot was taken.

Quoted 8 lines

Or, if you really meant, that the tracking words on a host cluster level, what clusters would be included in the bitmap? Would only the virtual disk be included? VM state? Any metadata?

It seems we definitely need a new section on dirty tracking bitmaps that describes what the bitmap actually means and how it's supposed to work with snapshots. I guess we could also talk about how it works with other changes to the image like resizing.

Bitmaps track guest clusters.
Backup is not related to snapshots, so, I think set all bits in the dirty bitmap on snapshot switch is ok. Of course we can't just ignore snapshot switch, as it changes how user (and backup) sees the virtual disk. Deleting the bitmap on snapshot switch is not good option too I think.

Bitmap size is defined to be equal to virtual disk size. So on resize, the one who resize must resize the bitmap too, to maintain accordance of the image file and the spec.

I'll think about such section, ok.

Quoted 113 lines
Quoted 35 lines
docs/specs/qcow2.txt +13 -0
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support

granularity_bits

docs/specs/qcow2.txt +6 -0
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the

image

docs/specs/qcow2.txt +6 -0
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values

greater than

docs/specs/qcow2.txt +6 -0
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined,

extra_data_size is

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 3 lines
docs/specs/qcow2.txt +3 -0
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

We talked about this in the other subthread.

Quoted 68 lines

+ variable: The name of the bitmap (not null terminated).
Must be
+ unique among all bitmap names within the bitmaps
extension.

docs/specs/qcow2.txt +2 -0
+
+        variable:   Padding to round up the bitmap directory entry

size to the

docs/specs/qcow2.txt +6 -0
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the

mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

docs/specs/qcow2.txt +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap

directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image

docs/specs/qcow2.txt +5 -0
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are

non-zero.

docs/specs/qcow2.txt +7 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be

aligned to
+ a cluster boundary. If the offset is 0, the
cluster is
+ unallocated; in that case, bit 0 determines how
this

docs/specs/qcow2.txt +7 -0
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters,

described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into

docs/specs/qcow2.txt +7 -0
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are

zero (see

docs/specs/qcow2.txt +3 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's

granularity, the

docs/specs/qcow2.txt +7 -0
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size
Quoted 4 lines

then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

In which order are the bits stored in the bitmap?

Quoted 3 lines
Quoted 1 line

What do you mean?

He means BE or LE. You can intuit it from the formula, but it's not explicitly stated, I think

byte ordering? But it make sense when we deal with integers, which occupies more then 1 byte. Here is plain byte sequence. Bitmap. Like a string.. What should I write here additionally?

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

Kevin

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
#20 lore raw

Am 21.01.2016 um 09:22 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 351 lines · John Snow
Quoted 350 lines · Vladimir Sementsov-Ogievskiy
Quoted 348 lines · Kevin Wolf
Quoted 213 lines

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 71 lines

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 172
++++++++++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +0 -0
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                      write to an image with unknown auto-clear

features if it
clears the respective bits from this field first.
- Bits 0-63: Reserved (set to 0)

docs/specs/qcow2.txt +2 -0
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for

the bitmaps

docs/specs/qcow2.txt +3 -0
+                                extension data.
+
+                                It is an error if this bit is set

without the

docs/specs/qcow2.txt +3 -0
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present
Quoted 4 lines

but this
+ bit is unset, the bitmaps extension
data is
+ inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is potentially inconsistent".

Quoted 20 lines
docs/specs/qcow2.txt +2 -0
+
+                    Bits 1-63:  Reserved (set to 0)
             96 -  99:  refcount_order
                      Describes the width of a reference count block

entry (width

docs/specs/qcow2.txt +0 -0
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like

the following:

                          0x00000000 - End of the header extension area
                          0xE2792ACA - Backing file format name
                          0x6803f857 - Feature name table

+ 0x23852875 - Bitmaps extension
other - Unknown header extension, can
be safely
ignored
@@ -166,6 +178,34 @@ the header extension data. Each entry look
like this:
terminated if it has full length)
+== Bitmaps extension ==

docs/specs/qcow2.txt +2 -0
+
+The bitmaps extension is an optional header extension. It provides
Quoted 6 lines

the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never really defines what a dirty tracking bitmap even contains. It has a few hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies they track guest clusters rather than host clusters.

Quoted 38 lines
docs/specs/qcow2.txt +2 -0
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features

above.

docs/specs/qcow2.txt +5 -0
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image.

Must be

docs/specs/qcow2.txt +3 -0
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535

bitmaps per

docs/specs/qcow2.txt +4 -0
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the

cumulative

docs/specs/qcow2.txt +4 -0
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap

directory

docs/specs/qcow2.txt +3 -0
+                   starts. Must be aligned to a cluster boundary.
+
+
  == Host cluster management ==
    qcow2 manages the allocation of host clusters by maintaining a

reference count

docs/specs/qcow2.txt +0 -0
@@ -360,3 +400,133 @@ Snapshot table entry:
            variable:   Padding to round up the snapshot table entry

size to the
next multiple of 8.

docs/specs/qcow2.txt +5 -0
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to
Quoted 3 lines

store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

s/related/related to/

Quoted 37 lines
docs/specs/qcow2.txt +5 -0
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory

entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These

docs/specs/qcow2.txt +6 -0
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap

table
+ (described below) for the bitmap starts. Must be
aligned to

docs/specs/qcow2.txt +4 -0
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the

bitmap.

docs/specs/qcow2.txt +9 -0
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the
Quoted 7 lines

virtual
+ disk by any application that would write to
this qcow2
+ file (including writes, snapshot switching,
etc.). The
+ type of this bitmap must be 'dirty tracking
bitmap'.

This suggests that we can represent snapshot switching in a dirty tracking bitmap. Which is almost impossible if we track guest clusters, except if loading a snapshot should mean that all bits in the bitmap are set. However, that feels a bit useless and dirty tracking across snapshots doesn't seem to make a whole lot of sense anyway.

Maybe what would make more sense is that a bitmap is tied to a specific snapshot or bitmaps are included in a snapshot, so that you can revert to the dirty status as it was when the snapshot was taken.

Quoted 8 lines

Or, if you really meant, that the tracking words on a host cluster level, what clusters would be included in the bitmap? Would only the virtual disk be included? VM state? Any metadata?

It seems we definitely need a new section on dirty tracking bitmaps that describes what the bitmap actually means and how it's supposed to work with snapshots. I guess we could also talk about how it works with other changes to the image like resizing.

Bitmaps track guest clusters.
Backup is not related to snapshots, so, I think set all bits in the dirty bitmap on snapshot switch is ok. Of course we can't just ignore snapshot switch, as it changes how user (and backup) sees the virtual disk. Deleting the bitmap on snapshot switch is not good option too I think.

Bitmap size is defined to be equal to virtual disk size. So on resize, the one who resize must resize the bitmap too, to maintain accordance of the image file and the spec.

I'll think about such section, ok.

Quoted 113 lines
Quoted 35 lines
docs/specs/qcow2.txt +13 -0
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support

granularity_bits

docs/specs/qcow2.txt +6 -0
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the

image

docs/specs/qcow2.txt +6 -0
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values

greater than

docs/specs/qcow2.txt +6 -0
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined,

extra_data_size is

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 3 lines
docs/specs/qcow2.txt +3 -0
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

We talked about this in the other subthread.

Quoted 68 lines

+ variable: The name of the bitmap (not null terminated).
Must be
+ unique among all bitmap names within the bitmaps
extension.

docs/specs/qcow2.txt +2 -0
+
+        variable:   Padding to round up the bitmap directory entry

size to the

docs/specs/qcow2.txt +6 -0
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the

mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

docs/specs/qcow2.txt +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap

directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image

docs/specs/qcow2.txt +5 -0
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are

non-zero.

docs/specs/qcow2.txt +7 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be

aligned to
+ a cluster boundary. If the offset is 0, the
cluster is
+ unallocated; in that case, bit 0 determines how
this

docs/specs/qcow2.txt +7 -0
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters,

described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into

docs/specs/qcow2.txt +7 -0
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are

zero (see

docs/specs/qcow2.txt +3 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's

granularity, the

docs/specs/qcow2.txt +7 -0
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size
Quoted 4 lines

then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

In which order are the bits stored in the bitmap?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

What do you mean?

He means BE or LE. You can intuit it from the formula, but it's not explicitly stated, I think

byte ordering? But it make sense when we deal with integers, which occupies more then 1 byte. Here is plain byte sequence. Bitmap. Like a string.. What should I write here additionally?

I meant sub-byte ordering. Whether bit 0 is the LSB or the MSB. I assume you intend to use the LSB for bit 0, but it's not written anywhere.

Actually, that's an assumption that is made for bitmaps in the whole spec (including e.g. feature flags), even though it is only explicitly spelt out for the refcount blocks. I guess we should move it to the 'General' section.

Kevin

#21 lore raw
Quoted 357 lines · Kevin Wolf

Am 21.01.2016 um 09:22 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 350 lines · John Snow
Quoted 349 lines · Vladimir Sementsov-Ogievskiy
Quoted 348 lines · Kevin Wolf
Quoted 213 lines

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 71 lines

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It provides an ability to store virtual disk related bitmaps in a qcow2 image. For now there is only one type of such bitmaps: Dirty Tracking Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, should be stored in this qcow2 file. The size of each bitmap (considering its granularity) is equal to virtual disk size.

---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change it, if we decide allowing managing app to specify any extra data, by defining some magic value as a top of user extra data.. So, for now non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.

v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

   docs/specs/qcow2.txt | 172
++++++++++++++++++++++++++++++++++++++++++++++++++-
   1 file changed, 171 insertions(+), 1 deletion(-)
docs/specs/qcow2.txt +0 -0
index 121dfc8..997239d 100644
@@ -103,7 +103,18 @@ in the description of a field.
                       write to an image with unknown auto-clear

features if it
clears the respective bits from this field first.
- Bits 0-63: Reserved (set to 0)

docs/specs/qcow2.txt +2 -0
+                    Bit 0:      Bitmaps extension bit
+                                This bit indicates consistency for

the bitmaps

docs/specs/qcow2.txt +3 -0
+                                extension data.
+
+                                It is an error if this bit is set

without the

docs/specs/qcow2.txt +3 -0
+                                bitmaps extension present.
+
+                                If the bitmaps extension is present
Quoted 4 lines

but this
+ bit is unset, the bitmaps extension
data is
+ inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is potentially inconsistent".

Quoted 20 lines
docs/specs/qcow2.txt +2 -0
+
+                    Bits 1-63:  Reserved (set to 0)
              96 -  99:  refcount_order
                       Describes the width of a reference count block

entry (width

docs/specs/qcow2.txt +0 -0
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like

the following:

                           0x00000000 - End of the header extension area
                           0xE2792ACA - Backing file format name
                           0x6803f857 - Feature name table

+ 0x23852875 - Bitmaps extension
other - Unknown header extension, can
be safely
ignored
@@ -166,6 +178,34 @@ the header extension data. Each entry look
like this:
terminated if it has full length)
+== Bitmaps extension ==

docs/specs/qcow2.txt +2 -0
+
+The bitmaps extension is an optional header extension. It provides
Quoted 6 lines

the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never really defines what a dirty tracking bitmap even contains. It has a few hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies they track guest clusters rather than host clusters.

Quoted 38 lines
docs/specs/qcow2.txt +2 -0
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features

above.

docs/specs/qcow2.txt +5 -0
+
+The fields of the bitmaps extension are:
+
+          0 -  3:  nb_bitmaps
+                   The number of bitmaps contained in the image.

Must be

docs/specs/qcow2.txt +3 -0
+                   greater than or equal to 1.
+
+                   Note: Qemu currently only supports up to 65535

bitmaps per

docs/specs/qcow2.txt +4 -0
+                   image.
+
+          4 -  7:  bitmap_directory_size
+                   Size of the bitmap directory in bytes. It is the

cumulative

docs/specs/qcow2.txt +4 -0
+                   size of all (nb_bitmaps) bitmap headers.
+
+          8 - 15:  bitmap_directory_offset
+                   Offset into the image file at which the bitmap

directory

docs/specs/qcow2.txt +3 -0
+                   starts. Must be aligned to a cluster boundary.
+
+
   == Host cluster management ==
     qcow2 manages the allocation of host clusters by maintaining a

reference count

docs/specs/qcow2.txt +0 -0
@@ -360,3 +400,133 @@ Snapshot table entry:
             variable:   Padding to round up the snapshot table entry

size to the
next multiple of 8.

docs/specs/qcow2.txt +5 -0
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to
Quoted 3 lines

store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.

s/related/related to/

Quoted 37 lines
docs/specs/qcow2.txt +5 -0
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory

entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These

docs/specs/qcow2.txt +6 -0
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+    Byte 0 -  7:    bitmap_table_offset
+                    Offset into the image file at which the bitmap

table
+ (described below) for the bitmap starts. Must be
aligned to

docs/specs/qcow2.txt +4 -0
+                    a cluster boundary.
+
+         8 - 11:    bitmap_table_size
+                    Number of entries in the bitmap table of the

bitmap.

docs/specs/qcow2.txt +9 -0
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap must reflect all changes of the
Quoted 7 lines

virtual
+ disk by any application that would write to
this qcow2
+ file (including writes, snapshot switching,
etc.). The
+ type of this bitmap must be 'dirty tracking
bitmap'.

This suggests that we can represent snapshot switching in a dirty tracking bitmap. Which is almost impossible if we track guest clusters, except if loading a snapshot should mean that all bits in the bitmap are set. However, that feels a bit useless and dirty tracking across snapshots doesn't seem to make a whole lot of sense anyway.

Maybe what would make more sense is that a bitmap is tied to a specific snapshot or bitmaps are included in a snapshot, so that you can revert to the dirty status as it was when the snapshot was taken.

Quoted 8 lines

Or, if you really meant, that the tracking words on a host cluster level, what clusters would be included in the bitmap? Would only the virtual disk be included? VM state? Any metadata?

It seems we definitely need a new section on dirty tracking bitmaps that describes what the bitmap actually means and how it's supposed to work with snapshots. I guess we could also talk about how it works with other changes to the image like resizing.

Bitmaps track guest clusters.
Backup is not related to snapshots, so, I think set all bits in the dirty bitmap on snapshot switch is ok. Of course we can't just ignore snapshot switch, as it changes how user (and backup) sees the virtual disk. Deleting the bitmap on snapshot switch is not good option too I think.

Bitmap size is defined to be equal to virtual disk size. So on resize, the one who resize must resize the bitmap too, to maintain accordance of the image file and the spec.

I'll think about such section, ok.

Quoted 113 lines
Quoted 35 lines
docs/specs/qcow2.txt +13 -0
+                    Bits 2 - 31 are reserved and must be 0.
+
+             16:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      1: Dirty tracking bitmap
+
+                    Values 0, 2 - 255 are reserved.
+
+             17:    granularity_bits
+                    Granularity bits. Valid values: 0 - 63.
+
+                    Note: Qemu currently doesn't support

granularity_bits

docs/specs/qcow2.txt +6 -0
+                    greater than 31.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    A bitmap's granularity is how many bytes of the

image

docs/specs/qcow2.txt +6 -0
+                    accounts for one bit of the bitmap.
+
+        18 - 19:    name_size
+                    Size of the bitmap name. Must be non-zero.
+
+                    Note: Qemu currently doesn't support values

greater than

docs/specs/qcow2.txt +6 -0
+                    1023.
+
+        20 - 23:    extra_data_size
+                    Size of type-specific extra data.
+
+                    For now, as no extra data is defined,

extra_data_size is

docs/specs/qcow2.txt Snapshot table entry: · line 476

Quoted 3 lines
docs/specs/qcow2.txt +3 -0
+                    reserved and must be zero.
+
+        variable:   Type-specific extra data for the bitmap.

We talked about this in the other subthread.

Quoted 68 lines

+ variable: The name of the bitmap (not null terminated).
Must be
+ unique among all bitmap names within the bitmaps
extension.

docs/specs/qcow2.txt +2 -0
+
+        variable:   Padding to round up the bitmap directory entry

size to the

docs/specs/qcow2.txt +6 -0
+                    next multiple of 8.
+
+=== Bitmap table ===
+
+Bitmaps are stored using a one-level structure (as opposed to two-level
+structure like for refcounts and guest clusters mapping) for the

mapping of
+bitmap data to host clusters. This structure is called the bitmap table.

docs/specs/qcow2.txt +2 -0
+
+Each bitmap table has a variable size (stored in the bitmap

directory Entry)
+and may use multiple clusters, however, it must be contiguous in the image

docs/specs/qcow2.txt +5 -0
+file.
+
+Structure of a bitmap table entry:
+
+    Bit       0:    Reserved and must be zero if bits 9 - 55 are

non-zero.

docs/specs/qcow2.txt +7 -0
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of the host cluster offset. Must be

aligned to
+ a cluster boundary. If the offset is 0, the
cluster is
+ unallocated; in that case, bit 0 determines how
this

docs/specs/qcow2.txt +7 -0
+                    cluster should be treated when read from.
+
+        56 - 63:    Reserved and must be zero.
+
+=== Bitmap data ===
+
+As noted above, bitmap data is stored in separate clusters,

described by the
+bitmap table. Given an offset (in bytes) into the bitmap data, the offset into

docs/specs/qcow2.txt +7 -0
+the image file can be obtained as follows:
+
+    image_offset =
+        bitmap_table[bitmap_data_offset / cluster_size] +
+            (bitmap_data_offset % cluster_size)
+
+This offset is not defined if bits 9 - 55 of bitmap table entry are

zero (see

docs/specs/qcow2.txt +3 -0
+above).
+
+Given an offset byte_nr into the virtual disk and the bitmap's

granularity, the

docs/specs/qcow2.txt +7 -0
+bit offset into the bitmap can be calculated like this:
+
+    bit_offset =
+        image_offset(byte_nr / granularity / 8) * 8 +
+            (byte_nr / granularity) % 8
+
+If the size of the bitmap data is not a multiply of cluster size
Quoted 4 lines

then the last
+cluster of the bitmap data contains some unused tail bits. These bits must be
+zero.

In which order are the bits stored in the bitmap?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

What do you mean?

He means BE or LE. You can intuit it from the formula, but it's not explicitly stated, I think

byte ordering? But it make sense when we deal with integers, which occupies more then 1 byte. Here is plain byte sequence. Bitmap. Like a string.. What should I write here additionally?

Quoted 7 lines

I meant sub-byte ordering. Whether bit 0 is the LSB or the MSB. I assume you intend to use the LSB for bit 0, but it's not written anywhere.

Actually, that's an assumption that is made for bitmaps in the whole spec (including e.g. feature flags), even though it is only explicitly spelt out for the refcount blocks. I guess we should move it to the 'General' section.

Understand now. Then it should be separate patch.

Quoted 2 lines

Kevin

Signature · 3 lines
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.