Skip to contentSkip to conversation

docs: qcow2: introduce compression type feature

PATCH In review Vladimir Sementsov-Ogievskiy 4 participants 14 messages Jan 31, 2020 – Feb 6, 2020

lore
Share & tools
lore
#0 lore raw

v11 cover docs: qcow2: introduce compression type feature

v11:
01: grammar/wording [Eric]
add Eric's r-b
02: empty lines between paragraphs [Eric]

Vladimir Sementsov-Ogievskiy (2):
docs: improve qcow2 spec about extending image header
docs: qcow2: introduce compression type feature

 docs/interop/qcow2.txt | 62 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 59 insertions(+), 3 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH v11 1/2 docs: improve qcow2 spec about extending image header

Make it more obvious how to add new fields to the version 3 header and how to interpret them.

The specification is adjusted so that for new defined optional fields:

1. Software may support some of these optional fields and ignore the others, which means that features may be backported to downstream Qemu independently.
2. If we want to add incompatible field (or a field, for which some its values would be incompatible), it must be accompanied by incompatible feature bit.

Also the concept of "default is zero" is clarified, as it's strange to say that the value of the field is assumed to be zero for the software version which don't know about the field at all and don't know how to treat it be it zero or not.

---
 docs/interop/qcow2.txt | 45 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 42 insertions(+), 3 deletions(-)
docs/interop/qcow2.txt +42 -3
index af5711e533..823cc266e0 100644
@@ -79,9 +79,9 @@ The first cluster of a qcow2 image contains the file header:
                     Offset into the image file at which the snapshot table
                     starts. Must be aligned to a cluster boundary.
 
-If the version is 3 or higher, the header has the following additional fields.
-For version 2, the values are assumed to be zero, unless specified otherwise
-in the description of a field.
+For version 2, the header is exactly 72 bytes in length, and finishes here.
+For version 3 or higher, the header length is at least 104 bytes, including
+the next fields through header_length.
 
          72 -  79:  incompatible_features
                     Bitmask of incompatible features. An implementation must
@@ -164,6 +164,45 @@ in the description of a field.
         100 - 103:  header_length
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
+                    For version 3 it's at least 104 bytes and must be a multiple
+                    of 8.
+
+
+=== Additional fields (version 3 and higher) ===
+
+In general, these fields are optional and may be safely ignored by the software,
+as well as filled by zeros (which is equal to field absence), if software needs
+to set field B, but does not care about field A which precedes B. More
+formally, additional fields have the following compatibility rules:
+
+1. If the value of the additional field must not be ignored for correct
+handling of the file, it will be accompanied by a corresponding incompatible
+feature bit.
+
+2. If there are no unrecognized incompatible feature bits set, an unknown
+additional field may be safely ignored other than preserving its value when
+rewriting the image header.
+
+3. An explicit value of 0 will have the same behavior as when the field is not
+present*, if not altered by a specific incompatible bit.
+
+*. A field is considered not present when header_length is less than or equal
+to the field's offset. Also, all additional fields are not present for
+version 2.
+
+        < ... No additional fields in the header currently ... >
+
+
+=== Header padding ===
+
+@header_length must be a multiple of 8, which means that if the end of the last
+additional field is not aligned, some padding is needed. This padding must be
+zeroed, so that if some existing (or future) additional field will fall into
+the padding, it will be interpreted accordingly to point [3.] of the previous
+paragraph, i.e.  in the same manner as when this field is not present.
+
+
+=== Header extensions ===
 
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
Signature · 1 line
2.21.0
#2 lore raw

Re: [PATCH v11 1/2] docs: improve qcow2 spec about extending image header

Quoted 9 lines · Vladimir Sementsov-Ogievskiy

Make it more obvious how to add new fields to the version 3 header and how to interpret them.

The specification is adjusted so that for new defined optional fields:

1. Software may support some of these optional fields and ignore the others, which means that features may be backported to downstream Qemu independently.
2. If we want to add incompatible field (or a field, for which some its

s/some/some of/

Quoted 13 lines

values would be incompatible), it must be accompanied by incompatible feature bit.

Also the concept of "default is zero" is clarified, as it's strange to say that the value of the field is assumed to be zero for the software version which don't know about the field at all and don't know how to treat it be it zero or not.

---
  docs/interop/qcow2.txt | 45 +++++++++++++++++++++++++++++++++++++++---
  1 file changed, 42 insertions(+), 3 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v11 1/2
#3 lore raw

Re: [PATCH v11 1/2] docs: improve qcow2 spec about extending image header

In reply to Vladimir Sementsov-Ogievskiy
Quoted 12 lines · Vladimir Sementsov-Ogievskiy

Make it more obvious how to add new fields to the version 3 header and how to interpret them.

The specification is adjusted so that for new defined optional fields:

1. Software may support some of these optional fields and ignore the others, which means that features may be backported to downstream Qemu independently.
2. If we want to add incompatible field (or a field, for which some its values would be incompatible), it must be accompanied by incompatible feature bit.

Quoted 7 lines

Also the concept of "default is zero" is clarified, as it's strange to say that the value of the field is assumed to be zero for the software version which don't know about the field at all and don't know how to treat it be it zero or not.

Berto

#4 lore raw

PATCH v11 2/2 docs: qcow2: introduce compression type feature

In reply to Vladimir Sementsov-Ogievskiy

The patch adds a new additional field to the qcow2 header: compression_type, which specifies compression type. If field is absent or zero, default compression type is set: ZLIB, which corresponds to current behavior.

New compression type (ZSTD) is to be added in further commit.

---
 docs/interop/qcow2.txt | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
docs/interop/qcow2.txt +18 -1
index 823cc266e0..3a8d5c0c6e 100644
@@ -109,6 +109,11 @@ the next fields through header_length.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
+                    Bit 3:      Compression type bit.  If this bit is set,
+                                a non-default compression is used for compressed
+                                clusters. The compression_type field must be
+                                present and not zero.
+
                     Bits 3-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
@@ -190,7 +195,19 @@ present*, if not altered by a specific incompatible bit.
 to the field's offset. Also, all additional fields are not present for
 version 2.
 
-        < ... No additional fields in the header currently ... >
+              104:  compression_type
+
+                    Defines the compression method used for compressed clusters.
+                    All compressed clusters in an image use the same compression
+                    type.
+
+                    If the incompatible bit "Compression type" is set: the field
+                    must be present and non-zero (which means non-zlib
+                    compression type). Otherwise, this field must not be present
+                    or must be zero (which means zlib).
+
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/>
 
 
 === Header padding ===
Signature · 1 line
2.21.0
#5 lore raw
Quoted 17 lines · Vladimir Sementsov-Ogievskiy

The patch adds a new additional field to the qcow2 header: compression_type, which specifies compression type. If field is absent or zero, default compression type is set: ZLIB, which corresponds to current behavior.

New compression type (ZSTD) is to be added in further commit.

---
  docs/interop/qcow2.txt | 19 ++++++++++++++++++-
  1 file changed, 18 insertions(+), 1 deletion(-)
docs/interop/qcow2.txt +0 -0
index 823cc266e0..3a8d5c0c6e 100644

docs/interop/qcow2.txt the next fields through header_length. · line 116

Quoted 8 lines
docs/interop/qcow2.txt +5 -0
@@ -109,6 +109,11 @@ the next fields through header_length.
                                  An External Data File Name header extension may
                                  be present if this bit is set.
 
+                    Bit 3:      Compression type bit.  If this bit is set,
+                                a non-default compression is used for compressed
+                                clusters. The compression_type field must be
+                                present and not zero.
+

I'm still not sold on this. What's wrong with:

Bit 3: Compression type bit. If this bit is set,
a non-default compression may be used for compressed clusters, and the compression_type field must be present.

Quoted 8 lines
docs/interop/qcow2.txt +0 -1
                      Bits 3-63:  Reserved (set to 0)
 
           80 -  87:  compatible_features
@@ -190,7 +195,19 @@ present*, if not altered by a specific incompatible bit.
  to the field's offset. Also, all additional fields are not present for
  version 2.
 
-        < ... No additional fields in the header currently ... >

docs/interop/qcow2.txt present*, if not altered by a specific incompatible bit. · line 207

Quoted 10 lines
docs/interop/qcow2.txt +10 -0
+              104:  compression_type
+
+                    Defines the compression method used for compressed clusters.
+                    All compressed clusters in an image use the same compression
+                    type.
+
+                    If the incompatible bit "Compression type" is set: the field
+                    must be present and non-zero (which means non-zlib
+                    compression type). Otherwise, this field must not be present
+                    or must be zero (which means zlib).

If the incompatible bit "Compression type" is set: the field must be present. Otherwise, this field must not be present or must be zero (which means zlib). It is recommended but not required that the "Compression type" bit be clear if this field is present but zero.

Quoted 7 lines
docs/interop/qcow2.txt +3 -0
+
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/>
 
 
  === Header padding ===
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#6 lore raw
Quoted 8 lines · Eric Blake

docs/interop/qcow2.txt present*, if not altered by a specific incompatible bit. · line 207

Quoted 4 lines
docs/interop/qcow2.txt +4 -0
+                    If the incompatible bit "Compression type" is set: the field
+                    must be present and non-zero (which means non-zlib
+                    compression type). Otherwise, this field must not be present
+                    or must be zero (which means zlib).

If the incompatible bit "Compression type" is set: the field must be present. Otherwise, this field must not be present or must be zero (which means zlib).

But "not being present" and "being zero" is equivalent (as described in the previous commit).

And if the incompatible bit is not present then the field can be safely ignored (i.e. whether it is zero or not is irrelevant).

Let's try again:

Defines the compression method used for compressed clusters. All compressed clusters in an image use the same type.

The value of this field should only be used when the incompatible bit "Compression type" is set. If that bit is unset then this field is not used and the compression method is zlib.

Berto

#7 lore raw
Quoted 16 lines · Alberto Garcia
Quoted 8 lines · Eric Blake

docs/interop/qcow2.txt present*, if not altered by a specific incompatible bit. · line 207

Quoted 4 lines
docs/interop/qcow2.txt +4 -0
+                    If the incompatible bit "Compression type" is set: the field
+                    must be present and non-zero (which means non-zlib
+                    compression type). Otherwise, this field must not be present
+                    or must be zero (which means zlib).

If the incompatible bit "Compression type" is set: the field must be present. Otherwise, this field must not be present or must be zero (which means zlib).

But "not being present" and "being zero" is equivalent (as described in the previous commit).

And if the incompatible bit is not present then the field can be safely ignored (i.e. whether it is zero or not is irrelevant).

Quoted 8 lines

Let's try again:

Defines the compression method used for compressed clusters. All compressed clusters in an image use the same type.

The value of this field should only be used when the incompatible bit "Compression type" is set. If that bit is unset then this field is not used and the compression method is zlib.

I like that wording.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#8 lore raw
Quoted 19 lines · Eric Blake
Quoted 18 lines · Alberto Garcia
Quoted 8 lines · Eric Blake
Quoted 4 lines

+                    If the incompatible bit "Compression type" is set: the field +                    must be present and non-zero (which means non-zlib +                    compression type). Otherwise, this field must not be present +                    or must be zero (which means zlib).

             If the incompatible bit "Compression type" is set: the field must be present. Otherwise, this field must not be present or must be zero (which means zlib).

But "not being present" and "being zero" is equivalent (as described in the previous commit).

And if the incompatible bit is not present then the field can be safely ignored (i.e. whether it is zero or not is irrelevant).

Let's try again:

Quoted 9 lines
Quoted 6 lines

    Defines the compression method used for compressed clusters. All compressed clusters in an image use the same type.

    The value of this field should only be used when the incompatible bit "Compression type" is set. If that bit is unset then this field is not used and the compression method is zlib.

I like that wording.

I'm OK with it too, as well as I'm OK with the stricter variant, when we don't allow incompatible images with zlib set. I don't see any serious difference.

But I need this to land somehow. Max likes stricter variant and he is maintainer of qcow2..

Max, will you merge it as is, or did you change your mind, or should we ask Kevin for his opinion?

Signature · 2 lines
Best regards,
Vladimir
#9 lore raw
Quoted 3 lines · Vladimir Sementsov-Ogievskiy

I'm OK with it too, as well as I'm OK with the stricter variant, when we don't allow incompatible images with zlib set. I don't see any serious difference.

I also think both options are fine.

Berto

#10 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Eric Blake
Quoted 22 lines · Alberto Garcia
Quoted 8 lines · Eric Blake
Quoted 7 lines

+                    If the incompatible bit "Compression type" is
set: the field
+                    must be present and non-zero (which means non-zlib
+                    compression type). Otherwise, this field must not be present
+                    or must be zero (which means zlib).

Quoted 4 lines

             If the incompatible bit "Compression type" is set: the field
             must be present. Otherwise, this field must not be present or must be zero (which means zlib).

But "not being present" and "being zero" is equivalent (as described in the previous commit).

And if the incompatible bit is not present then the field can be safely ignored (i.e. whether it is zero or not is irrelevant).

Let's try again:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

    Defines the compression method used for compressed clusters. All compressed clusters in an image use the same type.

    The value of this field should only be used when the incompatible bit
    "Compression type" is set. If that bit is unset then this field is not used and the compression method is zlib.

This doesn’t fully make sense to me. Maybe with s/should/must/, because as it is it means that for non-zlib compression methods, you *should* set the compression type bit and add this header extension; but you may also just add the extension and not set the compression bit.

All in all, I didn’t see anyone disagreeing on the fact that there are only two cases that make any sense:
(1) Have the bit unset and the extension not present or zero: zlib.
(2) Have the bit set and an extension present and non-zero: not zlib.

If those are the only sensible choices, I don’t see any practical argument for allowing anything else[1]. (However, I do see an argument for forbidding anything else, namely to ensure that everyone follows these sensible guidelines.)

Quoted 7 lines
Quoted 2 lines

I like that wording.

I'm OK with it too, as well as I'm OK with the stricter variant, when we don't allow incompatible images with zlib set. I don't see any serious difference.

Quoted 5 lines

But I need this to land somehow. Max likes stricter variant and he is maintainer of qcow2..

Max, will you merge it as is, or did you change your mind, or should we ask Kevin for his opinion?

I’m currently preparing a pull request (without this series), but after that I’m planning to merge the stricter variant.

As far as I’ve seen, the argument for making it less strict was still accompanied by “Sure, nobody would set this flag for zlib-compressed images because that doesn’t make sense”. So if nobody would do that, we might as well just forbid it and thus ensure that everyone indeed does the sensible thing.

Max

[1] Besides “The specification would be one restriction shorter”, which I don’t think is a very good argument. Because without that sentence, anyone who implements qcow2 has to think about the problem anyway and figure out that quasi-restriction by themselves. If they don’t, it’d be a bit bad because they’d produce incompatible zlib-compressed images for no reason. Hence why I don’t see the restriction as a burden to the reader but as a helpful guideline (that must be followed).

#11 lore raw
Quoted 10 lines · Max Reitz
Quoted 9 lines

I'm OK with it too, as well as I'm OK with the stricter variant, when we don't allow incompatible images with zlib set. I don't see any serious difference.

But I need this to land somehow. Max likes stricter variant and he is maintainer of qcow2..

Max, will you merge it as is, or did you change your mind, or should we ask Kevin for his opinion?

Quoted 8 lines

I’m currently preparing a pull request (without this series), but after that I’m planning to merge the stricter variant.

As far as I’ve seen, the argument for making it less strict was still accompanied by “Sure, nobody would set this flag for zlib-compressed images because that doesn’t make sense”. So if nobody would do that, we might as well just forbid it and thus ensure that everyone indeed does the sensible thing.

Fair enough; I'm happy to live with your decision as maintainer's prerogative.

I _do_ hope that the actual implementation series gets merged soon, though, and that as part of that series, you remember to tweak the optional 'Feature Name' extension header to name the new incompatible bit introduced in this patch. And there's the simultaneous patches from my qcow2 autoclear-all-zeroes bit that touch the same files, so we may have some rebasing fun ahead of us...

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#12 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 11 lines · Vladimir Sementsov-Ogievskiy

v11:
01: grammar/wording [Eric]
add Eric's r-b
02: empty lines between paragraphs [Eric]

Vladimir Sementsov-Ogievskiy (2):
docs: improve qcow2 spec about extending image header
docs: qcow2: introduce compression type feature

 docs/interop/qcow2.txt | 62 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 59 insertions(+), 3 deletions(-)

Thanks, I’ve fixed patch 1’s commit message and applied the series to my block branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block

Max

#13 lore raw
Quoted 10 lines · Max Reitz
Quoted 9 lines · Vladimir Sementsov-Ogievskiy

v11:
01: grammar/wording [Eric]
add Eric's r-b
02: empty lines between paragraphs [Eric]

Vladimir Sementsov-Ogievskiy (2):
docs: improve qcow2 spec about extending image header
docs: qcow2: introduce compression type feature

Quoted 8 lines
Quoted 2 lines
  docs/interop/qcow2.txt | 62 ++++++++++++++++++++++++++++++++++++++++--
  1 file changed, 59 insertions(+), 3 deletions(-)

Thanks, I’ve fixed patch 1’s commit message and applied the series to my block branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block

Thanks!

Signature · 2 lines
Best regards,
Vladimir