Skip to contentSkip to conversation

qcow2: add zstd cluster compression

PATCH Superseded Vladimir Sementsov-Ogievskiy 2 participants 8 messages Dec 16, 2019 – Jan 20, 2020

v9 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v9 cover qcow2: add zstd cluster compression

Hi all!

Here is my proposal, about how to correctly update qcow2 specification to introduce new field, keeping in mind currently existing images and downstream Qemu instances.

v9: Merge 01 and 02
Change wordings
Require header alignment

Vladimir Sementsov-Ogievskiy (2):
docs: improve qcow2 spec about extending image header
docs: qcow2: introduce compression type feature

 docs/interop/qcow2.txt | 53 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 50 insertions(+), 3 deletions(-)
Signature · 1 line
2.18.0
#1 lore raw

PATCH v9 1/2 docs: improve qcow2 spec about extending image header

Make it more obvious how to add new fields to the version 3 header and how to interpret them.

The specification is adjusted so for new defined optional fields:

1. Software may support some of these optional fields and ignore the others, which means that features may be backported to downstream Qemu independently.
2. If we want to add incompatible field (or a field, for which some its values would be incompatible), it must be accompanied by incompatible feature bit.

Also the concept of "default is zero" is clarified, as it's strange to say that the value of the field is assumed to be zero for the software version which don't know about the field at all and don't know how to treat it be it zero or not.

---
 docs/interop/qcow2.txt | 39 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)
docs/interop/qcow2.txt +36 -3
index af5711e533..d92c827763 100644
@@ -79,9 +79,9 @@ The first cluster of a qcow2 image contains the file header:
                     Offset into the image file at which the snapshot table
                     starts. Must be aligned to a cluster boundary.
 
-If the version is 3 or higher, the header has the following additional fields.
-For version 2, the values are assumed to be zero, unless specified otherwise
-in the description of a field.
+For version 2, the header is exactly 72 bytes in length, and finishes here.
+For version 3 or higher, the header length is at least 104 bytes, including
+the next fields through header_length.
 
          72 -  79:  incompatible_features
                     Bitmask of incompatible features. An implementation must
@@ -164,6 +164,39 @@ in the description of a field.
         100 - 103:  header_length
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
+                    For version 3 it's at least 104 bytes and must be a multiply
+                    of 8.
+
+Additional fields (version 3 and higher)
+
+In general, these fields are optional and may be safely ignored by the software,
+as well as filled by zeros (which is equal to field absence), if software needs
+to set field B, but don't want to care about field A, which precedes B. More
+formally, additional fields have the following compatibility rules:
+
+1. If the value of the additional field must not be ignored for correct
+handling of the file, it will be accompanied by a corresponding incompatible
+feature bit.
+
+2. If there are no unrecognized incompatible feature bits set, an unknown
+additional field may be safely ignored other than preserving its value when
+rewriting the image header.
+
+3. An explicit value of 0 will have the same behavior as when the field is not
+present*, if not altered by specific incompatible bit.
+
+*. Field is not present when header_length is less or equal to field's offset.
+Also, all additional fields are not present for version 2.
+
+        < ... No additional fields in the header currently ... >
+
+Header padding
+
+@header_length must be a multiply of 8, which means that if last additional field
+end is not aligned, some padding is needed. This padding must be zeroed, so that,
+if some existing (or future) additional field will fall into the padding, it
+will be interpreted accordingly to point [3.] of the previous paragraph, i.e.
+in same manner as when this field is not present.
 
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
Signature · 1 line
2.18.0
#2 lore raw

Re: [PATCH v9 1/2] docs: improve qcow2 spec about extending image header

Quoted 17 lines · Vladimir Sementsov-Ogievskiy

Make it more obvious how to add new fields to the version 3 header and how to interpret them.

The specification is adjusted so for new defined optional fields:

1. Software may support some of these optional fields and ignore the others, which means that features may be backported to downstream Qemu independently.
2. If we want to add incompatible field (or a field, for which some its values would be incompatible), it must be accompanied by incompatible feature bit.

Also the concept of "default is zero" is clarified, as it's strange to say that the value of the field is assumed to be zero for the software version which don't know about the field at all and don't know how to treat it be it zero or not.

Quoted 4 lines
---
 docs/interop/qcow2.txt | 39 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

I put review of this off for so long because I always waited for Eric to give his R-b, but maybe not.

I generally think that he’s stricter on what to write in documentation, and accordingly I only have nit picks on spelling and structure:

Quoted 18 lines
docs/interop/qcow2.txt +3 -3
index af5711e533..d92c827763 100644
@@ -79,9 +79,9 @@ The first cluster of a qcow2 image contains the file header:
                     Offset into the image file at which the snapshot table
                     starts. Must be aligned to a cluster boundary.
 
-If the version is 3 or higher, the header has the following additional fields.
-For version 2, the values are assumed to be zero, unless specified otherwise
-in the description of a field.
+For version 2, the header is exactly 72 bytes in length, and finishes here.
+For version 3 or higher, the header length is at least 104 bytes, including
+the next fields through header_length.
 
          72 -  79:  incompatible_features
                     Bitmask of incompatible features. An implementation must

docs/interop/qcow2.txt in the description of a field. · line 167

Quoted 4 lines
docs/interop/qcow2.txt +1 -0
@@ -164,6 +164,39 @@ in the description of a field.
         100 - 103:  header_length
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
+                    For version 3 it's at least 104 bytes and must be a multiply

s/multiply/multiple/

docs/interop/qcow2.txt in the description of a field. · line 170

Quoted 3 lines
docs/interop/qcow2.txt +3 -0
+                    of 8.
+
+Additional fields (version 3 and higher)

If this is supposed to be a heading, maybe it should enclosed by “===” on both sides.

docs/interop/qcow2.txt in the description of a field. · line 174

Quoted 4 lines
docs/interop/qcow2.txt +4 -0
+
+In general, these fields are optional and may be safely ignored by the software,
+as well as filled by zeros (which is equal to field absence), if software needs
+to set field B, but don't want to care about field A, which precedes B. More

s/don't/does not/ (or maybe s/don't want/does not/)

docs/interop/qcow2.txt in the description of a field. · line 186

Quoted 12 lines
docs/interop/qcow2.txt +12 -0
+formally, additional fields have the following compatibility rules:
+
+1. If the value of the additional field must not be ignored for correct
+handling of the file, it will be accompanied by a corresponding incompatible
+feature bit.
+
+2. If there are no unrecognized incompatible feature bits set, an unknown
+additional field may be safely ignored other than preserving its value when
+rewriting the image header.
+
+3. An explicit value of 0 will have the same behavior as when the field is not
+present*, if not altered by specific incompatible bit.

s/by specific/by a specific/

docs/interop/qcow2.txt in the description of a field. · line 188

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+
+*. Field is not present when header_length is less or equal to field's offset.

s/Field/A field/, s/field's/the field's/

(maybe also +considered, as in "A field is considered not present...")

docs/interop/qcow2.txt in the description of a field. · line 191

Quoted 3 lines
docs/interop/qcow2.txt +3 -0
+Also, all additional fields are not present for version 2.
+
+        < ... No additional fields in the header currently ... >

This looks a bit weird to me, but the next patch will remove it again, so who cares.

docs/interop/qcow2.txt in the description of a field. · line 193

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+Header padding

Same heading note here (I’d make this “=== Header padding ===”).

docs/interop/qcow2.txt in the description of a field. · line 194

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+
+@header_length must be a multiply of 8, which means that if last additional field

s/multiply/multiple/

docs/interop/qcow2.txt in the description of a field. · line 196

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+end is not aligned, some padding is needed. This padding must be zeroed, so that,

I think s/last additional field end/the last additional field’s end/, or maybe s/last additional field end/the end of the last additional field/.

docs/interop/qcow2.txt in the description of a field. · line 199

Quoted 3 lines
docs/interop/qcow2.txt +3 -0
+if some existing (or future) additional field will fall into the padding, it
+will be interpreted accordingly to point [3.] of the previous paragraph, i.e.
+in same manner as when this field is not present.

s/in same/in the same/

Quoted 1 line

I think there should be a new heading here
(“=== Header extensions ===”).

Max

Quoted 3 lines

Directly after the image header, optional sections called header extensions can be stored. Each extension has a structure like the following:

#3 lore raw

Re: [PATCH v9 1/2] docs: improve qcow2 spec about extending image header

Quoted 23 lines · Max Reitz
Quoted 17 lines · Vladimir Sementsov-Ogievskiy

Make it more obvious how to add new fields to the version 3 header and how to interpret them.

The specification is adjusted so for new defined optional fields:

1. Software may support some of these optional fields and ignore the others, which means that features may be backported to downstream Qemu independently.
2. If we want to add incompatible field (or a field, for which some its values would be incompatible), it must be accompanied by incompatible feature bit.

Also the concept of "default is zero" is clarified, as it's strange to say that the value of the field is assumed to be zero for the software version which don't know about the field at all and don't know how to treat it be it zero or not.

Quoted 4 lines
---
  docs/interop/qcow2.txt | 39 ++++++++++++++++++++++++++++++++++++---
  1 file changed, 36 insertions(+), 3 deletions(-)
Quoted 5 lines

I put review of this off for so long because I always waited for Eric to give his R-b, but maybe not.

I generally think that he’s stricter on what to write in documentation, and accordingly I only have nit picks on spelling and structure:

This is very helpful too, thanks.

I'll resend now with your suggestions, to make it easier to read for others.

Quoted 100 lines
Quoted 19 lines
docs/interop/qcow2.txt +3 -3
index af5711e533..d92c827763 100644
@@ -79,9 +79,9 @@ The first cluster of a qcow2 image contains the file header:
                      Offset into the image file at which the snapshot table
                      starts. Must be aligned to a cluster boundary.
 
-If the version is 3 or higher, the header has the following additional fields.
-For version 2, the values are assumed to be zero, unless specified otherwise
-in the description of a field.
+For version 2, the header is exactly 72 bytes in length, and finishes here.
+For version 3 or higher, the header length is at least 104 bytes, including
+the next fields through header_length.
 
           72 -  79:  incompatible_features
                      Bitmask of incompatible features. An implementation must

docs/interop/qcow2.txt in the description of a field. · line 167

Quoted 4 lines
docs/interop/qcow2.txt +1 -0
@@ -164,6 +164,39 @@ in the description of a field.
          100 - 103:  header_length
                      Length of the header structure in bytes. For version 2
                      images, the length is always assumed to be 72 bytes.
+                    For version 3 it's at least 104 bytes and must be a multiply

s/multiply/multiple/

docs/interop/qcow2.txt in the description of a field. · line 170

Quoted 3 lines
docs/interop/qcow2.txt +3 -0
+                    of 8.
+
+Additional fields (version 3 and higher)

If this is supposed to be a heading, maybe it should enclosed by “===” on both sides.

docs/interop/qcow2.txt in the description of a field. · line 174

Quoted 4 lines
docs/interop/qcow2.txt +4 -0
+
+In general, these fields are optional and may be safely ignored by the software,
+as well as filled by zeros (which is equal to field absence), if software needs
+to set field B, but don't want to care about field A, which precedes B. More

s/don't/does not/ (or maybe s/don't want/does not/)

docs/interop/qcow2.txt in the description of a field. · line 186

Quoted 12 lines
docs/interop/qcow2.txt +12 -0
+formally, additional fields have the following compatibility rules:
+
+1. If the value of the additional field must not be ignored for correct
+handling of the file, it will be accompanied by a corresponding incompatible
+feature bit.
+
+2. If there are no unrecognized incompatible feature bits set, an unknown
+additional field may be safely ignored other than preserving its value when
+rewriting the image header.
+
+3. An explicit value of 0 will have the same behavior as when the field is not
+present*, if not altered by specific incompatible bit.

s/by specific/by a specific/

docs/interop/qcow2.txt in the description of a field. · line 188

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+
+*. Field is not present when header_length is less or equal to field's offset.

s/Field/A field/, s/field's/the field's/

(maybe also +considered, as in "A field is considered not present...")

docs/interop/qcow2.txt in the description of a field. · line 191

Quoted 3 lines
docs/interop/qcow2.txt +3 -0
+Also, all additional fields are not present for version 2.
+
+        < ... No additional fields in the header currently ... >

This looks a bit weird to me, but the next patch will remove it again, so who cares.

docs/interop/qcow2.txt in the description of a field. · line 193

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+Header padding

Same heading note here (I’d make this “=== Header padding ===”).

docs/interop/qcow2.txt in the description of a field. · line 194

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+
+@header_length must be a multiply of 8, which means that if last additional field

s/multiply/multiple/

docs/interop/qcow2.txt in the description of a field. · line 196

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+end is not aligned, some padding is needed. This padding must be zeroed, so that,

I think s/last additional field end/the last additional field’s end/, or maybe s/last additional field end/the end of the last additional field/.

docs/interop/qcow2.txt in the description of a field. · line 199

Quoted 3 lines
docs/interop/qcow2.txt +3 -0
+if some existing (or future) additional field will fall into the padding, it
+will be interpreted accordingly to point [3.] of the previous paragraph, i.e.
+in same manner as when this field is not present.

s/in same/in the same/

Quoted 1 line

I think there should be a new heading here
(“=== Header extensions ===”).

Max

Quoted 3 lines

Directly after the image header, optional sections called header extensions can be stored. Each extension has a structure like the following:

Signature · 2 lines
Best regards,
Vladimir
#4 lore raw

PATCH v9 2/2 docs: qcow2: introduce compression type feature

In reply to Vladimir Sementsov-Ogievskiy

The patch add new additional field to qcow2 header: compression_type, which specifies compression type. If field is absent or zero, default compression type is set: ZLIB, which corresponds to current behavior.

New compression type (ZSTD) is to be added in further commit.

---
 docs/interop/qcow2.txt | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
docs/interop/qcow2.txt +15 -1
index d92c827763..77146b5169 100644
@@ -109,6 +109,11 @@ the next fields through header_length.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
+                    Bit 3:      Compression type bit.  If this bit is set,
+                                non-default compression is used for compressed
+                                clusters. compression_type field must be
+                                present and not zero.
+
                     Bits 3-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
@@ -188,7 +193,16 @@ present*, if not altered by specific incompatible bit.
 *. Field is not present when header_length is less or equal to field's offset.
 Also, all additional fields are not present for version 2.
 
-        < ... No additional fields in the header currently ... >
+              104:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    If incompatible compression type bit is set: the field must
+                    be present and non-zero (which means non-zlib compression type)
+                    If incompatible compression type bit is unset: the field
+                    may not exist or it must be zero (which means zlib).
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/>
 
 Header padding
 
Signature · 1 line
2.18.0
#5 lore raw

Re: [PATCH v9 2/2] docs: qcow2: introduce compression type feature

Quoted 17 lines · Vladimir Sementsov-Ogievskiy

The patch add new additional field to qcow2 header: compression_type, which specifies compression type. If field is absent or zero, default compression type is set: ZLIB, which corresponds to current behavior.

New compression type (ZSTD) is to be added in further commit.

---
 docs/interop/qcow2.txt | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
docs/interop/qcow2.txt +0 -0
index d92c827763..77146b5169 100644

docs/interop/qcow2.txt the next fields through header_length. · line 113

Quoted 5 lines
docs/interop/qcow2.txt +2 -0
@@ -109,6 +109,11 @@ the next fields through header_length.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
+                    Bit 3:      Compression type bit.  If this bit is set,
+                                non-default compression is used for compressed

s/non-default/a non-default/

docs/interop/qcow2.txt the next fields through header_length. · line 115

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+                                clusters. compression_type field must be
+                                present and not zero.

s/compression_type field/The compression_type field/

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 199

Quoted 13 lines
docs/interop/qcow2.txt +5 -1
+
                     Bits 3-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
@@ -188,7 +193,16 @@ present*, if not altered by specific incompatible bit.
 *. Field is not present when header_length is less or equal to field's offset.
 Also, all additional fields are not present for version 2.
 
-        < ... No additional fields in the header currently ... >
+              104:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.

Sounds a bit too complicated. Maybe „All compressed clusters in an image use the same compression type.” instead? (Or s/an/the same/)

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 200

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+                    If incompatible compression type bit is set: the field must

Hmm, this sounds like there was an “incompatible compression type” bit, instead of an incompatible bit called “compression type”. So maybe “If the incompatible bit "compression type" is set, this field must...”?

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 201

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+                    be present and non-zero (which means non-zlib compression type)

s/$/./

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 202

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+                    If incompatible compression type bit is unset: the field

I’d just make this “Otherwise, this field...”

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 203

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+                    may not exist or it must be zero (which means zlib).

“must not be present or must be zero”?

(“exist” sounds a bit weird; the spec only defined “not present” so far.
As for the “may not”, that isn’t in RFC 2119. :-))

Max

Quoted 6 lines
docs/interop/qcow2.txt +2 -0
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/>
 
 Header padding
 
#6 lore raw

Re: [PATCH v9 2/2] docs: qcow2: introduce compression type feature

Quoted 39 lines · Max Reitz
Quoted 17 lines · Vladimir Sementsov-Ogievskiy

The patch add new additional field to qcow2 header: compression_type, which specifies compression type. If field is absent or zero, default compression type is set: ZLIB, which corresponds to current behavior.

New compression type (ZSTD) is to be added in further commit.

---
  docs/interop/qcow2.txt | 16 +++++++++++++++-
  1 file changed, 15 insertions(+), 1 deletion(-)
docs/interop/qcow2.txt +0 -0
index d92c827763..77146b5169 100644

docs/interop/qcow2.txt the next fields through header_length. · line 113

Quoted 5 lines
docs/interop/qcow2.txt +2 -0
@@ -109,6 +109,11 @@ the next fields through header_length.
                                  An External Data File Name header extension may
                                  be present if this bit is set.
 
+                    Bit 3:      Compression type bit.  If this bit is set,
+                                non-default compression is used for compressed

s/non-default/a non-default/

docs/interop/qcow2.txt the next fields through header_length. · line 115

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+                                clusters. compression_type field must be
+                                present and not zero.

s/compression_type field/The compression_type field/

Quoted 8 lines
docs/interop/qcow2.txt +1 -0
+
                      Bits 3-63:  Reserved (set to 0)
 
           80 -  87:  compatible_features
@@ -188,7 +193,16 @@ present*, if not altered by specific incompatible bit.
  *. Field is not present when header_length is less or equal to field's offset.
  Also, all additional fields are not present for version 2.
 
Quoted 8 lines

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 199

Quoted 5 lines
docs/interop/qcow2.txt +4 -1
-        < ... No additional fields in the header currently ... >
+              104:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.

Sounds a bit too complicated. Maybe „All compressed clusters in an image use the same compression type.” instead? (Or s/an/the same/)

Sounds better, yes.

Quoted 6 lines

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 200

Quoted 2 lines
docs/interop/qcow2.txt +1 -0
+                    If incompatible compression type bit is set: the field must

Hmm, this sounds like there was an “incompatible compression type” bit, instead of an incompatible bit called “compression type”. So maybe “If the incompatible bit "compression type" is set, this field must...”?

OK)

Quoted 7 lines

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 201

Quoted 2 lines
docs/interop/qcow2.txt +1 -0
+                    be present and non-zero (which means non-zlib compression type)

s/$/./

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 202

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+                    If incompatible compression type bit is unset: the field
Quoted 8 lines

I’d just make this “Otherwise, this field...”

docs/interop/qcow2.txt present*, if not altered by specific incompatible bit. · line 203

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+                    may not exist or it must be zero (which means zlib).

“must not be present or must be zero”?

(“exist” sounds a bit weird; the spec only defined “not present” so far.
As for the “may not”, that isn’t in RFC 2119. :-))

Agreed, thanks.

Quoted 11 lines

Max

Quoted 6 lines
docs/interop/qcow2.txt +2 -0
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/>
 
  Header padding
 
Signature · 2 lines
Best regards,
Vladimir
#7 lore raw
In reply to Vladimir Sementsov-Ogievskiy

ping

Quoted 17 lines · Vladimir Sementsov-Ogievskiy

Hi all!

Here is my proposal, about how to correctly update qcow2 specification to introduce new field, keeping in mind currently existing images and downstream Qemu instances.

v9: Merge 01 and 02
Change wordings
Require header alignment

Vladimir Sementsov-Ogievskiy (2):
docs: improve qcow2 spec about extending image header
docs: qcow2: introduce compression type feature

  docs/interop/qcow2.txt | 53 +++++++++++++++++++++++++++++++++++++++---
  1 file changed, 50 insertions(+), 3 deletions(-)
Signature · 2 lines
Best regards,
Vladimir