Skip to contentSkip to conversation

qcow2: add zstd cluster compression

PATCH Superseded Denis Plotnikov 2 participants 9 messages Aug 19, 2019 – Aug 27, 2019 Block layer

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover qcow2: add zstd cluster compression

v3:
* relax the compression type setting requirement when
the compression type is not zlib [Eric, Kevin]
* add compression type values to the spec [Eric]
* fix wording in the spec and descriptions [Eric]
* fix functions descriptions [Max]
* fix zstd (de)compression functions flaws [Max]
* fix zstd related parts of configure file [Max]
* rebased to v4.1.0-rc5 and chenged the series version aiming to 4.2

v2:
* relax the compression type setting restriction in the spec
* fix qcow2 header size checking
* fix error processing and messaging
* fix qcow2 image specific info reporting
* set Qcow2CompressionType zstd config dependant
* add zstd compressed cluster format description to the spec

v1:
* extend qcow2 header instead of adding a new incompatible extension header specification re-written accordingly
* enable zstd compression via config
* fix zstd (de)compression functions
* fix comments/description
* fix function naming

---
The goal of the patch-set is to enable qcow2 to use zstd compression for clusters. ZSTD provides better (de)compression performance than currently used ZLIB. Using it will improve perforamnce (reduce compression time) when the compressed clusters is used, e.g backup scenarios.

Also, the patch-set extends qcow2 specification by adding compression_type feature. The feature enables adding ZSTD and another compression algorithms in the future.

Here is some measurements ZSTD vs ZLIB:

The test:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

The results:
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0

Both ZLIB and ZSTD gave the same compression ratio: ~1.5 compressed image size in both cases: ~1.4G

Denis Plotnikov (3):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression

 block/qcow2-threads.c     | 172 ++++++++++++++++++++++++++++++++++----
 block/qcow2.c             | 100 ++++++++++++++++++++++
 block/qcow2.h             |  26 ++++--
 configure                 |  34 ++++++++
 docs/interop/qcow2.txt    |  39 ++++++++-
 include/block/block_int.h |   1 +
 qapi/block-core.json      |  23 ++++-
 7 files changed, 371 insertions(+), 24 deletions(-)
Signature · 1 line
2.17.0
#1 lore raw

PATCH v3 1/3 qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image.

The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLIB. It works roughly 2x faster than ZLIB providing a comparable compression ratio and therefore provides a performance advantage in backup scenarios.

The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.

---
 block/qcow2.c             | 94 +++++++++++++++++++++++++++++++++++++++
 block/qcow2.h             | 26 ++++++++---
 docs/interop/qcow2.txt    | 19 +++++++-
 include/block/block_int.h |  1 +
 qapi/block-core.json      | 22 ++++++++-
 5 files changed, 152 insertions(+), 10 deletions(-)
block/qcow2.c +94 -0
index 039bdc2f7e..4e07b7e9ec 100644
@@ -1197,6 +1197,32 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
     return ret;
 }
 
+static int check_compression_type(BDRVQcow2State *s, Error **errp)
+{
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        error_setg(errp, "qcow2: unknown compression type: %u",
+                   s->compression_type);
+        return -ENOTSUP;
+    }
+
+    /*
+     * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+     * the incompatible feature flag must be set
+     */
+
+    if (s->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB &&
+        !(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
+            error_setg(errp, "qcow2: Invalid compression type setting");
+            return -EINVAL;
+    }
+
+    return 0;
+}
+
 /* Called with s->lock held.  */
 static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
                                       int flags, Error **errp)
@@ -1312,6 +1338,35 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
     s->compatible_features      = header.compatible_features;
     s->autoclear_features       = header.autoclear_features;
 
+    /*
+     * Handle compression type
+     * Older qcow2 images don't contain the compression type header.
+     * Distinguish them by the header length and use
+     * the only valid (default) compression type in that case
+     */
+    if (header.header_length > offsetof(QCowHeader, compression_type)) {
+        /* sanity check that we can read a compression type */
+        size_t min_len = offsetof(QCowHeader, compression_type) +
+                         sizeof(header.compression_type);
+        if (header.header_length < min_len) {
+            error_setg(errp,
+                       "Could not read compression type, "
+                       "qcow2 header is too short");
+            ret = -EINVAL;
+            goto fail;
+        }
+
+        header.compression_type = be32_to_cpu(header.compression_type);
+        s->compression_type = header.compression_type;
+    } else {
+        s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+    }
+
+    ret = check_compression_type(s, errp);
+    if (ret) {
+        goto fail;
+    }
+
     if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
         void *feature_table = NULL;
         qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2516,6 +2571,12 @@ int qcow2_update_header(BlockDriverState *bs)
     total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
     refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
+    ret = check_compression_type(s, NULL);
+
+    if (ret) {
+        goto fail;
+    }
+
     *header = (QCowHeader) {
         /* Version 2 fields */
         .magic                  = cpu_to_be32(QCOW_MAGIC),
@@ -2538,6 +2599,7 @@ int qcow2_update_header(BlockDriverState *bs)
         .autoclear_features     = cpu_to_be64(s->autoclear_features),
         .refcount_order         = cpu_to_be32(s->refcount_order),
         .header_length          = cpu_to_be32(header_length),
+        .compression_type       = cpu_to_be32(s->compression_type),
     };
 
     /* For older versions, write a shorter header */
@@ -2635,6 +2697,11 @@ int qcow2_update_header(BlockDriverState *bs)
                 .bit  = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
                 .name = "lazy refcounts",
             },
+            {
+                .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+                .bit  = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
+                .name = "compression type",
+            },
         };
 
         ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
@@ -3202,6 +3269,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         .refcount_table_offset      = cpu_to_be64(cluster_size),
         .refcount_table_clusters    = cpu_to_be32(1),
         .refcount_order             = cpu_to_be32(refcount_order),
+        .compression_type           = cpu_to_be32(QCOW2_COMPRESSION_TYPE_ZLIB),
         .header_length              = cpu_to_be32(sizeof(*header)),
     };
 
@@ -3221,6 +3289,24 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
             cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
     }
 
+    if (qcow2_opts->has_compression_type &&
+        qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+        switch (qcow2_opts->compression_type) {
+        case QCOW2_COMPRESSION_TYPE_ZLIB:
+            break;
+
+        default:
+            error_setg_errno(errp, -EINVAL, "Unknown compression type");
+            goto out;
+        }
+
+        header->compression_type = cpu_to_be32(qcow2_opts->compression_type);
+
+        header->incompatible_features |=
+            cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);
+    }
+
     ret = blk_pwrite(blk, 0, header, cluster_size, 0);
     g_free(header);
     if (ret < 0) {
@@ -3402,6 +3488,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
         { BLOCK_OPT_ENCRYPT,            BLOCK_OPT_ENCRYPT_FORMAT },
         { BLOCK_OPT_COMPAT_LEVEL,       "version" },
         { BLOCK_OPT_DATA_FILE_RAW,      "data-file-raw" },
+        { BLOCK_OPT_COMPRESSION_TYPE,   "compression-type" },
         { NULL, NULL },
     };
 
@@ -4598,6 +4685,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
             .data_file          = g_strdup(s->image_data_file),
             .has_data_file_raw  = has_data_file(bs),
             .data_file_raw      = data_file_is_raw(bs),
+            .compression_type   = s->compression_type,
         };
     } else {
         /* if this assertion fails, this probably means a new version was
@@ -5163,6 +5251,12 @@ static QemuOptsList qcow2_create_opts = {
             .help = "Width of a reference count entry in bits",
             .def_value_str = "16"
         },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",
+            .def_value_str = "zlib"
+        },
         { /* end of list */ }
     }
 };
block/qcow2.h +19 -7
index fc1b0d3c1e..9a241e4b9a 100644
@@ -140,6 +140,7 @@ typedef struct QCowHeader {
 
     uint32_t refcount_order;
     uint32_t header_length;
+    uint32_t compression_type;
 } QEMU_PACKED QCowHeader;
 
 typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -203,16 +204,20 @@ enum {
 
 /* Incompatible feature bits */
 enum {
-    QCOW2_INCOMPAT_DIRTY_BITNR      = 0,
-    QCOW2_INCOMPAT_CORRUPT_BITNR    = 1,
-    QCOW2_INCOMPAT_DATA_FILE_BITNR  = 2,
-    QCOW2_INCOMPAT_DIRTY            = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
-    QCOW2_INCOMPAT_CORRUPT          = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
-    QCOW2_INCOMPAT_DATA_FILE        = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_DIRTY_BITNR            = 0,
+    QCOW2_INCOMPAT_CORRUPT_BITNR          = 1,
+    QCOW2_INCOMPAT_DATA_FILE_BITNR        = 2,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
+    QCOW2_INCOMPAT_DIRTY                  = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
+    QCOW2_INCOMPAT_CORRUPT                = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
+    QCOW2_INCOMPAT_DATA_FILE              = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE       =
+        1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
 
     QCOW2_INCOMPAT_MASK             = QCOW2_INCOMPAT_DIRTY
                                     | QCOW2_INCOMPAT_CORRUPT
-                                    | QCOW2_INCOMPAT_DATA_FILE,
+                                    | QCOW2_INCOMPAT_DATA_FILE
+                                    | QCOW2_INCOMPAT_COMPRESSION_TYPE,
 };
 
 /* Compatible feature bits */
@@ -359,6 +364,13 @@ typedef struct BDRVQcow2State {
 
     bool metadata_preallocation_checked;
     bool metadata_preallocation;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * The only way to change the compression type is to convert the image
+     * with the desired compression type set
+     */
+    uint32_t compression_type;
 } BDRVQcow2State;
 
 typedef struct Qcow2COWRegion {
docs/interop/qcow2.txt +18 -1
index af5711e533..e1be8bd5c3 100644
@@ -109,7 +109,12 @@ in the description of a field.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit. The bit must be set if
+                                the compression type differs from default of zlib.
+                                If the compression type is default the bit should
+                                be unset.
+
+                    Bits 4-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
                     Bitmask of compatible features. An implementation can
@@ -165,6 +170,18 @@ in the description of a field.
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
 
+        104 - 107:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    The compression type is set on image creation only.
+                    The default compression type is zlib (value: 0).
+                    When the compression type differs from the default
+                    the compression type bit (incompatible feature bit 3)
+                    must be set.
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/> (default)
+
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
 
include/block/block_int.h +1 -0
index 3aa1e832a8..4b254802e5 100644
@@ -58,6 +58,7 @@
 #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
 #define BLOCK_OPT_DATA_FILE         "data_file"
 #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
 #define BLOCK_PROBE_BUF_SIZE        512
 
qapi/block-core.json +20 -2
index 0d43d4f37c..2c002ca6a9 100644
@@ -78,6 +78,8 @@
 #
 # @bitmaps: A list of qcow2 bitmap details (since 4.0)
 #
+# @compression-type: the image cluster compression method (since 4.2)
+#
 # Since: 1.7
 ##
 { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
       '*corrupt': 'bool',
       'refcount-bits': 'int',
       '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      'compression-type': 'Qcow2CompressionType'
   } }
 
 ##
@@ -4274,6 +4277,18 @@
   'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib:  zlib compression, see <http://zlib.net/>
+#
+# Since: 4.2
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
 ##
 # @BlockdevCreateOptionsQcow2:
 #
@@ -4297,6 +4312,8 @@
 #                   allowed values: off, falloc, full, metadata)
 # @lazy-refcounts   True if refcounts may be updated lazily (default: off)
 # @refcount-bits    Width of reference counts in bits (default: 16)
+# @compression-type The image cluster compression method
+#                   (default: zlib, since 4.2)
 #
 # Since: 2.12
 ##
@@ -4312,7 +4329,8 @@
             '*cluster-size':    'size',
             '*preallocation':   'PreallocMode',
             '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type': 'Qcow2CompressionType' } }
 
 ##
 # @BlockdevCreateOptionsQed:
Signature · 1 line
2.17.0
#2 lore raw

Re: [Qemu-devel] [PATCH v3 1/3] qcow2: introduce compression type feature

Quoted 140 lines · Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image.

The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLIB. It works roughly 2x faster than ZLIB providing a comparable compression ratio and therefore provides a performance advantage in backup scenarios.

The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.

---
  block/qcow2.c             | 94 +++++++++++++++++++++++++++++++++++++++
  block/qcow2.h             | 26 ++++++++---
  docs/interop/qcow2.txt    | 19 +++++++-
  include/block/block_int.h |  1 +
  qapi/block-core.json      | 22 ++++++++-
  5 files changed, 152 insertions(+), 10 deletions(-)
block/qcow2.c +68 -0
index 039bdc2f7e..4e07b7e9ec 100644
@@ -1197,6 +1197,32 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
      return ret;
  }
 
+static int check_compression_type(BDRVQcow2State *s, Error **errp)
+{
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        error_setg(errp, "qcow2: unknown compression type: %u",
+                   s->compression_type);
+        return -ENOTSUP;
+    }
+
+    /*
+     * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+     * the incompatible feature flag must be set
+     */
+
+    if (s->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB &&
+        !(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
+            error_setg(errp, "qcow2: Invalid compression type setting");
+            return -EINVAL;
+    }
+
+    return 0;
+}
+
  /* Called with s->lock held.  */
  static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
                                        int flags, Error **errp)
@@ -1312,6 +1338,35 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
      s->compatible_features      = header.compatible_features;
      s->autoclear_features       = header.autoclear_features;
 
+    /*
+     * Handle compression type
+     * Older qcow2 images don't contain the compression type header.
+     * Distinguish them by the header length and use
+     * the only valid (default) compression type in that case
+     */
+    if (header.header_length > offsetof(QCowHeader, compression_type)) {
+        /* sanity check that we can read a compression type */
+        size_t min_len = offsetof(QCowHeader, compression_type) +
+                         sizeof(header.compression_type);
+        if (header.header_length < min_len) {
+            error_setg(errp,
+                       "Could not read compression type, "
+                       "qcow2 header is too short");
+            ret = -EINVAL;
+            goto fail;
+        }
+
+        header.compression_type = be32_to_cpu(header.compression_type);
+        s->compression_type = header.compression_type;
+    } else {
+        s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+    }
+
+    ret = check_compression_type(s, errp);
+    if (ret) {
+        goto fail;
+    }
+
      if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
          void *feature_table = NULL;
          qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2516,6 +2571,12 @@ int qcow2_update_header(BlockDriverState *bs)
      total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
      refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
+    ret = check_compression_type(s, NULL);
+
+    if (ret) {
+        goto fail;
+    }
+
      *header = (QCowHeader) {
          /* Version 2 fields */
          .magic                  = cpu_to_be32(QCOW_MAGIC),
@@ -2538,6 +2599,7 @@ int qcow2_update_header(BlockDriverState *bs)
          .autoclear_features     = cpu_to_be64(s->autoclear_features),
          .refcount_order         = cpu_to_be32(s->refcount_order),
          .header_length          = cpu_to_be32(header_length),
+        .compression_type       = cpu_to_be32(s->compression_type),
      };
 
      /* For older versions, write a shorter header */
@@ -2635,6 +2697,11 @@ int qcow2_update_header(BlockDriverState *bs)
                  .bit  = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
                  .name = "lazy refcounts",
              },
+            {
+                .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+                .bit  = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
+                .name = "compression type",
+            },
          };
 
          ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
@@ -3202,6 +3269,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
          .refcount_table_offset      = cpu_to_be64(cluster_size),
          .refcount_table_clusters    = cpu_to_be32(1),
          .refcount_order             = cpu_to_be32(refcount_order),
+        .compression_type           = cpu_to_be32(QCOW2_COMPRESSION_TYPE_ZLIB),
          .header_length              = cpu_to_be32(sizeof(*header)),
      };
 

block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3296

Quoted 8 lines
block/qcow2.c +5 -0
@@ -3221,6 +3289,24 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
              cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
      }
 
+    if (qcow2_opts->has_compression_type &&
+        qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+        switch (qcow2_opts->compression_type) {
+        case QCOW2_COMPRESSION_TYPE_ZLIB:

a bit confusing, as it's unreachable because of if (), and because if we are here we are going to do a wrong thing: to set incompatible feature (so, I firstly thought that this is a bug and then looked above at if() condition)

block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3307

Quoted 11 lines
block/qcow2.c +11 -0
+            break;
+
+        default:
+            error_setg_errno(errp, -EINVAL, "Unknown compression type");
+            goto out;
+        }
+
+        header->compression_type = cpu_to_be32(qcow2_opts->compression_type);
+
+        header->incompatible_features |=
+            cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);

cpu_to_be32 actually

block/qcow2.c static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt · line 3491

Quoted 10 lines
block/qcow2.c +3 -0
+    }
+
      ret = blk_pwrite(blk, 0, header, cluster_size, 0);
      g_free(header);
      if (ret < 0) {
@@ -3402,6 +3488,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
          { BLOCK_OPT_ENCRYPT,            BLOCK_OPT_ENCRYPT_FORMAT },
          { BLOCK_OPT_COMPAT_LEVEL,       "version" },
          { BLOCK_OPT_DATA_FILE_RAW,      "data-file-raw" },
+        { BLOCK_OPT_COMPRESSION_TYPE,   "compression-type" },

I think we don't need it. This array is commented as:
/* Change legacy command line options into QMP ones */

but compression-type is not a legacy option, it's a new one. Why to tolerate old notation for it?

Quoted 97 lines
block/qcow2.c +7 -0
          { NULL, NULL },
      };
 
@@ -4598,6 +4685,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
              .data_file          = g_strdup(s->image_data_file),
              .has_data_file_raw  = has_data_file(bs),
              .data_file_raw      = data_file_is_raw(bs),
+            .compression_type   = s->compression_type,
          };
      } else {
          /* if this assertion fails, this probably means a new version was
@@ -5163,6 +5251,12 @@ static QemuOptsList qcow2_create_opts = {
              .help = "Width of a reference count entry in bits",
              .def_value_str = "16"
          },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",
+            .def_value_str = "zlib"
+        },
          { /* end of list */ }
      }
  };
block/qcow2.h +19 -7
index fc1b0d3c1e..9a241e4b9a 100644
@@ -140,6 +140,7 @@ typedef struct QCowHeader {
 
      uint32_t refcount_order;
      uint32_t header_length;
+    uint32_t compression_type;
  } QEMU_PACKED QCowHeader;
 
  typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -203,16 +204,20 @@ enum {
 
  /* Incompatible feature bits */
  enum {
-    QCOW2_INCOMPAT_DIRTY_BITNR      = 0,
-    QCOW2_INCOMPAT_CORRUPT_BITNR    = 1,
-    QCOW2_INCOMPAT_DATA_FILE_BITNR  = 2,
-    QCOW2_INCOMPAT_DIRTY            = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
-    QCOW2_INCOMPAT_CORRUPT          = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
-    QCOW2_INCOMPAT_DATA_FILE        = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_DIRTY_BITNR            = 0,
+    QCOW2_INCOMPAT_CORRUPT_BITNR          = 1,
+    QCOW2_INCOMPAT_DATA_FILE_BITNR        = 2,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
+    QCOW2_INCOMPAT_DIRTY                  = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
+    QCOW2_INCOMPAT_CORRUPT                = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
+    QCOW2_INCOMPAT_DATA_FILE              = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE       =
+        1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
 
      QCOW2_INCOMPAT_MASK             = QCOW2_INCOMPAT_DIRTY
                                      | QCOW2_INCOMPAT_CORRUPT
-                                    | QCOW2_INCOMPAT_DATA_FILE,
+                                    | QCOW2_INCOMPAT_DATA_FILE
+                                    | QCOW2_INCOMPAT_COMPRESSION_TYPE,
  };
 
  /* Compatible feature bits */
@@ -359,6 +364,13 @@ typedef struct BDRVQcow2State {
 
      bool metadata_preallocation_checked;
      bool metadata_preallocation;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * The only way to change the compression type is to convert the image
+     * with the desired compression type set
+     */
+    uint32_t compression_type;
  } BDRVQcow2State;
 
  typedef struct Qcow2COWRegion {
docs/interop/qcow2.txt +6 -1
index af5711e533..e1be8bd5c3 100644
@@ -109,7 +109,12 @@ in the description of a field.
                                  An External Data File Name header extension may
                                  be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit. The bit must be set if
+                                the compression type differs from default of zlib.
+                                If the compression type is default the bit should
+                                be unset.
+
+                    Bits 4-63:  Reserved (set to 0)
 
           80 -  87:  compatible_features
                      Bitmask of compatible features. An implementation can

docs/interop/qcow2.txt in the description of a field. · line 173

Quoted 4 lines
docs/interop/qcow2.txt +1 -0
@@ -165,6 +170,18 @@ in the description of a field.
                      Length of the header structure in bytes. For version 2
                      images, the length is always assumed to be 72 bytes.
 
+        104 - 107:  compression_type

Why 4 bytes? 1 is enough and 2 are enough for sure. Or we need to align all fields to 4 bytes?

docs/interop/qcow2.txt in the description of a field.

Quoted 6 lines
docs/interop/qcow2.txt +4 -0
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
 
 
+                    The compression type is set on image creation only.

this sentence is not needed, why to abandon inplace conversion? And anyway, it's not about specification of format.

Quoted 84 lines
docs/interop/qcow2.txt +7 -0
+                    The default compression type is zlib (value: 0).
+                    When the compression type differs from the default
+                    the compression type bit (incompatible feature bit 3)
+                    must be set.
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/> (default)
+
  Directly after the image header, optional sections called header extensions can

be stored. Each extension has a structure like the following:

include/block/block_int.h +1 -0
index 3aa1e832a8..4b254802e5 100644
@@ -58,6 +58,7 @@
  #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
  #define BLOCK_OPT_DATA_FILE         "data_file"
  #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
  #define BLOCK_PROBE_BUF_SIZE        512
 
qapi/block-core.json +20 -2
index 0d43d4f37c..2c002ca6a9 100644
@@ -78,6 +78,8 @@
  #
  # @bitmaps: A list of qcow2 bitmap details (since 4.0)
  #
+# @compression-type: the image cluster compression method (since 4.2)
+#
  # Since: 1.7
  ##
  { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
        '*corrupt': 'bool',
        'refcount-bits': 'int',
        '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      'compression-type': 'Qcow2CompressionType'
    } }
 
  ##
@@ -4274,6 +4277,18 @@
    'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib:  zlib compression, see <http://zlib.net/>
+#
+# Since: 4.2
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
  ##
  # @BlockdevCreateOptionsQcow2:
  #
@@ -4297,6 +4312,8 @@
  #                   allowed values: off, falloc, full, metadata)
  # @lazy-refcounts   True if refcounts may be updated lazily (default: off)
  # @refcount-bits    Width of reference counts in bits (default: 16)
+# @compression-type The image cluster compression method
+#                   (default: zlib, since 4.2)
  #
  # Since: 2.12
  ##
@@ -4312,7 +4329,8 @@
              '*cluster-size':    'size',
              '*preallocation':   'PreallocMode',
              '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type': 'Qcow2CompressionType' } }
 
  ##
  # @BlockdevCreateOptionsQed:
Signature · 2 lines
Best regards,
Vladimir
#3 lore raw

Re: [Qemu-devel] [PATCH v3 1/3] qcow2: introduce compression type feature

Quoted 144 lines · Vladimir Sementsov-Ogievskiy
Quoted 143 lines · Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image.

The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLIB. It works roughly 2x faster than ZLIB providing a comparable compression ratio and therefore provides a performance advantage in backup scenarios.

The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.

---
   block/qcow2.c             | 94 +++++++++++++++++++++++++++++++++++++++
   block/qcow2.h             | 26 ++++++++---
   docs/interop/qcow2.txt    | 19 +++++++-
   include/block/block_int.h |  1 +
   qapi/block-core.json      | 22 ++++++++-
   5 files changed, 152 insertions(+), 10 deletions(-)
block/qcow2.c +68 -0
index 039bdc2f7e..4e07b7e9ec 100644
@@ -1197,6 +1197,32 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
       return ret;
   }
 
+static int check_compression_type(BDRVQcow2State *s, Error **errp)
+{
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        error_setg(errp, "qcow2: unknown compression type: %u",
+                   s->compression_type);
+        return -ENOTSUP;
+    }
+
+    /*
+     * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+     * the incompatible feature flag must be set
+     */
+
+    if (s->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB &&
+        !(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
+            error_setg(errp, "qcow2: Invalid compression type setting");
+            return -EINVAL;
+    }
+
+    return 0;
+}
+
   /* Called with s->lock held.  */
   static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
                                         int flags, Error **errp)
@@ -1312,6 +1338,35 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
       s->compatible_features      = header.compatible_features;
       s->autoclear_features       = header.autoclear_features;
 
+    /*
+     * Handle compression type
+     * Older qcow2 images don't contain the compression type header.
+     * Distinguish them by the header length and use
+     * the only valid (default) compression type in that case
+     */
+    if (header.header_length > offsetof(QCowHeader, compression_type)) {
+        /* sanity check that we can read a compression type */
+        size_t min_len = offsetof(QCowHeader, compression_type) +
+                         sizeof(header.compression_type);
+        if (header.header_length < min_len) {
+            error_setg(errp,
+                       "Could not read compression type, "
+                       "qcow2 header is too short");
+            ret = -EINVAL;
+            goto fail;
+        }
+
+        header.compression_type = be32_to_cpu(header.compression_type);
+        s->compression_type = header.compression_type;
+    } else {
+        s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+    }
+
+    ret = check_compression_type(s, errp);
+    if (ret) {
+        goto fail;
+    }
+
       if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
           void *feature_table = NULL;
           qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2516,6 +2571,12 @@ int qcow2_update_header(BlockDriverState *bs)
       total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
       refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
+    ret = check_compression_type(s, NULL);
+
+    if (ret) {
+        goto fail;
+    }
+
       *header = (QCowHeader) {
           /* Version 2 fields */
           .magic                  = cpu_to_be32(QCOW_MAGIC),
@@ -2538,6 +2599,7 @@ int qcow2_update_header(BlockDriverState *bs)
           .autoclear_features     = cpu_to_be64(s->autoclear_features),
           .refcount_order         = cpu_to_be32(s->refcount_order),
           .header_length          = cpu_to_be32(header_length),
+        .compression_type       = cpu_to_be32(s->compression_type),
       };
 
       /* For older versions, write a shorter header */
@@ -2635,6 +2697,11 @@ int qcow2_update_header(BlockDriverState *bs)
                   .bit  = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
                   .name = "lazy refcounts",
               },
+            {
+                .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+                .bit  = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
+                .name = "compression type",
+            },
           };
 
           ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
@@ -3202,6 +3269,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
           .refcount_table_offset      = cpu_to_be64(cluster_size),
           .refcount_table_clusters    = cpu_to_be32(1),
           .refcount_order             = cpu_to_be32(refcount_order),
+        .compression_type           = cpu_to_be32(QCOW2_COMPRESSION_TYPE_ZLIB),
           .header_length              = cpu_to_be32(sizeof(*header)),
       };
 
@@ -3221,6 +3289,24 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
               cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
       }
 
Quoted 9 lines

block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3296

Quoted 5 lines
block/qcow2.c +5 -0
+    if (qcow2_opts->has_compression_type &&
+        qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+        switch (qcow2_opts->compression_type) {
+        case QCOW2_COMPRESSION_TYPE_ZLIB:

a bit confusing, as it's unreachable because of if (), and because if we are here we are going to do a wrong thing: to set incompatible feature (so, I firstly thought that this is a bug and then looked above at if() condition)

Yes, there should be QCOW2_COMPRESSION_TYPE_ZSTD being added in later patches in the series

Quoted 5 lines
Quoted 5 lines
block/qcow2.c +5 -0
+            break;
+
+        default:
+            error_setg_errno(errp, -EINVAL, "Unknown compression type");
+            goto out;
Quoted 8 lines

block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3307

Quoted 6 lines
block/qcow2.c +6 -0
+        }
+
+        header->compression_type = cpu_to_be32(qcow2_opts->compression_type);
+
+        header->incompatible_features |=
+            cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);

cpu_to_be32 actually

true

Quoted 117 lines

block/qcow2.c static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt · line 3491

Quoted 11 lines
block/qcow2.c +3 -0
+    }
+
       ret = blk_pwrite(blk, 0, header, cluster_size, 0);
       g_free(header);
       if (ret < 0) {
@@ -3402,6 +3488,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
           { BLOCK_OPT_ENCRYPT,            BLOCK_OPT_ENCRYPT_FORMAT },
           { BLOCK_OPT_COMPAT_LEVEL,       "version" },
           { BLOCK_OPT_DATA_FILE_RAW,      "data-file-raw" },
+        { BLOCK_OPT_COMPRESSION_TYPE,   "compression-type" },

I think we don't need it. This array is commented as:

   /* Change legacy command line options into QMP ones */

but compression-type is not a legacy option, it's a new one. Why to tolerate old notation for it?

Quoted 100 lines
block/qcow2.c +7 -0
           { NULL, NULL },
       };
 
@@ -4598,6 +4685,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
               .data_file          = g_strdup(s->image_data_file),
               .has_data_file_raw  = has_data_file(bs),
               .data_file_raw      = data_file_is_raw(bs),
+            .compression_type   = s->compression_type,
           };
       } else {
           /* if this assertion fails, this probably means a new version was
@@ -5163,6 +5251,12 @@ static QemuOptsList qcow2_create_opts = {
               .help = "Width of a reference count entry in bits",
               .def_value_str = "16"
           },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",
+            .def_value_str = "zlib"
+        },
           { /* end of list */ }
       }
   };
block/qcow2.h +19 -7
index fc1b0d3c1e..9a241e4b9a 100644
@@ -140,6 +140,7 @@ typedef struct QCowHeader {
 
       uint32_t refcount_order;
       uint32_t header_length;
+    uint32_t compression_type;
   } QEMU_PACKED QCowHeader;
 
   typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -203,16 +204,20 @@ enum {
 
   /* Incompatible feature bits */
   enum {
-    QCOW2_INCOMPAT_DIRTY_BITNR      = 0,
-    QCOW2_INCOMPAT_CORRUPT_BITNR    = 1,
-    QCOW2_INCOMPAT_DATA_FILE_BITNR  = 2,
-    QCOW2_INCOMPAT_DIRTY            = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
-    QCOW2_INCOMPAT_CORRUPT          = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
-    QCOW2_INCOMPAT_DATA_FILE        = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_DIRTY_BITNR            = 0,
+    QCOW2_INCOMPAT_CORRUPT_BITNR          = 1,
+    QCOW2_INCOMPAT_DATA_FILE_BITNR        = 2,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
+    QCOW2_INCOMPAT_DIRTY                  = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
+    QCOW2_INCOMPAT_CORRUPT                = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
+    QCOW2_INCOMPAT_DATA_FILE              = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE       =
+        1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
 
       QCOW2_INCOMPAT_MASK             = QCOW2_INCOMPAT_DIRTY
                                       | QCOW2_INCOMPAT_CORRUPT
-                                    | QCOW2_INCOMPAT_DATA_FILE,
+                                    | QCOW2_INCOMPAT_DATA_FILE
+                                    | QCOW2_INCOMPAT_COMPRESSION_TYPE,
   };
 
   /* Compatible feature bits */
@@ -359,6 +364,13 @@ typedef struct BDRVQcow2State {
 
       bool metadata_preallocation_checked;
       bool metadata_preallocation;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * The only way to change the compression type is to convert the image
+     * with the desired compression type set
+     */
+    uint32_t compression_type;
   } BDRVQcow2State;
 
   typedef struct Qcow2COWRegion {
docs/interop/qcow2.txt +6 -1
index af5711e533..e1be8bd5c3 100644
@@ -109,7 +109,12 @@ in the description of a field.
                                   An External Data File Name header extension may
                                   be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit. The bit must be set if
+                                the compression type differs from default of zlib.
+                                If the compression type is default the bit should
+                                be unset.
+
+                    Bits 4-63:  Reserved (set to 0)
 
            80 -  87:  compatible_features
                       Bitmask of compatible features. An implementation can
@@ -165,6 +170,18 @@ in the description of a field.
                       Length of the header structure in bytes. For version 2
                       images, the length is always assumed to be 72 bytes.
 
Quoted 3 lines

docs/interop/qcow2.txt in the description of a field. · line 173

Quoted 1 line
docs/interop/qcow2.txt +1 -0
+        104 - 107:  compression_type

Why 4 bytes? 1 is enough and 2 are enough for sure. Or we need to align all fields to 4 bytes?

Each field in the header is at least 4 bytes in size including "32-35 crypt_method" which also is not expected to have a big variety of values (now it's 3). I'm not sure but there should be some kind of reason for that.

Quoted 9 lines

docs/interop/qcow2.txt in the description of a field.

Quoted 6 lines
docs/interop/qcow2.txt +4 -0
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
 
 
+                    The compression type is set on image creation only.

this sentence is not needed, why to abandon inplace conversion? And anyway, it's not about specification of format.

But on the other hand, it makes the intentions clearer.>

Quoted 88 lines
Quoted 86 lines
docs/interop/qcow2.txt +7 -0
+                    The default compression type is zlib (value: 0).
+                    When the compression type differs from the default
+                    the compression type bit (incompatible feature bit 3)
+                    must be set.
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/> (default)
+
   Directly after the image header, optional sections called header extensions can

be stored. Each extension has a structure like the following:

include/block/block_int.h +1 -0
index 3aa1e832a8..4b254802e5 100644
@@ -58,6 +58,7 @@
   #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
   #define BLOCK_OPT_DATA_FILE         "data_file"
   #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
   #define BLOCK_PROBE_BUF_SIZE        512
 
qapi/block-core.json +20 -2
index 0d43d4f37c..2c002ca6a9 100644
@@ -78,6 +78,8 @@
   #
   # @bitmaps: A list of qcow2 bitmap details (since 4.0)
   #
+# @compression-type: the image cluster compression method (since 4.2)
+#
   # Since: 1.7
   ##
   { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
         '*corrupt': 'bool',
         'refcount-bits': 'int',
         '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      'compression-type': 'Qcow2CompressionType'
     } }
 
   ##
@@ -4274,6 +4277,18 @@
     'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib:  zlib compression, see <http://zlib.net/>
+#
+# Since: 4.2
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
   ##
   # @BlockdevCreateOptionsQcow2:
   #
@@ -4297,6 +4312,8 @@
   #                   allowed values: off, falloc, full, metadata)
   # @lazy-refcounts   True if refcounts may be updated lazily (default: off)
   # @refcount-bits    Width of reference counts in bits (default: 16)
+# @compression-type The image cluster compression method
+#                   (default: zlib, since 4.2)
   #
   # Since: 2.12
   ##
@@ -4312,7 +4329,8 @@
               '*cluster-size':    'size',
               '*preallocation':   'PreallocMode',
               '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type': 'Qcow2CompressionType' } }
 
   ##
   # @BlockdevCreateOptionsQed:
Signature · 2 lines
Best,
Denis
#4 lore raw

PATCH v3 2/3 qcow2: rework the cluster compression routine

In reply to Denis Plotnikov

The patch allow to process image compression type defined in the image header and choose an appropriate method for image clusters (de)compression.

---
 block/qcow2-threads.c | 78 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 64 insertions(+), 14 deletions(-)
block/qcow2-threads.c +64 -14
index 3b1e63fe41..14b5bd76fb 100644
@@ -73,8 +73,11 @@ typedef struct Qcow2CompressData {
     Qcow2CompressFunc func;
 } Qcow2CompressData;
 
+
 /*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
  *
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
@@ -83,8 +86,8 @@ typedef struct Qcow2CompressData {
  *          -ENOMEM destination buffer is not enough to store compressed data
  *          -EIO    on any other error
  */
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
-                              const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
 {
     ssize_t ret;
     z_stream strm;
@@ -119,19 +122,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
 }
 
 /*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
  *
  * Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
  *
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
  *
  * Returns: 0 on success
- *          -1 on fail
+ *          -EIO on fail
  */
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-                                const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
 {
     int ret = 0;
     z_stream strm;
@@ -144,7 +147,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
 
     ret = inflateInit2(&strm, -12);
     if (ret != Z_OK) {
-        return -1;
+        return -EIO;
     }
 
     ret = inflate(&strm, Z_FINISH);
@@ -154,7 +157,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
          * @src buffer may be processed partly (because in qcow2 we know size of
          * compressed data with precision of one sector)
          */
-        ret = -1;
+        ret = -EIO;
     }
 
     inflateEnd(&strm);
@@ -189,20 +192,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
     return arg.ret;
 }
 
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on fail
+ */
 ssize_t coroutine_fn
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
                   const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_compress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_compress;
+        break;
+
+    default:
+        return -ENOTSUP;
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on fail
+ */
 ssize_t coroutine_fn
 qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
                     const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_decompress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_decompress;
+        break;
+
+    default:
+        return -ENOTSUP;
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
 
Signature · 1 line
2.17.0
Reviewed-by on v3 2/3
#5 lore raw

Re: [Qemu-devel] [PATCH v3 2/3] qcow2: rework the cluster compression routine

Quoted 102 lines · Denis Plotnikov

The patch allow to process image compression type defined in the image header and choose an appropriate method for image clusters (de)compression.

---
  block/qcow2-threads.c | 78 +++++++++++++++++++++++++++++++++++--------
  1 file changed, 64 insertions(+), 14 deletions(-)
block/qcow2-threads.c +25 -12
index 3b1e63fe41..14b5bd76fb 100644
@@ -73,8 +73,11 @@ typedef struct Qcow2CompressData {
      Qcow2CompressFunc func;
  } Qcow2CompressData;
 
+
  /*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
   *
   * @dest - destination buffer, @dest_size bytes
   * @src - source buffer, @src_size bytes
@@ -83,8 +86,8 @@ typedef struct Qcow2CompressData {
   *          -ENOMEM destination buffer is not enough to store compressed data
   *          -EIO    on any other error
   */
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
-                              const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
  {
      ssize_t ret;
      z_stream strm;
@@ -119,19 +122,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
  }
 
  /*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
   *
   * Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
   *
   * @dest - destination buffer, @dest_size bytes
   * @src - source buffer, @src_size bytes
   *
   * Returns: 0 on success
- *          -1 on fail
+ *          -EIO on fail
   */
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-                                const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
  {
      int ret = 0;
      z_stream strm;
@@ -144,7 +147,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
 
      ret = inflateInit2(&strm, -12);
      if (ret != Z_OK) {
-        return -1;
+        return -EIO;
      }
 
      ret = inflate(&strm, Z_FINISH);
@@ -154,7 +157,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
           * @src buffer may be processed partly (because in qcow2 we know size of
           * compressed data with precision of one sector)
           */
-        ret = -1;
+        ret = -EIO;
      }
 
      inflateEnd(&strm);
@@ -189,20 +192,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
      return arg.ret;
  }
 
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on fail
+ */
  ssize_t coroutine_fn
  qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
                    const void *src, size_t src_size)
  {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_compress);

block/qcow2-threads.c qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, · line 220

Quoted 10 lines
block/qcow2-threads.c +10 -0
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_compress;
+        break;
+
+    default:
+        return -ENOTSUP;

it may be g_assert_not_reached() instead, as we never set state field to unsupported value.

Quoted 24 lines
block/qcow2-threads.c +16 -2
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
  }
 
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on fail
+ */
  ssize_t coroutine_fn
  qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
                      const void *src, size_t src_size)
  {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_decompress);

block/qcow2-threads.c qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, · line 252

Quoted 10 lines
block/qcow2-threads.c +10 -0
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_decompress;
+        break;
+
+    default:
+        return -ENOTSUP;

and here, and may be in previous patch (where call switch on s->compression_type)

But anyway, with this changed or without:

Quoted 7 lines
block/qcow2-threads.c +3 -0
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
  }
 
 
Signature · 2 lines
Best regards,
Vladimir
#6 lore raw

PATCH v3 3/3 qcow2: add zstd cluster compression

In reply to Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, at the moment, has been the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

---
 block/qcow2-threads.c  | 94 ++++++++++++++++++++++++++++++++++++++++++
 block/qcow2.c          |  6 +++
 configure              | 34 +++++++++++++++
 docs/interop/qcow2.txt | 20 +++++++++
 qapi/block-core.json   |  3 +-
 5 files changed, 156 insertions(+), 1 deletion(-)
block/qcow2-threads.c +94 -0
index 14b5bd76fb..85d04e6c2e 100644
@@ -28,6 +28,11 @@
 #define ZLIB_CONST
 #include <zlib.h>
 
+#ifdef CONFIG_ZSTD
+#include <zstd.h>
+#include <zstd_errors.h>
+#endif
+
 #include "qcow2.h"
 #include "block/thread-pool.h"
 #include "crypto.h"
@@ -165,6 +170,85 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
     return ret;
 }
 
+#ifdef CONFIG_ZSTD
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ *          -ENOMEM destination buffer is not enough to store compressed data
+ *          -EIO    on any other error
+ */
+
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    ssize_t ret;
+    uint32_t *c_size = dest;
+    /* steal some bytes to store compressed chunk size */
+    char *d_buf = ((char *) dest) + sizeof(*c_size);
+
+    if (dest_size < sizeof(*c_size)) {
+        return -ENOMEM;
+    }
+
+    dest_size -= sizeof(*c_size);
+
+    ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+    if (ZSTD_isError(ret)) {
+        if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+            return -ENOMEM;
+        } else {
+            return -EIO;
+        }
+    }
+
+    /* store the compressed chunk size in the very beginning of the buffer */
+    *c_size = ret;
+
+    return ret + sizeof(*c_size);
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          -EIO on any error
+ */
+
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{
+    ssize_t ret;
+    /*
+     * zstd decompress wants to know the exact length of the data
+     * for that purpose, on the compression the length is stored in
+     * the very beginning of the compressed buffer
+     */
+    const uint32_t *s_size = src;
+    const char *s_buf = ((const char *) src) + sizeof(*s_size);
+
+    ret = ZSTD_decompress(dest, dest_size, s_buf, *s_size);
+
+    if (ZSTD_isError(ret)) {
+        return -EIO;
+    }
+
+    return 0;
+}
+#endif
+
 static int qcow2_compress_pool_func(void *opaque)
 {
     Qcow2CompressData *data = opaque;
@@ -216,6 +300,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
         fn = qcow2_zlib_compress;
         break;
 
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+        fn = qcow2_zstd_compress;
+        break;
+#endif
     default:
         return -ENOTSUP;
     }
@@ -248,6 +337,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
         fn = qcow2_zlib_decompress;
         break;
 
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+        fn = qcow2_zstd_decompress;
+        break;
+#endif
     default:
         return -ENOTSUP;
     }
block/qcow2.c +6 -0
index 4e07b7e9ec..dfb7b52033 100644
@@ -1201,6 +1201,9 @@ static int check_compression_type(BDRVQcow2State *s, Error **errp)
 {
     switch (s->compression_type) {
     case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
         break;
 
     default:
@@ -3294,6 +3297,9 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
 
         switch (qcow2_opts->compression_type) {
         case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+        case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
             break;
 
         default:
configure +34 -0
index 714e7fb6a1..111c34878d 100755
@@ -441,6 +441,7 @@ opengl_dmabuf="no"
 cpuid_h="no"
 avx2_opt=""
 zlib="yes"
+zstd=""
 capstone=""
 lzo=""
 snappy=""
@@ -1358,6 +1359,10 @@ for opt do
   ;;
   --disable-lzfse) lzfse="no"
   ;;
+  --enable-zstd) zstd="yes"
+  ;;
+  --disable-zstd) zstd="no"
+  ;;
   --enable-guest-agent) guest_agent="yes"
   ;;
   --disable-guest-agent) guest_agent="no"
@@ -1812,6 +1817,7 @@ disabled with --disable-FEATURE, default is enabled if available:
                   (for reading bzip2-compressed dmg images)
   lzfse           support of lzfse compression library
                   (for reading lzfse-compressed dmg images)
+  zstd            support of zstd compression library
   seccomp         seccomp support
   coroutine-pool  coroutine freelist (better performance)
   glusterfs       GlusterFS backend
@@ -2407,6 +2413,30 @@ EOF
     fi
 fi
 
+#########################################
+# zstd check
+
+if test "$zstd" == "yes" ; then
+    if $pkg_config --exists libzstd; then
+        zstd_cflags=$($pkg_config --cflags libzstd)
+        zstd_libs=$($pkg_config --libs libzstd)
+        QEMU_CFLAGS="$zstd_cflags $QEMU_CFLAGS"
+        LIBS="$zstd_libs $LIBS"
+    else
+        cat > $TMPC << EOF
+#include <zstd.h>
+int main(void) { ZSTD_versionNumber(); return 0; }
+EOF
+        if compile_prog "" "-lzstd" ; then
+            LIBS="$LIBS -lzstd"
+        else
+            feature_not_found "zstd" "Install libzstd-devel"
+        fi
+    fi
+else
+    zstd="no"
+fi
+
 ##########################################
 # libseccomp check
 
@@ -6460,6 +6490,7 @@ echo "lzo support       $lzo"
 echo "snappy support    $snappy"
 echo "bzip2 support     $bzip2"
 echo "lzfse support     $lzfse"
+echo "zstd support      $zstd"
 echo "NUMA host support $numa"
 echo "libxml2           $libxml2"
 echo "tcmalloc support  $tcmalloc"
@@ -7306,6 +7337,9 @@ fi
 if test "$sheepdog" = "yes" ; then
   echo "CONFIG_SHEEPDOG=y" >> $config_host_mak
 fi
+if test "$zstd" = "yes" ; then
+  echo "CONFIG_ZSTD=y" >> $config_host_mak
+fi
 
 if test "$tcg_interpreter" = "yes"; then
   QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
docs/interop/qcow2.txt +20 -0
index e1be8bd5c3..4b0dc124f5 100644
@@ -181,6 +181,7 @@ in the description of a field.
                     must be set.
                     Available compression type values:
                         0: zlib <https://www.zlib.net/> (default)
+                        1: zstd <http://github.com/facebook/zstd>
 
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
@@ -536,6 +537,9 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
                     Another compressed cluster may map to the tail of the final
                     sector used by this compressed cluster.
 
+                    The layout of the compressed data depends on the compression
+                    type used for the image (see compressed cluster layout).
+
 If a cluster is unallocated, read requests shall read the data from the backing
 file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
 no backing file or the backing file is smaller than the image, they shall read
@@ -788,3 +792,19 @@ In the image file the 'enabled' state is reflected by the 'auto' flag. If this
 flag is set, the software must consider the bitmap as 'enabled' and start
 tracking virtual disk changes to this bitmap from the first write to the
 virtual disk. If this flag is not set then the bitmap is disabled.
+
+=== Compressed cluster layout ===
+
+The compressed cluster data may have a different layout depending on the
+compression type used for the image, and store specific data for the particular
+compression type.
+
+Compressed data layout for the available compression types:
+(x = data_space_length - 1)
+
+    zlib <http://zlib.net/>:
+        Byte  0 -  x:     the compressed data content
+                          all the space provided used for compressed data
+    zstd <http://github.com/facebook/zstd>:
+        Byte  0 -  3:     the length of compressed data
+              4 -  x:     the compressed data content
qapi/block-core.json +2 -1
index 2c002ca6a9..9e458d5b40 100644
@@ -4283,11 +4283,12 @@
 # Compression type used in qcow2 image file
 #
 # @zlib:  zlib compression, see <http://zlib.net/>
+# @zstd:  zstd compression, see <http://github.com/facebook/zstd>
 #
 # Since: 4.2
 ##
 { 'enum': 'Qcow2CompressionType',
-  'data': [ 'zlib' ] }
+  'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
 
 ##
 # @BlockdevCreateOptionsQcow2:
Signature · 1 line
2.17.0
Reviewed-by on v3 3/3
#7 lore raw
Quoted 80 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, at the moment, has been the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

---
  block/qcow2-threads.c  | 94 ++++++++++++++++++++++++++++++++++++++++++
  block/qcow2.c          |  6 +++
  configure              | 34 +++++++++++++++
  docs/interop/qcow2.txt | 20 +++++++++
  qapi/block-core.json   |  3 +-
  5 files changed, 156 insertions(+), 1 deletion(-)
block/qcow2-threads.c +24 -0
index 14b5bd76fb..85d04e6c2e 100644
@@ -28,6 +28,11 @@
  #define ZLIB_CONST
  #include <zlib.h>
 
+#ifdef CONFIG_ZSTD
+#include <zstd.h>
+#include <zstd_errors.h>
+#endif
+
  #include "qcow2.h"
  #include "block/thread-pool.h"
  #include "crypto.h"
@@ -165,6 +170,85 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
      return ret;
  }
 
+#ifdef CONFIG_ZSTD
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ *          -ENOMEM destination buffer is not enough to store compressed data
+ *          -EIO    on any other error
+ */
+
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    ssize_t ret;
+    uint32_t *c_size = dest;

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 201

Quoted 10 lines
block/qcow2-threads.c +10 -0
+    /* steal some bytes to store compressed chunk size */
+    char *d_buf = ((char *) dest) + sizeof(*c_size);
+
+    if (dest_size < sizeof(*c_size)) {
+        return -ENOMEM;
+    }
+
+    dest_size -= sizeof(*c_size);
+
+    ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);

Let's call this "5" to be something like QCOW2_DEFAUTLT_ZSTD_COMPRESSION_LEVEL, we'll possibly want to add an option for this in future.

Quoted 29 lines
block/qcow2-threads.c +29 -0
+
+    if (ZSTD_isError(ret)) {
+        if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+            return -ENOMEM;
+        } else {
+            return -EIO;
+        }
+    }
+
+    /* store the compressed chunk size in the very beginning of the buffer */
+    *c_size = ret;
+
+    return ret + sizeof(*c_size);
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          -EIO on any error
+ */
+
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 240

Quoted 10 lines
block/qcow2-threads.c +10 -0
+                                     const void *src, size_t src_size)
+{
+    ssize_t ret;
+    /*
+     * zstd decompress wants to know the exact length of the data
+     * for that purpose, on the compression the length is stored in
+     * the very beginning of the compressed buffer
+     */
+    const uint32_t *s_size = src;
+    const char *s_buf = ((const char *) src) + sizeof(*s_size);

but what if *s_size (or even src_size) is corrupted? I think we need to check it before calling ZSTD_decompress:

if (src_size < sizeof(*s_size) || src_size - sizeof(*s_size) < *s_size) {
    return -EIO;
}

with this (and optionally with QCOW2_DEFAUTLT_ZSTD_COMPRESSION_LEVEL):

Quoted 11 lines
block/qcow2-threads.c +11 -0
+
+    ret = ZSTD_decompress(dest, dest_size, s_buf, *s_size);
+
+    if (ZSTD_isError(ret)) {
+        return -EIO;
+    }
+
+    return 0;
+}
+#endif
+

[...]

Signature · 2 lines
Best regards,
Vladimir
#8 lore raw
In reply to Denis Plotnikov

Ping!

Quoted 80 lines · Denis Plotnikov

v3:
* relax the compression type setting requirement when
the compression type is not zlib [Eric, Kevin]
* add compression type values to the spec [Eric]
* fix wording in the spec and descriptions [Eric]
* fix functions descriptions [Max]
* fix zstd (de)compression functions flaws [Max]
* fix zstd related parts of configure file [Max]
* rebased to v4.1.0-rc5 and chenged the series version aiming to 4.2

v2:
* relax the compression type setting restriction in the spec
* fix qcow2 header size checking
* fix error processing and messaging
* fix qcow2 image specific info reporting
* set Qcow2CompressionType zstd config dependant
* add zstd compressed cluster format description to the spec

v1:
* extend qcow2 header instead of adding a new incompatible extension header specification re-written accordingly
* enable zstd compression via config
* fix zstd (de)compression functions
* fix comments/description
* fix function naming

---
The goal of the patch-set is to enable qcow2 to use zstd compression for clusters. ZSTD provides better (de)compression performance than currently used ZLIB. Using it will improve perforamnce (reduce compression time) when the compressed clusters is used, e.g backup scenarios.

Also, the patch-set extends qcow2 specification by adding compression_type feature. The feature enables adding ZSTD and another compression algorithms in the future.

Here is some measurements ZSTD vs ZLIB:

The test:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

The results:
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0

Both ZLIB and ZSTD gave the same compression ratio: ~1.5 compressed image size in both cases: ~1.4G

Denis Plotnikov (3):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression

  block/qcow2-threads.c     | 172 ++++++++++++++++++++++++++++++++++----
  block/qcow2.c             | 100 ++++++++++++++++++++++
  block/qcow2.h             |  26 ++++--
  configure                 |  34 ++++++++
  docs/interop/qcow2.txt    |  39 ++++++++-
  include/block/block_int.h |   1 +
  qapi/block-core.json      |  23 ++++-
  7 files changed, 371 insertions(+), 24 deletions(-)
Signature · 2 lines
Best,
Denis